neataptic

{ "docs": [ { "location": "/", "text": "", "title": "Home" }, { "location": "/docs/", "text": "Welcome to the documentation of Neataptic! If you are a rookie with neural networks:\ncheck out any of the tutorials on the left to get started. If you want more\ninformation about a certain part of Neataptic, it most probably is also in the\nmenu on the left. If it isn't, feel free to let me know by creating an \nissue\n.\n\n\nIf you want to implement a genetic neural network algorithm, but don't know how,\nfeel free to contact me at \nwagenaartje@protonmail.com\n!", "title": "Docs" }, { "location": "/docs/tutorials/tutorials/", "text": "In order to get you started on Neataptic, a few tutorials have been written that\ngive a basic overview on how to create, train and evolve your networks. It is\nrecommended to read all of them before you start digging in your own project.\n\n\n\n\nTraining\n\n\nEvolution\n\n\nNormalization\n\n\nVisualization\n\n\n\n\nIf you have any questions, feel free to create an issue \nhere\n.\nIf you feel like anything is missing, feel free to create a pull request!", "title": "Tutorials" }, { "location": "/docs/tutorials/training/", "text": "Training your network is not that hard to do - it's the preparation that is harder.\n\n\nThe training set\n\n\nFirst of all, you should normalize your data. That means you have to convert all your input and output data to values within a range of \n0\n to \n1\n. If you are unsure how to do this, visit the \nNormalization\n page. Each training sample in your training set should be an object that looks as follows:\n\n\n{ input: [], output: [] }\n\n\n\n\nSo an example of a training set would be (XOR):\n\n\nvar myTrainingSet = [\n { input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }\n];\n\n\n\n\nThe network architecture\n\n\nThere is no fixed rule of thumb for choosing your network architecture. Adding more layers makes your neural network recognize more abstract relationships, although it requires more computation. Any function can be mapped with just one (big) hidden layer, but I do not advise this. I advise to use any of the following architectures if you're a starter:\n\n\n\n\nPerceptron\n - a fully connected feed forward network\n\n\nLSTM\n - a recurrent network that can recognize patterns over very long time lags between inputs.\n\n\nNARX\n - a recurrent network that remembers previous inputs and outputs\n\n\n\n\nBut for most problems, a perceptron is sufficient. Now you only have to determine the size and amount of the network layers. I advise you to take a look at this \nStackExchange question\n for more help on deciding the hidden size.\n\n\nFor the training set I provided above (XOR), there are only 2 inputs and one output. I use the rule of thumb: \ninput size + output size = hidden size\n. So the creation of my network would like this:\n\n\nmyNetwork = architect.Perceptron(2, 3, 1);\n\n\n\n\nTraining the network\n\n\nFinally: we're going to train the network. The function for training your network is very straight-forward:\n\n\nyourNetwork.train(yourData, yourOptions);\n\n\n\n\nThere are \na lot\n of options. I won't go over all of them here, but you can check out the \nNetwork wiki\n for all the options.\n\n\nI'm going to use the following options:\n\n\n\n\nlog: 10\n - I want to log the status every 10 iterations\n\n\nerror: 0.03\n - I want the training to stop if the error is below 0.03\n\n\niterations: 1000\n - I want the training to stop if the error of 0.03 hasn't been reached after 1000 iterations\n\n\nrate: 0.3\n - I want a learning rate of 0.3\n\n\n\n\nSo let's put it all together:\n\n\nmyNetwork.train(myTrainingSet, {\n log: 10,\n error: 0.03,\n iterations: 1000,\n rate: 0.3\n});\n\n// result: {error: 0.02955628620843985, iterations: 566, time: 31}\n\n\n\n\nNow let us check if it \nactually\n works:\n\n\nmyNetwork.activate([0,0]); // [0.1257225731473885]\nmyNetwork.activate([0,1]); // [0.9371910625522613]\nmyNetwork.activate([1,0]); // [0.7770757408042104]\nmyNetwork.activate([1,1]); // [0.1639697315652196]\n\n\n\n\nAnd it works! If you want it to be more precise, lower the target error.\n\n\nHelp\n\n\nIf you need more help, feel free to create an issue \nhere\n!", "title": "Training" }, { "location": "/docs/tutorials/training/#the-training-set", "text": "First of all, you should normalize your data. That means you have to convert all your input and output data to values within a range of 0 to 1 . If you are unsure how to do this, visit the Normalization page. Each training sample in your training set should be an object that looks as follows: { input: [], output: [] } So an example of a training set would be (XOR): var myTrainingSet = [\n { input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }\n];", "title": "The training set" }, { "location": "/docs/tutorials/training/#the-network-architecture", "text": "There is no fixed rule of thumb for choosing your network architecture. Adding more layers makes your neural network recognize more abstract relationships, although it requires more computation. Any function can be mapped with just one (big) hidden layer, but I do not advise this. I advise to use any of the following architectures if you're a starter: Perceptron - a fully connected feed forward network LSTM - a recurrent network that can recognize patterns over very long time lags between inputs. NARX - a recurrent network that remembers previous inputs and outputs But for most problems, a perceptron is sufficient. Now you only have to determine the size and amount of the network layers. I advise you to take a look at this StackExchange question for more help on deciding the hidden size. For the training set I provided above (XOR), there are only 2 inputs and one output. I use the rule of thumb: input size + output size = hidden size . So the creation of my network would like this: myNetwork = architect.Perceptron(2, 3, 1);", "title": "The network architecture" }, { "location": "/docs/tutorials/training/#training-the-network", "text": "Finally: we're going to train the network. The function for training your network is very straight-forward: yourNetwork.train(yourData, yourOptions); There are a lot of options. I won't go over all of them here, but you can check out the Network wiki for all the options. I'm going to use the following options: log: 10 - I want to log the status every 10 iterations error: 0.03 - I want the training to stop if the error is below 0.03 iterations: 1000 - I want the training to stop if the error of 0.03 hasn't been reached after 1000 iterations rate: 0.3 - I want a learning rate of 0.3 So let's put it all together: myNetwork.train(myTrainingSet, {\n log: 10,\n error: 0.03,\n iterations: 1000,\n rate: 0.3\n});\n\n// result: {error: 0.02955628620843985, iterations: 566, time: 31} Now let us check if it actually works: myNetwork.activate([0,0]); // [0.1257225731473885]\nmyNetwork.activate([0,1]); // [0.9371910625522613]\nmyNetwork.activate([1,0]); // [0.7770757408042104]\nmyNetwork.activate([1,1]); // [0.1639697315652196] And it works! If you want it to be more precise, lower the target error.", "title": "Training the network" }, { "location": "/docs/tutorials/training/#help", "text": "If you need more help, feel free to create an issue here !", "title": "Help" }, { "location": "/docs/tutorials/evolution/", "text": "Neuro-evolution is something that is fairly underused in the machine learning\ncommunity. It is quite interesting to see new architectures develop for\ncomplicated problems. In this guide I will tell you how to set up a simple\nsupervised neuro-evolution process. If you want to do an unsupervised\nneuro-evolution process, head over to the \nNEAT\n page.\n\n\nThe training set\n\n\nYou always have to supply a training set for supervised learning. First of all,\nyou should normalize your data. That means you have to convert all your input and\noutput data to values within a range of \n0\n to \n1\n. If you are unsure how to do\nthis, visit the \nNormalization\n tutorial. Each training sample\nin your training set should be an object that looks as follows:\n\n\n{ input: [], output: [] }\n\n\n\n\nSo an example of a training set would be (XOR):\n\n\nvar myTrainingSet = [\n { input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }\n];\n\n\n\n\nThe network architecture\n\n\nYou can start off with \nany\n architecture. You can even use the evolution process\nto optimize already trained networks. However, I advise you to start with an empty\nnetwork, as originally described in the \nNEAT\n\n paper. The constructor of an empty network is as follows:\n\n\nvar myNetwork = new Network(inputSize, outputSize);\n\n\n\n\nSo for the training set I made above, the network can be constructed as follows:\n\n\nvar myNetwork = new Network(2, 1); // 2 inputs, 1 output\n\n\n\n\nNow we have our network. We don't have to tinker around anymore; the evolution\nprocess will do that \nfor\n us.\n\n\nEvolving the network\n\n\nBe warned: there are \na lot\n of options involved in the evolution of a process.\nIt's a matter of trial and error before you reach options that work really well.\nPlease note that most options are optional, as default values have been configured.\n\n\nPlease note that the \nevolve\n function is an \nasync\n function, so you need to wrap\nit in an async function to call \nawait\n.\n\n\nThe evolve function is as follows:\n\n\nyourNetwork.evolve(yourData, yourOptions);\n\n\n\n\nCheck out the evolution options \nhere\n and \nhere\n. I'm going to use the following options to evolve the network:\n\n\n\n\nmutation: methods.mutation.FFW\n - I want to solve a feed forward problem, so I supply all feed forward mutation methods. More info \nhere\n.\n\n\nequal: true\n - During the crossover process, parent networks will be equal. This allows the spawning of new network architectures more easily.\n\n\npopsize: 100\n - The default population size is 50, but 100 worked better for me.\n\n\nelitism: 10\n - I want to keep the fittest 10% of the population to the next generation without breeding them.\n\n\nlog: 10\n - I want to log the status every 10 iterations.\n\n\nerror: 0.03\n - I want the evolution process when the error is below 0.03;\n\n\niterations: 1000\n - I want the evolution process to stop after 1000 iterations if the target error hasn't been reached yet.\n\n\nmutationRate: 0.5\n - I want to increase the mutation rate to surpass possible local optima.\n\n\n\n\nSo let's put it all together:\n\n\nawait myNetwork.evolve(myTrainingSet, {\n mutation: methods.mutation.FFW,\n equal: true,\n popsize: 100,\n elitism: 10,\n log: 10,\n error: 0.03,\n iterations: 1000,\n mutationRate: 0.5\n});\n\n// results: {error: 0.0009000000000000001, generations: 255, time: 1078}\n// please note that there is a hard local optima that has to be beaten\n\n\n\n\nNow let us check if it \nactually\n works:\n\n\nmyNetwork.activate([0,0]); // [0]\nmyNetwork.activate([0,1]); // [1]\nmyNetwork.activate([1,0]); // [1]\nmyNetwork.activate([1,1]); // [0]\n\n\n\n\nNotice how precise the results are. That is because the evolution process makes\nfull use of the diverse activation functions. It actually uses the \nActivation.STEP\n\nfunction to get a binary \n0\n and \n1\n output.\n\n\nHelp\n\n\nIf you need more help, feel free to create an issue \nhere\n!", "title": "Evolution" }, { "location": "/docs/tutorials/evolution/#the-training-set", "text": "You always have to supply a training set for supervised learning. First of all,\nyou should normalize your data. That means you have to convert all your input and\noutput data to values within a range of 0 to 1 . If you are unsure how to do\nthis, visit the Normalization tutorial. Each training sample\nin your training set should be an object that looks as follows: { input: [], output: [] } So an example of a training set would be (XOR): var myTrainingSet = [\n { input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }\n];", "title": "The training set" }, { "location": "/docs/tutorials/evolution/#the-network-architecture", "text": "You can start off with any architecture. You can even use the evolution process\nto optimize already trained networks. However, I advise you to start with an empty\nnetwork, as originally described in the NEAT \n paper. The constructor of an empty network is as follows: var myNetwork = new Network(inputSize, outputSize); So for the training set I made above, the network can be constructed as follows: var myNetwork = new Network(2, 1); // 2 inputs, 1 output Now we have our network. We don't have to tinker around anymore; the evolution\nprocess will do that for us.", "title": "The network architecture" }, { "location": "/docs/tutorials/evolution/#evolving-the-network", "text": "Be warned: there are a lot of options involved in the evolution of a process.\nIt's a matter of trial and error before you reach options that work really well.\nPlease note that most options are optional, as default values have been configured. Please note that the evolve function is an async function, so you need to wrap\nit in an async function to call await . The evolve function is as follows: yourNetwork.evolve(yourData, yourOptions); Check out the evolution options here and here . I'm going to use the following options to evolve the network: mutation: methods.mutation.FFW - I want to solve a feed forward problem, so I supply all feed forward mutation methods. More info here . equal: true - During the crossover process, parent networks will be equal. This allows the spawning of new network architectures more easily. popsize: 100 - The default population size is 50, but 100 worked better for me. elitism: 10 - I want to keep the fittest 10% of the population to the next generation without breeding them. log: 10 - I want to log the status every 10 iterations. error: 0.03 - I want the evolution process when the error is below 0.03; iterations: 1000 - I want the evolution process to stop after 1000 iterations if the target error hasn't been reached yet. mutationRate: 0.5 - I want to increase the mutation rate to surpass possible local optima. So let's put it all together: await myNetwork.evolve(myTrainingSet, {\n mutation: methods.mutation.FFW,\n equal: true,\n popsize: 100,\n elitism: 10,\n log: 10,\n error: 0.03,\n iterations: 1000,\n mutationRate: 0.5\n});\n\n// results: {error: 0.0009000000000000001, generations: 255, time: 1078}\n// please note that there is a hard local optima that has to be beaten Now let us check if it actually works: myNetwork.activate([0,0]); // [0]\nmyNetwork.activate([0,1]); // [1]\nmyNetwork.activate([1,0]); // [1]\nmyNetwork.activate([1,1]); // [0] Notice how precise the results are. That is because the evolution process makes\nfull use of the diverse activation functions. It actually uses the Activation.STEP \nfunction to get a binary 0 and 1 output.", "title": "Evolving the network" }, { "location": "/docs/tutorials/evolution/#help", "text": "If you need more help, feel free to create an issue here !", "title": "Help" }, { "location": "/docs/tutorials/normalization/", "text": "Although Neataptic networks accepts non-normalized values as input, normalizing your input makes your network converge faster. I see a lot of questions where people ask how to normalize their data correctly, so I decided to make a guide.\n\n\nExample data\n\n\nYou have gathered this information, now you want to use it to train/activate a neural network:\n\n\n{ stock: 933, sold: 352, price: 0.95, category: 'drinks', id: 40 }\n{ stock: 154, sold: 103, price: 5.20, category: 'foods', id: 67 }\n{ stock: 23, sold: 5, price: 121.30, category: 'electronics', id: 150 }\n\n\n\n\nSo some information on the above data:\n\n \nstock\n: the amount of this item in stock\n\n \nsold\n: the amount of this item sold ( in the last month )\n\n \nprice\n: the price of this item\n\n \ncategory\n: the type of product\n* \nid\n: the id of the product\n\n\nNormalize\n\n\nSo we want to represent each of these inputs as a number between \n0\n and \n1\n, however, we can not change the relativity between the values. So we need to treat each different input the same (\nstock\n gets treated the same for every item for example).\n\n\nWe have two types of values in our input data: numerical values and categorical values. These should always be treated differently.\n\n\nNumerical values\n\n\nNumerical values are values where the distance between two values matters. For example, \nprice: 0.95\n is twice as small as \nprice: 1.90\n. But not all integers/decimals are numerical values. Id's are often represented with numbers, but there is no relation between \nid: 4\n and \nid: 8\n . So these should be treated as categorical values.\n\n\nNormalizing numerical values is quite easy, we just need to determine a maximum value we divide a certain input with. For example, we have the following data:\n\n\nstock: 933\nstock: 154\nstock: 23\n\n\n\n\nWe need to choose a value which is \n= 933\n with which we divide all the \nstock\n values. We could choose \n933\n, but what if we get new data, where the \nstock\n value is higher than \n933\n? Then we have to renormalize all the data and retrain the network.\n\n\nSo we need to choose a value that is \n=933\n, but also \n= future values\n and it shouldn't be a too big number. We could make the assumption that the \nstock\n will never get larger than \n2000\n, so we choose \n2000\n as our maximum value. We now normalize our data with this maximum value:\n\n\n// Normalize the data with a maximum value (=2000)\nstock: 933 -\n 933/2000 -\n 0.4665\nstock: 154 -\n 154/2000 -\n 0.077\nstock: 23 -\n 23/2000 -\n 0.0115\n\n\n\n\nCategorical data\n\n\nCategorical data shows no relation between different categories. So each category should be treated as a seperate input, this is called \none-hot encoding\n. Basically, you create a seperate input for each category. You set all the inputs to \n0\n, except for the input which matches the sample category. This is one-hot encoding for our above training data:\n\n\n\n \n\n \n\n \nSample\n\n \nDrinks\n\n \nFoods\n\n \nElectronics\n\n \n\n \n\n \n\n \n\n \n1\n\n \n1\n\n \n0\n\n \n0\n\n \n\n \n\n \n2\n\n \n0\n\n \n1\n\n \n0\n\n \n\n \n\n \n3\n\n \n0\n\n \n0\n\n \n1\n\n \n\n \n\n\n\n\n\nBut this also allows the addition of new categories over time: you just a need input. It has no effect on the performances of the network on the past training data as when the new category is set to \n0\n, it has no effect (\nweight * 0 = 0\n).\n\n\nNormalized data\n\n\nSo applying what I have explained above creates our normalized data, note that the relativity between inputs has not been changed. Also note that some values may be rounded in the table below.\n\n\n{ stock: 0.4665, sold: 0.352, price: 0.00317, drinks: 1, foods: 0, electronics: 0, id40: 1, id67: 0, id150: 0 }\n{ stock: 0.077, sold: 0.103, price: 0.01733, drinks: 0, foods: 1, electronics: 0, id40: 0, id67: 1, id150: 0 }\n{ stock: 0.0115, sold: 0.005, price: 0.40433, drinks: 0, foods: 0, electronics: 1, id40: 0, id67: 0, id150: 1 }\n\n\n\n\nMax values:\n\n\n\n\nstock\n: 2000\n\n\nsold\n: 1000\n\n\nprice\n : 300\n\n\n\n\nPlease note, that these inputs should be provided in arrays for neural networks in Neataptic:\n\n\n[ 0.4665, 0.352, 0.00317, 1, 0, 0, 1, 0, 0 ]\n[ 0.77, 0.103, 0.01733, 0, 1, 0, 0, 1, 0 ]\n[ 0.0115, 0.005, 0.40433, 0, 0, 1, 0, 0, 1 ]", "title": "Normalization" }, { "location": "/docs/tutorials/normalization/#example-data", "text": "You have gathered this information, now you want to use it to train/activate a neural network: { stock: 933, sold: 352, price: 0.95, category: 'drinks', id: 40 }\n{ stock: 154, sold: 103, price: 5.20, category: 'foods', id: 67 }\n{ stock: 23, sold: 5, price: 121.30, category: 'electronics', id: 150 } So some information on the above data: stock : the amount of this item in stock sold : the amount of this item sold ( in the last month ) price : the price of this item category : the type of product\n* id : the id of the product", "title": "Example data" }, { "location": "/docs/tutorials/normalization/#normalize", "text": "So we want to represent each of these inputs as a number between 0 and 1 , however, we can not change the relativity between the values. So we need to treat each different input the same ( stock gets treated the same for every item for example). We have two types of values in our input data: numerical values and categorical values. These should always be treated differently.", "title": "Normalize" }, { "location": "/docs/tutorials/normalization/#numerical-values", "text": "Numerical values are values where the distance between two values matters. For example, price: 0.95 is twice as small as price: 1.90 . But not all integers/decimals are numerical values. Id's are often represented with numbers, but there is no relation between id: 4 and id: 8 . So these should be treated as categorical values. Normalizing numerical values is quite easy, we just need to determine a maximum value we divide a certain input with. For example, we have the following data: stock: 933\nstock: 154\nstock: 23 We need to choose a value which is = 933 with which we divide all the stock values. We could choose 933 , but what if we get new data, where the stock value is higher than 933 ? Then we have to renormalize all the data and retrain the network. So we need to choose a value that is =933 , but also = future values and it shouldn't be a too big number. We could make the assumption that the stock will never get larger than 2000 , so we choose 2000 as our maximum value. We now normalize our data with this maximum value: // Normalize the data with a maximum value (=2000)\nstock: 933 - 933/2000 - 0.4665\nstock: 154 - 154/2000 - 0.077\nstock: 23 - 23/2000 - 0.0115", "title": "Numerical values" }, { "location": "/docs/tutorials/normalization/#categorical-data", "text": "Categorical data shows no relation between different categories. So each category should be treated as a seperate input, this is called one-hot encoding . Basically, you create a seperate input for each category. You set all the inputs to 0 , except for the input which matches the sample category. This is one-hot encoding for our above training data: \n \n \n Sample \n Drinks \n Foods \n Electronics \n \n \n \n \n 1 \n 1 \n 0 \n 0 \n \n \n 2 \n 0 \n 1 \n 0 \n \n \n 3 \n 0 \n 0 \n 1 \n \n But this also allows the addition of new categories over time: you just a need input. It has no effect on the performances of the network on the past training data as when the new category is set to 0 , it has no effect ( weight * 0 = 0 ).", "title": "Categorical data" }, { "location": "/docs/tutorials/normalization/#normalized-data", "text": "So applying what I have explained above creates our normalized data, note that the relativity between inputs has not been changed. Also note that some values may be rounded in the table below. { stock: 0.4665, sold: 0.352, price: 0.00317, drinks: 1, foods: 0, electronics: 0, id40: 1, id67: 0, id150: 0 }\n{ stock: 0.077, sold: 0.103, price: 0.01733, drinks: 0, foods: 1, electronics: 0, id40: 0, id67: 1, id150: 0 }\n{ stock: 0.0115, sold: 0.005, price: 0.40433, drinks: 0, foods: 0, electronics: 1, id40: 0, id67: 0, id150: 1 } Max values: stock : 2000 sold : 1000 price : 300 Please note, that these inputs should be provided in arrays for neural networks in Neataptic: [ 0.4665, 0.352, 0.00317, 1, 0, 0, 1, 0, 0 ]\n[ 0.77, 0.103, 0.01733, 0, 1, 0, 0, 1, 0 ]\n[ 0.0115, 0.005, 0.40433, 0, 0, 1, 0, 0, 1 ]", "title": "Normalized data" }, { "location": "/docs/tutorials/visualization/", "text": "This is a step-by-step tutorial aimed to teach you how to create and visualise neural networks using Neataptic.\n\n\nStep 1\n\nCreate a javascript file. Name it anything you want. But make sure to start it off with the following:\n\n\nvar Node = neataptic.Node;\nvar Neat = neataptic.Neat;\nvar Network = neataptic.Network;\nvar Methods = neataptic.Methods;\nvar Architect = neataptic.Architect;\n\n\n\n\nThis makes the whole developing a whole lot easier\n\n\nStep 2\n\nCreate a html file. Copy and paste this template if you want:\n\n\nhtml\n\n \nhead\n\n \nscript src=\nhttp://d3js.org/d3.v3.min.js\n/script\n\n \nscript src=\nhttp://marvl.infotech.monash.edu/webcola/cola.v3.min.js\n/script\n\n\n \nscript src=\nhttps://rawgit.com/wagenaartje/neataptic/master/dist/neataptic.js\n/script\n\n \nscript src=\nhttps://rawgit.com/wagenaartje/neataptic/master/graph/graph.js\n/script\n\n \nlink rel=\nstylesheet\n type=\ntext/css\n href=\nhttps://rawgit.com/wagenaartje/neataptic/master/graph/graph.css\n\n \n/head\n\n \nbody\n\n \ndiv class=\ncontainer\n\n \ndiv class=\nrow\n\n \nsvg class=\ndraw\n width=\n1000px\n height=\n1000px\n/\n\n \n/div\n\n \n/div\n\n \nscript src=\nyourscript.js\n/script\n\n \n/body\n\n\n/html\n\n\n\n\n\nStep 3\n Create a network. You can do that in any of the following ways:\n\n\nvar network = architect.Random(2, 20, 2, 2);\n\n\n\n\nvar network = architect.Perceptron(2, 10, 10, 2);\n\n\n\n\nOr if you want to be more advanced, construct your own:\n\n\nvar A = new Node();\nvar B = new Node();\nvar C = new Node();\nvar D = new Node();\nvar E = new Node();\nvar F = new Node();\n\nvar nodes = [A, B, C, D, E, F];\n\nfor(var i = 0; i \n nodes.length-1; i++){\n node = nodes[i];\n for(var j = 0; j \n 2; j++){\n var connectTo = nodes[Math.floor(Math.random() * (nodes.length - i) + i)];\n node.connect(connectTo);\n }\n}\n\nvar network = architect.Construct(nodes);\n\n\n\n\nStep 4\n Retrieve data and draw a graph\n\n\ndrawGraph(network.graph(1000, 1000), '.draw');\n\n\n\n\nSee a working example \nhere\n!\nSee more info on graphs \nhere\n!", "title": "Visualization" }, { "location": "/docs/important/important/", "text": "Train\n\n\nEvolve", "title": "Important functions" }, { "location": "/docs/important/train/", "text": "The train method allows you to train your network with given parameters. If this\ndocumentation is too complicated, I recommend to check out the\n\ntraining tutorial\n!\n\n\nConstructor\n\n\nInitiating the training process is similar to initiating the evolution process:\n\n\n\nmyNetwork.train(trainingSet, options)\n\n\n\n\nTraining set\n\n\nWhere set is an array containing objects in the following way: \n{ input: [input(s)], output: [output(s)] }\n. So for example, this is how you would train an XOR:\n\n\n\nvar network = new architect.Perceptron(2,4,1);\n\n// Train the XOR gate\nnetwork.train([{ input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }]);\n\nnetwork.activate([0,1]); // 0.9824...\n\n\n\n\nOptions\n\n\nOptions allow you to finetune the training process:\n\n\n\n\nlog\n - If set to \nn\n, will output the training status every \nn\n iterations (\nlog : 1\n will log every iteration)\n\n\nerror\n - The target error to reach, once the network falls below this error, the process is stopped. Default: \n0.03\n\n\ncost\n - The cost function to use. See \ncost methods\n. Default: \nmethods.cost.MSE\n\n\nrate\n - Sets the learning rate of the backpropagation process. Default: \n0.3\n.\n\n\ndropout\n - Sets the dropout of the hidden network nodes. Read more about it on the \nregularization\n page. Default: \n0\n.\n\n\nshuffle\n - When set to \ntrue\n, will shuffle the training data every iteration. A good option to use if your network is performing less in cross validation than in the real training set. Default: \nfalse\n\n\niterations\n - Sets the amount of iterations the process will maximally run, even when the target error has not been reached. Default: \nNaN\n\n\nschedule\n - You can schedule tasks to happen every \nn\n iterations. An example of usage is \nschedule : { function: function(data){console.log(Date.now, data.error)}, iterations: 5}\n. This will log the time and error every 5 iterations. This option allows for complex scheduled tasks during training.\n\n\nclear\n - If set to \ntrue\n, will clear the network after every activation. This is useful for training \nLSTM\n's, more importantly for timeseries prediction. Default: \nfalse\n\n\nmomentum\n - Sets the momentum of the weight change. More info \nhere\n. Default: \n0\n\n\nratePolicy\n - Sets the rate policy for your training. This allows your rate to be dynamic, see the \nrate policies page\n. Default: \nmethods.rate.FIXED()\n\n\nbatchSize\n - Sets the (mini-) batch size of your training. Default: \n1\n (online training)\n\n\n\n\nIf you want to use the default options, you can either pass an empty object or\njust dismiss the whole second argument:\n\n\nmyNetwork.evolve(trainingSet, {});\n\n// or\n\nmyNetwork.evolve(trainingSet);\n\n\n\n\nThe default value will be used for any option that is not explicitly provided\nin the options object.\n\n\nExample\n\n\nSo the following setup will train until the error of \n0.0001\n is reached or if the iterations hit \n1000\n. It will log the status every iteration as well. The rate has been lowered to \n0.2\n.\n\n\nvar network = new architect.Perceptron(2,4,1);\n\nvar trainingSet = [\n { input: [0,0], output: [1] },\n { input: [0,1], output: [0] },\n { input: [1,0], output: [0] },\n { input: [1,1], output: [1] }\n];\n\n// Train the XNOR gate\nnetwork.train(trainingSet, {\n log: 1,\n iterations: 1000,\n error: 0.0001,\n rate: 0.2\n});\n\n\n\n\nCross-validation\n\n\nThe last option is the \ncrossValidate\n option, which will validate if the network also performs well enough on a non-trained part of the given set. Options:\n\n\n\n\ncrossValidate.testSize\n - Sets the amount of test cases that should be assigned to cross validation. If set to \n0.4\n, 40% of the given set will be used for cross validation.\n\n\ncrossValidate.testError\n - Sets the target error of the validation set.\n\n\n\n\nSo an example of cross validation would be:\n\n\nvar network = new architect.Perceptron(2,4,1);\n\nvar trainingSet = [\n { input: [0,0], output: [1] },\n { input: [0,1], output: [0] },\n { input: [1,0], output: [0] },\n { input: [1,1], output: [1] }\n];\n\n// Train the XNOR gate\nnetwork.train(trainingSet, {\n crossValidate :\n {\n testSize: 0.4,\n testError: 0.02\n }\n});\n\n\n\n\nPS: don't use cross validation for small sets, this is just an example!", "title": "Train" }, { "location": "/docs/important/train/#constructor", "text": "Initiating the training process is similar to initiating the evolution process: \nmyNetwork.train(trainingSet, options)", "title": "Constructor" }, { "location": "/docs/important/train/#training-set", "text": "Where set is an array containing objects in the following way: { input: [input(s)], output: [output(s)] } . So for example, this is how you would train an XOR: \nvar network = new architect.Perceptron(2,4,1);\n\n// Train the XOR gate\nnetwork.train([{ input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }]);\n\nnetwork.activate([0,1]); // 0.9824...", "title": "Training set" }, { "location": "/docs/important/train/#options", "text": "Options allow you to finetune the training process: log - If set to n , will output the training status every n iterations ( log : 1 will log every iteration) error - The target error to reach, once the network falls below this error, the process is stopped. Default: 0.03 cost - The cost function to use. See cost methods . Default: methods.cost.MSE rate - Sets the learning rate of the backpropagation process. Default: 0.3 . dropout - Sets the dropout of the hidden network nodes. Read more about it on the regularization page. Default: 0 . shuffle - When set to true , will shuffle the training data every iteration. A good option to use if your network is performing less in cross validation than in the real training set. Default: false iterations - Sets the amount of iterations the process will maximally run, even when the target error has not been reached. Default: NaN schedule - You can schedule tasks to happen every n iterations. An example of usage is schedule : { function: function(data){console.log(Date.now, data.error)}, iterations: 5} . This will log the time and error every 5 iterations. This option allows for complex scheduled tasks during training. clear - If set to true , will clear the network after every activation. This is useful for training LSTM 's, more importantly for timeseries prediction. Default: false momentum - Sets the momentum of the weight change. More info here . Default: 0 ratePolicy - Sets the rate policy for your training. This allows your rate to be dynamic, see the rate policies page . Default: methods.rate.FIXED() batchSize - Sets the (mini-) batch size of your training. Default: 1 (online training) If you want to use the default options, you can either pass an empty object or\njust dismiss the whole second argument: myNetwork.evolve(trainingSet, {});\n\n// or\n\nmyNetwork.evolve(trainingSet); The default value will be used for any option that is not explicitly provided\nin the options object.", "title": "Options" }, { "location": "/docs/important/train/#example", "text": "So the following setup will train until the error of 0.0001 is reached or if the iterations hit 1000 . It will log the status every iteration as well. The rate has been lowered to 0.2 . var network = new architect.Perceptron(2,4,1);\n\nvar trainingSet = [\n { input: [0,0], output: [1] },\n { input: [0,1], output: [0] },\n { input: [1,0], output: [0] },\n { input: [1,1], output: [1] }\n];\n\n// Train the XNOR gate\nnetwork.train(trainingSet, {\n log: 1,\n iterations: 1000,\n error: 0.0001,\n rate: 0.2\n});", "title": "Example" }, { "location": "/docs/important/train/#cross-validation", "text": "The last option is the crossValidate option, which will validate if the network also performs well enough on a non-trained part of the given set. Options: crossValidate.testSize - Sets the amount of test cases that should be assigned to cross validation. If set to 0.4 , 40% of the given set will be used for cross validation. crossValidate.testError - Sets the target error of the validation set. So an example of cross validation would be: var network = new architect.Perceptron(2,4,1);\n\nvar trainingSet = [\n { input: [0,0], output: [1] },\n { input: [0,1], output: [0] },\n { input: [1,0], output: [0] },\n { input: [1,1], output: [1] }\n];\n\n// Train the XNOR gate\nnetwork.train(trainingSet, {\n crossValidate :\n {\n testSize: 0.4,\n testError: 0.02\n }\n}); PS: don't use cross validation for small sets, this is just an example!", "title": "Cross-validation" }, { "location": "/docs/important/evolve/", "text": "The evolve function will evolve the network to conform the given training set. If you want to perform neuro-evolution on problems without a training set, check out the \nNEAT\n wiki page. This function may not always be successful, so always specify a number of iterations for it too maximally run.\n\n\nView a whole bunch of neuroevolution algorithms set up with Neataptic here.\n\n\nConstructor\n\n\nInitiating the evolution of your neural network is easy:\n\n\nawait myNetwork.evolve(trainingSet, options);\n\n\n\n\nPlease note that \nawait\n is used as \nevolve\n is an \nasync\n function. Thus, you\nneed to wrap these statements in an async function.\n\n\nTraining set\n\n\nWhere \ntrainingSet\n is your training set. An example is coming up ahead. An example\nof a training set would be:\n\n\n// XOR training set\nvar trainingSet = [\n { input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }\n];\n\n\n\n\nOptions\n\n\nThere are \na lot\n of options, here are the basic options:\n\n\n\n\ncost\n - Specify the cost function for the evolution, this tells a genome in the population how well it's performing. Default: \nmethods.cost.MSE\n (recommended).\n\n\namount\n- Set the amount of times to test the trainingset on a genome each generation. Useful for timeseries. Do not use for regular feedfoward problems. Default is \n1\n.\n\n\n\n\ngrowth\n - Set the penalty you want to give for large networks. The penalty get's calculated as follows: \npenalty = (genome.nodes.length + genome.connectoins.length + genome.gates.length) * growth;\n\nThis penalty will get added on top of the error. Your growth should be a very small number, the default value is \n0.0001\n\n\n\n\n\n\niterations\n - Set the maximum amount of iterations/generations for the algorithm to run. Always specify this, as the algorithm will not always converge.\n\n\n\n\nerror\n - Set the target error. The algorithm will stop once this target error has been reached. The default value is \n0.005\n.\n\n\nlog\n - If set to \nn\n, will output every \nn\n iterations (\nlog : 1\n will log every iteration)\n\n\nschedule\n - You can schedule tasks to happen every \nn\n iterations. An example of usage is \nschedule : { function: function(){console.log(Date.now)}, iterations: 5}\n. This will log the time every 5 iterations. This option allows for complex scheduled tasks during evolution.\n\n\nclear\n - If set to \ntrue\n, will clear the network after every activation. This is useful for evolving recurrent networks, more importantly for timeseries prediction. Default: \nfalse\n\n\nthreads\n - Specify the amount of threads to use. Default value is the amount of cores in your CPU. Set to \n1\n if you are evolving on a small dataset.\n\n\n\n\nPlease note that you can also specify \nany\n of the options that are specified on\nthe \nneat page\n.\n\n\nAn example of options would be:\n\n\nvar options = {\n mutation: methods.mutation.ALL,\n mutationRate: 0.4,\n clear: true,\n cost: methods.cost.MSE,\n error: 0.03,\n log: 1,\n iterations: 1000\n};\n\n\n\n\nIf you want to use the default options, you can either pass an empty object or\njust dismiss the whole second argument:\n\n\nawait myNetwork.evolve(trainingSet, {});\n\n// or\n\nawait myNetwork.evolve(trainingSet);\n\n\n\n\nThe default value will be used for any option that is not explicitly provided\nin the options object.\n\n\nResult\n\n\nThis function will output an object containing the final error, amount of iterations, time and the evolved network:\n\n\nreturn results = {\n error: mse,\n generations: neat.generation,\n time: Date.now() - start,\n evolved: fittest\n};\n\n\n\n\nExamples\n\n\n\n \nXOR\n\n Activates the network. It will activate all the nodes in activation order and produce an output.\n\n\nasync function execute () {\n var network = new Network(2,1);\n\n\n// XOR dataset\n var trainingSet = [\n { input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }\n ];\n\n\nawait network.evolve(trainingSet, {\n mutation: methods.mutation.FFW,\n equal: true,\n elitism: 5,\n mutationRate: 0.5\n });\n\n\nnetwork.activate([0,0]); // 0.2413\n network.activate([0,1]); // 1.0000\n network.activate([1,0]); // 0.7663\n network.activate([1,1]); // -0.008\n}\n\n\nexecute();", "title": "Evolve" }, { "location": "/docs/important/evolve/#constructor", "text": "Initiating the evolution of your neural network is easy: await myNetwork.evolve(trainingSet, options); Please note that await is used as evolve is an async function. Thus, you\nneed to wrap these statements in an async function.", "title": "Constructor" }, { "location": "/docs/important/evolve/#training-set", "text": "Where trainingSet is your training set. An example is coming up ahead. An example\nof a training set would be: // XOR training set\nvar trainingSet = [\n { input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }\n];", "title": "Training set" }, { "location": "/docs/important/evolve/#options", "text": "There are a lot of options, here are the basic options: cost - Specify the cost function for the evolution, this tells a genome in the population how well it's performing. Default: methods.cost.MSE (recommended). amount - Set the amount of times to test the trainingset on a genome each generation. Useful for timeseries. Do not use for regular feedfoward problems. Default is 1 . growth - Set the penalty you want to give for large networks. The penalty get's calculated as follows: penalty = (genome.nodes.length + genome.connectoins.length + genome.gates.length) * growth; \nThis penalty will get added on top of the error. Your growth should be a very small number, the default value is 0.0001 iterations - Set the maximum amount of iterations/generations for the algorithm to run. Always specify this, as the algorithm will not always converge. error - Set the target error. The algorithm will stop once this target error has been reached. The default value is 0.005 . log - If set to n , will output every n iterations ( log : 1 will log every iteration) schedule - You can schedule tasks to happen every n iterations. An example of usage is schedule : { function: function(){console.log(Date.now)}, iterations: 5} . This will log the time every 5 iterations. This option allows for complex scheduled tasks during evolution. clear - If set to true , will clear the network after every activation. This is useful for evolving recurrent networks, more importantly for timeseries prediction. Default: false threads - Specify the amount of threads to use. Default value is the amount of cores in your CPU. Set to 1 if you are evolving on a small dataset. Please note that you can also specify any of the options that are specified on\nthe neat page . An example of options would be: var options = {\n mutation: methods.mutation.ALL,\n mutationRate: 0.4,\n clear: true,\n cost: methods.cost.MSE,\n error: 0.03,\n log: 1,\n iterations: 1000\n}; If you want to use the default options, you can either pass an empty object or\njust dismiss the whole second argument: await myNetwork.evolve(trainingSet, {});\n\n// or\n\nawait myNetwork.evolve(trainingSet); The default value will be used for any option that is not explicitly provided\nin the options object.", "title": "Options" }, { "location": "/docs/important/evolve/#result", "text": "This function will output an object containing the final error, amount of iterations, time and the evolved network: return results = {\n error: mse,\n generations: neat.generation,\n time: Date.now() - start,\n evolved: fittest\n};", "title": "Result" }, { "location": "/docs/important/evolve/#examples", "text": "XOR \n Activates the network. It will activate all the nodes in activation order and produce an output. \nasync function execute () {\n var network = new Network(2,1); // XOR dataset\n var trainingSet = [\n { input: [0,0], output: [0] },\n { input: [0,1], output: [1] },\n { input: [1,0], output: [1] },\n { input: [1,1], output: [0] }\n ]; await network.evolve(trainingSet, {\n mutation: methods.mutation.FFW,\n equal: true,\n elitism: 5,\n mutationRate: 0.5\n }); network.activate([0,0]); // 0.2413\n network.activate([0,1]); // 1.0000\n network.activate([1,0]); // 0.7663\n network.activate([1,1]); // -0.008\n} execute();", "title": "Examples" }, { "location": "/docs/builtins/builtins/", "text": "If you are unfamiliar with building networks layer by layer, you can use the\npreconfigured networks. These networks will also be built layer by layer behind\nthe screens, but for the user they are all a simple one line function. At this\nmoment, Neataptic offers 6 preconfigured networks.\n\n\n\n\nGRU\n\n\nHopfield\n\n\nLSTM\n\n\nNARX\n\n\nPerceptron\n\n\nRandom", "title": "Built-in networks" }, { "location": "/docs/builtins/perceptron/", "text": "This architecture allows you to create multilayer perceptrons, also known as feed-forward neural networks. They consist of a sequence of layers, each fully connected to the next one.\n\n\n\n\nYou have to provide a minimum of 3 layers (input, hidden and output), but you can use as many hidden layers as you wish. This is a \nPerceptron\n with 2 neurons in the input layer, 3 neurons in the hidden layer, and 1 neuron in the output layer:\n\n\nvar myPerceptron = new architect.Perceptron(2,3,1);\n\n\n\n\nAnd this is a deep multilayer perceptron with 2 neurons in the input layer, 4 hidden layers with 10 neurons each, and 1 neuron in the output layer\n\n\nvar myPerceptron = new architect.Perceptron(2, 10, 10, 10, 10, 1);", "title": "Perceptron" }, { "location": "/docs/builtins/lstm/", "text": "The \nlong short-term memory\n is an architecture well-suited to learn from experience to classify, process and predict time series when there are very long time lags of unknown size between important events.\n\n\n\n\nTo use this architecture you have to set at least one input node, one memory block assembly (consisting of four nodes: input gate, memory cell, forget gate and output gate), and an output node.\n\n\nvar myLSTM = new architect.LSTM(2,6,1);\n\n\n\n\nAlso you can set many layers of memory blocks:\n\n\nvar myLSTM = new architect.LSTM(2, 4, 4, 4, 1);\n\n\n\n\nThat LSTM network has 3 memory block assemblies, with 4 memory cells each, and their own input gates, memory cells, forget gates and output gates.\n\n\nYou can pass options if desired like so:\n\n\nvar options = {\n memoryToMemory: false, // default is false\n outputToMemory: false, // default is false\n outputToGates: false, // default is false\n inputToOutput: true, // default is true\n inputToDeep: true // default is true\n};\n\nvar myLSTM = new architect.LSTM(2, 4, 4, 4, 1, options);\n\n\n\n\nWhile training sequences or timeseries prediction to a LSTM, make sure you set the \nclear\n option to true while training. \nSee an example of sequence prediction here.\n\n\nThis is an example of character-by-character typing by an LSTM: \nJSFiddle", "title": "LSTM" }, { "location": "/docs/builtins/gru/", "text": "Please be warned: GRU is still being tested, it might not always work for your dataset.\n\n\n\n\nThe Gated Recurrent Unit network is very similar to the LSTM network. GRU networks have \u00f3ne gate less and no selfconnections. Similarly to LSTM's, GRU's are well-suited to classify, process and predict time series when there are very long time lags of unknown size between important events.\n\n\n\n\nTo use this architecture you have to set at least one input node, one gated recurrent unit assembly, and an output node. The gated recurrent unit assembly consists of seven nodes: input, update gate, inverse update gate, reset gate, memorycell, output and previous output memory.\n\n\nvar myLSTM = new architect.GRU(2,6,1);\n\n\n\n\nAlso you can set many layers of gated recurrent units:\n\n\nvar myLSTM = new architect.GRU(2, 4, 4, 4, 1);\n\n\n\n\nThe above network has 3 hidden layers, with 4 GRU assemblies each. It has two