Here is given an example of the "bipropagation" algorithm for learning of NN. It is written in MatLAB language (R2015a) and is as similar as possible to "Deep learning" example "AutoencoderDigitsExample.m" which is included in MatLAB's Neural Network Toolbox. So you can easily compare both algorithms. I believe that my algorithm have few advantages over autoencoder. Please tell me what do you think about it. Please cite me in your works. Thanks a lot.
Download demo
======================================================
%% Training a Deep Neural Network for Digit Classification
% This example shows how to use the Neural Network Toolbox(TM) to train a
% deep neural network to classify images of digits and is very similar to
% "AutoencoderDigitsExample.m" which is included in
% Neuronal Network ToolBox from MatLAB. This example is made for comparison
% of both algorithms.
%
% Neural networks with multiple hidden layers can be useful for solving
% classification problems with complex data, such as images. Each layer can
% learn features at a different level of abstraction. However, training
% neural networks with multiple hidden layers can be difficult in practice.
%
% One way to effectively train a neural network with multiple layers is by
% training one layer at a time. You can achieve this by training separate
% hidden layers with biopropagation algorithm.
%
% This example shows you how to train a neural network with two hidden
% layers to classify digits in images. First you train the hidden layers
% individually. Then you train a final softmax layer, and join the layers
% together to form a deep network, which is trained one final time.
%
% Copyright 2016, Bojan PLOJ PhD.
%% Data Set
% This example uses synthetic data throughout, for training and testing.
% The synthetic images have been generated by applying random affine
% transformations to digit images created using different fonts.
%
% Each digit image is 28-by-28 pixels, and there are 5,000 training
% examples. You can load the training data, and view some of the images.
% Load the training data into memory
[xTrainImages, tTrain] = digittrain_dataset;
%%
% The labels for the images are stored in a 10-by-5000 matrix, where in
% every column a single element will be 1 to indicate the class that the
% digit belongs to, and all other elements in the column will be 0. It
% should be noted that if the tenth element is 1, then the digit image is a
% zero.
%
% To use the images for training a neural network, you have to arrange them
% into a matrix where each column represents a single image. You can do
% this by stacking the columns of an image to form a vector, and then
% forming a matrix from these vectors.
% Get the number of pixels in each image
imageWidth = 28;
imageHeight = 28;
inputSize = imageWidth*imageHeight;
% Turn the training images into vectors and put them in a matrix
xTrain = zeros(inputSize, numel(xTrainImages));
for i = 1:numel(xTrainImages)
xTrain(:,i) = xTrainImages{i}(:);
end
% Now that the training images have been arranged into a matrix, you are
% ready to begin training the network.
%% Training the first layer
%
% Bipropagation is an algorithm which attempts gradualy transform the
% input signal thrue the layers to the desired output signal.
% The size of individual layers usualy remain the same as is the size of
% the input. Only few outputs are slightly diferent than inputs according
% to the belonging class of pattern. Weight matrix
% started with values of eye matrix. Thus, the learning error is
% very small from the beginning on and learnin run very fast.
% Set the size of the hidden layer for hidden layers.
hiddenSize1 = 784;
% Create the network. You can experiment by changing the number of training
% epochs, and the training function
layer1 = perceptron;
layer1.trainFcn = 'trainscg';
layer1.trainParam.epochs = 1;
% Do not use process functions at the input or output
layer1.inputs{1}.processFcns = {};
layer1.outputs{1}.processFcns = {};
% Set the transfer function for both layers to the logistic sigmoid
layer1.layers{1}.transferFcn = 'satlin';
% Use all of the data for training
layer1.divideFcn = 'dividerand';%dividetrain,dividerand
layer1.performFcn = 'mse';
%%
% You now train the 1st layer. It should be noted that for the
% Bipropagation, the input data and target data are almost identical.
% Correction of xTrain according to the patterns class is:
correction=zeros(inputSize,5000);
for i=1:5000
for j=1:10
correction(78*(j-1)+1:78*j,i)=tTrain(j,i)/10000000;
end
end
CorrectedxTrain=0.9999999*xTrain+correction*1;
layer1 = train(layer1,xTrain,xTrain);
layer1.IW{1,1}=eye(784);
layer1.b{1}=zeros(784,1);
rezultat=layer1(xTrain)-xTrain;
layer1.trainParam.epochs = 100;
layer1.trainParam.min_grad=0.0000000001;
layer1 = train(layer1,xTrain,CorrectedxTrain,'useGPU','yes','showResources','yes');
%% Visualizing the Results from the first layer
% After training the 1 st layer, you can gain an insight into the features
% it has learned by visualizing them. Each neuron in the hidden layer will
% have a vector of weights associated with it in the input layer which will
% be tuned to respond to a particular visual feature. By reshaping these
% weight vectors, we can view a representation of these features.
W1 = layer1.IW{1};
weightsImage = helperWeightsToImageGallery(W1,imageHeight,imageWidth,10,10);
imshow(weightsImage);
vmes=layer1(xTrain);
%% drugi sloj
% Create the network. You can experiment by changing the number of training
% epochs, and the training function
layer2= perceptron;
layer2.trainFcn = 'trainscg';
layer2.trainParam.epochs = 1;
% Do not use process functions at the input or output
layer2.inputs{1}.processFcns = {};
layer2.outputs{1}.processFcns = {};
% Set the transfer function for both layers to the logistic sigmoid
layer2.layers{1}.transferFcn = 'satlin';%satlin,logsig
% Use all of the data for training
layer2.divideFcn = 'dividerand';%dividetrain,dividerand
layer2.performFcn = 'mse';
%%
% You now train the 2nd layer. It should be noted that for an
% bipropagation, the input data and target data are almost identical.
CorrectedxTrain=0.9999998*vmes+correction*2;
% Train the autoencoder
layer2 = train(layer2,xTrain,xTrain);
%pause(2);
layer2.IW{1,1}=eye(784);
layer2.b{1}=zeros(784,1);
rezultat=layer2(xTrain)-xTrain;
layer2.trainParam.epochs = 100;
layer2.trainParam.min_grad=0.0000000001;
layer2 = train(layer2,xTrain,CorrectedxTrain,'useGPU','yes','showResources','yes');
%% Visualizing the Results from the first layer
% After training the layer, you can gain an insight into the features
% it has learned by visualizing them. Each neuron in the hidden layer will
% have a vector of weights associated with it in the input layer which will
% be tuned to respond to a particular visual feature. By reshaping these
% weight vectors, we can view a representation of these features.
W1 = layer2.IW{1,1};
weightsImage = helperWeightsToImageGallery(W1,imageHeight,imageWidth,10,10);
imshow(weightsImage);
vmes2=layer2(xTrain);
%% Training the final Softmax Layer
% You will create a softmax layer, and train it on the output from the
% hidden layer of the second layerr. As the softmax layer only
% consists of one layer, you create it manually.
% Create an empty network
finalSoftmax = network;
% Set the number of inputs and layers
finalSoftmax.numInputs = 1;
finalSoftmax.numLayers = 1;
% Connect the 1st (and only) layer to the 1st input, and connect the 1st
% layer to the output
finalSoftmax.inputConnect(1,1) = 1;
finalSoftmax.outputConnect = 1;
% Add a connection for a bias term to the first layer
finalSoftmax.biasConnect = 1;
% Set the size of the input and the 1st layer
finalSoftmax.inputs{1}.size = hiddenSize1;
finalSoftmax.layers{1}.size = 10;
% Use the softmax transfer function for the first layer
finalSoftmax.layers{1}.transferFcn = 'softmax';
% Use all of the data for training
finalSoftmax.divideFcn = 'dividerand';%dividetrain,dividerand
% Use the cross-entropy performance function
finalSoftmax.performFcn = 'crossentropy';
% You can experiment by the number of training epochs and the training
% function
finalSoftmax.trainFcn = 'trainscg';
finalSoftmax.trainParam.epochs = 200;
%%
% Next, you train the softmax layer.
finalSoftmax = train(finalSoftmax,vmes2,tTrain);
%%
% You join these layers together to form a multilayer neural network. You
% create the neural network manually, and then configure the settings, and
% copy the weights and biases from the bipropagation layers
% and softmax layer.
% Create an empty network
finalNetwork = network;
% Specify one input and three layers
finalNetwork.numInputs = 1;
finalNetwork.numLayers = 3;
% Connect the 1st layer to the input
finalNetwork.inputConnect(1,1) = 1;
% Connect the 2nd layer to the 1st layer
finalNetwork.layerConnect(2,1) = 1;
finalNetwork.layerConnect(3,2) = 1;
% Connect the output to the 3rd layer
finalNetwork.outputConnect(1,3) = 1;
% Add a connection for a bias term for each layer
finalNetwork.biasConnect = [1; 1; 1];
% Set the size of the input
finalNetwork.inputs{1}.size = inputSize;
% Set the size of the first layer to the same as the layer in autoencHid1
finalNetwork.layers{1}.size = hiddenSize1;
finalNetwork.layers{2}.size = hiddenSize1;
% Set the size of the third layer to the same as the layer in finalSoftmax
finalNetwork.layers{3}.size = 10;
% Set the transfer function for the first layer to the same as in
% autoencHid1
finalNetwork.layers{1}.transferFcn = layer1.layers{1}.transferFcn;%'satlin';
finalNetwork.layers{2}.transferFcn = layer2.layers{1}.transferFcn;%satlin
% Set the transfer function for the third layer to the same as in
% finalSoftmax
finalNetwork.layers{3}.transferFcn = 'softmax';
% Use all of the data for training
finalNetwork.divideFcn = 'dividetrain';
% Copy the weights and biases from the three networks that have already
% been trained
finalNetwork.IW{1,1} = layer1.IW{1,1};
finalNetwork.b{1} = layer1.b{1,1};
finalNetwork.LW{2,1} = layer2.IW{1,1};
finalNetwork.b{2} = layer2.b{1,1};
finalNetwork.LW{3,2} = finalSoftmax.IW{1,1};
finalNetwork.b{3} = finalSoftmax.b{1,1};
% Use the cross-entropy performance function
finalNetwork.performFcn = 'crossentropy';
% You can experiment by changing the number of training epochs and the
% training function
finalNetwork.trainFcn = 'trainscg';
finalNetwork.trainParam.epochs = 350;
finalNetwork.trainParam.min_grad=0.0000000000000001;
%%
% You can view a diagram of the multilayer network with the |view| command.
view(finalNetwork);
%%
% With the full deep network formed, you can compute the results on the
% test set. Before you can do this, you have to reshape the test images
% into a matrix, as was done for the training set.
% Load the test images
[xTestImages, tTest] = digittest_dataset;
% Turn the test images into vectors and put them in a matrix
xTest = zeros(inputSize, numel(xTestImages));
for i = 1:numel(xTestImages)
xTest(:,i) = xTestImages{i}(:);
end
%%
% You can visualize the results with a confusion matrix. The numbers in the
% bottom right hand square of the matrix will give the overall accuracy.
y = finalNetwork(xTest);
plotconfusion(tTest,y);
%% Fine tuning the Deep Neural Network
% The results for the deep neural network can be improved by performing
% backpropagation on the whole multilayer network. This process is often
% referred to as fine tuning.
%
% You fine tune the network by retraining it on the training data in a
% supervised fashion. You then view the results again using a confusion
% matrix.
finalNetwork = train(finalNetwork,xTrain,tTrain,'useGPU','yes','showResources','yes');
y = finalNetwork(xTest);
plotconfusion(tTest,y);
%% Summary
% This example showed how to train a deep neural network to classify digits
% in images using the Neural Network Toolbox(TM). The steps that have been
% outlined could be applied to other similar problems such as classifying
% images of letters, or even small images of objects of a specific
% category.
Download demo
======================================================
%% Training a Deep Neural Network for Digit Classification
% This example shows how to use the Neural Network Toolbox(TM) to train a
% deep neural network to classify images of digits and is very similar to
% "AutoencoderDigitsExample.m" which is included in
% Neuronal Network ToolBox from MatLAB. This example is made for comparison
% of both algorithms.
%
% Neural networks with multiple hidden layers can be useful for solving
% classification problems with complex data, such as images. Each layer can
% learn features at a different level of abstraction. However, training
% neural networks with multiple hidden layers can be difficult in practice.
%
% One way to effectively train a neural network with multiple layers is by
% training one layer at a time. You can achieve this by training separate
% hidden layers with biopropagation algorithm.
%
% This example shows you how to train a neural network with two hidden
% layers to classify digits in images. First you train the hidden layers
% individually. Then you train a final softmax layer, and join the layers
% together to form a deep network, which is trained one final time.
%
% Copyright 2016, Bojan PLOJ PhD.
%% Data Set
% This example uses synthetic data throughout, for training and testing.
% The synthetic images have been generated by applying random affine
% transformations to digit images created using different fonts.
%
% Each digit image is 28-by-28 pixels, and there are 5,000 training
% examples. You can load the training data, and view some of the images.
% Load the training data into memory
[xTrainImages, tTrain] = digittrain_dataset;
%%
% The labels for the images are stored in a 10-by-5000 matrix, where in
% every column a single element will be 1 to indicate the class that the
% digit belongs to, and all other elements in the column will be 0. It
% should be noted that if the tenth element is 1, then the digit image is a
% zero.
%
% To use the images for training a neural network, you have to arrange them
% into a matrix where each column represents a single image. You can do
% this by stacking the columns of an image to form a vector, and then
% forming a matrix from these vectors.
% Get the number of pixels in each image
imageWidth = 28;
imageHeight = 28;
inputSize = imageWidth*imageHeight;
% Turn the training images into vectors and put them in a matrix
xTrain = zeros(inputSize, numel(xTrainImages));
for i = 1:numel(xTrainImages)
xTrain(:,i) = xTrainImages{i}(:);
end
% Now that the training images have been arranged into a matrix, you are
% ready to begin training the network.
%% Training the first layer
%
% Bipropagation is an algorithm which attempts gradualy transform the
% input signal thrue the layers to the desired output signal.
% The size of individual layers usualy remain the same as is the size of
% the input. Only few outputs are slightly diferent than inputs according
% to the belonging class of pattern. Weight matrix
% started with values of eye matrix. Thus, the learning error is
% very small from the beginning on and learnin run very fast.
% Set the size of the hidden layer for hidden layers.
hiddenSize1 = 784;
% Create the network. You can experiment by changing the number of training
% epochs, and the training function
layer1 = perceptron;
layer1.trainFcn = 'trainscg';
layer1.trainParam.epochs = 1;
% Do not use process functions at the input or output
layer1.inputs{1}.processFcns = {};
layer1.outputs{1}.processFcns = {};
% Set the transfer function for both layers to the logistic sigmoid
layer1.layers{1}.transferFcn = 'satlin';
% Use all of the data for training
layer1.divideFcn = 'dividerand';%dividetrain,dividerand
layer1.performFcn = 'mse';
%%
% You now train the 1st layer. It should be noted that for the
% Bipropagation, the input data and target data are almost identical.
% Correction of xTrain according to the patterns class is:
correction=zeros(inputSize,5000);
for i=1:5000
for j=1:10
correction(78*(j-1)+1:78*j,i)=tTrain(j,i)/10000000;
end
end
CorrectedxTrain=0.9999999*xTrain+correction*1;
layer1 = train(layer1,xTrain,xTrain);
layer1.IW{1,1}=eye(784);
layer1.b{1}=zeros(784,1);
rezultat=layer1(xTrain)-xTrain;
layer1.trainParam.epochs = 100;
layer1.trainParam.min_grad=0.0000000001;
layer1 = train(layer1,xTrain,CorrectedxTrain,'useGPU','yes','showResources','yes');
%% Visualizing the Results from the first layer
% After training the 1 st layer, you can gain an insight into the features
% it has learned by visualizing them. Each neuron in the hidden layer will
% have a vector of weights associated with it in the input layer which will
% be tuned to respond to a particular visual feature. By reshaping these
% weight vectors, we can view a representation of these features.
W1 = layer1.IW{1};
weightsImage = helperWeightsToImageGallery(W1,imageHeight,imageWidth,10,10);
imshow(weightsImage);
vmes=layer1(xTrain);
%% drugi sloj
% Create the network. You can experiment by changing the number of training
% epochs, and the training function
layer2= perceptron;
layer2.trainFcn = 'trainscg';
layer2.trainParam.epochs = 1;
% Do not use process functions at the input or output
layer2.inputs{1}.processFcns = {};
layer2.outputs{1}.processFcns = {};
% Set the transfer function for both layers to the logistic sigmoid
layer2.layers{1}.transferFcn = 'satlin';%satlin,logsig
% Use all of the data for training
layer2.divideFcn = 'dividerand';%dividetrain,dividerand
layer2.performFcn = 'mse';
%%
% You now train the 2nd layer. It should be noted that for an
% bipropagation, the input data and target data are almost identical.
CorrectedxTrain=0.9999998*vmes+correction*2;
% Train the autoencoder
layer2 = train(layer2,xTrain,xTrain);
%pause(2);
layer2.IW{1,1}=eye(784);
layer2.b{1}=zeros(784,1);
rezultat=layer2(xTrain)-xTrain;
layer2.trainParam.epochs = 100;
layer2.trainParam.min_grad=0.0000000001;
layer2 = train(layer2,xTrain,CorrectedxTrain,'useGPU','yes','showResources','yes');
%% Visualizing the Results from the first layer
% After training the layer, you can gain an insight into the features
% it has learned by visualizing them. Each neuron in the hidden layer will
% have a vector of weights associated with it in the input layer which will
% be tuned to respond to a particular visual feature. By reshaping these
% weight vectors, we can view a representation of these features.
W1 = layer2.IW{1,1};
weightsImage = helperWeightsToImageGallery(W1,imageHeight,imageWidth,10,10);
imshow(weightsImage);
vmes2=layer2(xTrain);
%% Training the final Softmax Layer
% You will create a softmax layer, and train it on the output from the
% hidden layer of the second layerr. As the softmax layer only
% consists of one layer, you create it manually.
% Create an empty network
finalSoftmax = network;
% Set the number of inputs and layers
finalSoftmax.numInputs = 1;
finalSoftmax.numLayers = 1;
% Connect the 1st (and only) layer to the 1st input, and connect the 1st
% layer to the output
finalSoftmax.inputConnect(1,1) = 1;
finalSoftmax.outputConnect = 1;
% Add a connection for a bias term to the first layer
finalSoftmax.biasConnect = 1;
% Set the size of the input and the 1st layer
finalSoftmax.inputs{1}.size = hiddenSize1;
finalSoftmax.layers{1}.size = 10;
% Use the softmax transfer function for the first layer
finalSoftmax.layers{1}.transferFcn = 'softmax';
% Use all of the data for training
finalSoftmax.divideFcn = 'dividerand';%dividetrain,dividerand
% Use the cross-entropy performance function
finalSoftmax.performFcn = 'crossentropy';
% You can experiment by the number of training epochs and the training
% function
finalSoftmax.trainFcn = 'trainscg';
finalSoftmax.trainParam.epochs = 200;
%%
% Next, you train the softmax layer.
finalSoftmax = train(finalSoftmax,vmes2,tTrain);
%%
% You join these layers together to form a multilayer neural network. You
% create the neural network manually, and then configure the settings, and
% copy the weights and biases from the bipropagation layers
% and softmax layer.
% Create an empty network
finalNetwork = network;
% Specify one input and three layers
finalNetwork.numInputs = 1;
finalNetwork.numLayers = 3;
% Connect the 1st layer to the input
finalNetwork.inputConnect(1,1) = 1;
% Connect the 2nd layer to the 1st layer
finalNetwork.layerConnect(2,1) = 1;
finalNetwork.layerConnect(3,2) = 1;
% Connect the output to the 3rd layer
finalNetwork.outputConnect(1,3) = 1;
% Add a connection for a bias term for each layer
finalNetwork.biasConnect = [1; 1; 1];
% Set the size of the input
finalNetwork.inputs{1}.size = inputSize;
% Set the size of the first layer to the same as the layer in autoencHid1
finalNetwork.layers{1}.size = hiddenSize1;
finalNetwork.layers{2}.size = hiddenSize1;
% Set the size of the third layer to the same as the layer in finalSoftmax
finalNetwork.layers{3}.size = 10;
% Set the transfer function for the first layer to the same as in
% autoencHid1
finalNetwork.layers{1}.transferFcn = layer1.layers{1}.transferFcn;%'satlin';
finalNetwork.layers{2}.transferFcn = layer2.layers{1}.transferFcn;%satlin
% Set the transfer function for the third layer to the same as in
% finalSoftmax
finalNetwork.layers{3}.transferFcn = 'softmax';
% Use all of the data for training
finalNetwork.divideFcn = 'dividetrain';
% Copy the weights and biases from the three networks that have already
% been trained
finalNetwork.IW{1,1} = layer1.IW{1,1};
finalNetwork.b{1} = layer1.b{1,1};
finalNetwork.LW{2,1} = layer2.IW{1,1};
finalNetwork.b{2} = layer2.b{1,1};
finalNetwork.LW{3,2} = finalSoftmax.IW{1,1};
finalNetwork.b{3} = finalSoftmax.b{1,1};
% Use the cross-entropy performance function
finalNetwork.performFcn = 'crossentropy';
% You can experiment by changing the number of training epochs and the
% training function
finalNetwork.trainFcn = 'trainscg';
finalNetwork.trainParam.epochs = 350;
finalNetwork.trainParam.min_grad=0.0000000000000001;
%%
% You can view a diagram of the multilayer network with the |view| command.
view(finalNetwork);
%%
% With the full deep network formed, you can compute the results on the
% test set. Before you can do this, you have to reshape the test images
% into a matrix, as was done for the training set.
% Load the test images
[xTestImages, tTest] = digittest_dataset;
% Turn the test images into vectors and put them in a matrix
xTest = zeros(inputSize, numel(xTestImages));
for i = 1:numel(xTestImages)
xTest(:,i) = xTestImages{i}(:);
end
%%
% You can visualize the results with a confusion matrix. The numbers in the
% bottom right hand square of the matrix will give the overall accuracy.
y = finalNetwork(xTest);
plotconfusion(tTest,y);
%% Fine tuning the Deep Neural Network
% The results for the deep neural network can be improved by performing
% backpropagation on the whole multilayer network. This process is often
% referred to as fine tuning.
%
% You fine tune the network by retraining it on the training data in a
% supervised fashion. You then view the results again using a confusion
% matrix.
finalNetwork = train(finalNetwork,xTrain,tTrain,'useGPU','yes','showResources','yes');
y = finalNetwork(xTest);
plotconfusion(tTest,y);
%% Summary
% This example showed how to train a deep neural network to classify digits
% in images using the Neural Network Toolbox(TM). The steps that have been
% outlined could be applied to other similar problems such as classifying
% images of letters, or even small images of objects of a specific
% category.
Komentarji
Objavite komentar