|
Unit-02/Lecture-01 |
||||||||
|
Basic Concept of
Neural Network (Jun-2012) Artificial
neural networks (ANNs) are biologically inspired computer programs designed
to simulate the way in which the human brain processes information. ANNs
gather their knowledge by detecting the patterns and relationships in data
and learn (or are trained) through experience, not from programming. An ANN
is formed from hundreds of single units, artificial neurons or processing elements
(PE), connected with coefficients (weights), which constitute the neural
structure and are organised in layers. The power of neural computations comes
from connecting neurons in a network. Each PE has weighted inputs, transfer
function and one output. The behavior of a neural network is determined by
the transfer functions of its neurons, by the learning rule, and by the
architecture itself. The weights are the adjustable parameters and, in that
sense, a neural network is a parameterized system. The weighed sum of the
inputs constitutes the activation of the neuron. The activation signal is
passed through transfer function to produce a single output of the neuron. Neural
neworks are typically organized in layers. Layers are made up of a number of
interconnected 'nodes' which contain an 'activation function'. Patterns are
presented to the network via the 'input layer', which communicates to one or
more 'hidden layers' where the actual processing is done via a system of
weighted 'connections'. The hidden layers then link to an 'output layer'
where the answer is output as shown in the graphic below.
Most ANNs
contain some form of 'learning rule' which modifies the weights of the
connections according to the input patterns that it is presented with. ANNs
learn by example as do their biological counterparts; a child learns to
recognize dogs from examples of dogs. biological
neural network
The brain
is principally composed of about 10 billion neurons, each connected to about
10,000 other neurons. Each of the yellow blobs in the picture above are
neuronal cell bodies (soma), and the lines are the input and output channels
(dendrites and axons) which connect them.
Each neuron receives electrochemical inputs from other neurons at the
dendrites. If the sum of these electrical inputs is sufficiently
powerful to activate the neuron, it transmits an electrochemical signal along
the axon, and passes this signal to the other neurons whose dendrites are
attached at any of the axon terminals. These attached neurons may
then fire. It is important to note that a neuron fires only if the total
signal received at the cell body exceeds a certain level. The neuron
either fires or it doesn't, there aren't different grades of firing. The human
brain can be described as a biological neural networkan interconnected web
of neurons transmitting elaborate patterns of electrical signals. Dendrites
receive input signals and, based on those inputs, fire an output signal via
an axon. Or something like that. How the human brain actually works is an
elaborate and complex mystery, one that we certainly are not going to attempt
to tackle in rigorous detail in this chapter.
A neuron
operates by receiving signals from other neurons through connections, called synapses.
The combination of these signals, in excess of a certain threshold or activation
level, will result in the neuron firing, that is sending a signal on
to other neurons connected to it. Some signals act as excitations and
others as inhibitions to a neuron firing. What we call thinking is
believed to be the collective effect of the presence or absence of firings in
the pattern of synaptic connections between neurons.
OR The
fundamental element of the neural network is called a neuron. As shown in
figure . a neuron mainly consists of three parts: dendrites, soma, and axon.
Dentrites are the tree-like structure that receives the signal from
surrounding neurons, where each line is connected to one neuron. Axon is a
thin cylinder that transmits the signal from one neuron to others. At the end
of axon, the contact to the dendrites is made through a synapse. The
inter-neuronal signal at the synapse is usually chemical diffusion but
sometimes electrical impulses. A neuron fires an electrical impulse only if
certain condition is met The incoming impulse signal from each synapse to the
neuron is either excitatory or inhibitory, which means helping or hindering
firing. The condition of causing firing is that the excitatory signal should
exceed the inhibitory signal by a certain amount in a short period of time,
called the period of latent summation. As we assign a weight to each incoming
impulse signal, the excitatory signal has positive weight and the inhibitory
signal has negative weight. This way, we can say, ``A neuron fires only if
the total weight of the synapses that receive impulses in the period of
latent summation exceeds the threshold."
|
|
Unit-02/Lecture-02 |
||||||||||||
|
Difference
between ANN and human brain(Jun2014)
Characteristics of neural networks (i)
The
NNs exhibit mapping capabilities, that is, they can map input patterns to
their associated output patterns. (ii)
The
NNs learn by examples. Thus, NN architectures can be trained with known
examples of a problem before they are tested for their inference capability
on unknown instances of the problem. They can, therefore, identify new
objects previously untrained. (iii)
The
NNs possess the capability to generalize. Thus, they can predict new
outcomes from past trends. (iv)
The
NNs are robust systems and are fault tolerant. They can, therefore, recall
full patterns from incomplete, partial or noisy patterns. (v)
The
NNs can process information in parallel, at high speed, and in a distributed
manner.
|
|
Unit-02/Lecture-03 |
||||||||||||
|
Single Layer Feed-forward Network (Jun 2014) The Single Layer Feed-forward Network
consists of a single layer of weights
, where the inputs are directly connected to the outputs, via a series of weights. The synaptic links
carrying weights connect every input to every output , but not other way.
This way it is considered a network of
feed-forward type. The sum of the products of the weights and the
inputs is calculated in each neuron
node, and if the value is above some threshold (typically 0) the neuron fires and takes
the activated value (typically
1); otherwise it takes the
deactivated value (typically -1)
Fig: Single layer
feed forward neural network Perceptron
Learning Algorithm(Jun 2014) The perceptron learning rule was
originally developed by Frank Rosenblatt in the late 1950s. Training
patterns are presented to the network's inputs; the output is computed.
Then the connection weights wjare modified by an amount
that is proportional to the product of
The algorithm is as follows:
where
Notice that learning only occurs when
an error is made, otherwise the weights are left unchanged. During training, it is often useful
to measure the performance of the network as it attempts to find the optimal
weight set. A common error measure or cost function used is
sum-squared error. It is computed over all of the input vector/output vector
pairs in the training set and is given by the equation below:
where p is the number of input/output
vector pairs in the training set.
|
|
Unit-02/Lecture-04 |
|
Linear Separability: A linear threshold neuron implements a partition
of input vectors into two classes ( i.e. divides up the input vectors into
two classes or categories depending on their weighted sum value). The
boundary between these classes is defined by: WT X =
b where b is the threshold ( or bias ). The boundary
separating the two classes is a hyperplane in n dimensions.
Graphically:
Widrow Hoff Learning rule : The WIDROW-HOFF Learning rule is very similar
to the perception Learning rule. However the origins are
different. The units with linear activation functions are
called linear units. A network with a single linear unit is called as adaline
(adaptive linear neuron). That is in an ADALINE, the input-output
relationship is linear. Adaline uses bipolar activation for its input signals
and its target output. The weights between the input and the output are
adjustable. Adaline is a net which has only one output unit. The adaline
network may be trained using the delta learning rule. The delta learning rule
may also b called as least mean square (LMS) rule or Widrow-Hoff
rule. This learning rule is found to minimize the mean-squared error between
the activation and the target value Delta Learning rule
Hebb Learning rule: It is an algorithm developed for training of
pattern association networks. The hebb learning rule is widely used for finding
the weights of an associative neural net. The training vector pairs here are
denoted as s:t. The algorithm steps are given below: Step0: set all the initial weights to 0 wij = 0 Step1: for each training target input out output vector pairs s:t, perform steps 2-4 Step2: activate the input layer units to current
training input. xi = si (for i = 1 to n) Step3: activate the output layer units to current
target output, yi = tj (for j = 1 to m) Step4: start the weight adjustment wij(new) = wij (old) + xiyi (for i =1 to n, j = 1
to m) |
|
Unit-02/Lecture-05 |
||||||||
|
ADALINE: (2013 Jun) It is Known as Adaptive Linear Neuron.Adaline
is a network with a single linear unit.The Adaline network is trained using
the delta rule Architecture As already stated Adaline is a
single-unit neuron, which receives input from several units and also from one
unit, called bias. An Adeline model consists of trainable weights. The inputs
are of two values (+1 or -1) and the weights have signs (positive or
negative). Initially random weights are
assigned. The net input calculated is applied to a quantizer transfer
function (possibly activation function) that restores the output to +1 or -1.
The Adaline model compares the actual output with the target output and with
the bias and the adjusts all the weights
Training
Algorithm The Adaline network training
algorithm is as follows: Step 0: weights and bias are to be set to some
random values but not zero. Set the learning rate parameter α. Step1: perform steps 2-6 when stopping condition
is false. Step2: perform steps 3-5 for each bipolar training
pair s:t Step3: set activations foe input units i=1 to n. Step4: calculate the net input to the output unit. Step5: update the weight and bias for i=1 to n Step6: if the highest weight change that
occurred during training is smaller than a specified tolerance then stop the training
process, else continue. This is the test for the stopping condition of a
network. Testing
Algorithm It is very essential to perform the
testing of a network that has been trained. When the training has been
completed, the Adaline can be used to classify input patterns. A step
function is used to test the performance of the network. The testing
procedure for the Adaline network is as follows: Step0: initialize the weights. (The weights are
obtained from the training algorithm.) Step1: perform steps 2-4 for each bipolar input
vector x. Step2: set the activations of the input units to x. Step3: calculate the net input to the output units Step4: apply the activation function over the net
input calculated. Madaline
Architecture It consists of n units of input
layer and m units of adaline layer and 1Unit of the Madaline layer.Each
neuron in the adaline and madaline layers has a bias of excitation 1 The Adaline layer is present between
the input layer and the madaline layer; the adaline layer is considered as
the hidden layer. Uses The use of hidden layer gives the net
computational capability which is not found in the single-layer nets, but
this complicates the training process to some extent. Training Algorithm: In this training algorithm, only the
weights between the hidden layers are adjusted, and the weights for the
output units are fixed. The weights v1, v2
vm and the bias b0 that enter
into output unit Y are determined so that the response of unit Y is 1
|
|
Unit-02/Lecture-06 |
||||||||
|
AI v/s ANN The key difference is that neural
networks are a stepping stone in the search for artificial intelligence. (i)
Artificial intelligence is a vast
field that has the goal of creating intelligent machines, something that has
been achieved many times depending on how you define intelligence. Despite
the fact that we have computers that can win at "Jeopardy" and beat
chess champions, the goal of AI is generally seen as a quest for general
intelligence, or intelligence that can be applied to diverse and unrelated
situational problems. (ii)
Many of the AIs built up to this
point have been built with a purpose, such as running a ping pong playing
robot or dominating at "Jeopardy". This is the inevitable result
when computer scientists sit down and create something to do a specific task
they end up with something that can do that task and not much else. (iii)
To get around this problem of
task-orientated AIs, computer scientists started playing around with
artificial neural networks. Our generally intelligent brains are made up of biological
neural networks that make connections based on our perceptions and outside
stimulus. (iv)
Artificial neural networks try to
recreate this learning system on computers by constructing a simple framework
program to respond to a problem and receive feedback on how it does. A
computer can optimize its response by doing the same problem thousands of
times and adjusting its response according to the feedback it receives. The
computer can then be given a different problem, which it can approach in the
same way as it learned from the previous one. By varying the problems and the
number of approaches to solving them that the computer has learned, computer
scientists can teach a computer to be a generalist. (v)
Although this conjures up images of
computers taking over the world and harvesting humans as seen in Hollywood
movies like "The Martrix," we are still a long way from neural
networking our way to artificial intelligence. The problems being tested on
neural networks are all expressed mathematically. You cant hold a flower up
to a computer and tell it to guess the color by the smell, because the smell
would have to be expressed in numbers and then the computer would have to
catalog those numbers in memory, along with images of flowers emitting that
smell. (vi)
Artificial neural networks that can
be given more inputs of things like smell - and the capacity to learn from
all those inputs may be on track to produce the first artificial
intelligence that meets the standards of even the most hardcore AI
enthusiast. (vii)
In essence, artificial neural
networks are models of human neural networks that are designed to help
computers learn. Artificial intelligence is the Holy Grail some computer
scientists are trying to achieve using techniques like mimicking neural
networks .
|
|
Unit-02/Lecture-07 |
|||||||||
|
The basic element of the Multi Layer
Perceptron (MLP) neural network, is the artificial neuron. An artificial
neuron, is a unit that performs a simple mathematical operation on its
inputs. In the figure below, the neuron is graphically presented.
The input, x, of the neuron
consists of the variables x1 ... xn and a bias term, known as the momentum
constant, which is equal to 1. Each of the input values is multiplied by
a weight, wi , after which the results are added. On the result, a simple
mathematical function, f (x), is performed. This function is also known as
the activation function. The calculations the neuron performs are thus
given by: y = f ( w0 + x1*w1 + ... + xn*wn ) Numerous choices for the functions
exist. Frequently used implementations are the Sigmoid functions: f(u) = 1 / (1 + e-u )
f (u) = c1 * tanh ( c2 * u)
(in the picture c1 = c2 = 1) MLP Network An MLP network, as any type of back-propagation network
can consist of many neurons, which are ordered in layers. The neurons in the
hidden layers do the actual processing, while the neurons in the input and
output layer merely distribute and collect the signals. Although many hidden
layers can be used, it has been shown that an MLP with one hidden layer can
approximate any continuous function. Therefore in 20-sim, the MLP networks
only have one hidden layer. Training the MLP
network The MLP network is trained by
adapting the weights. During training the network output is compared with a
desired output. The error between these two signals is used to adapt the
weights. This rate of adaptation is controlled by the learning rate. A high learning rate will make the network adapt
its weights quickly, but will make it potentially unstable. Setting the
learning rate to zero, will make the network keep its weights constant
|
|
Unit-02/Lecture-08 |
||||||||||||||||||||||||
|
Figure
2‑5 Bipolar sigmoid function Activation functions for the hidden
units are needed to introduce non-linearity into the networks. The reason is
that a composition of linear functions is again a linear function. However,
it is the non-linearity (i.e., the capability to represent nonlinear
functions) that makes multi-layer networks so powerful. Almost any nonlinear
function does the job, although for back-propagation learning it must be
differentiable and it helps if the function is bounded (see Section 3.4).
The sigmoid functions are the most common choices [5]. For the output units, activation
functions should be chosen to be suited to the distribution of the target
values. We have already seen that for binary [0,1] outputs, the sigmoid
function is an excellent choice. For continuous-valued targets with a bounded
range, the sigmoid functions are again useful, provided that either the
outputs or the targets to be scaled to the range of the output activation
function. But if the target values
have no known bounded range, it is better to use an unbounded activation
function, most often the identity function (which amounts to no activation
function). If the target values are positive but have no known upper bound,
an exponential output activation function can be used
|
|
Unit-02/Lecture-09 |
||||||||
|
Backpropagation ( Error back propagation algorithm)
(Jun 2012): an abbreviation for "backward
propagation of errors", is a common method of training artificial neural
networks used in conjunction with an optimization method
such as gradient descent. The method calculates the gradient
of a loss function with respects to all the weights in
the network. The gradient is fed to the optimization method which in turn
uses it to update the weights, in an attempt to minimize the loss function. Backpropagation requires a known,
desired output for each input value in order to calculate the loss function
gradient. It is therefore usually considered to be a supervised learning
method, although it is also used in some unsupervised
networks such as autoencoders. It is a generalization of the delta rule
to multi-layered feedforward networks,
made possible by using the chain rule to iteratively compute gradients for
each layer. Backpropagation requires that the activation function
used by the artificial neurons
(or "nodes") be differentiable. The backpropagation learning
algorithm can be divided into two phases: propagation and weight update. Phase 1:
Propagation Each propagation involves the
following steps:
Phase 2: Weight
update For each weight-synapse follow the
following steps:
This ratio (percentage) influences
the speed and quality of learning; it is called the learning rate. The
greater the ratio, the faster the neuron trains; the lower the ratio, the
more accurate the training is. The sign of the gradient of a weight indicates
where the error is increasing, this is why the weight must be updated in the
opposite direction. Momentum, limitation, characteristics and application of
EBPA A single-layer neural network has
many restrictions. This network can accomplish very limited classes of tasks.
Minsky and Papert (1969) showed that a two layer feed-forward network can
overcome many restrictions, but they did not present a solution to the
problem as "how to adjust the weights from input to hidden layer" ? An answer to this question was
presented by Rumelhart, Hinton and Williams in 1986. The central idea behind
this solution is that the errors for the units of the hidden layer are
determined by back-propagating the errors of the units of the output layer.
This method is often called the Back-propagation learning rule . Back-propagation can also be
considered as a generalization of the delta rule for non-linear activation
functions and multi-layer networks. Back-propagation is a systematic
method of training multi-layer artificial neural networks Real world
is faced with a situations where data is incomplete or noisy. To make reasonable predictions about
what is missing from the information available is a difficult task when
there is no a good theory available that may to help reconstruct the
missing data. It is in such situations the Back-propagation (Back-Prop) networks
may provide some answers. A
BackProp network consists of at least three layers of units : -an input layer, -at least one intermediate hidden
layer, and -an output layer. Typically,
units are connected in a feed-forward fashion with input units fully
connected to units in the hidden layer and hidden units fully connected to
units in the output layer. When
a BackProp network is cycled, an input pattern is propagated forward to the
output units through the intervening input-to-hidden and hidden-to-output
weights. The
output of a BackProp network is interpreted as a classification decision With BackProp networks, learning
occurs during a training phase. The steps followed during learning
are : −
each input pattern in a training set
is applied to the input units and then propagated forward. −
the pattern of activation arriving at
the output layer is compared with the correct (associated) output pattern to
calculate an error signal. −
the error signal for each such target
output pattern is then back-propagated from the outputs to the inputs in
order to appropriately adjust the weights in each layer of the network. −
after a BackProp network has learned
the correct classification for a set of inputs, it can be tested on a second
set of inputs to see how well it classifies untrained patterns
|