Unit-03/Lecture-01

Counterpropagation Network: (Jun-2012)

Counterpropagation Network is supervised neural network that can be used for multimodal processing, but is not trained using the backpropagation rule is the counterpropagation network. This network has been specifically developed to provide bidirectional mapping between input and output training patterns. For this reason it can be used in multimodal processing to associate different modal data sets. Counterpropagation networks typically converge much more quickly than multilayer perceptron neural networks. Consequently, counterpropagation networks are preferred over multilayer perceptron neural networks when the data sets to be processed are large and time is critical.

The counterpropagation network architecture is an example of the third category of supervised crossmodal neural networks in which there are no separate modal layers, and all processing is carried out by a common multimodal hidden layer. In the counterpropagation network, the hidden layer is trained using Kohonen’s self-organising learning rule and the output layer, which maps the output of the hidden layer to target output values, is trained using Grossberg’s outstar learning algorithm. The hidden layer is referred to as the Kohonen layer whilst the output layer is referred to as the Grossberg layer.

There are two types of counterpropagation networks: full and forward-only. Full counterpropagation networks are designed to learn bidirectional mappings between two sets of vectors whilst forward-only counterpropagation networks are trained to provide the mapping in only one direction. We consider the training of the forward-only network first, followed later by a description of the full counterpropagation network.

Forward-only counterpropagation

Kohonen’s Layer Grossberg’s Layer

Figure : Forward only counterpropagation architecture

To train the forward-only counterpropagation network (Figure ), examples of the desired

mapping are presented to the network. Each example consists of the input vector = Ân´1 x and output vector =Âm´1 y . The weights in the Kohonen and Grossberg layer are trained independently.

First, input vectors are applied to the network, and the neurons in the hidden layer are trained through Kohonen’s self-organising learning. For each input, the neuron with weights closest to the input pattern wins, and its weight vector wj is updated in accordance with the equation.

S.NO	RGPV QUESTIONS	Year	Marks
Q.1	What are the difference in learning approach of Counter Propagation Network (CPN) to feed forward network?	Jun-2012	10
Q.2	Explain function and characters of counter propagation network	Jun-2014	7

Unit-03/Lecture-02

Full-counterpropagation

The full counterpropagation network differs from the forward only counterpropagation network in that both the hidden layer and output layer have two sets of weights, one for the x input vectors and the other set for the y output vectors.

Figure : Full counterpropagation network (adapted from Ham and Kostanic, 2001)

The Kohonen learning rule is used to update the hidden layer neuron weights .

As in the forward counterpropagation network, the winner generates an output 1, with the output of the other neurons being set to 0.

are learning rate parameters and i is the index of the winning neuron.

As in the forward counter propagation network, the winner generates an output 1, with the output of the other neurons being set to 0.

Forward-only counterpropagation

Figure : Forward only counterpropagation architecture

To train the forward-only counterpropagation network examples of the desired mapping are presented to the network. Each example consists of the input vector x and output vector = y . The weights in the Kohonen and Grossberg layer are trained independently. First, input vectors are applied to the network, and the neurons in the hidden layer are trained through Kohonen’s self-organising learning. For each input, the neuron with weights closest to the input pattern wins, and its weight vector wj is updated in accordance with the equation:

(1)

where a is the learning rate. Other neurons in the hidden layer do not adjust their weights.

After the Kohonen layer has been trained, input-output vector pairs are then applied to the network,and the output layer is trained in accordance with Grossberg’s learning rule. For each applied input,winner take all competition ensues between the Kohonen layer neurons and the winner generates anoutput 1, with the output of the other neurons being set to 0. The weights in the output layer are then updated in accordance with the Grossberg learning rule:

(2)

where u_ji is the weight between the jt^h second layer neuron and the i neuron in the output layer; b is the learning rate parameter for the output layer, and i z is the output of the th i neuron in the Kohonen layer.

S.NO	RGPV QUESTIONS	Year	Marks
Q.1	Full CPN is more efficient than the forward only CPN .Justified?	Jun-2013	10
Q.2

Unit-03/Lecture-03

Hopfield network ;(Jun-2013)

A Hopfield network is a form of recurrent artificial neural network invented by John Hopfield in 1982. Hopfield nets serve as content-addressable memory systems with binary threshold nodes. A Hopfield network is a class of artificial neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behaviour. Unlike feedforward neural networks, RNNs can use their internal memory to process arbitrary sequences of inputs. This makes them applicable to tasks such as unsegmented connected handwriting recognition, where they have achieved the best known results. Hopfield networks also provide a model for understanding human memory.

CONFIGURATION

The units in Hopfield nets are binary threshold units, i.e. the units only take on two different values for their states and the value is determined by whether or not the units' input exceeds their threshold. Hopfield nets normally have units that take on values of 1 or -1, and this convention will be used throughout the article. However, other literature might use units that take values of 0 and 1.

Every pair of units i and j in a Hopfield network have a connection that is described by the connectivity weight . In this sense, the Hopfield network can be formally described as a complete undirected graph , where is a set of McCulloch-Pitts neurons and is a function that links pairs of nodes to a real value, the connectivity weight.

The connections in a Hopfield net typically have the following restrictions:

· (no unit has a connection with itself)

· (connections are symmetric)

The requirement that weights be symmetric is typically used, as it will guarantee that the energy function decreases monotonically while following the activation rules, and the network may exhibit some periodic or chaotic behaviour if non-symmetric weights are used. However, Hopfield found that this chaotic behavior is confined to relatively small parts of the phase space, and does not impair the network's ability to act as a content-addressable associative memory system

Updation of Weight

Updating one unit (node in the graph simulating the artificial neuron) in the Hopfield network is performed using the following rule:

where:

·  is the strength of the connection weight from unit j to unit i (the weight of the connection).

·  is the state of unit j.

·  is the threshold of unit i.

Updates in the Hopfield network can be performed in two different ways:

· Asynchronous: Only one unit is updated at a time. This unit can be picked at random, or a pre-defined order can be imposed from the very beginning.

· Synchronous: All units are updated at the same time. This requires a central clock to the system in order to maintain synchronization. This method is less realistic, since biological or physical systems lack a global clock that keeps track of time.

Energy

Hopfield nets have a scalar value associated with each state of the network referred to as the "energy", E, of the network, where:

This value is called the "energy" because the definition ensures that when units are randomly chosen to update, the energy E will either lower in value or stay the same. Furthermore, under repeated updating the network will eventually converge to a state which is a local minimum in the energy function (which is considered to be a Lyapunov function). Thus, if a state is a local minimum in the energy function, it is a stable state for the network. Note that this energy function belongs to a general class of models in physics, under the name of Ising models; these in turn are a special case of Markov networks, since the associated probability measure, the Gibbs measure, has the Markov property

In this Fig Energy Landscape of a Hopfield Network, highlighting the current state of the network (up the hill), an attractor state to which it will eventually converge, a minimum energy level and a basin of attraction shaded in green. Note how the update of the Hopfield Network is always going down in Energy.

S.NO	RGPV QUESTIONS	Year	Marks
Q.1	the limitation and application of Hopfield Network and Boltzman machine.	Jun-2013	10

Unit-03/Lecture-04

Learning rules

There are various different learning rules that can be used to store information in the memory of the Hopfield Network. It is desirable for a learning rule to have both of the following two properties:

· Local: A learning rule is local if each weight is updated using information available to neurons on either side of the connection that is associated with that particular weight.

· Incremental: New patterns can be learned without using information from the old patterns that have been also used for training. That is, when a new pattern is used for training, the new values for the weights only depend on the old values and on the new pattern.^[1]

These properties are desirable, since a learning rule satisfying them is more biologically plausible. For example, since the human brain is always learning new concepts, one can reason that human learning is incremental. A learning system that would not be incremental would generally be trained only once, with a huge batch of training data.

Hebbian learning rule for Hopfield networks

The Hebbian Theory was introduced by Donald Hebb in 1949, in order to explain "associative learning", in which simultaneous activation of neuron cells leads to pronounced increases in synaptic strength between those cells.^[2] It is often summarized as "Neurons that fire together, wire together. Neurons that fire out of sync, fail to link".

The Hebbian rule is both local and incremental. For the Hopfield Networks, it is implemented in the following manner, when learning binary patterns:

where represents bit i from pattern .

If the bits corresponding to neurons i and j are equal in pattern , then the product will be positive. This would, in turn, have a positive effect on the weight and the values of i and j will tend to become equal. The opposite happens if the bits corresponding to neurons i and j are different

S.NO	RGPV QUESTIONS	Year	Marks
Q.1		Jun-2014	7
Q.2		Jun-2014	7
Q.3		Jun-2013	10

Unit-03/Lecture-05

Associative Memories : (Jun-2012,14)

Description A content-addressable memory is a type of memory that allows for the recall of data based on the degree of similarity between the input pattern and the patterns stored in memory. It refers to a memory organization in which the memory is accessed by its content as opposed to an explicit address like in the traditional computer memory system. Therefore, this type of memory allows the recall of information based on partial knowledge of its contents. Suppose we are given a memory of names of several people as shown in the figure below. If the given memory is content-addressable, using the erroneous string "Crhistpher Columbos" as key is sufficient to retrieve the correct name "Christopher Colombus." In this sense, this type of memory is robust and fault-tolerant, as this type of memory exhibits some form of error-correction capability

Fig : A Content-addressable memory , Input and Output

An associative memory is a content-addressable structure that maps specific input representations to specific output representations. It is a system that “associates” two patterns (X, Y) such that when one is encountered, the other can be recalled. Typically, XÎ {-1, +1}m, Y Î {-1, +1}n and m and n are the length of vectors X and Y, respectively. The components of the vectors can be thought of as pixels when the two patterns are considered as bitmap images. There are two classes of associative memory: autoassociative and heteroassociative. An autoassociative memory is used to retrieve a previously stored pattern that most closely resembles the current pattern, i.e., X = Y. On the other hand, in a hetero associative memory, the retrieved pattern is, in general, different from the input pattern not only in content but possibly also different in type and format, i.e., X ¹Y. Artificial neural networks can be used as associative memories. One of the simplest artificial neural associative memory is the linear associator. The Hopfield model and bidirectional associative memory (BAM) models are some of the other popular artificial neural network models used as associative memories.

Linear Associator

The linear associator is one of the simplest and first studied associative memory model. Below is the network architecture of the linear associator.

Linear Associator

It is a feedforward type network where the output is produced in a single feedforward computation. In the figure, all the m input units are connected to all the n output units via the connection weight matrix W = [wij]m x n where wij denotes the synaptic strength of the unidirectional connection from the ith input unit to the jth output unit. It is the connection weight matrix that stores the p different associated pattern pairs {(Xk, Yk) | k = 1, 2, ..., p} where Xk Î {-1, +1}m and Yk Î {-1, +1}n in a distributed representation.

S.NO	RGPV QUESTIONS	Year	Marks
Q.1.	What are the short notes on the following: (i) Associative Memory (ii) Hopfiled network (iii) Recuurent Network (iv) Boltzman machine	Jun-2012	10
Q.2	Explain the working of Associative memory?	Jun-2014	7

Unit-03/Lecture-06

Boltzmann machine: (Jun-2012,13)

A Boltzmann machine is a type of stochastic recurrent neural network invented by Geoffrey Hinton and Terry Sejnowski in 1985. Boltzmann machines can be seen as the stochastic, generative counterpart of Hopfield nets. They were one of the first examples of a neural network capable of learning internal representations, and are able to represent and (given sufficient time) solve difficult combinatoric problems. They are theoretically intriguing because of the locality and Hebbian nature of their training algorithm, and because of their parallelism and the resemblance of their dynamics to simple physical processes. Due to a number of issues discussed below, Boltzmann machines with unconstrained connectivity have not proven useful for practical problems in machine learning or inference, but if the connectivity is properly constrained, the learning can be made efficient enough to be useful for practical problems.

They are named after the Boltzmann distribution in statistical mechanics, which is used in their sampling function.

Fig: A graphical representation of an example Boltzmann machine. Each undirected edge represents dependency. In this example there are 3 hidden units and 4 visible units.

Configuration

A graphical representation of a Boltzmann machine with a few weights labeled. Each undirected edge represents dependency and is weighted with weight . In this example there are 3 hidden units (blue) and 4 visible units (white).

A Boltzmann machine, like a Hopfield network, is a network of units with an "energy" defined for the network. It also has binary units, but unlike Hopfield nets, Boltzmann machine units are stochastic. The global energy, , in a Boltzmann machine is identical in form to that of a Hopfield network:

Where:

· is the connection strength between unit and unit .

· is the state, , of unit .

· is the bias of unit in the global energy function. ( is the activation threshold for the unit.)

The connections in a Boltzmann machine have two restrictions:

· . (No unit has a connection with itself.)

· . (All connections are symmetric.)

Often the weights are represented in matrix form with a symmetric matrix , with zeros along the diagonal.

S.NO	RGPV QUESTIONS	Year	Marks
Q.1	What are the short notes on the following: (i) Associative Memory (ii) Hopfiled network (iii) Recuurent Network (iv) Boltzman machine	Jun-2012	10
Q.2	the limitation and application of Hopfield Network and Boltzman machine.	Jun-2013	10

Unit-03/Lecture-07

Training

The units in the Boltzmann Machine are divided into 'visible' units, V, and 'hidden' units, H. The visible units are those which receive information from the 'environment', i.e. our training set is a set of binary vectors over the set V. The distribution over the training set is denoted .

As is discussed above, the distribution over global states converges as the Boltzmann machine reaches thermal equilibrium. We denote this distribution, after we marginalize it over the hidden units, as .

Our goal is to approximate the "real" distribution using the which will be produced (eventually) by the machine. To measure how similar the two distributions are, we use the Kullback–Leibler divergence, :

where the sum is over all the possible states of . is a function of the weights, since they determine the energy of a state, and the energy determines , as promised by the Boltzmann distribution. Hence, we can use a gradient descent algorithm over , so a given weight, is changed by subtracting the partial derivative of with respect to the weight.

There are two phases to Boltzmann machine training, and we switch iteratively between them. One is the "positive" phase where the visible units' states are clamped to a particular binary state vector sampled from the training set (according to ). The other is the "negative" phase where the network is allowed to run freely, i.e. no units have their state determined by external data. Surprisingly enough, the gradient with respect to a given weight, , is given by the very simple equation (proved in Ackley et al.^[1]):

where:

· is the probability of units i and j both being on when the machine is at equilibrium on the positive phase.

· is the probability of units i and j both being on when the machine is at equilibrium on the negative phase.

· denotes the learning rate

This result follows from the fact that at thermal equilibrium the probability of any global state when the network is free-running is given by the Boltzmann distribution(hence the name "Boltzmann machine").Remarkably, this learning rule is fairly biologically plausible because the only information needed to change the weights is provided by "local" information. That is, the connection (or synapse biologically speaking) does not need information about anything other than the two neurons it connects. This is far more biologically realistic than the information needed by a connection in many other neural network training algorithms, such as backpropagation.

The training of a Boltzmann machine does not use the EM algorithm, which is heavily used in machine learning. By minimizing the KL-divergence, it is equivalent to maximizing the log-likelihood of the data. Therefore, the training procedure performs gradient ascent on the log-likelihood of the observed data. This is in contrast to the EM algorithm, where the posterior distribution of the hidden nodes must be calculated before the maximization of the expected value of the complete data likelihood during the M-step.

Training the biases is similar, but uses only single node activity:

S.NO	RGPV QUESTIONS	Year	Marks
Q.1	What are the difference in learning approach of Counter Propagation Network (CPN) to feed forward network?	Jun-2012	10
Q.2

Unit-03/Lecture-08

Adaptive Resonance Theory

The adaptive resonance theory (ART) has been developed to avoid the stability-plasticity dilemma in competitive networks learning. The stability-plasticity dilemma addresses how a learning system can preserve its previously learned knowledge while keeping its ability to learn new patterns. ART architecture models can self-organize in real time producing stable recognition while getting input patterns beyond those originally stored.

Classification

ART is a family of different neural architecture.

ART 1:-

The first and most basic architecture is ART1 (Carpenter and Grossberg, 1987). ART1 can learn and recognize binary patterns. It is the simplest variety of ART networks, accepting only binary inputs. ART 2 ART 2 extends network capabilities to support continuous inputs.

ART 2:-

ART2 (Carpenter and Grossberg, 1987) is a class of architectures categorizing arbitrary sequences of analog input patterns. It is a streamlined form of ART-2 with a drastically accelerated runtime, and with qualitative results being only rarely inferior to the full ART-2 implementation.

ART 3 :-

It builds on ART-2 by simulating rudimentary neurotransmitter regulation of synaptic activity by incorporating simulated sodium (Na+) and calcium (Ca2+) concentrations into the system’s equations, which results in a more physiologically realistic means of partially inhibiting categories that trigger mismatch resets.

Fuzzy ART :-

It implements fuzzy logic into ART’s pattern recognition, thus enhancing generalizability. An optional (and very useful) feature of fuzzy ART is complement coding, a means of incorporating the absence of features into pattern classifications, which goes a long way towards preventing inefficient and unnecessary category proliferation.

ARTMAP :-

It is also known as Predictive ART, combines two slightly modified ART-1 or ART-2 units into a supervised learning structure where the first unit takes the input data and the second unit takes the correct output data, then used to make the minimum possible adjustment of the vigilance parameter in the first unit in order to make the correct classification.

Fuzzy ARTMAP:- It is merely ARTMAP using fuzzy ART units, resulting in a corresponding increase in efficacy. An ART system consists of two subsystems, an attentional subsystem and an orienting subsystem. The stabilization of learning and activation occurs in the attentional subsystem

by matching bottom-up input activation and top-down expectation. The orienting subsystem controls the attentional subsystem when a mismatch occurs in the attentional subsystem. In other words, the orienting subsystem works like a novelty detector.

Properties

An ART system has four basic properties. The first is the self-scaling computational units. The attentional subsystem is based on competitive learning enhancing pattern features but suppressing noise. The second is self-adjusting memory search. The system can search memory in parallel and adaptively change its search order. Third, already learned patterns directly access their corresponding category. Finally, the system can adaptively modulate attentional vigilance using the environment as a teacher. If the environment disapproves the current recognition of the system, it changes this parameter to be more vigilant.

There are two models of ART1, a slow-learning and a fast-learning one. The slow learning model is described by in terms of differential equations while the fast learning model uses the results of convergence in the slow learning model. In this chapter we will not show a full implementation on ART1, instead an implementation of the fast learning model will be more efficient and sufficient to show the ART1 architecture behaviour

S.NO	RGPV QUESTIONS	Year	Marks
Q.1	Briefly describe the Architecture of an ART2 Netrwork	Jun-2013	10
Q.2

Unit-03/Lecture-09

Model Description

ART1 is the simplest ART learning model specifically designed for recognizing binary

patterns. The ART1 system consists of an attentional subsystem and an orienting subsystem as shown in figure

The attentional subsystem consists of two competitive networks, the comparison layer F1 and the recognition layer F2, and two control gains, Gain 1 and Gain 2. The orienting subsystem contains the reset layer for controlling the attentional subsystem overall dynamics.

The comparison layer receives the binary external input passing it to the recognition layer responsible for matching it to a classification category. This result is passed back to the comparison layer to find out if the category matches that of the input vector. If there is a match a new input vector is read and the cycle starts again. If there is a mismatch the orienting system is in charge of inhibiting the previous category in order to get a new category match in the recognition layer. The two gains control the activity of the recognition and comparison layer, respectively.

A processing element x₁i in layer F₁ is shown in below figure

The excitatory input to x1i in layer F1 comes from three sources: (1) the external input vector Ii, (2) the control gain G1and (3) the internal network input V1i made of the output from F2 multiplied appropriate connections weights. There is no inhibitory input to the neuron. The output of the neuron is fed to the F2 layer as well as the orient subsystem.

A processing element x2j in layer F2 is shown in below figure

S.NO	RGPV QUESTIONS	Year	Marks
Q.1	Explain the architecture of Adaptive Resonance Theory (ART) with the help of a Schematic diagram ?	Jun-2012	10
Q.2	Explain the classical ARTnetwork with example?	Jun-2014	7