Neural Networks: The Boltzmann Machine Learning Algorithm

A detailed exploration of Lecture 12a, covering the goals of learning, challenges, and surprising facts about weight updates in Boltzmann Machines.

The Goal of Learning in Boltzmann Machines

We aim to maximize the product of probabilities that the Boltzmann machine assigns to the binary vectors in the training set. This is equivalent to maximizing the sum of the log probabilities that the Boltzmann machine assigns to the training vectors.

It is also equivalent to maximizing the probability that we would obtain exactly the N training cases if we did the following: Let the network settle to its stationary distribution N different times with no external input, then sample the visible vector once each time.

Why the Learning Can Be Difficult

Consider a chain of units with visible units at the ends. If the training set consists of (1,0) and (0,1), we desire the product of all weights (w1, w2, w3, w4, w5) to be negative. To determine how to adjust an individual weight like w1 or w5, knowledge of intermediary weights like w3 becomes crucial.

This interconnectedness highlights the complexity of local weight updates in such architectures, making the learning process non-trivial as changes in one weight can have cascading effects across the network.

A Very Surprising Fact

In the context of Boltzmann Machines, a remarkable discovery reveals that everything one weight needs to know about the other weights and the data is encapsulated within the difference of two specific correlations.

This elegant simplification underpins the core of the learning algorithm, providing a powerful insight into how complex probabilistic models can efficiently update their internal parameters based on observed data.

Neural Networks: The Boltzmann Machine Learning Algorithm

The Goal of Learning in Boltzmann Machines

Why the Learning Can Be Difficult

A Very Surprising Fact

Ready to delve into Machine Learning?