A role for sleep and dreaming in neural networks

dreamingWhen training neural networks it is a good idea to have a training set which has examples that are randomly ordered. We want to ensure that any sequence of training set examples, long or short, has statistics that are representative of the whole. During training we will be adjusting weights, often by using stochastic gradient descent, and so we ideally would like the source statistics to remain stationary.

During on-line training, such as with a robot, or when people learn, adjacent training examples are highly correlated. Visual scenes have temporal coherence and people spend a long time at specific tasks, such as playing a card game, where their visual input, over perhaps hours, is not representative of the general statistics of natural scenes. During on-line training we would expect that a neural net weights would become artificially biased by having highly correlated consecutive training examples so that the network would not be as effective at tasks requiring balanced knowledge of the whole training set.
Continue reading

Man vs AI: Ethics and the Future of Machines

mind_machine_by_neodecayRecent strides in artificial intelligence from big name players such as Google, Facebook, and Baidu, as well as increasingly successful heterogenous systems like IBM’s Watson have provoked fear and excitement amongst the intelligentsia in equal measures. Public figures, such as Steven Hawking, are concerned, and not surprisingly the popular press are excited to cover it. Recently, Elon Musk has become worried that AI might eventually spell doom for the human race. He donated $10 million to fund the Future of Life organization whose stated goal is to ensure AI remains beneficial and does not threaten our wellbeing. An open letter by this organization, titled “Research Priorities for Robust and Beneficial Artificial Intelligence,” was signed by hundreds of research leaders. Influential futurist, Ray Kurzweil, has popularized the idea of the technological singularity where intelligent systems surpass human capabilities and leave us marginalized at best.
Continue reading

Regression and Conspiracy Theories

RegressionThis post is about fitting a curve through a set of points. This is called regression: It is also the classic problem of coming up with a generalization from a discrete training data set in machine learning. We have a set of points that are observations at specific places and we want to make a system that predicts what the likely observations should be at all places within the domain of interest. We use the given observations (a training set) to train a model and then when we get more observations (a test set) we can evaluate how much error there is in our Continue reading

Keeping Up with Innovation

http://worldaftermidnight.com/wam-video/

I wonder about this.

Watching that video makes me think that the person who made it is probably… getting old.

I don’t think that young people worry that the world is changing too fast. After all what makes the author think that the rules of business suddenly changed 15 years ago. Maybe that was when they started in the business and now they feel they don’t understand things Continue reading

Boundaries and Connections in Physics and Perception

Apple MandalaMany people are familiar with the maxim from chaos theory that the fluttering of a butterfly in one part of the world can lead to a hurricane in another. Causes and effects are ubiquitous. Mystics tell us that one of the basic revelations is that everything is connected, there is no you and me, and we are one with the cosmos. Let us shine the spotlight of science onto this concept and play a little.

As we go about our day doing ordinary things, we think in terms of tasks that we need to do and the objects and tools that will be useful. We talk to other people who seem distinct Continue reading

Formal Indecision

Godel

A formal system in mathematics is a system which contains a set of axioms (unquestionable statements of truth), and then adds to these a set of production rules by which new true statements can be generated. The process of production can go on forever allowing a never-ending list of truths to be derived.

A famous result called Gödel’s incompleteness theorem shows that in such a system, there are some true statements that are nevertheless un-provable (undecidable) using the rules of the system. It shows that a self-consistent Continue reading

The Raven and the Shoe

Raven
The hypothesis that all ravens are black is logically equivalent to the statement that all non black things are non ravens, and this is supported by the observation of a white shoe.

This is a paraphrasing of a famous paradox due to Hempel. There is a lot of fairly impenetrable discussion on the Wikipedia page about this paradox, some of which I believe to be incorrect, and so I include a readable resolution based on a Bayesian perspective here, and I also relate this issue to our ability to know the truth Continue reading