Bayesian “Intuition”

As search engines strive to produce ever more human-like results for reasons such as recommendations, they would do well to effectively utilize Bayesian style correlation measures and conditional probability.  Psychological research done collaboratively between two professors at Brown University and MIT, (discussed in The Economist here) has discovered that the mind can produce significantly accurate models of correlation with only one variable if the proper priors are known. These conditional values were not given explicitly but rather appear to have been formed from the diversity of knowledge accumulated by people throughout their lives.  Humans were able to effectively give values for data not just adhering to normal distributions but “are also the Poisson distribution, the Erlang distribution, the power-law distribution and many even weirder ones that are not the consequence of simple mathematical equations”

The strength of computers and networks often discussed is their adeptness with large numbers (or nodes) and the links between them. As algorithms are sought that more accurately reflect human thought it seems they will need to be programmed to emulate what we would often categorize as intuition. This “knowing” without statistical proof is a pervading quality of human thought, and it seems in many numerical instances to be fairly accurate. These unknown but accurate prior values may be able to be derived from the sheer number of successful search results since these are of course provided by human minds. However there may be unknown prior values that have evaded the grasp of data because of the uniqueness of their linkage. Instead these may be hiding in a multitude of currently uncorrelated long tails. Whichever search engine best derives the prior values of correlation for the searches with a low number of instances may prove to provide the most accurate, here meaning the most human, responses to our queries.