The Wisdom of Crowds vs The Black Swan

I am fascinated by the potential of dynamically integrating data. No surprise here I suppose as WorldAPP's business is focused on developing IDEAS (Integrated Data Engineering Applications).

There are some dimensions to data integration however that are truly unique to this moment in man's history. And it all has to do with the web.

One is the compression of time between events and another is a universally accessible compendium of collective responses. A frivolous example of this would be millions of fans voting on American Idol during the show. A more gripping example, one that I have availed of, are the real time tweets coming in from the streets of Tehran as terrified Iranian citizens are being beaten up by the notoriously brutal Basij militia.

Sitting by the quiet waters of Rhode Island, I now know what is happening on an exact street in Tehran as it is happening.

Creatively integrating such dynamic data into meaningful composites will be the great "game" of this new "Thumbs" generation. Many amazingly fertile amalgams will be happening both across social networks and within business processes.

My interest in all of this does not rest solely at the data engineering flash point, but also at the data validation point. Gathering, integrating and presenting data from multiple data repositories is becoming extraordinarily simple. However, making wise decisions about the "meaning" of this data has not seen the equivalent exponential advance.

Technology is limited here. At the end of the day, computers are still dumb. We are still sniffing our way out of the cave and trying to sense the whereabouts of menacing predators.

Some interesting efforts however are now surfacing to try and leverage this "wisdom of crowds" in order to ratchet down our level of risk. Piqqem being one. Piqqem is an online stock selection service that leverages the wisdom of crowds. Launched last year by early-stage Apple investor Mike Markula and TellMe Networks founder Mike McCue, Piqqem encourages users to make predictions on stock prices with simple graphical icons or using online graphing charts.

Nice IDEA. From such "wisdom", the theory is that we will get qualitatively better stock predictions.

However there is always another side to the coin.

Such an effort would be vocally disputed by Nassim Taleb, author of "The Black Swan." Taleb contends that in such deliberations we will always omit "the highly improbable consequential event". Or more precisely our inability to predict outliers will always imply our inability to predict the course of history. (The current global meltdown would one such example that no one saw coming).

Data being such a beautifully fungible commodity, who is right in this dispute, is probably less important to WorldAPP, and to you, than what is pragmatically the best course of action.

Being a battle-scarred entrepreneur, I tend to side with Taleb, and would advise you don't bet the farm, even if you have conjured up the most elegant IDEA the world has ever seen.