The Wisdom of Crowds vs The Black Swan

I am fascinated by the potential of dynamically integrating data. No surprise here I suppose as WorldAPP's business is focused on developing IDEAS (Integrated Data Engineering Applications).

There are some dimensions to data integration however that are truly unique to this moment in man's history. And it all has to do with the web.

One is the compression of time between events and another is a universally accessible compendium of collective responses. A frivolous example of this would be millions of fans voting on American Idol during the show. A more gripping example, one that I have availed of, are the real time tweets coming in from the streets of Tehran as terrified Iranian citizens are being beaten up by the notoriously brutal Basij militia.

Sitting by the quiet waters of Rhode Island, I now know what is happening on an exact street in Tehran as it is happening.

Creatively integrating such dynamic data into meaningful composites will be the great "game" of this new "Thumbs" generation. Many amazingly fertile amalgams will be happening both across social networks and within business processes.

My interest in all of this does not rest solely at the data engineering flash point, but also at the data validation point. Gathering, integrating and presenting data from multiple data repositories is becoming extraordinarily simple. However, making wise decisions about the "meaning" of this data has not seen the equivalent exponential advance.

Technology is limited here. At the end of the day, computers are still dumb. We are still sniffing our way out of the cave and trying to sense the whereabouts of menacing predators.

Some interesting efforts however are now surfacing to try and leverage this "wisdom of crowds" in order to ratchet down our level of risk. Piqqem being one. Piqqem is an online stock selection service that leverages the wisdom of crowds. Launched last year by early-stage Apple investor Mike Markula and TellMe Networks founder Mike McCue, Piqqem encourages users to make predictions on stock prices with simple graphical icons or using online graphing charts.

Nice IDEA. From such "wisdom", the theory is that we will get qualitatively better stock predictions.

However there is always another side to the coin.

Such an effort would be vocally disputed by Nassim Taleb, author of "The Black Swan." Taleb contends that in such deliberations we will always omit "the highly improbable consequential event". Or more precisely our inability to predict outliers will always imply our inability to predict the course of history. (The current global meltdown would one such example that no one saw coming).

Data being such a beautifully fungible commodity, who is right in this dispute, is probably less important to WorldAPP, and to you, than what is pragmatically the best course of action.

Being a battle-scarred entrepreneur, I tend to side with Taleb, and would advise you don't bet the farm, even if you have conjured up the most elegant IDEA the world has ever seen.

Pure genius in data display

I love the challenges we are faced with when customers ask us to develop an IDEA (Integrated Data Engineering Application) to solve difficult business process problems. It’s fascinating stuff. The IDEAs come in all shapes and sizes and often end up inside a sexy data drill down presentation layer using our valued partner QlikView.

But there is one particular technology we haven’t tapped into yet and I am chomping at the bit to see it deployed. It was developed by Gapminder.org and the technology has subsequently been bought by Google (no surprise there).

It’s so amazing; I call it “spawning intelligence”.

The technology displays data over time as it dynamically unfolds along two axis. The video I recommend from Hans Rosling shows how this technology can reveal fresh insights into the relationship between child mortality rates and world population growth over decades of historical data.

The intellectual stimulation from witnessing these variables as they morph through time is a new paradigm of learning and provocation that must be classed as a game changer.

Data is only beginning to have its day in the sun. Pundits and pulpits beware. Facts alone, presented at this level of mesmerizing acuity, are going to rule this new millennium.

You can see Rosling strut his stuff here: http://www.ted.com/index.php/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html

Social Networking - Integrating People Engineering Applications

As you may know, WorldAPP is the originator of IDEAS - Integrated Data Engineering Applications.

But maybe there is room out there for another company to focus on IPEAS 0r Integrated People Engineering Applications?

Sure, it is not nearly as sexy an acronym, but social networking is certainly now the new red hot space. Yesterday over lunch, one of our business analysts was proclaiming how he had just succumbed and become a "Twitter" the previous night. His friend had finally managed to explain to him the value of looking at himself as a "product" and using Twitter to keep those who might be interested in him as a "product" up-to-date – all this in easy-to-digest 140 character bites of information.

Certainly, here at WorldAPP, we are both aware and excited that companies will soon be asking us to integrate relevant information from Linked In, Facebook and other data repositories into their next IDEA.

Data is data after all and it is the quality and value of data, not the database or structure that houses it, that truly matters from a business efficiencies perspective.