Big Data Hype

If you are bored in a meeting with the droning and moaning, you can always play ‘Buzzword Bingo’.  This is a game where you write down the usual cadre of buzzwords being used in industry.  The guru-speak and ‘visionary’, ‘transformational’ leadership gibberish will surely make an appearance in a meeting of any length exceeding 12 minutes.

Big Data is a key Buzzword in the current IT landscape.  ‘Big’ means ‘large’.  Big Data Gurus discuss data being of ‘large volume, moving with velocity’.  Big and Fast.  Big IT firms are selling Big IT Projects around Big IT hype.  Apparently tweets and Farcebook posts need to be ‘data mined’ so we can uncover ‘correlations’ about products and services.  Since firms have enough trouble with structured, table-based data, and scrubbing and cleaning data with defined relationships, it only makes sense (in the fervid minds of sales people) to further degrade data and information by tossing in unstructured, uncorrelated and irrelevant data.  Yes, I am really going to change the strategic direction of my chocolate-making company due to made up correlations from Farcebook and Twitter. Truly.

70% of Big Data Projects fail.  60% of Agile-Dev Ops have no impact on their organizations.  70% or more of ‘Big’ ERP projects failed.  So if the attitude is ‘lets increase our failure rate’, then by all means use Dev-Ops (which most engineers and managers don’t understand) within a ‘Big Data Project’.  It basically doubles the chance of failure.

Question:  Why would I used ‘Big Data’ if I don’t understand my ‘small data’, have not documented my ‘small data’ and have not produced models of my ‘small data’?

A major issue with IT is hyperbole and hype.  You pay Gartner or Forrester Research enough money and you will be placed into a magic box, or given ‘leadership’ status within a sub-domain.  Since Big IT firms have Big stock prices to support and Big director bonuses to pay; they need to up-sell industry and government on the ‘latest’ and ‘most important’ fad.  Apparently the white noise and rubbish on social media is going to ‘transform’ the company, and if we just use ‘Dev ops’ on the ‘Humongous Data’ project, we will be ‘cool’, ‘leading edge’ and able to flick our hair at elite dinner parties whilst saying ‘well of course we are Big Data tolerant darling, and yes we are certainly Agile.’

The reality of data is this:  it has to be relevant, clean, known, related and structured.  Most firms do not understand their pre-Big Data IS.  Throwing in unstructured data via failed projects is a distraction not a benefit.  Using Agile dev-ops in a ‘Big Data’ project,  that doesn’t need such a method, and which is not understood or supported by skills and resources, is just guaranteeing failure.

Having good, usable and relevant data is the whole point of IS.  Not hyperbole, hype and paying large IT firms ‘Humongous’ sums of money for failed ‘Big Data’ endeavors.

Back to basics. Avoid the Hype.