Data is Omnipresent

With the explosion of Social Media channels, even small businesses are capturing large volumes of data. We hoard terabytes of data, but is there something valuable in it? The clutter is mind boggling. A “Big Data” project seems to be what the Doctor ordered. But read on to get a quick primer on what you should care about.

First, let’s quickly define what makes up “Big Data” and what the traits of “Big Data” are.

Structured Data: Generally speaking, anything that you could manipulate in Excel by way of summarizing, aggregating, averaging or be able to apply some computation on – can be called Structured Data.   Sales generate most of it. Financial, CRM and ERP applications have it. Except for very large retail organizations, such data, amounts only to tens and hundreds of gigabytes. Also, the growth of Structured Data is generally in a small incremental steps.

Unstructured Data: Every other type of data can be thought of as Unstructured Data. Note that nearly all of the applications that primarily hold Structured Data contain some Unstructured Data as well. Text comments, item descriptions, customer chat transcripts are common types of Unstructured Data found in such applications.

Unstructured Data also include Emails, application logs, multi-media files, social data such as Facebook posts, tweets, extremely detailed website visitor data that includes mouse movements, clicks, views and machine-generated data.

What makes “Big Data” Big?

It is the Unstructured Data that really makes the difference. This Data is growing exponentially. Even small and medium-sized organizations are seeing data explode around them, primarily driven by Unstructured Data. Every Organization is adding terabytes of data each year. Petabyte scales are becoming common place.

There are 3 characteristics of this Unstructured Data that makes it overwhelming to manage:

• Volume: Websites, application logs, machine sensors and social media generate extremely large amounts of data.
• Velocity: This data comes in fast and furious. As your marketing efforts drive more visitors to your website, more the data comes your way. Additional web pages, logging events, applications – all exponentially increase the velocity of data.
• Variety: Not only is the data unstructured, but is of great variety. Emails, PDFs, audio & video files, images, drawings, make up a mind boggling variety of data.

But to get most out of Big Data, you really need to understand the Value and Veracity, the two additional Vs of the Big Data

The 4th Characteristic – Value:

Depending on the use, certain types of data add more value to the Organization than others. For instance, consider the machine data generated from an Electric Meter. That data is of primary interest to the Electrical distribution company and not the business at which the meter was installed. Similarly, detailed logs from office equipment such as photocopiers and printers are of value for the office equipment maintenance company but not to the businesses; the equipment has been leased to. In short, data that is captured and analyzed should be fully aligned with business needs.

The 5th Characteristic – Veracity:

Veracity refers to the quality of data. Information related to the customer is spread across multiple places. Some parts are in internal applications such as ERP and CRM. Others are in social media such as Facebook and Twitter. Being able to combine all the data from all the sources is important to realizing the value of Big Data. Without this ability, investments in Big Data will not be successful.

Plan your “Big Data” initiatives carefully

Despite the positioning of Big Data as the cure-all, the ugly truth is that “Big Data” needs investments in Data Quality and Master Data Management. This is true of any data related initiative, including “Big Data.” Keeping these characteristics while planning a “Big Data” initiative will go a long way in ensuring success.