Thought Leadership

Big Data: The Trend Beneath the Buzzword


Apr 30

John Sumser, Principal Analyst, HRxAnalysts

The subject of Big Data is on everyone’s short list. The trend beneath the buzzword is so important that is as hard to imagine as the web was in 1993. It’s driven by smart tools, cloud architectures, cheap processing, cheap storage, broadening access to data and the search for new ways to gain productivity.

The idea is big enough to support an array of conflicting notions. A scan through the search results finds topics that range from analytics to large scale data integrations to novel correlations. The Big Data tent includes all of these somewhat contradictory ideas.

Part of the reason for the confusion is that the Big Data idea is really about the tools and techniques that can be applied to very large data problems. At this point, the subject is about something like a carpenter’s toolbox. All of the fun happens when you start building houses.

Here are the broad categories of things that Big Data tools are being applied to:

1. Big Data Means More Data That Fits

Analytics are nothing new. The 80s saw data begin to drive processes as organizations embraced Total Quality and Statistical Process Control. The first wave of the information age saw the democratization of chart making and lots of stuff turned into pie charts and bar charts. The underlying sources of information always had problems making custom reports.

To this day, people spend tons of time assembling unique reports by hand. The tools of Big Data include the ability to develop repeatable processes from disparate databases that live under your roof. From this perspective, Big Data is more of the same old number crunching. It’s just faster, easier and more repeatable.

2. Big Data Means Consuming the Social Graph and, More Data That Doesn’t Fit

Some questions have remained impossible to imagine answering. Collecting and analyzing the data required to actually manage a market based employment branding campaign on a job by job basis of 5,000 jobs (which is how you ought to do it) was beyond comprehension. Now it’s possible for a non-programmer to assemble the data in away that leads to process automation.

Huge chunks of data that are bigger than the storage capacities of old fashioned enterprise tools are popping out of the woodwork. As employers increasingly find it important to own a copy (or two) of all employees’ social data and be able to digest it, questions that involve cross-referencing in-house data with external data are becoming normal. With no database actually capable of holding all the data, organizations will use Big Data tools to have large chunks of data sync up.

This is particularly acute in the Recruiting operation where it is useful to have (or have access to) massive data on the labor market and the candidate pool.

3. Big Data Means Integrated Data, All Data Singing Like a Choir

The holy grail in legacy enterprise software solutions is a single integrated database that has all of the silos within the organization talking to and learning from each other. Part of the reason that HR has failed to demonstrate is alignment with business goals is that the holy grail never materialized. Making correlations across the organization has always required the involvement of the IT department.

With tools that behave like intelligent overlays, rich internal data sets can be combined, compared and contrasted to unearth gems of insight about what really drives performance both for humans and for the enterprise itself. Big Data solves the historic data logistics bottleneck and makes it possible for new questions to emerge.

4. Big Data Means Dis-Integrated Data, Pulling Pieces to Answer Your Question

At the very same time, one-off questions can be unearthed by pulling micro chunks of data to focus on very specific situations. Big Data includes tools for effective navigation of huge data sets individually or in combination. You can see the impact of a decision on all of sales or of offices with less than three employees who all have an average of more than 10 years experience. This is where much of the power of Big Data really exists. The discovery of high-leverage, unique opportunities to increase organizational performance is how major gains are tweaked out of functioning systems.

5. Big Data Means More Insight Rediscovering the Obvious

Expect a lot of grumbling as people get accustomed to using these new tools. Unearthing brilliant discoveries like people who live in the rain belt get to work later on rainy days than people who live in the banana belt. Cross referencing external databases with internal data will produce some amazing stories.

6. Big Data Means More Insight from the Tiny

The Butterfly Effect is often used to explain Chaos Theory. In the Butterfly Effect, A butterfly causes a hurricane. Small variations in inputs can cause large variations in outputs. Twentieth century computing architecture was designed to save the very resources that are now in abundance. As a result, computing systems usually focus on the meaning of large trends in databases rather than tiny increments.

A great example of this is web server statistics analysis. While most contemporary tools focus on big chunks of traffic, the answer is sometimes a tiny thing. While it’s interesting to know that 15,000 of your website visitors came from Botswana, it’s probably more important to know that Bill Gates visited. Contemporary systems can’t do this but Big Data makes it possible.

7. Big Data Means New Correlations That Drive Performance

Lots of questions never get asked because it seems impossible to answer them. The dawn of the Big Data era will create many opportunities to understand the connections between things that don’t seem to be connected. Think local gas price fluctuations and attendance or sophomore year classes and performance.

While it’s easy to focus on the data part of Big Data, the trend is important because it defines the next generation of work. Where the information age brought standardization and proceduralization to all sorts of jobs, the next step is to restore curiosity. The possibilities of Big Data are mostly limited by the curiosity of the people who are handling it. We currently don’t have good tools and techniques for developing curiosity in people who have had transactional jobs.

In the coming years, much of the HR workforce will need training to make the shift from transaction processing to the development of good questions. It’s not an easy shift.

If you want to get up to speed on Big Data, here’s a good starting list:

Tagged With: , , , , , , , , ,

John Sumser has been chronicling and creating the state of the art in Recruiting for nearly 20 years. A philosophy major with slim employment prospects, Sumser joined the Defense establishment in DC is the late 1970s (after several years of taking over student governments at DC Universities). Finding his niche as an engineer, Sumser navigated Hopkins and Loyola grad schools while becoming an R&D executive. In those days, staffing was likely to be done by the hiring manager. At 36, John packed up his family and headed to Silicon Valley just before the internet moved from Defense R&D labs into the public eye. His post-defense career began with the operation of the first business that allowed personal internet accounts. in 1993, he founded interbiznet, the first fiirm to evaluate, chronicle and prod the nascent Online Recruiting Industry. Today, as the Editor of the HRExaminer and as the Principal Analyst at HRxAnalysts, Sumser continues to break new ground. Currently, he is leading a team to evaluate the emerging social media (recruiting and HR) market, building a comprehensive curriculum in social media effectiveness for Recruiters and collaborating on the development of employment related economic indices for several clients. His ongoing analysis of influence in the HR and Recruiting industry continues to provoke thought and conversation.