The Value of Big Data

Companies of all sizes are analyzing vast volumes of data to gain new insight and make better business decisions.

The term “big data” has emerged in recent years to describe the vast amounts of information being generated by a wide range of systems and applications. All of that data is pouring in at an ever-faster rate, and smart organizations are mining it for business insight.

In retail and hospitality, for example, companies are analyzing sales transaction data to better understand and target their customers. Retailers are also using data from social media, web searches and other sources to predict product sales, which helps them optimize their inventory and promotional campaigns.

However, big data isn’t just about increasing revenue. In fact, most organizations are using big data to cut costs, increase productivity, improve business processes, speed time-to-market and enhance customer service. According to a recent IDC report, the banking, manufacturing, federal government and professional services sectors will spend $72.4 billion on big data analytics solutions this year. They will also be the largest spenders in 2020 when their total investment will be $101.5 billion.

“The three industries that comprise the financial services sector — banking, insurance, and securities and investment services — all show great promise for future spending on big data and business analytics. This technology can be applied across key use cases throughout these financial institutions, from fraud detection and risk management to enhancing and optimizing the customer’s journey,” said IDC Program Director Jessica Goepfert.

“Outside of financial services, several other industries present compelling opportunities. Within telecommunications, for instance, big data and analytics are applied to help retain and gain new customers as well as for network capacity planning and optimization.”

All but two of the industries covered in IDC’s report plan to significantly increase their spending on big data through 2020. These organizations are betting on big data analytics to help them make better decisions by detecting patterns in real-time and historical data.

Beyond Business Intelligence

Crunching data to improve decision-making is hardly a new concept. For years, organizations have been capturing vast stores of transactional information in data warehouses and using business intelligence (BI) tools to slice and dice the data for strategic planning. The latest BI applications can even support real-time analysis, and be accessed by business users throughout the organization for spotting trends and predicting future performance.

Big data analytics goes beyond traditional BI, however. While BI tools are designed to answer predefined questions that are oriented to a consistent style of reporting, big data analytics offers a more ad-hoc approach. The focus is on enabling end-users to create on-the-fly searches to uncover new patterns and insights.

Another key difference is the ability to extract information from data that isn’t recorded in neat rows and columns in a database or spreadsheet. The vast majority of big data is unstructured and scattered across the web or stored in text documents. Organizations are also using big data analytics to make sense of the huge volumes of data generated by sensors and other Internet of Things (IoT) devices.

A once-small, open-source project called Hadoop has become a leading platform for analyzing these large datasets. Hadoop is a storage technology that enables batch processing of unstructured and structured data across hundreds or thousands of computing nodes operating in parallel. As a result, Hadoop facilitates the rapid transfer of data between nodes, and enables the clustering of commodity servers to provide the processing power needed to handle big data analytics tasks.

A recent Forrester Research report found that almost 40 percent of firms were implementing or expanding big data technology in 2016, and another 30 percent planned to adopt big data in the next 12 months. Thirty percent of respondents had implemented Hadoop.

Not Just for Enterprises

Large and very large companies (those with more than 1,000 employees) will be responsible for more than 60 percent of big data analytics spending through 2020, according to IDC. However, the research firm says that small to midsize businesses (SMBs) will be significant contributors, with nearly a quarter of the worldwide revenues coming from companies with fewer than 500 employees.

Midmarket firms in particular have made big data analytics a top IT priority. According to research firm TechAisle, 77 percent of midmarket firms (those with 100 to 999 employees) have implemented big data analytics solutions, and 91 percent plan to do so at some point in the future. Although limited IT resources and poor data quality tend to inhibit big data initiatives, the availability of more cost-effective solutions and cloud-based analytics tools are helping to drive adoption in the midmarket.

As technologies such as cloud computing, social media and the IoT contribute to massive data growth, organizations need the ability to extract meaning from data to navigate shifting market conditions. Big data analytics tools are capturing the attention of organizations looking to gain deeper visibility into structured and unstructured data and improve decision-making processes. That’s why IDC forecasts worldwide revenues for big data analytics will grow from $150.8 billion in 2017 to more than $210 billion in 2020, a compound annual growth rate of 11.9 percent.

“After years of traversing the adoption S-curve, big data and business analytics solutions have finally hit mainstream,” said Dan Vesset, IDC group vice president, Analytics and Information Management. “Big data analytics as an enabler of decision support and decision automation is now firmly on the radar of top executives. This category of solutions is also one of the key pillars of enabling digital transformation efforts across industries and business processes globally.”