Due to the increasing diffusion of electronic devices and data-led services, the total amount of information stored by humanity doubles every two years. Big data, “chunks of data too big to be processed through common software packages”, is getting bigger and ubiquitous. However, the offer of big data software and solutions has also reached maturity, and a number of opportunities arise for businesses who want to take advantage of this technology.
This article summarises the state-of-the-art of big data solutions and suggests avenues for further development of the technology.
From Data To Metadata
Nowadays, the focus of big data technology is no more on the narrow informative content, but on the broader context in which the data has been collected. This means that data will cover only a minor role in the immediate future and metadata, which describes the data and provides additional value, will increasingly become important and complex. A big data application can be imagined similar to a shopping cart. One can easily see the data (i.e. the content of the cart) at any point in time.
However, little value emerges from looking at a single purchase. Basic information such as price, weight, size and content of the item is immediately available, but interesting pieces of information are harder to collect. In the shopping cart of big data, the item purchased is not at the centre of attention. Instead, additional information (i.e. metadata) enables one to answer several questions: who purchased the item? When? Where?
Thus, one can imagine big data as a smart shopping cart capable of keeping track of and elaborating extra information about the purchases.
Aggregation And Meaning
Nevertheless, metadata alone is not enough to build competitive big data applications. Metadata constitutes the first layer of operation and serves as a basis for aggregating multiple items in a relevant way.
Sets of data related by comparable meta-data can be grouped and analysed together to obtain significant insights on customer behaviour and sales forecasts. In a grocery store, purchases could be grouped under several metrics. For example, time series of a single item’s sales can be used to predict future sales of that item, or an analysis of past consumer purchases can yearn inferences on his or her preferences.
Or again, items purchased together can be bundled to offer customer-oriented discounts. Once aggregated, data becomes simpler and easier to understand. One can summarise information and treat groups of data rather than individual items, saving computational power and time.
This gives rise to second-level meta-data that describes and categorises a group of items based on some preliminary analysis. For instance, items and customers can be clustered according to the frequency of purchase, seasonality, tastes and a myriad of other patterns. Cross-combination of data sources, the amount of data produced each year grows faster and faster than devices’ computational power.
Using Big Data
In a world of information, managers must choose the most relevant data sources to aid decision-making and focus the limited resources of the company. The power of big data can be truly harnessed only with a clear objective in mind and access to adequate complimentary sources. One should find his own way in the labyrinth of data available online and understand when additional pieces can be generated or combined to create additional value.
The applications for a shopping cart could be limitless. For example, it might connect to the customer’s social media accounts and map purchasing habits of groups of friends, or suggest to the user how to optimise his or her path within the store. Moreover, non-conventional parameters can be included to answer sophisticated questions.
What do customers do before and after visiting the store? How do weather conditions affect purchasing behaviour? What is the best way to position items, improve efficiency and maximise sales? The emerging Internet of things and big data go hand in hand and contribute to each other’s potential.
An Ever-Expanding Sector
With more devices connected to the internet and at a low price, the mole of available data will expand further to include GPS tracking coordinates, medical records, user-generated content, governmental statistics, multimedia files, and so on. One can easily keep track of every activity happening in a business ecosystem. Data is ubiquitous, and every industry has its own sources of distribution.
Data-driven applications have also grown exponentially in the last years, fulfilling a broader variety of customer needs and specific problems. The landscape of big-data solutions includes software dedicated to banking, logistics, retailing, healthcare – just to name a few. Furthermore, prices of data management software have fallen dramatically, with complete suites retailing for as little as €10 per month.
If, on the one hand, specific and expensive solutions tailored for complex problems have driven market growth, on the other hand, mass-market applications will determine revenues and market leadership. Bringing the power of big data to a broader audience is a key step towards establishing a standard and a recognised dominant brand. Nevertheless, the challenges ahead are not easy to overcome.
With the increasing availability of data sources, several tools allow businesses to combine information and craft their own dashboards tailored to individual needs. Moreover, standardisation could be complicated given the high diversity of datasets and imperfections in many of them. Finally, several specialised players dominate protected niches and benefit from a significant advantage over new entrants.
Competing in 2017
After the first wave of trials and errors, today big data applications can provide effective real-time suggestions to tackle complex business problems. New applications emerge every day, fulfilling a long series of needs at an increasingly lower cost. For software producers, a general rule applies: one either has the means to target the mass market with a cheap and satisfying solution or should produce pricey high-end packages for customers with a compelling reason to buy. In a nutshell, businesses and users of big data solutions will be the true winners this year.
Companies should engage with big data through an active approach, hiring their own data specialists, building their own dashboards and monitoring the evolution of trends and forecasts on a daily basis. Organisations with limited budgets will also gain from this scenario, taking advantage of a dynamic competitive landscape and affordable, complete software applications.