How is smart data transforming work?

The last ten years have been what many data scientists would call the Age of Big Data. Put simply, the digital transformation of work, manufacturing, commerce, shopping has allowed for the creation of huge volumes of data. Everything from spending patterns to footfall, mechanical performance and subtle demographic trends have all been brought to life by data.

But while volume still counts, creating and capturing data isn’t the beginning and end of the data revolution. Now that regulatory changes like GDPR have been implemented, the purpose of data collection has shifted from quantity to quality: less about Big Data, and more about Smart Data.

There’s no accepted definition of Smart Data, but Duncan Stoddard, founder and MD of DS Analytics, says in his view, Big Data and the various progressions in machine and deep learning have been super valuable in some ways, “But the use cases for those types of models and analyses are actually quite limited. They are valuable where they’ve been used, but for the vast majority of organisations they aren’t really applicable.”

This was echoed recently by Campbell Brown, CEO & Co-Founder at PredictHQ, who told the Forbes Technology Council.

“It’s time that we accept the limitations of big data and embrace the need for Smart Data”

Brown called for businesses to set their data scientists free to do the work they dream about: “Not collecting, aggregating and cleaning, but building models to tap into signals over noise for core processes such as labour optimisation and price forecasting.”

Smarter, better

Stoddard agrees, and stresses that there are a huge number of applications of data science that help in decision making and improve relationships with customers or manage your supply chain better. And to deliver that, data must be smart. Businesses must therefore look to deploy their data scientists to focus on, ‘Refining or transforming data that is messy or not amenable to analysis to make a decision.’

A growing consensus among tech thinkers is emerging that believes that by refining or modelling messy data and converting it into something that can be used for other models or other insights is the next really interesting horizon, and one that will open up an exciting new range of opportunities for those working in data science.

Data science skills are hard to find. Great jobs shouldn't be. Discover a range of opportunities through our technology recruitment brand Computer Futures.

Creating and using Smart Data could involve a scenario where text is extracted from news articles. Being human-generated, this data is inherently unstructured, messy in its construction and content. By using applications such as natural language processing (NLP), raw unstructured data that can’t be used in a modelling framework can be transformed and enriched, allowing data scientists to extract features and insight from it.

Mindset shift

“The output of that process is data itself,” Stoddard explains. “It might be a table that has features that relate to original source text, that can be used in another context – in this case to cluster users based on areas of interest, or predict what articles might be of interest, or to predict what new customers or subscribers might want from your service.”

Setting data scientists free to roam through datasets to extract insight will challenge some organisations that are used to more rigid approaches, but it is the way of the future, with the concept of data sharing across organisations also coming to the fore. This would involve mutual data trading where data can be enriched by sharing – an approach the UK government is looking to support.

“So imagine for instance if you’re trying to make recommendations for a movie that someone wants to watch, if you shared data with a company that knows all about that user’s music taste or travel history then the recommendations could be improved by that,” Stoddard explains.

Skills gap

Of course, keeping pace with the rate of change as Big Data gives way to Smart Data will require businesses to constantly monitor whether they have the right skills in their teams. And while that may demand investment of time and resources, embracing smart data techniques may in fact confer a competitive advantage.

“I think it’s definitely true that most data scientists would prefer to spend their time building models that simulate certain real-world processes and address real world problems – like optimising prices or predicting the spread of infectious diseases,” Stoddard points out.

“But for many smart data scientists and software engineers, that’s a tiny part of their job. A lot of data science work in many organisations is data processing, querying vast, complex databases or tweaking the parameters of a monster algorithm that’s been developed by someone else.

As Smart Data takes precedent, however, the greater insight it can provide will create new opportunities for data scientists to tackle real-life problems. For those with the right skills, the future looks bright.