How is smart data transforming work?

The Age of Big Data has helped many organisations unearth great insights, but in the future will smart data drive deeper insights?

Graphic of a brain showing network connectors on black background with blue lines

The last ten years have been what many data scientists would call the Age of Big Data. Put simply, the digital transformation of work, manufacturing, commerce, shopping has allowed for the creation of huge volumes of data. Everything from spending patterns to footfall, mechanical performance and subtle demographic trends have all been brought to life by data.

But while volume still counts, creating and capturing data isn’t the beginning and end of the data revolution. Now that regulatory changes like GDPR have been implemented, the purpose of data collection has shifted from quantity to quality: less about Big Data, and more about Smart Data.

There’s no accepted definition of Smart Data, but Duncan Stoddard, founder and MD of DS Analytics, says in his view, Big Data and the various progressions in machine and deep learning have been super valuable in some ways, “But the use cases for those types of models and analyses are actually quite limited. They are valuable where they’ve been used, but for the vast majority of organisations they aren’t really applicable.”

This was echoed recently by Campbell Brown, CEO & Co-Founder at PredictHQ, who told the Forbes Technology Council.

“It’s time that we accept the limitations of big data and embrace the need for Smart Data”

Brown called for businesses to set their data scientists free to do the work they dream about: “Not collecting, aggregating and cleaning, but building models to tap into signals over noise for core processes such as labour optimisation and price forecasting.”

Smarter, better

Stoddard agrees, and stresses that there are a huge number of applications of data science that help in decision making and improve relationships with customers or manage your supply chain better. And to deliver that, data must be smart. Businesses must therefore look to deploy their data scientists to focus on, ‘Refining or transforming data that is messy or not amenable to analysis to make a decision.’

A growing consensus among tech thinkers is emerging that believes that by refining or modelling messy data and converting it into something that can be used for other models or other insights is the next really interesting horizon, and one that will open up an exciting new range of opportunities for those working in data science.

Data science skills are hard to find. Great jobs shouldn't be. Discover a range of opportunities through our technology recruitment brand Computer Futures.

Creating and using Smart Data could involve a scenario where text is extracted from news articles. Being human-generated, this data is inherently unstructured, messy in its construction and content. By using applications such as natural language processing (NLP), raw unstructured data that can’t be used in a modelling framework can be transformed and enriched, allowing data scientists to extract features and insight from it.

Mindset shift

“The output of that process is data itself,” Stoddard explains. “It might be a table that has features that relate to original source text, that can be used in another context – in this case to cluster users based on areas of interest, or predict what articles might be of interest, or to predict what new customers or subscribers might want from your service.”

Setting data scientists free to roam through datasets to extract insight will challenge some organisations that are used to more rigid approaches, but it is the way of the future, with the concept of data sharing across organisations also coming to the fore. This would involve mutual data trading where data can be enriched by sharing – an approach the UK government is looking to support.

“So imagine for instance if you’re trying to make recommendations for a movie that someone wants to watch, if you shared data with a company that knows all about that user’s music taste or travel history then the recommendations could be improved by that,” Stoddard explains.

Skills gap

Of course, keeping pace with the rate of change as Big Data gives way to Smart Data will require businesses to constantly monitor whether they have the right skills in their teams. And while that may demand investment of time and resources, embracing smart data techniques may in fact confer a competitive advantage.

“I think it’s definitely true that most data scientists would prefer to spend their time building models that simulate certain real-world processes and address real world problems – like optimising prices or predicting the spread of infectious diseases,” Stoddard points out.

“But for many smart data scientists and software engineers, that’s a tiny part of their job. A lot of data science work in many organisations is data processing, querying vast, complex databases or tweaking the parameters of a monster algorithm that’s been developed by someone else.

As Smart Data takes precedent, however, the greater insight it can provide will create new opportunities for data scientists to tackle real-life problems. For those with the right skills, the future looks bright.

Find out more about our global house of specialist recruitment brands

Read more insights

Close up of a male biologist looking an screen of molecules with purple/pink hue overlay
09 December 2022

The biotechnology that made the world safer

Covid-preventing vaccines become a game-changing reality yet were developed in only a year - here's how

Close up of drinking bottles being blown on production line
07 December 2022

Seven challenges employers in STEM face in the next 10 years

With STEM roles becoming ever more crucial to the global economy, what are the key issues facing companies operating in that environment?

Female executive with her arms crossed, wearing black glasses and black suit smiling out window of office looking at the horizon
30 November 2022

Global hotspots for R&D investment

Governments and organisations need to invest in R&D programmes to ensure advancement in the future. Discover more in out world map

Network engineer tidying messy cables in server room
08 November 2022

Global report

In our global report we look at what influences early career talent into STEM, and how can employers attract the talent of the future

Close up of young smiling woman at a conference
08 November 2022

European report

What influences early career European talent into STEM, and how can employers in Europe attract the talent of the future?

Early-career employees huddling around a laptop smiling

United States report

What influences early career American talent into STEM, and how can employers in America attract the talent of the future?

Young smiling female Asian student sat in study group
08 November 2022

Asia Pacific report

What influences early career Asian talent in STEM, and how can employers in the Asia Pacific region attract the talent of the future?

Early-career STEM talent, huddled around mobile phone pointing at screen
08 November 2022

African snapshot

What influences early career African talent in STEM, and how can employers in Africa attract the talent of the future?

Team of Bio scientists in lab analysing samples
01 November 2022

The future of digital healthcare

Digitalisation is transforming how healthcare is delivered, fuelling demand for new and specialised skills.