How is smart data transforming work?

The Age of Big Data has helped many organisations unearth great insights, but in the future will smart data drive deeper insights?

Graphic of a brain showing network connectors on black background with blue lines

The last ten years have been what many data scientists would call the Age of Big Data. Put simply, the digital transformation of work, manufacturing, commerce, shopping has allowed for the creation of huge volumes of data. Everything from spending patterns to footfall, mechanical performance and subtle demographic trends have all been brought to life by data.

But while volume still counts, creating and capturing data isn’t the beginning and end of the data revolution. Now that regulatory changes like GDPR have been implemented, the purpose of data collection has shifted from quantity to quality: less about Big Data, and more about Smart Data.

There’s no accepted definition of Smart Data, but Duncan Stoddard, founder and MD of DS Analytics, says in his view, Big Data and the various progressions in machine and deep learning have been super valuable in some ways, “But the use cases for those types of models and analyses are actually quite limited. They are valuable where they’ve been used, but for the vast majority of organisations they aren’t really applicable.”

This was echoed recently by Campbell Brown, CEO & Co-Founder at PredictHQ, who told the Forbes Technology Council.

“It’s time that we accept the limitations of big data and embrace the need for Smart Data”

Brown called for businesses to set their data scientists free to do the work they dream about: “Not collecting, aggregating and cleaning, but building models to tap into signals over noise for core processes such as labour optimisation and price forecasting.”

Smarter, better

Stoddard agrees, and stresses that there are a huge number of applications of data science that help in decision making and improve relationships with customers or manage your supply chain better. And to deliver that, data must be smart. Businesses must therefore look to deploy their data scientists to focus on, ‘Refining or transforming data that is messy or not amenable to analysis to make a decision.’

A growing consensus among tech thinkers is emerging that believes that by refining or modelling messy data and converting it into something that can be used for other models or other insights is the next really interesting horizon, and one that will open up an exciting new range of opportunities for those working in data science.

Data science skills are hard to find. Great jobs shouldn't be. Discover a range of opportunities through our technology recruitment brand Computer Futures.

Creating and using Smart Data could involve a scenario where text is extracted from news articles. Being human-generated, this data is inherently unstructured, messy in its construction and content. By using applications such as natural language processing (NLP), raw unstructured data that can’t be used in a modelling framework can be transformed and enriched, allowing data scientists to extract features and insight from it.

Mindset shift

“The output of that process is data itself,” Stoddard explains. “It might be a table that has features that relate to original source text, that can be used in another context – in this case to cluster users based on areas of interest, or predict what articles might be of interest, or to predict what new customers or subscribers might want from your service.”

Setting data scientists free to roam through datasets to extract insight will challenge some organisations that are used to more rigid approaches, but it is the way of the future, with the concept of data sharing across organisations also coming to the fore. This would involve mutual data trading where data can be enriched by sharing – an approach the UK government is looking to support.

“So imagine for instance if you’re trying to make recommendations for a movie that someone wants to watch, if you shared data with a company that knows all about that user’s music taste or travel history then the recommendations could be improved by that,” Stoddard explains.

Skills gap

Of course, keeping pace with the rate of change as Big Data gives way to Smart Data will require businesses to constantly monitor whether they have the right skills in their teams. And while that may demand investment of time and resources, embracing smart data techniques may in fact confer a competitive advantage.

“I think it’s definitely true that most data scientists would prefer to spend their time building models that simulate certain real-world processes and address real world problems – like optimising prices or predicting the spread of infectious diseases,” Stoddard points out.

“But for many smart data scientists and software engineers, that’s a tiny part of their job. A lot of data science work in many organisations is data processing, querying vast, complex databases or tweaking the parameters of a monster algorithm that’s been developed by someone else.

As Smart Data takes precedent, however, the greater insight it can provide will create new opportunities for data scientists to tackle real-life problems. For those with the right skills, the future looks bright.

Find out more about our global house of specialist recruitment brands

Read more insights

New thinking is needed to tackle drug shortages
02 May 2023

New thinking is needed to tackle drug shortages

Advanced manufacturing, smart supply chains and data have important roles to play in improving market access.

How companies can capitalise on the global megatrends
27 April 2023

How companies can capitalise on the global megatrends

There are three key areas of focus that can make firms victorious in the war for talent, find out what they are

Five global megatrends fuelling STEM recruitment
25 April 2023

Five global megatrends fuelling STEM recruitment

Demand for STEM talent is on the rise, but what is driving the increase?

What will developers do with 40% more time?
20 April 2023

What will developers do with 40% more time?

Developers are set to get a lot of time back when AI writes the boring and repetitive code

Window Cleaner to Director: A journey through the career of Lawrence Doe
31 March 2023

Window Cleaner to Director: A journey through the career of Lawrence Doe

Confidence, curiosity and a willingness to learn are all characteristics that will serve you well in your career at SThree.

Our role in a wider society
29 March 2023

Our role in a wider society

Supporting carbon offsetting initiatives with Earthly.

Breaking the Glass (BTG) programme helps to address gender disparity in STEM
15 March 2023

Breaking the Glass (BTG) programme helps to address gender disparity in STEM

Empowering women to access roles and promotions in STEM through our Breaking the Glass community programme.

Diversity in engineering
01 March 2023

Diversity in engineering

Advising and supporting our clients on how to hire without bias.

Carbon dioxide removal is increasing demand for STEM expertise
28 February 2023

Carbon dioxide removal is increasing demand for STEM expertise

Capturing and storing carbon dioxide is set to play an important part in efforts to halt climate change. What jobs and skills will be nee...