How is smart data transforming work?

The Age of Big Data has helped many organisations unearth great insights, but in the future will smart data drive deeper insights?

Graphic of a brain showing network connectors on black background with blue lines

The last ten years have been what many data scientists would call the Age of Big Data. Put simply, the digital transformation of work, manufacturing, commerce, shopping has allowed for the creation of huge volumes of data. Everything from spending patterns to footfall, mechanical performance and subtle demographic trends have all been brought to life by data.

But while volume still counts, creating and capturing data isn’t the beginning and end of the data revolution. Now that regulatory changes like GDPR have been implemented, the purpose of data collection has shifted from quantity to quality: less about Big Data, and more about Smart Data.

There’s no accepted definition of Smart Data, but Duncan Stoddard, founder and MD of DS Analytics, says in his view, Big Data and the various progressions in machine and deep learning have been super valuable in some ways, “But the use cases for those types of models and analyses are actually quite limited. They are valuable where they’ve been used, but for the vast majority of organisations they aren’t really applicable.”

This was echoed recently by Campbell Brown, CEO & Co-Founder at PredictHQ, who told the Forbes Technology Council.

“It’s time that we accept the limitations of big data and embrace the need for Smart Data”

Brown called for businesses to set their data scientists free to do the work they dream about: “Not collecting, aggregating and cleaning, but building models to tap into signals over noise for core processes such as labour optimisation and price forecasting.”

Smarter, better

Stoddard agrees, and stresses that there are a huge number of applications of data science that help in decision making and improve relationships with customers or manage your supply chain better. And to deliver that, data must be smart. Businesses must therefore look to deploy their data scientists to focus on, ‘Refining or transforming data that is messy or not amenable to analysis to make a decision.’

A growing consensus among tech thinkers is emerging that believes that by refining or modelling messy data and converting it into something that can be used for other models or other insights is the next really interesting horizon, and one that will open up an exciting new range of opportunities for those working in data science.

Data science skills are hard to find. Great jobs shouldn't be. Discover a range of opportunities through our technology recruitment brand Computer Futures.

Creating and using Smart Data could involve a scenario where text is extracted from news articles. Being human-generated, this data is inherently unstructured, messy in its construction and content. By using applications such as natural language processing (NLP), raw unstructured data that can’t be used in a modelling framework can be transformed and enriched, allowing data scientists to extract features and insight from it.

Mindset shift

“The output of that process is data itself,” Stoddard explains. “It might be a table that has features that relate to original source text, that can be used in another context – in this case to cluster users based on areas of interest, or predict what articles might be of interest, or to predict what new customers or subscribers might want from your service.”

Setting data scientists free to roam through datasets to extract insight will challenge some organisations that are used to more rigid approaches, but it is the way of the future, with the concept of data sharing across organisations also coming to the fore. This would involve mutual data trading where data can be enriched by sharing – an approach the UK government is looking to support.

“So imagine for instance if you’re trying to make recommendations for a movie that someone wants to watch, if you shared data with a company that knows all about that user’s music taste or travel history then the recommendations could be improved by that,” Stoddard explains.

Skills gap

Of course, keeping pace with the rate of change as Big Data gives way to Smart Data will require businesses to constantly monitor whether they have the right skills in their teams. And while that may demand investment of time and resources, embracing smart data techniques may in fact confer a competitive advantage.

“I think it’s definitely true that most data scientists would prefer to spend their time building models that simulate certain real-world processes and address real world problems – like optimising prices or predicting the spread of infectious diseases,” Stoddard points out.

“But for many smart data scientists and software engineers, that’s a tiny part of their job. A lot of data science work in many organisations is data processing, querying vast, complex databases or tweaking the parameters of a monster algorithm that’s been developed by someone else.

As Smart Data takes precedent, however, the greater insight it can provide will create new opportunities for data scientists to tackle real-life problems. For those with the right skills, the future looks bright.

Find out more about our global house of specialist recruitment brands

Read more insights

09 June 2022

Empowering our colleagues to help our planet

Our Dutch teams took advantage of our paid volunteering leave to do their bit to help the environment. Find out what they got up to

Abstract picture of pink bubbles of blood cells on a blue background closeup
22 February 2022

How can companies provide meaning to their employees

The next Work in Progress documentary explores the topic of purpose in work and how companies can provide meaning to the work they do. 

Senator John A. Nejedly Bridge in Northern California
25 November 2021

Building a diverse workforce in the Asia Pacific region

Assisting our financial services client to map the market and source candidates from diverse backgrounds.

Solar and wind farm through the mist at dawn
26 October 2021

Being a climate conscious business

How climate change is affecting our business and the STEM markets we operate in with independent sustainability consultancy Avieco.

volunteers working at a plant recycling centre
22 October 2021

Empowering our colleagues to help the environment

Everyone at SThree gets 40 paid volunteering hours a year. Read about how our teams around the world are helping the environment.

Close up of a child getting vaccinated in the arm
24 September 2021

Providing access to medical testing through mobile and logistics technology

This health tech start-up uses mobile and logistics technology to provide access to medical testing, including COVID-19 in rural Nigeria.

Close up of bionic hand and its electrical components
24 September 2021

Improving mobility and quality of life through 3D printed prosthetics

Improving mobility and quality of life through 3D printed prosthetics

close up of the inside of a plastic bottle lying discarded on the ground
24 September 2021

Digitising water utility operations to reduce water shortages in Africa

How the WayPoints team developed software that brings water utility operations in Africa into the 21st century.

Combine harvester at dawn sparing fertilizer on crops
24 September 2021

Tackling bacterial diseases in agriculture and beyond

The Uniphage team are hoping their innovative STEM business concept will create platform technology to cure bacterial diseases in agricul...

 Young Indian farmer with agronomist at banana field
24 September 2021

Alleviating single-use plastic in the fast food sector

A team of students from India developed sustainable food packaging to reduce the amount of plastic packaging being produced

birds eye view of a dam full of water releasing small volume  down the valley
24 September 2021

What if you could turn single-use face masks into clean energy?

Living Energy are hoping their innovative STEM business concept will turn our global single-use plastic problem into a clean energy source.

New road construction with steam roller and other machinery
24 September 2021

Transforming the construction market with a low-carbon alternative

A team of young entrepreneurs from India have developed a low-carbon alternative that could transform the global construction market.

Women inspecting quality of plant crops wearing goggles and holding clipboard
24 September 2021

Who knew cows loved the salty taste of seaweed?

Based in the UK, a team of students have developed a cow feed that uses seaweed to reduce the amount of methane entering the atmosphere

Wind turbine maintenance engineer working during daylight
15 September 2021

How expertise in niche fields kept a US wind farm construction project on track

Global renewable energy construction company benefits from key hires with specialist skills in wind turbines.

Male robotics scientist standing next to computer screens and robot mimicking his actions
04 June 2021

Now is the time to energise the future of work

We've partnered with Samuel Durand on a new documentary exploring the future of work

Gettyimages 154511418 RT1
10 October 2020

It's okay to not be okay - why should you care about your mental health?

This article looks at why you should care about your mental health, especially during a pandemic this World Mental Health day.

hand touching pint fiber optics
25 June 2020

The future of the workplace is now

Experts discuss the future of work and what it means for the fourth industrial revolution.

Birdseye view of two cargo ships at port
07 August 2019

Corporate social responsibility in recruitment

Head of CSR Gemma Branney discusses the importance of ethical procurement in the recruitment industry