The last ten years have been what many data scientists would call the Age of Big Data. Put simply, the digital transformation of work, manufacturing, commerce, shopping has allowed for the creation of huge volumes of data. Everything from spending patterns to footfall, mechanical performance and subtle demographic trends have all been brought to life by data.
But while volume still counts, creating and capturing data isn’t the beginning and end of the data revolution. Now that regulatory changes like GDPR have been implemented, the purpose of data collection has shifted from quantity to quality: less about Big Data, and more about Smart Data.
There’s no accepted definition of Smart Data, but Duncan Stoddard, founder and MD of DS Analytics, says in his view, Big Data and the various progressions in machine and deep learning have been super valuable in some ways, “But the use cases for those types of models and analyses are actually quite limited. They are valuable where they’ve been used, but for the vast majority of organisations they aren’t really applicable.”
This was echoed recently by Campbell Brown, CEO & Co-Founder at PredictHQ, who told the Forbes Technology Council.
“It’s time that we accept the limitations of big data and embrace the need for Smart Data”
Brown called for businesses to set their data scientists free to do the work they dream about: “Not collecting, aggregating and cleaning, but building models to tap into signals over noise for core processes such as labour optimisation and price forecasting.”
Stoddard agrees, and stresses that there are a huge number of applications of data science that help in decision making and improve relationships with customers or manage your supply chain better. And to deliver that, data must be smart. Businesses must therefore look to deploy their data scientists to focus on, ‘Refining or transforming data that is messy or not amenable to analysis to make a decision.’
A growing consensus among tech thinkers is emerging that believes that by refining or modelling messy data and converting it into something that can be used for other models or other insights is the next really interesting horizon, and one that will open up an exciting new range of opportunities for those working in data science.
Data science skills are hard to find. Great jobs shouldn't be. Discover a range of opportunities through our technology recruitment brand Computer Futures.
Creating and using Smart Data could involve a scenario where text is extracted from news articles. Being human-generated, this data is inherently unstructured, messy in its construction and content. By using applications such as natural language processing (NLP), raw unstructured data that can’t be used in a modelling framework can be transformed and enriched, allowing data scientists to extract features and insight from it.
“The output of that process is data itself,” Stoddard explains. “It might be a table that has features that relate to original source text, that can be used in another context – in this case to cluster users based on areas of interest, or predict what articles might be of interest, or to predict what new customers or subscribers might want from your service.”
Setting data scientists free to roam through datasets to extract insight will challenge some organisations that are used to more rigid approaches, but it is the way of the future, with the concept of data sharing across organisations also coming to the fore. This would involve mutual data trading where data can be enriched by sharing – an approach the UK government is looking to support.
“So imagine for instance if you’re trying to make recommendations for a movie that someone wants to watch, if you shared data with a company that knows all about that user’s music taste or travel history then the recommendations could be improved by that,” Stoddard explains.
Of course, keeping pace with the rate of change as Big Data gives way to Smart Data will require businesses to constantly monitor whether they have the right skills in their teams. And while that may demand investment of time and resources, embracing smart data techniques may in fact confer a competitive advantage.
“I think it’s definitely true that most data scientists would prefer to spend their time building models that simulate certain real-world processes and address real world problems – like optimising prices or predicting the spread of infectious diseases,” Stoddard points out.
“But for many smart data scientists and software engineers, that’s a tiny part of their job. A lot of data science work in many organisations is data processing, querying vast, complex databases or tweaking the parameters of a monster algorithm that’s been developed by someone else.
As Smart Data takes precedent, however, the greater insight it can provide will create new opportunities for data scientists to tackle real-life problems. For those with the right skills, the future looks bright.Find out more about our global house of specialist recruitment brands
Read more insights
Empowering our colleagues to help our planet
Our Dutch teams took advantage of our paid volunteering leave to do their bit to help the environment. Find out what they got up to
How can companies provide meaning to their employees
The next Work in Progress documentary explores the topic of purpose in work and how companies can provide meaning to the work they do.
Building a diverse workforce in the Asia Pacific region
Assisting our financial services client to map the market and source candidates from diverse backgrounds.
Being a climate conscious business
How climate change is affecting our business and the STEM markets we operate in with independent sustainability consultancy Avieco.
Empowering our colleagues to help the environment
Everyone at SThree gets 40 paid volunteering hours a year. Read about how our teams around the world are helping the environment.
Providing access to medical testing through mobile and logistics technology
This health tech start-up uses mobile and logistics technology to provide access to medical testing, including COVID-19 in rural Nigeria.
Improving mobility and quality of life through 3D printed prosthetics
Improving mobility and quality of life through 3D printed prosthetics
Digitising water utility operations to reduce water shortages in Africa
How the WayPoints team developed software that brings water utility operations in Africa into the 21st century.
Tackling bacterial diseases in agriculture and beyond
The Uniphage team are hoping their innovative STEM business concept will create platform technology to cure bacterial diseases in agricul...
Alleviating single-use plastic in the fast food sector
A team of students from India developed sustainable food packaging to reduce the amount of plastic packaging being produced
What if you could turn single-use face masks into clean energy?
Living Energy are hoping their innovative STEM business concept will turn our global single-use plastic problem into a clean energy source.
Transforming the construction market with a low-carbon alternative
A team of young entrepreneurs from India have developed a low-carbon alternative that could transform the global construction market.
Who knew cows loved the salty taste of seaweed?
Based in the UK, a team of students have developed a cow feed that uses seaweed to reduce the amount of methane entering the atmosphere
How expertise in niche fields kept a US wind farm construction project on track
Global renewable energy construction company benefits from key hires with specialist skills in wind turbines.
Now is the time to energise the future of work
We've partnered with Samuel Durand on a new documentary exploring the future of work
It's okay to not be okay - why should you care about your mental health?
This article looks at why you should care about your mental health, especially during a pandemic this World Mental Health day.
The future of the workplace is now
Experts discuss the future of work and what it means for the fourth industrial revolution.
Corporate social responsibility in recruitment
Head of CSR Gemma Branney discusses the importance of ethical procurement in the recruitment industry