Big Data


Using The Right Data To Align Your Products & Services

Integrated thinking – unlocking human insights

The business world today is data centric. Aligning products and services with what customers want and need is a common goal. However, there is still a lot of debate around the best data source to generate customer and market insights. A lot of the discussion focuses on the limitations of each method; big data vs. survey data, quantitative vs. qualitative. It’s time we transcended our areas of specialism and preference in order to recognise the power of integrated thinking.

All types of data have limitations

Weighing up the pros and cons of different data collection methods is one of the first things you learn when studying any subject that utilises data. That’s not limited to those who are studying; no matter what your role is, you could be a marketer, data scientist or product designer, you are guaranteed to have first hand experience of working with the limitations of different kinds of data.

Big data, for example, can only tell you what has happened. Data scientists are using that data to create predictive models of future behaviour, but depending on the variety of data available, they can only go so far. Big data typically lacks context which means there could be missing lurking variables that are strongly associated with the outcomes we’re trying to explain.

The limitation of survey data is that it relies on people’s memory, which can often be poor. How do we know we have the right inputs when designing the questionnaire? And can we really be sure people are paying attention?

With qualitative data, the sample sizes are very small, so how do we know the findings are representative of the population? Furthermore the high volume of unstructured data is time consuming to analyse and relies on highly objective and perceptive researchers to draw out incisive findings.

When the conversation is centred on why one method is better than another we miss the opportunity to think about how we could use these methods in tandem. It’s in the intersection of these differing data types that we truly start working with human data.

Integrating Big Data & Survey Data

In many respects big data and survey data complement each other really well. Combining the two allows us to overcome many of the limitations detailed above. For example, we do not need to ask behavioural or transactional questions if they are already being observed in the big data set. Much more context can be added in a survey and we can begin to fill the big data gaps to avoid missing the lurking variables. By integrating big data and survey data, predictive models are likely to be much more accurate. Using the integrated data as training data and then re-applying the models to the full big data set has the potential to yield more rounded insights and impactful outcomes.

Staying in touch with reality

Working with data can be really interesting. Not just in a geeky way, but in a way that advances our understanding of how individuals and groups think and behave. Yet now, more than ever, it is important for businesses to get their heads out of spreadsheets and statistical software and start speaking to real people. Watching how people behave, understanding their motivations and observing how they interact with different products or services in different situations is critical to understanding our customers. Creating brand communications that resonate on an emotional level will not come from big data.

We are no closer to the “end of theory” in 2017 than we were back in 2008 when Chris Anderson first published his controversial provocation. Without theory and rich qualitative information about how people live, the inputs to more quantitative research will be deeply distorted through bias, lack of plurality and a disconnection with reality.

The path to human insights

We are reaching a point where we can approach data and insight generation in much more sophisticated ways. The worlds of big data and market research are changing. They share the same objectives but have largely been seen as separate approaches competing with each other in a beauty contest. The job-to-be-done for business leaders is to make the best decisions to grow its business. Most of the time data can help with this. However, collecting masses of data will not deliver the best outcome if you are not collecting the right data. Smart teams, using tools that offer the right data, from a mixture of optimal sources is the future of market research.