BIG News & Reports Archive (19)
Here you will find the BIG News & Reports Archive
A small but appreciative audience braved the rain on October 6th to hear the award-winning BIG Conference paper by Jon Puleston of Lightspeed GMI and Amy Cashman of TNS Technology & Finance, “It’s not business, it’s personal”.
The presentation dealt with a survey of SMEs about their feelings towards banks but, before Amy Cashman discussed the stories emerging from the findings, Jon Puleston gave a fascinating description of the approach taken to the design of this online survey, and the principles underlying this approach.
Jon’s starting point was that surveys have traditionally evolved out of spoken interview methods and simply involve a long list of questions that the client wants answered. For online surveys, this approach will not do as it is so easy for respondents to simply drop out or to tick boxes mechanically without thinking. We must enter into the mind of the respondent and ask ourselves why the respondent should be bothered to read the questions and answer them thoughtfully.
Jon’s approach is to draw on other disciplines outside market research. Firstly, advertising. Advertising is all about motivating you to notice and absorb the ad, even when it involves reading lengthy text, by making the process fun, surprising or intriguing. To do this, advertisers may use striking visual imagery, humour, titillation as well as intellectual pleasures, as Jon illustrated with some well known examples of successful ads. Jon also remarked on the skill of copywriters in not just telling you what they want to tell you but finding the right words to make a connection with the consumer. The implication for survey researchers is that they must design questions that intrigue and challenge respondents and make them think.
The first page of an online survey is especially important as this, in a sense, is the ad for the survey that will draw respondents into completing it. The use of visual icons in questions also acts as an ad and helps to stimulate response: Jon presented data to show that respondents select more options in a multiple-option question when images are shown with each option.
Jon turned next to the film business. Film-makers have to hold our attention for hours and they do so in three main ways; by basing the script around a great question (for example, ‘ What if every day were the same?’ in Groundhog Day), by having a strong narrative structure, and by using heros with whom one can identify.
The third profession that Jon considered was publishing. Publishers mix both advertising and editorial content – that is, they include content that readers want to look at as well as content that advertisers want them to look at. For the researcher, this means asking questions in a way that respondents find engaging and intellectually rewarding, perhaps including supplementary questions that are not strictly related to those that the client wants, in order to get the answers that the client does want.
To do this, understanding of gaming becomes important for the way that computer games force people to think hard but deliver rewards to participants. Using quizzes, asking respondents to predict how other people will respond and providing feedback to respondents are all helpful.
On Weds 24th June 2015 we welcomed the 2nd BIG / ICG Northern Forum of 2015, hosted in the Pioneer Suite with a presentation from Andrew Vincent – what better way to spend a Wednesday evening than networking & learning just how to master ‘simplexity’.
Andrew began by stating that as an industry – our deliverables can always get better for our end client, regardless of our methodology or the technology used during the research – the client wants their problem solved.
Andrew discussed the issue of delivering insight in a ‘distraction environment’ one we’re all familiar with – people ducking in and out of meetings, checking their emails and struggling through the vast amount of information sitting alongside any research – is anyone really listening?
Playwright Richard Foreman made a great observation, that we risk turning in to ‘pancake people – spread wide and thin as we connect with that vast network of information accessed by the mere touch of a button’. Andrew stated, even in this complex world where the gap between suppliers and consumers draws ever closer with the help of technology, quality research is more important than ever, by successfully cutting through to the end user we can try to understand them, therefore improving the service or product in question.
Everything Should Be Made as Simple as Possible, But Not Simpler – Einstein
With most research projects, the findings can be the most time consuming aspect – how accurately should we present what we find? How can we be sure not to overcomplicate feedback – too much information can be confusing and unhelpful, but we don’t want to oversimplify either? Coming to the right compromise when delivering to a client is crucial, Andrew debated that more can be less but not always. This is where he has coined the phrase ‘simplexity’ from. A mixture of simplicity, requiring less effort but not necessarily conveying enough information and complexity, putting across the message without trimming down and editing the wheat from the chaff. Ultimately ‘simplexity’ requires strong language skills, and the ability to re think the message you are putting across – not just shorter but simpler – if I reduce this further do I reduce the meaning, if I expand do I give any more?
Andrew used an extremely interesting case study to illustrate the struggle of presenting results that may not necessarily be what the client was expecting. The research involved understanding Type 2 Diabetes and its diagnosis from a different angle, just why do poor lifestyle choices continue after patients are given their results. Andrew discussed that the results were much as he expected, but he didn’t want to present this to the client, so delved further. For some people the diagnosis of Diabetes was a relief, it reduced their likelihood to change their diet or exercise, it explained their past weight issues and was almost a comfort to them. A counter intuitive behavioural segment such as this is a hard one to approach a client with – how can you convey results that show someone feels better when given a diagnosis they will carry for the rest of their life? In this Andrew felt it was important that the story was told from the perspective of the decision that has to be made, by reframing and rethinking the results, focussing on new data rather than what was already in the field, he was able to provide the client with a strong conclusion to their research. It is the success of balancing simple & complex that means we are able to deliver rich information to all our end clients, whether this be internally – in briefings or de briefs or externally, presenting research findings.
A big thank you to Andrew & everyone who attended – we look forward to seeing you next time!
If you wish to view the event, please click here:
Dave Skelsey of Strictly Financial chaired the last Forum session of the current season when Adam Drummond from Opinium and Kieran Pedley from GFK NOP discussed the hotly contested topic from the run up & follow on from this year’s general election - just how did the pollsters get the predicted results so wrong? In a question & answer panel format, an audience of researchers were invited to pose some hard hitting questions with some insightful results.
As an enquiry begins into how the Conservative vote was so underestimated and with the future of political polling in the balance one question remains – are polls still relevant in the run up to elections? With claims being made that the results were too big to call until the exit poll, conducted with 20,000 voters, was complete, the type of people pollsters were talking to is being called into question. As we all know, those who participate in surveys willingly are engaged, they’re pro-active and ready to share their views, does this mean that those not so forthcoming are reflected in such studies? Did they have to be online savvy, those who just didn’t know who they were going to vote for- how were their views recorded? Those that did take part – were they happy to admit their allegiances to the Conservative Party? Many different theories for the shock election outcome have been raised in recent weeks ranging from weather conditions on the day to questioning whether Thursday was an ideal day to vote, the enquiry currently underway looks to answer all these issues.
A query was raised in the debate as to whether the experts felt that the right questions were being asked in the lead up? The polls were mainly conducted in two forms – one by using a ballot box type format, the other worked to gauge a voter’s thoughts on the current economy with a view to uncovering whom they would potentially be supporting. Both are deemed to be as accurate as the other, however these forms of data collection do only gather responses for open votes – how are silent voters represented?
It was suggested that the polls shaped the media as well as political agenda, they equally pounced when it was revealed that the results were so drastically out of whack with the expert’s predictions. This then pushed the matter of whether polls were even necessary? Is it just a number crunching exercise or as the experts say ‘an art and a science in which we need to experiment to get the right answer’? Without the official polls, its highly likely that newspapers would create their own, meaning that the results almost certainly wouldn’t be representative, information would be leaked and could be swayed depending on the publications political swing.
With insights showing no shift during the campaigns, all polls pointed to the outcome being neck and neck and the result appearing quite obvious – should less of an emphasis have been put on this initial data and more on an exit poll methodology? The experts commented that the exit method works well as it’s interviewing actual voters, those who have taken the time to go to the polling station and will hopefully offer a reliable insight. When the exit poll didn’t match up with their previous estimates, the consultants admitted they were bewildered but had to accept the numbers as they were. When asked how they went about choosing their sample, both pollsters used the same methodology, dividing the population up and randomly sampling from within those areas. Surely this meant there was some influence between the major polls, eyeing up their results and looking back at their own and vice versa. The representatives stated that while yes, numbers can look different to other polls you have to go ahead and not let it affect the numbers shown by the data you have collated.
As previously mentioned many different theories have been implemented in the time following the election including ‘lazy Labour’ voters not voting, there being a late swing due to the campaigns and most bizarrely the proposal that the weather conditions of the day would favour the Tories. The belief that sunshine entices Labour voters to the polling booth opposes this and Stephen Fisher, a politics lecturer at Oxford University who has studied the relationship between the weather and turnout stated that "If you made a statistical correlation and scored the weather according to how good it was and compiled a graph showing voter turnout, over the last 15 elections you don't see a correlation." There are good reasons to believe there would be a link, he says, because it is less pleasant to go outside in the rain, but there is little evidence that it is true.
Will the current enquiry sort out the polling dilemma?
Pollsters suggested everything is on the table and up for debate in the investigation and hopefully some clarification would be given over the glaring differences between the initial polls and the exit poll. We can assume that the difference between the polls and the result wasn’t a random sample error but a possible deeper methodological failing.
Re- contact surveys the day after voting found no significant evidence of a late swing big enough to create a seven point change in 24 hours and this is something that will be examined, as well as how well current sampling methods are reflecting the British population. All pollsters involved in this year’s election made it clear that they will take the time fully to examine all the possible causes of error and put them right for future surveys.
The presentation focussed around three areas: the common challenges faced in conducting international b2b research; best practice in overcoming these challenges; as well as taking a brief look at how we can account for cultural differences in b2b research.
We discussed the fact that most challenges in conducting international research in b2b markets fall roughly into four categories:
Sampling - such as varying sample availability in different countries, variable online penetration, different quality of client list in different countries, the importance of focus on number of countries included
Method - which data collection method to use based on the countries involved, the importance of native language fieldwork, again the role of online penetration in this area
Client needs - managing the differing needs of different client country teams, varying levels of engagement required based on culture, the difference in reporting requirements (strategic vs tactical) based on culture
Culture - cultural differences (difference working weeks, levels of formality, etc.), differences in answering questions, difficulty of projective techniques in certain countries, cultural nuances, different incentive requirements
In line with the above challenges, we discussed the importance of focus in multi-country studies, the importance of ensuring that international fieldwork is conducted in native language to ensure detailed and honest answers, and the importance of considering cultural nuances in the project design.
We also spoke about the importance of ensuring that, as researchers, we need to ensure we engage fully with each country team within our client's organisation. This certainly helps with engagement and uptake of any recommendation and actions across the organisation, and steers us away from any countries feeling like they are 'frozen out'. However we also have to ensure that mean scores are looked at country-by-country by our clients rather than compared across the board. With this in mind we should ensure country teams based in locations where respondents typically give lower scores, are not 'beaten around the head' with these scores when compared to those in typically higher scoring countries.
In the last part of the session we had a brief look at some of the findings from the white paper "Comparing apples to pommes: understanding and accounting for cultural bias in global b2b research" by B2B International's Conor Wilcock (full paper can be viewed here: https://www.b2binternational.com/publications/understanding-accounting-cultural-bias-global-b2b-research/ ). Here we looked at the differences across countries in how respondents typically distribute their responses on scaled answers (are they polarised to the extremities of the scale, or do they cluster round the middle), and also the level of respondent acquiescence (the level to which respondents are agreeable towards the interviewer). With the learnings from this in mind, we settled on the point that we feel it is best to be aware of cultural difference, and bear these differences in mind when in interpreting data and making recommendations, rather than trying to normalise any differences in answers.
Slides from the presentation are also available to download by members. You must log into your membership to access these - they are in the News and Reports Section of the website.
Northern Forum: Wednesday 11 March. Online Qualitative Research Tools: Spanner, monkey wrench or mole grip?Written by Fiona Roberts-Miller
On Wednesday 11th March Acumen Fieldwork & the Pioneer Suite hosted this year’s first BIG Northern Forum, in association with the ICG. Our speaker for this event was Ken Parker, Chair of the AQR and Co-Founder & Chair of Discovery Research, Spectrum Viewing Facilities and The Thinking Shed. Ken’s thought provoking talk was around the topic of Online Qual versus Traditional Qual. Ken posed the question as to whether the two research methods compete or complement each other, highlighting the pros & cons in each and ways in which he himself has used them in his work.
He investigated the depth & rich insight you can gain from an online methodology, as well as the tried and tested results achieved through traditional face to face interviewing. There’s something to be said for the gaps filled by using online forums, Ken noted, and the ease in accessing a much wider audience, being able to engage with B2B participants and other demographics who may not normally have the time to be able to partake in research at this level. However we cannot get away from the non verbal /non written communication that can be picked up on in a group setting, the body language, eye contact and behaviour that aid us in understanding people’s thoughts and behaviour in order the answer the ultimate question – why?
Ken concluded that it is not a situation where one method of research ‘outdoes’ the other, but more that online qual is no more than another tool at our disposal as researchers, both can produce rich results and aid us in creating a fuller picture of the participant.
A big thank you to Ken and all those who came out to support the Northern Forum & we look forward to seeing you at the next event!
The London Forum on Tuesday 10th February was very well attended - slides from the session are available for members to download, but you must be logged in to access them. They are here in the BIG News and Reports section. We will also have a report soon.
On Wednesday 26th November, Acumen Fieldwork hosted the final BIG Northern Forum of 2014, in association with the ICG. Our speaker was Roddy Glen, one of the most experienced and respected qualitative researchers in the UK who has been in the industry for nearly 40 years.
Not so much a presentation, but more of an interactive session, Roddy asked us, the audience, as many questions as we asked him. He spoke about the changes that he’s seen in the industry, the importance of good branding and told us a few funny stories from his moderating days, one of which involved a rather boisterous dog!
The forum was brilliantly entertaining and engaging and we can’t thank Roddy enough for travelling down from Edinburgh to share his anecdotes and advice with us.
Thank you to everyone who has supported the Northern Forums of 2014, please watch this space for details of our first 2015 event!
Betty Adamou, Research through Gaming: The Evolution of Gamification Research
Gamification is the use of game thinking and game mechanics in non-game contexts to engage users in solving problems. Gamification has been studied and applied in several domains, with some of the main purposes being to engage, teach, entertain, measure, and to improve the perceived ease of use of information systems.
Please log into your membership to view a report on the London Forum on Tuesday 14th October, with a link to a presentation by Betty Adamou of the session.