Category Archives: Market research

Honesty in market research

I just swapped emails with Tony Hufflett of The Fat Group. They’re trying to shake up the dull old world of market research; it certainly needs shaking up.

I thought this was an interesting point they make on their site about the honesty of responses to surveys.

There is room for respondent dishonesty in any questioning environment. The level of dishonesty grows with the increase in boredom or disengagment.

Our experience tells us that dishonesty (in any environment) occurs when the questioner either bores the respondent or the tone is wrong. This is even more true on the net when respondents are not having to act in an acceptable manner before another person (in a face-to-face encounter or over the phone (which creates its own artificiality)) instead they are able to be truly themselves and show how they feel straight away.

So the key to on-line honesty is to be creative, make it fun and don’t ask too many questions. If you still doubt on-line honesty you have only to look at the recent U.S and U.K elections. In both instances the web research came closest to the actual result. The reason: people couldn’t articulate face-to-face or over a phone something they felt the questioner might find unacceptable (such as under 35 and voting Tory in the last election) but would be honest about with their impersonal screen and in the voting booth.

When I do surveys, I recognise the point where boredom kicks in and I stop trying to articulate my real views and start rushing to get to the end.

Market research pratfalls

Tom Hamilton had me laughing this morning with this comment.

I spent much of yesterday travelling from London to Leeds and back by train and on the return journey I was handed a market research questionnaire for me to fill in to tell GNER what I thought of various aspects of their service. It included the following question:

Please tell us how far you agree with the following statements about GNER catering:

1) GO EAT from GNER provides fresh local food.

Local food? I’m on an intercity train. I’m travelling hundreds of miles at high speed. I have no idea what it would mean to say that the food I’ve bought in the buffet car is local (from Yorkshire? from London? from somewhere in between?); I’m inclined to assume that it isn’t local or that even if it was five minutes ago it isn’t any more; and I don’t care.

He also points to Chris Dillow’s related post containing this coverage from The Times of another case of the perils of market research:

Unveiling the final episode of the current run yesterday, Russell T. Davies, the writer, revealed that pre-transmission market research suggested that the BBC was heading for a £10 million disaster. He said: The research found that no one wanted to watch Doctor Who. Kids said it was a programme for their parents. The parents said it was a dead show. I expected it to die a death after one year. The research paper, based on interviews with viewers, is now gathering dust in a BBC marketing executive’s drawer. It found that viewers thought Doctor Who was a niche series for science fiction geeks, far from the family audience BBC One was seeking. The flop Thunderbirds feature-film revival was raised as a discouraging comparison. But the series has attracted seven million viewers, obliterating ITV1’s Saturday night competition, while remaining a critical success.

Measuring emotion?

The always interesting ecustomerserviceworld newsletter (I think that link is valid for two weeks) talks about Gallup’s survey methods.

One Gallup study demonstrated that a mid-sized bank could add $256 million to its total of customer balances on deposit by drawing higher (emotional) attachment scores from 50 000 customers.

Reading this reminds me of Denis Healey’s comment about watching your mother-in-law drive off a cliff… in your new car.

Should I be glad that here is some ammunition to show companies that emotions matter?

Or frustrated, because this is such a specious figure?

I tend to the latter. I question the idea that intangibles can be measured in this way. In life, much is unknowable and we must continuously use judgement and intution to progress. We don’t really need statistics to be good at dealing with people. Instead of spending a lot of money on measurements like this, I’d suggest a company spend more time really talking to its customers. Not “surveying” them.

Some will say, ah yes, but these figures are useful for dealing with hard-nosed businessmen who only value things with a price on them. That’s a classic shadow conversation. To the extent I want to engage with this, my guess is this: a businessman who doesn’t believe in valuing feelings isn’t going to look at this figure, having a damascene conversion and suddenly decide to be emotionally intelligent.

[UPDATE] Tom Guarriello expands on this theme with his usual eloquence:

Not that these experiences can’t be described empirically, just that they can’t be reduced to numbers and still retain their human qualities, which is, after all, what all of us are interested in learning about.

Market research: a waste of talent?

I took part in a market research street interview on Friday. I was ushered into a room over a pub and was sat in front of a Tablet PC.

It seems these days, after you’ve been recruited, they don’t even interview you. You have to sit and answer multiple choice questions on a PC. The process is pretty dull, and the questions reflect the clumsy mindset of brand management: “is such-and-such a brand for people like you?” Of course, soon I’m ticking answers without much care as I want to get finished, collect my free Mars bar, and leave!

I always feel a bit sorry for the people (mostly women) who do this work, I got the feeling they were bright people doing a boring job. Not the execs who do the powerpoint at the end of it all, but the street interviewers. The ground troops.

In fact, I always have this feeling about the folks at the sharp end of doing research – friendly, persistent, capable people doing work of little real value. Just how excited can someone get asking people to sit at a computer and say if they’ve seen ads of a tortoise with a diet coke on its back? OK, you could treat it as a zen exercise in attention I suppose, but apart from that?

Of course, the end product of all this effort are some crisp looking numbers. I can see the Powerpoint now: Fantastic! Another 3% slightly agree that the Ford Focus is a car!

Sure companies want, maybe need, to track awareness yadda yadda. But it all seems a bit of a waste to talent to me.

What would I do instead? I think someone at GM has a few thoughts.

The flaws of focus groups

Continuing my thoughts on the downsides of focus groups. I’m not saying they are inherently wrong but that there’s a lot that’s not good about the way they are often used.

Leapfrog Research had a good article in the MR industry magazine recently. Here’s a copy (word format). I love the descriptions of what goes on in the client side room (the room where the clients sit watching the group take place)

The viewing facility as business centre

Mobile phones recharged, laptops plugged in

Opining for Money

Last night I was paid to take part in a panel of experts (who me?) to give feedback to a financial brand about the trends we saw, their future strategy, blah blah. I met some interesting people and we talked and talked. Most of it was conducted focus-group style behind a one-way mirror and the best bit was when we came out and actually met the client at the end.

This allowed me to vent my concerns about the excessive amount of mediated communication in marketing. Too often, companies experience their customers at one remove, filtered through professional researchers, and then communicate back at one remove, through professional communicators. This has its uses but I think often just deadens the relationship and is responsible for a lot of very dull advertising. Sometimes these “communications professionals” just get in the way.

In a way, the format of the evening was a symptom of the problem. Although I’m perfectly happy to be paid to talk, I’d much rather have a real conversation than be dealing in abstractions through a one-way mirror.

Blogging at tipping point?

Jennifer Rice first pointed to these stats on blogging a few days ago. Jennifer and I were chatting about them on the phone yesterday, and Robert Paterson highlights the same story with the heading Blogging – Tipped? The stats are interesting.

As a professional researcher I take all of these numbers with a pinch of salt. So there may be 2.5 million U.S. bloggers or there may be 8.8 million. The real question such numbers help answer is, “Is it bigger than a bread box? Are we talking small, medium or large?”

Remarkably, when the Pew study first came out, AP spun the story as 2 percent being a surprisingly small number of bloggers (CNN.com attached the headline to that story: “Study: Very few bloggers on Net”.) Yet, blogger Rogers Cadenhead notes that Pew’s low-end estimate of the number of blogs is more than the 2.2 million copies that USA Today prints on regular weekdays, the country’s biggest newspaper.

I think something pretty cool is starting to happen here. Jennifer has asked me to support her on an international branding project – and we only know each other through trading our thoughts and ideas in blogs. I didn’t expect that to happen… and it’s really interesting to reflect on the impact on this work of us both being out here in public blogging. I realise that it creates a working relationship in which openness becomes especially important and valued.

That feels pretty significant. We’re creating a working relationship where there is multiplier for openness. And handling a branding project that – a few years ago – could really only have been done by a conventional agency. If this sort of thing becomes more established, that must have interesting implications for these businesses.

Overwhelm? What overwhelm? (And more on complexity)

Ton Zijlstra has done an eloquent blog Every Signal Starts Out As Noise. He argues, provocatively,

Why do we call information and data coming to us noise? Because we know not all that stuff is useful, we label the unuseful stuff as noise. And because of the tilted signal to noise ratio we perceive, i.e. what little we actually use from what comes at us, we say we suffer from information overload. I say that this is rubbish.

There is no such thing as information overload. It does not exist.

The whole piece is well worth a read. Referring to Dave Snowden’s thinking, Ton argues that in a complex world the priority is not analysis, requiring us to read and know everything, He identifies six rules of thumb, all of which I like.

Look at what you see, not at what you don’t see

You are your own filter, important stuff will bubble to the surface at some point

It’s about the few actions you do take, not all the actions you could have.

Skim, not read, all available info, don’t judge yet

Combine what strikes you at first as possible patterns (barriers and attractors), and examine those more closely

Build upon the patterns, and choose one or two to cultivate and act upon

To me what’s central to dealing with the the symptoms of information overwhelm is to break off from looking to outside knowledge for one’s sense of self. I find it very easy to feel ill-informed in a world of so much information, and find it useful to remind myself that I know enough. Indeed, that I am enough

Ton’s material has prompted me to think again about how marketing needs to think and work quite differently if we accept the paradigm of a complex world, one in which it’s not easy to distinguish the signal from the noise, nor to predict with any certainty the consequences of our actions. One aspect that particularly strikes me is market research, where vast sums are spent in an effort to penetrate deep into the customer mind, as if to find a few key triggers that can turn markets. With this goes the old adman’s holy grail of the “unique consumer insight”. It represents an attachment to analysis as if this will usefully guide future behaviour. So often such “insights” cost a lot of money and turn out either to be either bland or misleading. (I blogged at length on this point here).

Along with this is the delusion that by massive promotional expenditure, you can impact in some predictable way how markets will operate… that somehow a brand is created by the brand manager, based on his/her alleged deep insights into their market. A feature of complexity is that cause-and-effect can only be discerned retrospectively and cannot be used to predict the future. Many an adman will cite questionable case histories, out of context, to justify big expenditure now. Such approaches often blind management to the subtle signals out there in the real world where real brands are created in a myriad of conversations. (Leading me back to Ton and his arguments for the value of blogging as means of participating in those conversations, rather than hypothesising about them.)