The reliability and accuracy of market research conducted in Germany has been called into question after the news magazine Der Spiegel gained access to thousands of internal documents from a number of market research companies. According to Der Spiegel, the documents, which include emails, questionnaires, instructions for interviewers and data sets, point to systematic manipulation in the sector.
The magazine reviewed documents from approximately 600 market research studies covering a 10-year period up to late 2017, as well as speaking with former and current workers in the industry. Der Spiegel said that the documents support the notion that a number of the 50 biggest market research institutes in Germany are at least partly using unsatisfactory methods.
Der Spiegel highlighted a survey commissioned in 2016 and 2017 by Merck & Co., which the news magazine said hired market research company Infratest, part of the Kantar Group, to field a questionnaire to oncologists about the cancer immunotherapy Keytruda. Der Spiegel noted that Kantar Health subcontracted the work to the field institute CSI International, which in turn was reported as having passed on the majority of the survey to another company that allegedly conducts the interviews using a range of freelance workers.
According to Der Spiegel, only a few doctors were called and interviewed, with these genuine responses used to fabricate further questionnaires with similar results to compile the full data set. The magazine noted that Merck later requested a second wave of the survey, with CSI issuing instructions that the data set for the new questionnaire should not deviate too strongly from the first wave. According to Spiegel, the documents show that CSI paid 10 euros ($12) per interview. Oncologists completing such a survey would normally be paid 10 times that amount, when not considerably more.
Der Spiegel indicated that the overall turnover for the market research sector has remained flat for a number of years at around 2.6 billion euros ($3.2 billion) annually, while the prices for surveys and market studies have been falling. "Some offers are simply too cheap. No real interviews can be performed for these prices," said Thomas Bäumer, professor for economic psychology at the Stuttgart University of Applied Sciences..
In response to the investigation, Merck are said to have told Der Spiegel that in the returned data set, there were "isolated implausible results." Meanwhile, Kantar are said to have confirmed that there have been problems with the quality of data from CSI, noting that for the Keytruda survey, "we repeated the questionnaire in our own company."
"Manipulation and deception in studies are very important issues in the sector," remarked Bäumer. Although he suggested "that most institutes comply with the International ESOMAR Code," he also said there would be a number of reasons for research companies to employ questionable practices, including cost. Other industry people have previously cast doubt on the quality of surveys conducted at low prices, with Joerg Ermert in 2013, then director of fundamental research with Deutsche Telekom, noting "rumours were constantly being circulated that certain institutes weren't being very particular about conducting proper interviews."
Der Spiegel highlighted problems with the use of subcontractors, which the majority of research institutes use, including Gesellschaft für Konsumforschung (GfK) and Kantar. The news magazine noted that prices for subcontractors, such as field institutes, can differ considerably for the same project. "If someone has two offers where one is 20 percent cheaper than the other, how is he likely to decide?" asks Bäumer, adding "in both cases he gets the results of a study as a data set... But you don't see the results, how they were reached."
According to Der Spiegel, when Kantar confirmed that there were problems with the data quality from some contractors, they attributed some of these problems to price pressures in the sector. "There are instances," they were reported to have said, "where we no longer quote because it is impractical from a cost perspective."