Just as all cars can be expected to have wheels and an engine, so too should all online omnibus companies be expected to field your questions and retrieve useful data. The types of questions one might ask when shopping for an online omnibus supplier are the same one might ask when shopping for a car: “What do I really want?” and “If what I want is not standard, can I get it as an option?”
Whether it’s cars or research, the truth is we don’t always know what we want or know what’s available. Ten years ago having a GPS come as standard equipment in a new car would have been unthinkable, but today it’s a reality in several models. Learning what to ask is just one step in getting what you want. Learning what’s available is the other.
When it comes to online omnibus research, there are several areas you’ll want to be familiar with in order to maximize the value of your investment. The questions you should consider are:
- Is the sample sufficient enough for you to get the insights you need?
- Is the questionnaire design process easy and accommodating?
- If the incoming sample is different than what is expected, can the data be weighted to reflect the population?
- Will you receive data in a format that will be useful and intuitive to understand?
- Will most of the thinking be incumbent upon you? Or will there be experts whom you can rely upon?
Today’s post focuses only on elements that pertain to research sample.
The data in any research study is only as good as the respondents who are willing to participate in the research. As the saying goes “garbage in, garbage out”. Making sure the sample comes from a reliable source is essential. Ask your vendor to explain where their sample comes from and what kind of quality controls they have in place to ensure the integrity of the output. If a vendor can’t provide reasonably good answers about the quality process in a timely fashion, you might want to look elsewhere.
The next step is to determine how much sample you actually need to meet your objectives, keeping in mind future needs as well as current ones. Shifting from vendor to vendor can be time consuming. If you plan to use omnibus research on a semi-regular basis, find an omnibus company that can anticipate your needs and that you believe will consistently meet or exceed your expectations.
You can find online omnibus companies that offer standard samples ranging from 500 completed interviews to 5,000, though most offer samples sizes in the 1,000 to 2,000 range. Choosing the vendor with the right sample size can be tricky as the size of the sample you’ll need almost always will depend upon whether you can get the right number of completes on the back end. Irrespective of sample size, you’ll need to determine if an online omnibus service is right (or any kind of omnibus service for that matter), or would you be better served with a targeted custom survey. A responsible vendor will likely help you work through the numbers. To illustrate the point, imagine you want to learn more about the behavior of widget buyers. Let’s say that widget buyers represent 10% of the population and you want at least 200 widget buyers. If the standard sample consists of 1,000 general population respondents, you might reasonably expect to obtain about 100 respondents who buy widgets. Since 100 is only about half of what you need, starting with only 1,000 respondents poses a problem. On the other hand, a sample company that starts with 2,000 respondents should be able to deliver what you need.
Another important thing to consider regarding sample size is how you intend to analyze the data. If you plan to analyze data across multiple segments, the sample will have to be able to support those analyses. Suppose that widgets have a 50% incidence rate, but you want to compare attitudes and usage between loyal widget buyers and buyers who switch between widget brands. Let’s also say that loyal buyers constitute about 30% of the widget buyer base while switchers constitute the remaining 70% and you need at least 200 respondents per buyer group. If we start with a sample of 1,000, multiply it by 50% for the incidence rate and then another 30% for the loyalty classification, we wind up with only 150 loyal widget buyers, short of the 200 needed.
So as we see from the widget examples, using a company that offers a starting sample of 1,000 respondents won’t meet the project’s objectives. That being said, you may still have options and should find out whether your omnibus company can:
- Field the study twice and have the data aggregated in a timely manner
- Increase the number of completes it typically offers
As with any business decision, the choice of whom to use comes down to what is doable and cost-effective. If you continually get more sample than your analyses require, you could be paying for something you don’t need. The only way to know for sure is to check the price differences against competing services and weigh it against the other benefits you might get. As a general rule, it is wise to start with an online omnibus company that has the right basic sample “horsepower” for the majority of your omnibus needs, but also has the capability to reach back and get you a little something extra from time to time.
While size does matter, it’s not all that matters. Find out what your supplier can do with their sample. Can they split it into multiple cells? Being able to randomly assign respondents to different cells is critical if you plan to use an omnibus to test advertising or product concepts. Since sample size in an omnibus survey is usually fixed, the number of concepts that will get tested affects the sample size for any given concept. In other words, the more concepts there are, the smaller the base sizes will be per concept; so identifying what kind of sample options your supplier can offer will help you determine which provider is right for your business.
Sample policies are just one of the several factors to consider when choosing an omnibus service. Our future posts will highlight additional factors that will aid you in your decision-making when it comes to choosing between omnibus providers.