Definition of the target population that you need information from is critical. If the population, for example is AFN members, that makes definition easy. If the population is residents of the USA for another example, that makes for a bigger project to put it mildly. But whatever your target population is, it has to be defined clearly up front.
Once you know who you are interested in, you need to figure out how to sample them (it is rarely feasible to contact them all). This can be complicated for a variety of reasons, and is more often than not the reasons that polls and surveys fail. Just for example, (not getting into politics here, just using something recent that peeps here might have noticed) multiple polls underestimated Trump's likely election success because apparently those most likely to vote for him were least likely to respond to pollsters. Even if one were to sample AFN members, how to sample them becomes complicated. Many members are not currently active, are they excluded, and what would that do to interpretation of the results? There are an almost unlimited number of ways that sampling populations of people can fail due to unanticipated bias, and good surveys are careful to explain what has been done to avoid the obvious pitfalls.
Once you figure out who exactly your population is, and how to sample it, you need to figure out what to ask. Questions must legitimately generate information relevant to the issue being explored. But beyond that the language used is critical and must be sufficiently neutral to avoid introducing bias. I imagine we have all (those of us who have ever listened to the questions) experienced polling obviously designed to obtain the desired answers. I rarely partcipate in polls, often for this reason (although I have been known to strategically lie rather than leaving an obviously biased poll - feels good, but probably accomplishes little
)
Bottom line, survey design is everything, absent bulletproof design from beginning to end, the output is not useful and could be harmful. Surveys on a country wide scale are huge, difficult, and expensive projects, and nonetheless regularly fail in spite of expert effort due to their complexity.
So, back to the ILGM report/survey, IMO it fails miserably on several counts:
1. The population is undefined. Exactly which home growers are part of the population? ILGM members only? Any US resident? Residents of legal US states only? Home growers who never ever sell anything? "Home growers" with what more accurately be described as a smallish grow up for illegal sale? Etc. Etc.
2. The sampling methods are undescribed. This is huge, especially if the population is intended to be country wide and not part of ILGM. What means were used to contact people? At what time of day? The list goes on and on.
3. Not much need be said about question design, complete fails on #1 and #2 render the details irrelevant.
Anyone who gets this far in this missive is granted several extra smoke breaks, not to mention my admiration for patience.
