As a membership manager, your members’ satisfaction and level of engagement is front of mind. You want your offering to align with their needs and expectations. The member satisfaction survey is one of the best tools to measure member engagement and this post will help you build and implement an effective survey.
Why use a member survey?
Information on growth and retention can easily be obtained from standard membership reports but information on member satisfaction is much harder to come by. This information cannot be found in numbers and spreadsheets; it must be gleaned from your members directly. This is where a well timed and meticulously crafted membership survey comes in.
Can I use a survey template?
Sadly there is no definitive template for surveying your members. The best performing surveys are tailored to your audience. That said, this post is packed with examples and pointers so that you don’t need to start from scratch.
How to create a successful member satisfaction survey
The rest of this post will list the 9 concepts you should understand to ensure your membership survey is a success.
1. Pick a single focus
Great surveys start with the end in mind. What is the specific overarching question you wish to answer?
- Do you want more information about what hooked your new members in?
- Are you keen to know which part of your offering is most valuable to your members?
- Do you have a hunch that there is a group of less engaged members who need different services?
Keeping one question in mind allows you to keep your survey short (people hate long surveys) and actionable (what’s the point if it doesn’t lead to action?).
2. Define your audience rigorously
To get a reliable survey result you need to think carefully about the group of people to whom the survey is relevant. In surveying terminology, this is your ‘population’.
Often your population will be your entire membership base but in some cases, you may run a survey that is only relevant to a sub-group. For example, if some of your members are students you may do a special survey just for them. In that case, your population would be just your student members.
The power of samples to avoid bias
One counterintuitive technique in surveying that can dramatically improve your success is sampling. Sampling is where you invite only a portion of your population to complete the survey.
The temptation is often to send your survey to everyone in your population because this may get the highest number of responses. But a high response rate is often a misleading metric to chase because it will compromise the reliability of your survey data.
For example, if you invite 2,000 people (all your members) to participate and receive 200 responses your sample will be biased because it will only include the 10% of your audience most likely to respond to surveys. These keen recipients are unlikely to be representative of your overall target population.
A better approach would be to reduce your target sample size by choosing 500 members at random from your population and put more effort into obtaining a much higher response rate. If you could get up to a 50% response rate this data would be more reliable. See the section below for tips on increasing the response rate.
It’s easy to get a sense of the best sample size to avoid bias using an online calculator like this one.
3. Keep it short
Now you have a distinct overarching question and a defined audience, you need to design your questions.
The shorter your survey, the more responses you are likely to receive. The best rule of thumb is to make your surveys feel uncomfortably short.
Another useful filter when shortening your draft survey is to focus on actionability. Where it is unclear what you would do differently as a result of the response, the question can probably be removed.
4. Choose the right question type
There are two types of questions: open and closed. Choose your question type wisely because each is appropriate to different situations.
Open questions for detailed feedback and new ideas
Open questions provide a comment box so respondents can provide any answer they like.
Benefits:
✔ Discover new information or ideas
✔ Let members speak freely in their own words (and thus understand how they feel)
✔ Solicit detailed opinions
Challenges:
- Time-consuming to review
- Interpreting answers can be subjective
- Fringe issues or complaints can be distracting
- They take respondents more time to complete
Example of a good open question:
“What topics would you recommend for our discussion forum series?”
Closed questions for validating existing ideas
Closed questions provide a defined list of responses.
Benefits:
✔ Easy to aggregate responses
✔ No room for subjective interpretation or distraction
✔ It can be easier to take action based on closed questions
Challenges:
- It can be easy to ask leading questions
- It is possible a respondent may not resonate with any of the response options
- It is easy to respond without thinking so hard so answers may not be truly representative of how the member feels
Example of a good closed question:
When would you be most likely to join a 30-minute webinar?
- Before work
- During work hours
- Lunchtime
- Evening
- Weekend
- None of the above
How to mix open and closed questions
It’s often good to mix open and closed questions and over time you can let these inform each other.
For example, an open question like “What led you to register as a member of our organisation?” is great for your first few surveys. It will build your awareness of the range of options and over time you will likely be able to move to a closed question with a set of defined response options instead.
Provide a ‘none of the above’ option
Don’t forget to provide people with a ‘get out’ for your questions. It can be frustrating for respondents to feel pushed into a point of view that isn’t quite how they feel. You can avoid this frustration by making answers non-compulsory or, even better, you can provide an extra option with one or more of the following statements: ‘None of the above.’ ‘Prefer not to say.’ ‘Not applicable.’ ‘All of the above.’ With these statements added to your survey response, if someone does not want to provide a specific answer at least you know why.
This option is particularly helpful when refining your survey. If many people reply ‘none of the above’ or ‘not applicable’ this tells you that you have not pitched your survey correctly and that you need more open questions to understand your members.
5. Try a pilot survey
If you are planning a big survey you can give yourself an extra chance to get the questions right by using a pilot survey. This is where you try out your survey on a small number of people to check that others read the questions as you intend them to be read. You may pick up mistakes in the survey like missing a check-box or using jargon that your members don’t understand. You may also find that some questions elicit a predictable response and can be removed.
You can send the pilot survey to people within your organisation to check over your work, or you could choose a few people at random from your population (not those who you intend to send the final survey to!) as survey guinea pigs – you don’t need to tell them it is a pilot!
Now you know how people answer your survey, you can make final adjustments and complete your survey template.
6. The power (danger) of suggestion
It can be quite easy to inadvertently ask subtly leading questions in your survey. Consider the four different ways of asking your members about how they subscribed to your organisation:
- “What led you…”
- “What made you…”
- “What encouraged you…”
- “Why did you…”
The words ‘led’, ‘made’ and ‘encouraged’ have slightly different connotations. To be ‘led’ refers to signposting, to be ‘made’ implies coercion, ‘encourage’ has paternalist undertones. “Why did you…” is far more direct, but in some contexts may sound accusatory.
Questions may also lead respondents to answer a certain way in following questions by planting ideas in respondents’ minds that develop over the course of the survey. For instance, a survey may focus on the events offered by your organisation but end with a question like “What member benefits do you find most useful?” Since the whole of the survey has focussed on events, you might expect that a more than average number of respondents would reply that ‘events’ was the most significant member benefit.
Once you have drafted your response, read through it again and look for examples of leading questions.
7. Increase response rate
You have your sample, you have your survey and people have responded to your survey. But, only half of them! Is a 50% response rate good enough?
The answer to this question depends on what you are trying to achieve. In some cases, any feedback is useful. But for a rigorous approach that reduces the chance of bias you should be aiming for an 80% response rate.
Working toward a high response rate is important because the people who responded initially may be unrepresentative. By chasing those less keen to respond you may unearth valuable information that you would not have got from the keen early respondents.
Example:
Your survey asks why people signed up to your membership. Those who didn’t reply to your survey might be the respondents who signed up through your recent offer. They are a subgroup of your sample who respond only to incentives, and your survey request wasn’t worth their while. If you give up too early you may miss this insight.
Practical ideas to increase response rate:
✔ Focus on a narrow sample of your membership base (as discussed in section 2 above)
✔ Offer incentives for longer surveys (e.g. a few months free subscription, a referral link for friends with mutual benefits, free trials or a prize draw)
✔ Offer incentives only to those who do not reply immediately
✔ Send follow up emails to the target sample and if possible exclude those who have already replied (this may be possible through your membership management software).
✔ Send follow up emails.
✔ Give people a call. They may be happy to answer the questions right away.
Finally, know when to give up. Some people may just really not have the headspace for a survey at this time.
8. Always follow up
It is always good practice to thank respondents and let them know how their contribution helped you. This will increase the chances that they participate again.
At first, this doesn’t need to be in too much detail. A little nod is all that is needed: “Thank you, you helped us improve the way new members find us and join our membership community.”
However, the best follow up is always to explain the results of the survey and how they are changing the way you work with your members. It is this kind of feedback that makes members feel like their voice matters!
Taking the time to summarise results and corresponding actions will not only increase the chances that your membership offering improves but it will also increase member engagement by increasing members’ sense of participation.
Here is an example of how you might acknowledge some of the specific responses.
“Twenty percent of our new members said they wished that they could use direct debit. We’ve taken this on board and updated our membership functionality in line with this and we now offer Direct Debit as a payment method. You can click here to update your payment method at any time. Thanks again for your valuable input.”
9. Going again
Well done, you have successfully completed a survey of your members! You’ve learned more about how they think and you’ve implemented some of their suggestions.
Before you move on to the next thing, these three tips will help you build a long term survey strategy that continuously increases member engagement:
✔ Capture outstanding questions to explore in the next survey
✔ Carefully record the sample you have surveyed so that you don’t over-survey them; perhaps focus on a different sample next time
✔ In some circumstances consider repeating a similar survey after a year has elapsed to evaluate whether the actions you took have had the desired impact
