Language selection

Government of Canada / Gouvernement du Canada

Search

How to analyze comments: Canada.ca design

On this page

Who should analyze comments?

This should be done by people who know the subject quite well.

People should be adept at spotting patterns and themes in data.

It’s best to have someone who is bilingual.

If more than one person is sharing the task of reading comments, having a shared understanding of the issues (and how you will group comments) is very important.

It’s good to get into the habit of looking at user comments regularly to identify any emerging or persistent issues affecting task success.

How many comments to read

There is no magic number for how many comments you need.

With comments, you are looking for enough to sufficiently describe an issue or answer a research question. There is a point of diminishing return when reading more comments does not lead to additional insights. This is called “saturation”.

Start with a small timeframe of data to see if you can identify issues and trends in the comments. Expand your date range to include more data and gain more clarity.

Manual comment analysis

Grouping comments with similar issues together with tags is useful for both small and large datasets. It helps you be more efficient with analysis by having smaller sets of data to analyze.

A small dataset may only need a few tags to make sense of the comments. A large dataset may require two levels tags to understand specific content issues.

Best practices for choosing how to group and tag comments

Familiarize yourself with your data

Read through a sample of comments and try to spot recurring patterns. Jot them down to get a rough overview of WHAT tasks, topics, or issues people are talking about.

Not every comment will be useful - sometimes it will be too unclear to use or be completely about another topic.

Consider tags based on a task or issue

Task-based tags are recommended when analyzing comments for groups of pages where there are multiple user tasks.

To identify tasks, ask yourself why the user came to the site. What were they trying to do, or what question were they trying to answer?

Example tagging model by task
Task tag User task Topics included
Vaccine safety Is the vaccine safe for me? Pre-existing conditions, ingredients/allergies, side effects
Getting vaccinated How do I get vaccinated? Eligibility, when, where, how to register
Proof of vaccination How do I get a copy of my vaccine record? Vaccine records, provincial apps, federal vaccine proof

Issue-based tags may be a better strategy when analyzing comments on a single page, single topic, or where a single task dominates your feedback.

Example tagging model by issue
Issue tag Issue experienced
No appointments No appointments, no appointments for several weeks

For large datasets you may find a second level of tags is needed to add precision. This can be done at the same time you tag the comments OR when you are ready to analyze a smaller set of comments.

Limit the number of tags being used

Start with broad tags and only include those for which you have multiple examples. Your goal with this first review is to succinctly group recurring topics/issues.

Aim to keep your set of tags to under 15 for the task. Limiting the number of tags will help surface the issues that need the most assistance.

“Other” is a tag too! Tag one-offs or low-frequency comments as “Other” until there are enough for them to graduate into having a tag of their own.

Avoid using overlapping or ambiguous tags

Make sure each tag is clearly differentiated from the others. Your aim is to reduce doubt about which tag a comment should get.

Be prepared to tweak your choice of tags

As you read more of your dataset, review your initial tag choices. Are they clear and unambiguous? Does one tag alone cover the majority of comments? Do you need to divide it into separate tags?

There’s no one-size-fits-all strategy. As you collect more comments, you may find you need to adjust your choice of tags.

Document and test your tagging strategy

Document your choice of tags with examples. This is especially useful if more than one person will share the responsibility for reviewing comments.

Ask others to review your tag choices to make sure that the tags are clear to other people. This is especially critical if more than one person will be helping to analyze comments. Agreeing on a common set of tags in the beginning (and when adjusting tags) avoids comments being tagged poorly between people.

Download a tagging strategy template (Excel, 61KB)

What to avoid when tagging comments

Mixing types of tags

If you want to add additional ways to analyse your dataset, it’s best to create new columns in your spreadsheet to note different kinds of facets. For example, adding a status or specifying a sub-issue.

Trying to be overly-precise

The purpose of tagging is to help you identify user priorities and group comments into smaller datasets to analyze. A “good enough” approach to defining and assigning tags will do.

If you have more comments than you can manage to review, classify and analyze, adjust your strategy: choose a specific task or time frame to focus on.

Tools and templates for comment analysis

For small datasets, any spreadsheet software should be adequate (Excel, Google Sheets) to sort and review comments.

For larger datasets, it’s helpful to use a tool that has more advanced functionality to sort, filter, and tag. If you have a data science specialist, they may prefer or have access to other specialized software.

Templates

Include other supporting data sources

Include other data sources in your reporting to build a more complete picture, confirm your insights, or add urgency from sources such as:

Page details

Date modified: