Turning Card Sort Lemons into Content Strategy Lemonade

The first rule of card sorting is…

No. I can’t do it. I refuse to do the Fight Club joke.

There are rules, though. Unwritten rules, yes, but rules all the same. And that first rule is the one we’re all taught to revere from the beginning: you don’t facilitate a remote card sort unless you absolutely have to.

Card sorting is a science that requires interaction and arguing and probably a few sandwiches and Diet Cokes. You do it in a room, on a table, on a chalkboard, behind smoky glass, out in the open; WHATEVER. Just make sure it’s live.

In theory, yes. This makes perfect sense.

But. BUT.

Budget, distance and sample size often get in the way, making the dream of in-person, get-to-know-you card sorting unattainable.

So you turn to remote card sorting, and you shield your eyes. Because while you’re going to get a lot of answers for a lower cost, you’re also going to get some serious stinkers. There are going to be some “Just get me through this so I can get a chance to win an Amazon.com gift card” stinkers. There are going to be some “I don’t really understand this process” stinkers. Most of all, there are going to be some “Who really gives a crap?” stinkers.

Good. Embrace them. We need to accept the fact that, no matter what, there are people who just don’t give a shit. Because those people are users, too, and we STILL need to sell them widgets.

“We Don’t Know” vs. “We Don’t Care”

This issue came up recently on a project with one of our clients. They’re known for huge scoreboards, but their product line includes everything from basketball shot clocks to arena-sized audio systems.

In order to determine how users research their products, we set up a card sort for the sports scoreboard section of the site. We facilitated the sort across three groups: the marketing department, regional sales managers, and site customers.

One group – the organization’s marketing people – worked through their card sort in person. A second group – regional sales managers – were poised to give more user-centered categorizations, but were too far scattered to work in person, so we settled on a very focused remote card sort.

Then we took the card sort to the streets by asking past and potential customers how THEY would sort categories.

We also did this one remotely, and we got a lot of feedback. A lot of it was good.

A lot of it was also pretty bad. We got a lot of people sorting cards into catch-all categories just to get through the sort. Categories like “Other.” “Miscellaneous.” “Never.” And my favorite: “Stuff We Don’t Care About.”

The answers were, at first, disappointing. That is, until we got over what we EXPECTED and focused on what we HAD. “There it is!” we said. “The silver lining!” we gasped.

Most of these users weren’t shirking their duties and refusing to categorize content categories – they were telling us that they don’t want that extra cruft. They want to find their product and move on. Who has time to worry about the other sections?

They weren’t saying, “We don’t know.” They were saying, “We don’t care.”

Those Lemons? They Become Lemonade, Yo.

What this means, of course, is that when in the comfort of their own home, card sort participants will be brutally honest. We’ve been conditioned to believe that live card sorting is good and remote card sorting is only acceptable, but by including non-plussed, narrow-focused users we’re hitting closer to home.

These aren’t people who are working as a group to find a solution – these are people in their natural state, warily searching a new site, ready to bolt for the first sign of familiarity.

Live card sorts give us structure and discussion. Remote card sorts give us strength in numbers. Flippant or bad individual remote card sort entries give us a touch of detail and a look inside the minds of our users.

We have to confirm that we’re able to effectively interpret these bad entries, though. Those quick steps are as follows:

1. Make sure you’re able to separate your data by groups.

Confirm that the remote software will allow you to ignore certain results. We use Optimal Workshop’s OptimalSort, which allows us to select groups of sorters and reload results based on user type.

2. Pull out I Don’t Care entries.

When you come across an entry that is obviously by someone who doesn’t care – think “75% of cards placed in an Other category” or “nonsensical category names with no real organization” – take note. Leave them out of the synthesized results.

Do not delete them, though. Just take them out of the analysis pool. We’re not discounting their contribution, but we ARE making sure they don’t skew the results.

3. Begin looking for trends.

Here’s where those “too busy to finish” jerks can be of some help. Are there cards or groupings that are commonly ignored? When someone says “Other,” is it because they’re not understanding the content or because it simply doesn’t apply to them?

You’re going to find some card sorters will sort one section of cards and leave the rest for dead in an “Other” category. How often is this happening?

4. Match those trends to the overall results.

Refer back to the stats and groupings. Are there unspoken issues that are suddenly being spoken in the bad entries? Are there chunks of content that the group sort doesn’t come to agreement on? How often do those chunks correspond with a bad entry’s lack of attention?

And, like, um… is there actually a need for an “Other” category? Is the hive mind too polite to break the rules and say, “these things don’t fit anywhere else?”

Want a Moral?

The moral: there isn’t one.

Like everything in content strategy, information architecture, web development, television sitcoms, tabloid newspapers and life itself, the value in a process or concept is what you get out of it. Card sorting remotely is usually not a Best-Bet-A+++-Would-Buy-Again prospect, but it’s often the ONLY prospect.

If there’s one skill that is too often overlooked in the web strategy industry, it’s the act of continuing to find useful information in what seems to be rubbish data. You focus on what you get, sure. But you also focus on what’s NOT there.

Clients who don’t care? That’s a sign. Users who won’t answer? That’s a sign, too. We work in counseling and psychology as much as we work in spreadsheets. This is one way we can bridge the gap.