You've made it to the third and final part. Incredible. I applaud you.

• Part 1: The Product and the Trivia Itself
• Part 2: The Business Side of WCT
• Part 3: The Technical Side of WCT ( >> you're reading this part)

The Technical Stuff

We love writing trivia and we love turning that trivia-love into trivia-money, but we also love making software. At the end of the day, without software I’m just a man yelling out random trivia facts into my webcam.

Here’s the four key technical areas where we invested Ryan’s brainpower:

1. Transitioning to Webflow
2. A/B Testing Trivia Difficulty Settings
3. Digging the Distribution of Slack and Teams
4. Automation and Algorithms

1. Transitioning to Webflow

Remember how earlier I wrote that co-founder Ryan had written 100% of our code? Well, we decided to make life easier by moving the front end “marketing site” off Ryan’s plate and onto mine. 

Inspired by this Tyler Tringas tweet, we started looking into what it would take to migrate our existing “marketing site” onto a “no-code” platform like Webflow. Well, not like Webflow. Specifically Webflow. 

We found a contractor through Webflow’s market of “Experts” and in two weeks we had fully migrated our site.

Our site. Not our entire app. We created a subdomain so that everything behind “auth” (i.e. logging in) remains app.watercoolertrivia.com. That is our web application, and Ryan builds those pages with our other engineering contributors.

Moving the marketing site to my plate meant I could build new pages quickly, run promotions, and set up design tests without slowing down Ryan from building more important and more impactful features. Plus I got to up-skill. I’m no Webflow expert, but I can scratch together a fun side project every now and then on the platform now.

2. A/B Testing Trivia Difficulty Settings

A/B testing is one of the many deities in the polytheistic church of Silicon Valley. Or something. So we hopped aboard.

Specifically, we’d heard for a few months that groups were disengaged during their free four-week trial of Water Cooler Trivia because the trivia questions were too hard. We heard this feedback, but didn’t act. That makes us bad entrepreneurs. Oops!

Then we did act. 

1. We tweaked what it means to be an “Easy” quiz. It used to mean that we give you questions with an average 70% correct. We moved that to a floor instead of an average.
2. For half of new trials, we defaulted the group into the Easy setting rather than the Medium difficulty setting we had traditionally used as default.
   

Making that change to half the groups meant we could measure the impact.

The results

Ultimately, the easier difficulty setting had a statistically-significant impact on increasing the share of free trials that converted to paying Water Cooler Trivia customers. Specifically, 

• For groups with easier quizzes, the share submitted increased by 5 percentage points (+10% relative)
• For groups with easier quizzes, the # of correct responses per quiz increased from 5.1 to 5.8 per quiz (most quizzes are 10 questions). Getting more questions right feels good.
• For groups with harder quizzes, they actively changed their difficulty setting away from the default at 2.5x the rate of groups with easier quizzes
  

So we rolled out the winning treatment. And eventually we launched even more difficulty settings. Accessibility rules.

3. Digging the Distribution of Slack and Teams

Technically our Slack integration was released in late 2019 and our Teams integration was released in early 2021. Editor’s note: the integration is in the final stages of Microsoft’s approval process as of Jan 25, 2021. However, we still consider integrations a key part of our year. 

Groups that sign up via Slack engage with Water Cooler Trivia about 2.5x more than groups that sign up via email. Realizing that, we added “Sign up with Slack” directly to our homepage. The popularity of both Slack and Teams exploded during the shift to remote work, so we doubled down on our product marketing and experience in the Slack and Teams apps. 

4. Automation and Algorithms

Free-text trivia rules. It’s more satisfying to pull that European capital or Indonesian ingredient from the back of your mind than randomly guessing multiple choice option C.

Of course, grading free-text responses is significantly more challenging than multiple-choice and true-false. We increase the challenge by declaring “spelling doesn’t matter” on each quiz. It’s cool with us if you don’t know the difference between Schwarzenegger and Schwartzenegger.

Historically, we’ve used a few clever tricks to keep false negatives and positives low (<0.5% of all responses). As the business has grown, we’ve had more manual grading on our plate, often requiring grading in the middle of the night.

We worked with a data scientist to assess eight different free-text matching algorithms on a set of thousands of our own hand-graded trivia responses. Some of the concepts:

• Vowel-stripping
• Levenshtein distance
• Token sort ratio
• Token set ratio
  

Unsurprisingly, we quickly discovered a tradeoff between 

• Increasing the share of responses “autograded” and
• Minimizing error rate
  

At one extreme, the “fuzzy partial ratio” matching algorithm (more detail here) was the most “aggressive” variant.  Although it would have reduced our manual grading by ~50%, it was likely to introduce hundreds of grading mistakes per week. 

We opted for a more conservative approach: “fuzzy token sort ratio.” We expect it to reduce our manual grading burden by ~10-15% without adding an increase in errors. 

Wrapping Up

Well, in way more detail than you wanted to know, that was our 2020 at Water Cooler Trivia HQ. We’re thrilled to have you with us as we roll out more features and keep the weekly trivia fun rolling for your team in 2021.