You've made it to the third and final part. Incredible. I applaud you.
We love writing trivia and we love turning that trivia-love into trivia-money, but we also love making software. At the end of the day, without software Iâm just a man yelling out random trivia facts into my webcam.
Hereâs the four key technical areas where we invested Ryanâs brainpower:
Remember how earlier I wrote that co-founder Ryan had written 100% of our code? Well, we decided to make life easier by moving the front end âmarketing siteâ off Ryanâs plate and onto mine.Â
Inspired by this Tyler Tringas tweet, we started looking into what it would take to migrate our existing âmarketing siteâ onto a âno-codeâ platform like Webflow. Well, not like Webflow. Specifically Webflow.Â
We found a contractor through Webflowâs market of âExpertsâ and in two weeks we had fully migrated our site.
Our site. Not our entire app. We created a subdomain so that everything behind âauthâ (i.e. logging in) remains app.watercoolertrivia.com. That is our web application, and Ryan builds those pages with our other engineering contributors.
Moving the marketing site to my plate meant I could build new pages quickly, run promotions, and set up design tests without slowing down Ryan from building more important and more impactful features. Plus I got to up-skill. Iâm no Webflow expert, but I can scratch together a fun side project every now and then on the platform now.
A/B testing is one of the many deities in the polytheistic church of Silicon Valley. Or something. So we hopped aboard.
Specifically, weâd heard for a few months that groups were disengaged during their free four-week trial of Water Cooler Trivia because the trivia questions were too hard. We heard this feedback, but didnât act. That makes us bad entrepreneurs. Oops!
Then we did act.Â
Making that change to half the groups meant we could measure the impact.
Ultimately, the easier difficulty setting had a statistically-significant impact on increasing the share of free trials that converted to paying Water Cooler Trivia customers. Specifically,Â
So we rolled out the winning treatment. And eventually we launched even more difficulty settings. Accessibility rules.
Technically our Slack integration was released in late 2019 and our Teams integration was released in early 2021. Editorâs note: the integration is in the final stages of Microsoftâs approval process as of Jan 25, 2021. However, we still consider integrations a key part of our year.Â
Groups that sign up via Slack engage with Water Cooler Trivia about 2.5x more than groups that sign up via email. Realizing that, we added âSign up with Slackâ directly to our homepage. The popularity of both Slack and Teams exploded during the shift to remote work, so we doubled down on our product marketing and experience in the Slack and Teams apps.Â
Free-text trivia rules. Itâs more satisfying to pull that European capital or Indonesian ingredient from the back of your mind than randomly guessing multiple choice option C.
Of course, grading free-text responses is significantly more challenging than multiple-choice and true-false. We increase the challenge by declaring âspelling doesnât matterâ on each quiz. Itâs cool with us if you donât know the difference between Schwarzenegger and Schwartzenegger.
Historically, weâve used a few clever tricks to keep false negatives and positives low (<0.5% of all responses). As the business has grown, weâve had more manual grading on our plate, often requiring grading in the middle of the night.
We worked with a data scientist to assess eight different free-text matching algorithms on a set of thousands of our own hand-graded trivia responses. Some of the concepts:
Unsurprisingly, we quickly discovered a tradeoff betweenÂ
At one extreme, the âfuzzy partial ratioâ matching algorithm (more detail here) was the most âaggressiveâ variant. Although it would have reduced our manual grading by ~50%, it was likely to introduce hundreds of grading mistakes per week.Â
We opted for a more conservative approach: âfuzzy token sort ratio.â We expect it to reduce our manual grading burden by ~10-15% without adding an increase in errors.Â
Well, in way more detail than you wanted to know, that was our 2020 at Water Cooler Trivia HQ. Weâre thrilled to have you with us as we roll out more features and keep the weekly trivia fun rolling for your team in 2021.