Aaron O'Connell

Read this first

The Math of Split Testing Part 3: The Chance of being Similar

tl;dr

Sometimes we want to verify that a new design will convert at nearly the same rate as an old design. Split tests of this type are not intended to find conversion rate wins, but rather to ensure that the new design is not “too much worse” than the old design. Here, we demonstrate how to analyze a split test of this sort and present an approximate formula to quickly calculate the chance that there is an acceptable difference between the conversion rates of the new and old designs.

Let’s say that we think that the design of our website is starting to look a little outdated and needs to be redone. In this scenario, we’re not really looking to increase conversion rate. We’d be happy to keep our current conversion rate and just make the website look a little more hip. To verify that our new design (branch B) converts at a similar rate as our old design (branch A) we should run a...

Continue reading →


Node FTW

tl;dr

Want to crunch some numbers? Node is 10x faster than Ruby.

I used to do a lot of scientific computing. My work environment had both Matlab and NumPy/SciPy all tricked out. These days I don’t do too much number crunching, spending most of my time with Ruby and JavaScript.

Recently I’ve been interested in the math of split testing so what do you think I turned to when it was time to do some simulations? That’s right, Ruby… I didn’t really think about what I was doing before I jumped in.

I had reached for Ruby even though the I coding up a numerical simulation instead of some web app. I wanted to simulate 100,000 coin flips as one trial, and then repeat it for a bunch of trials to decrease the statistical sampling error. Here’s the results:

$ ruby aa.rb 100

Trials: 100

Total Time: 21.243 seconds

But I needed more accuracy

$ ruby aa.rb 300

Trials: 300

Total Time: 62.260...

Continue reading →


The Math of Split Testing Part 2: Chance of Being Better

tl;dr

Split tests are a powerful way to determine if a new design has outperformed an old design. Here, we present the math necessary to empirically state the chance that a new design (branch B) is better than an old design (branch A). We walk through the problem with numerical simulations and present an approximate formula to easily determine the winner of an A/B test.

Split testing is the preferred way to collect data for fact based decision making. Longitudinal studies are subject to underlying market trends and are not well suited for conversion rate experiments. If, for example, you roll out a new version of your website in April and notice a decrease in conversion rate, you may be tempted to say that your new website design is a failure. But maybe April was just a bad month. If instead you had set up a split test for your new design, you could eliminate this concern. While...

Continue reading →


The Math of Split Testing Part 1: Statistical Sampling Uncertainty

tl;dr

Here we introduce the idea of statistical sampling uncertainty in the content of conversion rate optimization for website design. We begin by illustrating how successive trials of coin flips result in the binomial mass function. Then, we turn the problem on its head and show how to describe the conversion rate of a funnel event (the true bias of a weighted coin) by examining the results of a series of conversions. Finally, we discuss the validity of the Wald and Wilson confidence interval approximations to further quantify the conversion rate confidence interval.

This series of blog posts focuses on the math needed to make fact based decisions as quickly as possible. Many websites have clear actions they wish their visitors would take. A commerce website, for example, would like it if their visitors would purchase a product. While it may be reasonable to focus solely on the...

Continue reading →


Conditional Custom Templates with Action Pack Variants

tl;dr

Variants were introduced in Rails 4.1 making it possible to conditionally render a view template. Before Rails 4.1 we employed a custom technique dubbed MultipleTemplates to provide similar functionality. Here we present our MultipleTemplates approach and why we’re switching to Action Pack variants.

We’ve used conditional templates in our Rails API backend for quite some time now. Our main use for conditional templates is to reduce rendering times by including only the minimal amount of data. A careful profile of our response times initially led us to this approach.

Our website is comprised of two parts, a Rails app on the back-end that acts as an API for our data, and a separate Rails App on the front-end that converts the raw JSON into HTML pages to ship to the client. A while ago, our website was responding rather slowly because our design was very data heavy. We noticed...

Continue reading →


Sever-Side Rendering of Single Page Apps using PhantomJS and Node.js

tl;dr

We present a simple approach to server-side rendering of JavaScript heavy pages using PhantomJS and Node.js. When we receive a request from a bot, we use Nginx to route it to a special Node server. The Node server then spawns a Phantom process to render the page. Once Phantom has rendered the page, Node responds with the fully rendered page.

I recently ran a test to see how well the Googlebot can index a single page app (SPA). It turns out that the Googlebot isn’t all that great. It did a decent job discovering the link structure, but failed to properly read page content. The content of most of the pages that it indexed appeared blank. Having Google index blank pages is really bad, so the rest of this guide is devoted to fixing they underlying cause.

The problem with SPAs is that they often start with a blank page and fill in their content using JavaScript. Since the...

Continue reading →


I Coded the Angular Tutorial App in Backbone and it Took 260% More Code

tl;dr

The people at AngularJS created their PhoneCat tutorial app with 48 lines of JavaScript . When we coded the same app using Backbone instead of Angular, we found it took 171 lines of JavaScript – 260% more code. Here, we present a step-by-step tutorial of how we replicated the Angular tutorial in Backbone.

A while ago I decided to check out Angular since everyone’s been talking about it. I went to the Angular website and worked through their fantastic tutorial. I could see the merits of their approach, but I was left with the nagging feeling that if I coded the same application in Backbone it would only take me a few extra lines of code – a binding here and a callback there and I’d be done, right?

It turns out that I was wrong. After actually reproducing the Angular tutorial in Backbone, I found that it took significantly more JavaScript as well as a good bit of finesse. I...

Continue reading →


ctrl+s keyboard shortcut in svbtle

Mac
open the console cmd + option + J

$(window).on('keydown', function (event) {
    if (event.metaKey && event.which === 83) {
        event.preventDefault();
        $('.save_button a').click();
    }
});

close the console

Linux
open the console ctrl + shift + J

$(window).on('keydown', function (event) {
    if (event.ctrlKey && event.which === 83) {
        event.preventDefault();
        $('.save_button a').click();
    }
});

close the console

View →


How to Structure and Render Views in Backbone

tl;dr

Limited coupling between parent and child views is accomplished by making child views responsible for rendering their own content. Parent views communicate with child views by triggering collection events. Memory leaks are avoided by listening to router events and removing view listeners when appropriate.

The beauty of Backbone is that it gives you lots of freedom. I really appreciate the fact that Backbone strives to find the minimal set of useful primitives instead of deciding everything for you. This is why I was drawn to Backbone in the first place. Backbone solved problems that I actually had without forcing me down a path I didn’t understand. Most of the time Backbone’s lack of rigidity is a breath of fresh air. The downside, of course, is that you have to spend time figuring out how to do things on your own. And since coding client-side JavaScript apps isn’t always...

Continue reading →