Secret Mustache

Adventures of a Scruffy Nerf Herder

Evaluating Heroku usage

30 March 2016 is an application that is a vital ingredient to Mozilla QA’s community strategy. At it's start, the project was originally initiated and designed by the community to address the problem of discovering meaningful ways to contribute to Mozilla QA. In its simplest form, allows community members to browse project tasks and upon finding one that resonates with them, begin working on it.

One and Done: task manager

The development of the project as well as continued shaping of its features are handled both by the community as well as by the QA team itself. Project ideation and community engagement is handled by Rebecca Billings, Karl Thiessen, and Bob Silverberg. Code management is accomplished on get The deployment pipeline (stage, production, etc) and hosting are handled by Feature triage and ideation, in bugzilla.

For additional information see the One and Done team wiki.

Heroku vs. a different solution

For a project of One and Done’s complexity and development requirements, Heroku was a natural fit. The creation and management of deployment workflows as well as one-click installation of add-ons are cleanly implemented.

Deployment pipeline

Going into 2016 we have seen many projects at Mozilla standardizing around AWS as it's web services delivery mechanism. Given this and that QA owns we felt it would be advantageous to consider a move to this platform.

The problem space we wanted to explore

  • Continue to use our application as a platform to understand CD and the deployment environments used by our developers
  • Build on our knowledge base so we can offer timely suggestions of best practices to projects that are in the process of moving to AWS or may have already made the switch.

Approaching the problem(s)

  • Investigate the current workflow cases employs and map them to potential AWS equivalents.
  • Create a working document that summarizes these findings, includes a pros and cons list.
  • Identify teams that have domain knowledge and form relationships that should we want to move off of Heroku in the future, we can leverage their experience.

What we discovered

Heroku’s platform is quite slick, particularly for projects — like One and Done — that need flexibility and are self-supported by small teams. Heroku also scales quite readily if needed, that said our usage of it follows simple patterns and a low amount of traffic.

Amazon's large array of services often times matched 1:1 the offerings presented by Heroku. However there is a bit of additional mastery that is required to use, configure, and deploy using AWS.

Deployment pipeline


  • Creation of deployment workflow via a GUI is straightforward and quick.
  • For pull requests that contain complex or experimental ideas, Review Apps are easy to deploy. One and Done employs the use of review apps instead of a dev instance.
  • The ability to employ travis-ci for pre-deploy hooks if wanted.
  • In instances where a rollback is desired and the database hasn’t been affected, it’s a one-click operation.
  • Ease of setting up automatic backups of a database.
  • Dataclips - the ability to create short or long-lived queries that return parseable results saves developer time. In some instances this can stand in for a read-only API.
  • One-click install of add-ons - currently uses Heroku Postgres, Heroku Scheduler, Memcached Cloud, Postmark, Papertrail, and SSL.
  • Ease of ssh’ing into an instance to diagnose an issue
  • Preconfigured dashboards, lot's of them.

Amazon's Services

  • Amazon has a competing product called CodePipeline
  • There is no direct synonym for Heroku's Review Apps.
  • The ability to employ travis-ci for pre-deploy hooks if desired.
  • In instances where a rollback is desired and the database hasn’t been affected, it’s a one-click operation.
  • Ease of setting up automatic backups of a database.
  • Choice of setting up build pipelines and use of third party frameworks is near infinite.

Pros and cons

The reasons for considering a move to an AWS solution is fairly straightforward.


  • We're comfortable with Heroku. Using its platform offers a low barrier of entry for our team.
  • The ecosystem of extensions for Heroku meets our needs.
  • Application manifests allow teams to create a template of applications.
  • A platform API exists.
  • On the negative, very few of the projects we collaborate on use Heroku and it's workflows.
  • Both a pro and a con - there is one right way to configure a build pipeline.
  • Self supported by the Web QA team.


  • Transitioning to a solution that is powered by AWS would initially be a fairly high barrier to overcome.
  • The ecosystem of extensions and external services match what is offered in Heroku.
  • Leveraging AWS in our deployment and hosting workflows would bring us in line with what the majority of our other projects are using.
  • We can follow patterns that other teams have used when combining Jenkins and other bits into their pipelines.
  • AWS has CodePipeline - their version of a continuous delivery service. This isn’t used by our teams at Mozilla, thus I don’t think we should invest in it at this time.
  • Puppet works well with AWS - our Operations team uses Puppet.
  • Configuration to use third-party CI pipelines is near infinite.
  • Mozilla is standardizing around AWS centric solutions. On this merit alone, as a team we should discuss if there is value for us to do so as well.
  • Support from the Ops team as well as reproducible deployment patterns already exist at Mozilla.

This entry was tagged as testing mozilla

blog comments powered by Disqus