Engineering Club

Website Project Structure

April 17, 2013

We set out to create a template for web development that has enough space to grow features, without compromising the simplicity of static sites and one-step deployments. Adding pages shouldn’t require a programmer when you have a tech savvy designer. Ideally a designer with CSS chops can maintain the site completely.

QuizUp, our core product, does not utilize this style of development directly. Rather this is a structure that makes sense for smaller supporting apps.

Our goals:

  • Easy to bootstrap and maintain
  • Easy to deploy, even for designers
  • Increased complexity of requirements should have obvious solutions
  • A real template engine
  • The production site should pass Y!Slow tests

Technology Overview

  • Frontend

    • Backbone.js
    • CoffeeScript
    • Stylus

  • Server

    • Heroku
    • Flask
    • S3 + CloudFront

  • Development

    • Brunch
    • Honcho
    • Make

Repository

The file trees below are actually marged into one, but split up here for clarity. It’d be cleaner to nest the Brunch related code under frontend/, but we’ll later see why this isn’t possible (yet).

Basics

.
├── .gitignore
├── .slugignore
├── .env
├── Makefile
├── Procfile
└── Procfile.dev

Server

.
├── app.py
├── requirements.txt
├── dev-requirements.txt
├── templates/
└── venv/

Frontend

.
├── config.coffee
├── package.json
├── app
│   ├── application.coffee
│   ├── application.styl
│   ├── assets
│   ├── styles
└── vendor
    ├── scripts
    └── styles

Subsequent sections of this post will explain how to develop and deploy changes, and how Brunch and Flask fit together.

Clone our wireframe repo if you want to follow along: plain-vanilla-games/site-wireframe.

Development

During development, common actions are documented in a Makefile. This is a practise I picked up quite late in my career, assuming make was an archaic system to compile C code. However, Makefiles are simple and readable documents for project commands.

Before we go on, we’ll need a Python virtualenv and NPM package tree:

$ make bootstrap

This command assumes you have npm and virtualenv installed locally and runs these commands:

$ virtualenv --distribute venv
$ venv/bin/pip install -r dev-requirements.txt
$ npm install -g brunch
$ npm install

Two utilities of note are installed from dev-requirements.txt:

  • Honcho: Python clone of Foreman. Used for running Procfiles under a clean environment. We found the Python variety to be better maintained. See section on Procfiles.
  • ssstatic: A publicly available command line utility we wrote to synchronize a folder to S3 with all the HTTP goodies you would want (content type and cache headers mainly, also gzipping big texty files). [GitHub]

Brunch

Brunch is a command line utility to manage compilation and pipeline processing of frontend code. You define vendor requirements and Brunch compiles bundles of CSS and JavaScript to ship with your app. Brunch supports a variety of abstract syntaxes and plugins assets. We’re fans of Stylus and CoffeeScript, but LESS and various others are supported as well. Define the order of dependencies, filenames and output paths in config.coffee:

exports.config =
  paths:
    public: "static"
  files:
    javascripts:
      joinTo:
        'javascripts/vendor.js': /^vendor/
        'javascripts/app.js': /^app/
      order:
        before: [
          'vendor/scripts/jquery.js'
          'vendor/scripts/lodash.js'
          'vendor/scripts/backbone.js'
        ]

    stylesheets:
      joinTo:
        'stylesheets/app.css': /^(app|vendor)/
      order:
        before: [
          'vendor/styles/normalize.css'
        ]

    templates:
      joinTo: 'javascripts/app.js'

Now test that brunch can compile.

$ brunch build

You should see reaffirming output from Brunch and a top level static/ folder filled with HTML5 juice in the project root. We don’t version control this directory as it’s just output from Brunch. Flask routes static/ by default, so we should be all set to move on …

Templates

What frontend frameworks lack is a real URL router and template engine. Single page apps make up for this with pushState URL’s, but we still wouldn’t have solved the server routing (for when someone shares a deep link and the server needs to deal with the URL) or SEO issues. For this purpose we are using Flask which has Jinja2; one of the best template engines out there baked in by default. With minimal boilerplate we have a fully functioning server to deal with templates and routing. In our experience it’s also a great framework to grow into when the app grows in complexity, with high quality plugins for ORM’s, forms processing and much more.

To make the designer capable of adding routes without touching the backend code we traverse the template directory and attach new routes based on filenames found:

def template_route(dirname, filename):
    dirname = dirname.strip(".").strip("/")
    template = os.path.join(dirname, filename)
    if filename not in ('index.html', 'base.html'):
        url, ext = template.rsplit(".", 1)
        url += '/'
    elif filename == 'index.html':
        url = dirname.strip("/")
    else:
        return
    viewname = url.strip('/').replace('/', '_')
    app.add_url_rule('/%s' % url, viewname, lambda: render_template(template))

template_root = 'templates/'

for dirpath, dirnames, filenames in os.walk(template_root):
    for filename in filenames:
        if not filename.endswith(".html"):
            continue
        dirname = os.path.relpath(dirpath, template_root)
        template_route(dirname, filename)

Now templates/jobs.html is presented on /jobs. Adding a route is a matter of creating a file and restarting the server. The snippet above covers a couple of edge cases, such as respecting index.html as a default and ignoring base.html files usually reserved for base templates to extend.

There’s some more stuff going on in app.py, but nothing out of the ordinary.

Procfiles

Using Procfiles is a great way to define and document processes used by a project in one place (workers, databases, etc.).

To define the environment processes are run, Honcho will look for .env in the same folder, parse it and set up as the process environment:

SECRET_KEY=...  # Flask, optional
DEBUG=true  # Flask
PORT=5000  # Flask
AWS_ACCESS_KEY_ID=  # ssstatic
AWS_SECRET_ACCESS_KEY=...  # ssstatic
STATIC_HOST=dcpud7ylnuki1.cloudfront.net  # ssstatic, optional

You will need a file like this for various commands, so create it now with your own values.

We won’t go into detail on deploying a Python app to Heroku. However, it’s worth having a look at our Procfile, which Heroku uses to start server processes:

Procfile:

web: gunicorn app:app -b 0.0.0.0:$PORT -w 4

We have another Procfile for local development. One process for Flask and another for Brunch:

Procfile.dev:

web: venv/bin/python app.py
brunch: brunch watch

Because our project root is also a Brunch project, we can run these commands side by side. Brunch keeps static/ up to date, recompiling on frontend source code changes. The Flask development server restarts itself on server code changes. To make things even smoother, Brunch reloads the browser on JavaScript changes, and loads in new styles on CSS changes. To start the processes:

$ make run

Deployment

Create a Heroku account and app if you haven’t already. You’ll need the Heroku toolbelt. For more help refer to their awesome documentation.

$ heroku create
$ heroku config:add SECRET_KEY=`openssl rand -base64 32`  # For Flask

A gotcha that bit us because of the hybrid structure is that Heroku detected a node.js app because package.json was present. This is easily fixed by adding Brunch related files to .slugignore. Heroku will then deploy your app with a Python buildpack.

package.json
app
vendor

Refer to the Heroku documentation for more info. It’s really great.

You will need to commission a S3 bucket and CloudFront distribution to host your static assets. Heroku can do this, but it’s wasteful. Update the Makefile with your S3 bucket name.

Note on hostnames: Default behavior of ssstatic is to report the S3 root folder URL after syncing (via stdout). When using CloudFront you will need to set `STATIC_HOST` in `.env` to override the S3 hostname. Set this variable to the CloudFront hostname. Otherwise the site URL’s will point to the S3 root URL instead of the CDN root URL.

Deployment is roughly as follows:

  • Calculate a unique checksum for the static/ directory
  • Upload assets to S3 with helpful HTTP headers
  • Push new code to Heroku
  • Update the Heroku config with a new STATIC_URL

Our server code is aware of the STATIC_URL environment variable. When present (and debug mode is off), the static function made available in the templates will output the CDN URL. Just make sure to use it in templates for all asset references:

<link rel="stylesheet" href="{{ static('stylesheets/app.css') }}" type="text/css">

Again, Makefile has memorized the commands for us:

$ git commit -a -m 'New feature'
$ make deploy

Drawbacks

This way of development gives you a proper frontend compiler and a great web framework. The complexity may seem high at first, but by starting out with this structure, you can evolve from single-page app style, to something in between, or a more static style. Whatever suits your project or preference.

  • High level of complexity to get started (registering S3 buckets, CloudFront distributions, Heroku etc.).
  • Not as fast as a fully static site.
  • Heroku processes aren’t free anymore.
  • ssstatic creates a new URL for all assets based on the checksum of the whole static folder. This invalidates all browser caches even for tiny changes. It’s quite possible an MD5 based URL per-file would be more efficient (this is what Tornado does). We’re currently looking into this.

Example

We threw together a sample application using this structure. It’s a tiled layout of Instagram snaps taken at our office.

  • Site: http://vanilla-partytv.herokuapp.com/
  • Source: https://github.com/plain-vanilla-games/vanilla-partytv

Summary

$ git clone https://github.com/plain-vanilla-games/site-wireframe
$ cd site-wireframe
$ make bootstrap
$ heroku create
$ heroku config:add SECRET_KEY=`openssl rand -base64 32`
$ # Create .env and update Makefile variables
$ make run
$ make sync