Tests are documentation

Years ago, I was asked why I spent so much time writing tests - my answer: "tests are documentation".

At the simplest level, that's true - you read the tests and you can see examples of how the code in question works.

But it also works at a higher level, if your test framework supports it,

RSpec has a bad reputation amongst many developers - it's verbose, the code can be quite complex to debug when the framework itself goes wrong and some people just hate the way it looks - with its describe and it statements.

I've always found that when I use RSpec I write better code.

I think this is exactly because of how it looks - by writing out a skeleton set of describe and it statements, I am effectively thinking through what I want the thing to do, in ruby-flavoured English, before I actually start working on it. I don't want to assert that a value is x, I expect to see some results - which is a semantic difference - but then, writing is all about semantics.

This can then be taken a step further. If the describe blocks and it statements are an outline of what I want the app to do then they actually become the documentation. Even better, they become documentation that is almost trivial to maintain and keep up to date as the system changes.

So for a recent project, which has a HTTP/JSON API, I used the RSpec tests themselves as the developer documentation.

The documentation looks like this:

Log in
  Generating an API Token.

  Post to /api/logins with your email address and password and if correct, this will return an API token (as the `reference` field).  You can then use the API token in subsequent API calls to identify yourself.  The API token will expire if it is not used for 28 days, at which point you will need to authenticate with your email address and password to obtain a new one.  
  
  To use the API Token, use HTTP Basic Authentication, passing the token as the "password" field.  (The "username" field is ignored so just pass a dummy value).  
  
  POST /api/logins
  PARAMS: { "email": "email@example.com", "password": "password123" }
  RESPONSE: { "reference": API_TOKEN }
    succeeds given a correct email address and password - status 200
    fails given an incorrect email address - status 401
    fails given an incorrect password - status 401

The RSpec itself is a "request" spec, which Rails calls an "integration" test - RSpec uses slightly different terminology to the in-built Rails testing tools, which is annoying, but then everyone seems to refer to all types of tests with different words.

A request spec (or integration test) sends HTTP requests to the Rails app - which goes through the middleware stack, the router, the controller, views and models, and eventually returns a response to the fake browser. This is similar to what Rails calls a "system" test - however, the difference is system tests are designed to simulate a human using the app. Hence they use Capybara and run an actual browser instance (headless Chrome or Firefox) to perform the tests. That way you can also trigger any Javascript on your pages in the same way a real user would.

In our case, as this is just an API test, we don't need that level of sophistication, so our request specs use Rack::Test, which generates the HTTP requests and deals with the HTTP responses without the overhead of a full browser. (Plus - as soon as Javascript gets involved, tests become brittle due to timing errors as the various processes communicate with each other).

The actual spec itself looks like this:

require "rails_helper"

RSpec.describe "Log in", type: :request do
  include Fixtures

  title = <<-TITLE

  Generating an API Token.

  Post to /api/logins with your email address and password and if correct, this will return an API token (as the `reference` field).  You can then use the API token in subsequent API calls to identify yourself.  The API token will expire if it is not used for 28 days (but this may change), at which point you will need to authenticate with your email address and password to obtain a new one.  
  
  To use the API Token, use HTTP Basic Authentication, passing the token as the "password" field.  (The "username" field is ignored so just pass a dummy value).  
  
  POST /api/logins
  PARAMS: { email: "email@example.com", password: "password123" }
  RESPONSE: { "reference": API_TOKEN }
  TITLE
  
  describe title do
    it "succeeds given a correct email address and password - status 200" do
      post "/api/logins", params: { email: me.email, password: "password123" }
      expect(response.status).to eq 200
      reference = JSON.parse(response.body)["reference"]
      expect(reference).to eq me.api_token
    end

    it "fails given an incorrect email address - status 401" do
      post "/api/logins", params: { email: "nobody@nowhere.com", password: "password123" }
      expect(response.status).to eq 401
      reference = JSON.parse(response.body)["reference"]
      expect(reference).to be_nil
    end

    it "fails given an incorrect password - status 401" do
      post "/api/logins", params: { email: me.email, password: "notthepassword" }
      expect(response.status).to eq 401
      reference = JSON.parse(response.body)["reference"]
      expect(reference).to be_nil
    end
  end
end

I use one spec file to group related API calls together (for example, in this app I have "login_spec", "generating_reports_spec", "taking_an_assessment_spec"). Then within each spec file, I have a separate describe block for each individual API call - one per route within the Rails app. So in this case, we have a "login_spec" that tests a single API call - POST /api/logins.

We use Ruby's heredoc syntax (<<-TITLE ... TITLE) to generate a multi-line title that describes the route that we're about to test. This title follows a standard format:

• a human-readable title
• a brief explanation of what's going on
• the URI we are describing
• the expected parameters
• the expected response

Within the describe block, we then test the common scenarios when calling this API. In this case:

• what happens if someone logs in successfully?
• what happens if they provide a non-existent email?
• what happens if they provide the wrong password?

Each of these it blocks includes a human-friendly description of the scenario, including the expected HTTP status code.

Finally, I have a simple shell script - bin/docs - that generates the RSpec in "documentation" format and then copies it to the app's public directory. This script is then called by my deployment system, so before every new version, the tests are run, the documentation file updated and then copied to the live system.

bin/rspec -I spec/requests --format=documentation > public/api.routes.txt

The danger of documentation is that it gets out of date but this way, if I'm making a change to the system, I'm actually writing and editing the documentation at the same time as I'm writing the code.