Outside In: the return

Recently I've returned to "outside in" style development.

This used to be really popular ten to fifteen years ago, but kind of vanished. I suspect the reason for this is because it's a style of development that does not fit when your application is split across multiple code-bases - not for micro-services and, more importantly, not for mobile applications with an API back-end.

But that's not what I'm writing. And the trend, in general, seems to be heading back towards server-side rendering and monolithic applications (with a small number of external services). So it's a style of development that makes sense.

The basic idea is that you start from the point of view of a user of the application - someone who is sat "outside" the system. You write a specification that states what they want to do, why they want to do it and the steps they take to make it happen.

Again, all of this is written from the end-user's point of view.

There's no mention of routes or end points or models or databases. End-users don't see those things - they see screens and buttons and menus and fields - so we describe the steps in their terms.

And what I've found is that working this way fits perfectly with YAGNI - "you aren't going to need it". We end up writing less, much simpler, code because, if it's not required to meet the specification, we don't need to write it.

Aside: one of the things I've noticed over the years is that, when listing their "requirements" for an application, people do not fully understand what they are asking for (and we, as developers, don't fully understand what they have said).

Something they thought was "high priority, it must do this" quickly becomes "oh, we don't need that" when they actually use the system.

And something that was a "nice to have" (or was never even imagined at the start) quickly becomes "we must have this" when they see the application in action.

So YAGNI - writing the minimum amount of code for the one thing we are working on right now - means we don't end up wasting time.

Of course, we still need to design and think through exactly what it is that we are building. And we must be careful to make sure that the database is structured correctly, because database structure changes in tables full of live data can be tricky.

But, as you'll see, writing tests at each step, means we have confidence, in the future, when we need to make amendments to previous work.

So, once we have a specification, we start to implement each step in turn.

For a web application, it's a browser that the user will be interacting with. So it makes sense to make our steps remote control a browser, making sure it shows the information the user wants to see and behaves in the way that the user is expecting.

For ruby apps, this means using Capybara and headless Chrome or Firefox. This approach does have issues, especially timing issues in Javascript causing flaky tests, but I've found that it's not much of a problem when using Turbo and Stimulus. Plus the flakiness can be minimised by running your application and headless browser in docker containers. This means that the browser is not affected by stuff going on happening to the browser instance on your desktop. I've also heard that using Playwright instead of Selenium improves reliability too.

The specification is split into "setup", "action" and "expectation" sections. When using the Gherkin specification language, these map to "Given", "When" and "Then" steps.

Feature: Logging in
  Scenario: Successful login 
    Given I am an administrator at an account
    When I log in
    Then I should see my dashboard

"Given" steps set up the environment into a known state. While we are writing these, we will have to start thinking about models. For example that first line in the "successful login" scenario implies that we have "accounts", "administrators" and a "user" of some kind.

In the spirit of YAGNI, we just write the bare minimum code - if we were using my Fabrik gem, I might write:

step "I am an administrator at an account" do 
  @me = Fabrik.db.users.create 
  @account = Fabrik.db.accounts.create
  @role = Fabrik.db.roles.create account: @Account, user: @user, role_type: "administrator"
end

I have not designed the database, or created these models yet - but at this point of time, it seems likely that this will be enough to make the specification work.

"When" steps are the actions that the user (or the system) take. Most of the time, these are our Capybara commands to remote control the browser - commands like visit "/some/page", click_on "The Menu", fill_in "Email address", with: "someone@example.com".

Again, note that we're not really designing anything - there's nothing about routes or controllers or views, beyond taking note of things that the user will see and can interact with.

step "I log in" do 
  visit root_path
  fill_in "Email address", with: @me.email_address
  fill_in "Password", with: "password123"
  click_on "Log in"
end

"Then" steps are then test that things have happened as we expected.

For the user, we test that the output displayed on screen matches what they are looking for. For the system, we can test the database or check if API calls were made. The user expectations are more important than the system expectations though.

step "I should see my dashboard" do 
  expect(page).to have_text "#{@me.name}'s Dashboard"
end

Once we have an outline set of steps, written in Ruby, we can run the spec. Of course, it will fail - the first step references models that don't exist.

It's only now that our design work begins and we create some skeleton models.

I do a quick sketch of some possible database tables and models and decide that a User and Account make sense, with a Role joining the two. I'll also violate YAGNI at this point and make all three of these tables soft-deletable. This is because I know, from experience, that, when using foreign keys with cascade deletes, deleting a user or an account can cause lots of live client data to be deleted that we really want to keep.

In a Rails application I'd use the Authentication Generator to build the User model (adding in additional first and last name fields). I'll add in an Account model that has a name string field and a Role model that belongs_to :account and belongs_to :user, with an enum :role_type, user: 0, administrator: 1. For soft-deletes, I'll add an enum :status, active: 0, deleted: -1 column to each of these models. Then I'll make sure the correct indexes are added to the migrations and add the validations, associations and normalisations to the ActiveRecord models.

I'll probably write model specs for those validations and normalisations too - because they are part of the business rules of the system. Email addresses must be in a valid format, otherwise you just get lots of errors appearing in your logs when sending notifications - someone will always type the address in incorrectly. This rule is so important that I want to make sure that it is documented in the user spec. Likewise, an account must always have a name - it's an invariant of the system, so we document it and make sure the specs enforce it.

Finally, I'll add a Fabrik configuration so that we can create users, roles and accounts without having to specify all the required fields every time.

Faker.db.configure do
  with User do
    unique :email_address
    first_name { Faker::Name.first_name }
    last_name { Faker::Name.last_name }
    email_address { |u| Faker::Internet.unique.email(name: u.first_name) }
    password "password123"
    password_confirmation "password123"
    status "active"
  end 
  with Account do 
    unique :name 
    name { Faker::Company.unique.name }
    status "active"
  end 
  with Role do 
    unique :account, :user 
    account { accounts.create }
    user { users.create }
    role_type "user"
    status "active"
  end
end

Now that first step passes - and the second step fails because it tries to get the browser to visit the root path, which we haven't created yet.

So we add the root path to the config/routes.rb file, pointing it at DashboardController#show. The Rails authentication generator has already added a login page and it automatically forces any new controllers require authentication. So when the browser visits root_path it redirects to new_session_path and shows the login form. Capybara fills in and submits the form (we may need to tidy it up to meet the specification and make it fit our application's style) and then the spec fails on the final step - it expects to find some text saying we are on the user's dashboard - but our dashboard does not even have a show action at this point.

Here I add in a controller spec. This is important because controllers represent the public access points to our system - we need to be sure that users can only read and write data that they have permission for; data security is our most important responsibility.

So I add in a controller spec that looks something like this:

RSpec.describe DashboardController do 
  include Login
  describe "showing the dashboard - GET /" do 
    it "shows the login page if not logged in" do
      get root_path
      expect(response).to redirect_to new_session_path
    end 
    it "shows the dashboard if logged in" do 
      @user = Fabrik.db.users.create 
      login_as @user 
      get root_path 
      expect(response).to have_http_status 200
    end
  end
end

This is really simple - if we try to view the dashboard without a login, it redirects. But if we are logged in then it renders a page. I've added in a Login module that simulates a login for a user (the implementation depends on how your controllers and test framework work - you could just set the session cookie, or you may need to go to the login page and perform an actual login).

However, the controller spec still fails - because our controller does not return any content - and hence no 200 status. We create an empty view for DashboardController#show, update the action to render that view and now the controller spec passes.

But our actual feature specification is still failing - it's looking for the text "Alice Aardvark's Dashboard". Again, we update the view, with <h1><%= Current.user.to_s %>'s Dashboard</h1> and now the feature spec passes. In real life I'd actually use I18n for the view, because it's easier to start with I18n than retrofit it later. Even if we are only ever working in English, there are differences between American English and English English that users will complain about, so we might as well use I18n from the start.

And there we have it - a working feature for logging in a user. It's completely bare-bones - there's no styling on the UI, the models barely store any data at all.

But we can ship it today and say "we can guarantee that this feature works as expected".

Now, on to the next feature.