Javascript tests using Selenium, Chromium, Docker, Ruby on Rails and Cucumber

The other day, someone on Mastodon (I'm @rahoulb@ruby.social if you're interested) said they didn't like test-driven development as it imposed constraints on them and they prefered to improvise, like jazz. I love a bit of improvisation - blues rather than jazz though. When I was a kid I taught my self to play guitar by making stuff up going along to blues records, which means I still have no idea how to play in a major key. And for the record BB King is the greatest guitarist there's ever been, putting more feeling into a single note than most people manage in a lifetime.

Anyway, I replied that constraints are important and that's why I like TDD. Or rather BDD; behaviour-driven development. And I also realised that I'd been all over the place with Standard Procedure - the original plan was to have something ready to demo to my clients in January - it's now May and I've been through three total rewrites with no actual progress.

So I fired up Cucumber again and wrote a proper BDD specification for my first client. The difference with BDD is you define your "tests" in terms that matter to the user, whereas TDD is about the code.

Here's an example spec for my client who runs an online store.

  Scenario: Receiving a standard order
    Given an account called "UKDS" loaded from "ukds.yml"
    When the website receives a new standard order to be processed
    Then the order should have a 24 hour alert set against it
    And the order should have a status of "incoming_order"
    And the order should be assigned to "Nichola"
    And "Nichola" should be notified
    When "Nichola" logs in
    Then she should see the newly received order
    When she places the order with the supplier
    Then the order should have a status of "order_placed"
    And the previous alert should be inactive
    And the order should have a 120 hour alert set against it
    When the order arrives at the office
    Then the order should have a status of "requires_dispatch"
    When she prepares the order for delivery and posts it
    Then the order should have a status of "dispatched"
    And the previous alert should be inactive
    And the order should have a 48 hour alert set against it
    When the alert has passed
    Then the order should have a status of "complete"

As you can see, the definition is written in formalised English and the actual steps are defined in terms that the user will understand, not in terms that the code will understand. I like this because I’m thinking about what the app needs to do, not how I’m going to implement it.  Cucumber basically acts as a fancy regular expression parser and locates the relevant "steps" which then define what the test actually means.

Then "she should see the newly received order" do
  visit "/"
  within "#notifications" do
    expect(page).to have_text @order.reference
    click_on @order.reference
  end
end

When "she places the order with the supplier" do
  click_sl_button "Place order with GRO"
  within "form" do
    fill_in_rich_text "command_gro_notes", with: "Reference 123"
    click_sl_button "Place order"
  end
  wait_until("waiting for order_placed") { @order.reload.status.reference == "order_placed" }
  expect(@order.gro_notes).to eq "Reference 123"
  expect(page).to_not have_css("form")
end

And this is where it gets tricky.

I use RSpec as my testing framework and for these high level tests (different people call them different things, but they test the whole stack - from UI through to database and back again), in ruby we use Capybara to drive a browser. By default, Capybara uses Rack::Test to send GET, POST and other requests to our app. But Rack::Test isn't a real browser, so does not deal with Javascript. Instead, we can use Selenium to drive an actual instance of Chrome or Firefox.

But I do all my work in docker containers now. And I also use Github Actions to run tests before deployment. So how do I get Selenium working with my setup?

This actually took me ages to get working - there were other people who had done the same, but everyone's configuration was slightly different, and none of the articles were using the exact same stack that I was. This is how I did it, but your mileage will vary.

The plan is to create a separate docker container that runs an instance of Chrome. Capybara and Selenium will then use Chromedriver to remote-control that instance, telling it to connect to the Rails app (which is running in its own container). The slightly confusing bit is that Capybara (and the tests) are running in their own process within the same container as the Rails app (which is a separate process).

So the test calls out from the app container to drive the browser running in the test_server container. The browser then calls back into the app container to get the Rails process to do some work that is "displayed" in the test_server container's browser. And finally the Capybara process, in the app container, checks the display output by reading from the test_server container.

A tangled web.

In my Gemfile, I define the gems we're going to need:

group :development, :test do
   gem "rspec-rails"
   gem "object-factory", require: "object_factory"
   gem "cucumber-rails", require: false
   gem "database_cleaner"
   gem "capybara"
   gem 'capybara-shadowdom'
   gem "selenium-webdriver"
end

Rails normally installs the webdrivers gem but we don't need that, replacing it with selenium-webdriver. I've also installed capybara-shadowdom as I'm using Shoelace components and need to look into their internals to test they're working OK. Because the Rails app and the Capybara tests run in separate processes (unlike when you're using Rack::Test), we need to use the database_cleaner gem to reset the database after each scenario. object-factory is a gem I wrote over ten years ago for quickly setting up the database. It needs a rewrite but it works and I quite like the syntax (well I did create it, so I'm allowed).

Next we set up the docker containers. I don't want to be firing up the Chrome container when I'm just working on stuff - I only want it when I'm running the Cucumber tests. So I have a basic docker-compose.yml that defines my app itself, plus the services it needs (in my case, mysql_db, redis_db, memcached and auth which is a Supertokens server - I'll write about that another day). This docker-compose file defines the runtime dependencies of my app, as well as listing all the environment variables it needs - so it is essentially a document describing what is needed to deploy the app.

For the cucumber tests, I have a separate docker-compose-test.yml which is very simple:

version: "3.9"
services:
   test_server:
      image: selenium/standalone-chrome:latest
      ports:
         - 5900:5900
      volumes:
         - /dev/shm:/dev/shm
   app:
      environment:
         - RAILS_ENV=test
      depends_on:
         - test_server

We define a container to run Chrome and open port 5900 (as you can VNC into the container to watch the browser doing its thing - although I've not managed to get this working yet). And we override some settings on our app - we change the Rails environment to test and we tell it to wait till the selenium container has started before the Rails server begins its work.

To run the actual tests, I have a script bin/cucumber that simply starts docker using both docker-compose files.

docker compose -f docker-compose.yml -f docker-compose-test.yml run app bin/rails cucumber

This is all good, and pretty standard docker container stuff.

But things get tricky when it comes to configuring Capybara.

Cucumber-Rails installs a features/support/env.rb file that sets things up. But that file is auto-generated, so I put my other configuration files in the same features/support folder - Cucumber automatically loads all .rb files in there for us.

Firstly, is features/support/driver.rb

Capybara.register_driver :chrome do |app|
   options = Selenium::WebDriver::Options.chrome(args: %w[headless disable-gpu whitelisted-ips=' ' no-sandbox disable-dev-shm-usage window-size=1400,1400])
   Capybara::Selenium::Driver.new(app, browser: :remote, url: "http://test_server:4444", options: options)
end

Capybara.app_host = "http://#{IPSocket.getaddress(Socket.gethostname)}:4000"
Capybara.server_host = IPSocket.getaddress(Socket.gethostname)
Capybara.server_port = 4000
Capybara.default_max_wait_time = 10
Capybara.default_driver = :chrome

Firstly, we register a new driver for this selenium-powered browser. We set up some options, telling it how to start Chrome on the other container and then build a driver that connects to http://test_server:4444. If you remember, in docker-compose-test.yml, the selenium container is called test_server and port 4444 is used by Chromedriver to accept remote commands.

We then configure Capybara - firstly we set the app_host, server_host and server_port. When Cucumber starts, it creates a Rails process that runs our app. By default, the rails server is bound to 127.0.0.1 which means it is not accessible on the network. But as docker containers are, in effect, separate Linux boxes running indepently on an internal docker network, this would mean that Chrome would be unable to connect to the Rails app. So the app_host tells the Rails server to bind to the app container's IP address and run on port 4000 - not the default 3000 as that would clash with my working, development, instance, if that were also running. Then the server_host and server_port are used when the browser is making requests - if the test says visit "/some/path", Capybara will tell the remote browser to connect to http://#{server_host}:#{server_port}/some/path - again, pointing to the IP address of the app container. Because lots of Javascript stuff happens asynchronously, we tell Capybara to wait a maximum of 10 seconds for things to complete and then we tell it to use the driver we created earlier.

Aside: many of the examples I saw used the container names instead of the IP addresses, as Docker will set up basic name resolution for all containers on the same network. But I couldn't get this working - I think there was a timing issue because the test_server started before the app container so therefore would complain that it couldn't find app. So IP addresses it is.

Now, when I run my test script, docker compose uses the two yml files to fire up the services needed for the app (MySql, Redis and so on), plus starts an instance of Chrome within the test_server container. Then it calls bin/rails cucumber within the app container, which fires up the Rails app and starts the Capybara process. Cucumber then goes through each step in the various scenarios, telling Capybara to remote-control Chrome (in test_server), which displays pages from Rails (in app) and then tests the output to see if it is what we expected.

One last thing. My client's system is going to have orders POSTed to it from their website. So to test that, rather than using Chrome, I switch back to Rack::Test - which is perfect for simulating API calls.

In features/support/api.rb I have some helper methods.

def encode api_key
  ActionController::HttpAuthentication::Basic.encode_credentials("username", api_key.to_param)
end

def use_api api_key
  session = Capybara::Session.new(:rack_test, Rails.application)
  session.driver.header "Authorization", encode(api_key)
  session.driver.header "Content-Type", "application/json"
  session
end
  
def get_json_from session, url, params = {}
  response = session.driver.get url, params
  JSON.parse(response.body)
end

def post_json_to session, url, params = {}
  response = session.driver.post url, params.to_json
  JSON.parse(response.body)
end

The use_api method creates a separate Capybara session that uses the Rack::Test driver, pointing it to my Rails application. Rack::Test runs in the same process as Capybara - so there are actually two copies of the application being tested at the same time. But as they both connect to the same database (and other services), it all works OK.

To use it, my step looks like this:

session = use_api @api_key

# Find the Customers folder
@folder_data = get_json_from session, "/accounts/#{@account.id}/folders.json", reference: "customers"
@customers_folder_id = @folder_data.first["id"]

# Find the Customer role
@role_data = get_json_from session, "/accounts/#{@account.id}/roles.json", reference: "customer"
@role_id = @role_data.first["id"]

# Create a new person, with role "customer", and add them to the "customers" folder
@post_data = {
  command: {
    name: "Role::AddPersonJob",
    params: {
      parent_id: @customers_folder_id,
      first_name: "Alice",
      last_name: "Aadvark",
      email: "alice@example.com",
      addresses: [{line_1: "123 Fake Street", city: "Springfield", postcode: "SP1 FLD", country: "United Kingdom"}]
    }
  }
}
@customer_data = post_json_to(session, "/roles/#{@role_id}/perform_command.json", @post_data)