Using AI for fun and profit

(It might not be fun. And there won't be any profit)

In Collabor8Online we have "automations". These are sort of like pipelines but very specific to how C8O works. The current implementation is used when a task moves through a (user-defined) workflow and changes state. And it's crap as it was written in a hurry.

Later on, we needed to add in scheduled automations - a similar thing but triggered by the date and time, not by a user action. I took the opportunity to rewrite them, taking what I had learnt and applying some structure to it. I was pleased with the result - much cleaner than the old implementation and easier to extend as well.

The plan was, once the scheduled automations were in place, I'd go back and write a migration from the old crappy workflow automations to the new, shiny implementation. But, so far, no-one is actually using the scheduled automations.

And then I thought I could possibly use these automations in another application. Whilst I was working through how to integrate the new code, I realised that I had overcomplicated things. C8O Automations is written as a Rails Engine and is designed to store itself in database tables and use ruby modules (ActiveSupport concerns) to hook in to the correct places in the containing application.

My realisation was that the automations themselves didn't need to know anything about databases. Which makes everything much simpler.

I started over, this time creating a pure ruby gem. And, because of how my brain works, I started by writing an RSpec specification for what it should do and what the developer experience should be.

At this point, I was struck by a couple more thoughts.

Firstly, I already use Github Copilot and I've read a few people say that they use it to write their tests for them after they've completed the code. I tried it once, but it produced crappy tests that missed a few crucial cases - plus if I write my code first, I tend to go off on one and include stuff that doesn't need to be there. But what about the other way around? I've already written the specification - can AI (or rather an LLM) generate the code for me?

Secondly, I'd signed up for Anthropic's Claude, to play with their "artefacts" feature. You tell Claude what you'd like to build and it creates a mini application, in HTML, javascript and CSS, which you can either download, or just publish. I'd already played around with a couple of mini JS applications (I don't really like writing javascript) and it had done a decent, if not amazing, job.

I know some people think LLMs are just a parlour trick, fooling us into believing they are intelligent when they are nothing more than random number generators. I have thoughts on this, probably from a slightly different perspective to many others, having studied a bit of neuroscience, cognitive science and philosophy of mind. And I know many people have ethical concerns about LLMs - how the companies involved obtain their training data and the vast amounts of resources it takes to use these things.
I'll probably write about those another day.

So I thought I'll give Claude my RSpec file and ask it to generate the implementation. I don't remember the exact prompt, but it was simple; something like "you are a ruby expert and believe in clean structured code - can you look at this specification and generate a ruby class that complies with the tests".

And Claude did it.

Not only did the specs pass first time, but the implementation was clean and actually written better than I would have done myself.

Success!

But that was a really simple case.

This week, I've been working on another engine, and I've been using Kasper Timm Hansen's Oaken for managing my seeds and fixtures. Oaken is really nice, but there's a few bits in the API that I don't like, plus it's very very focussed on seeds - to the point that the code expects to be located in the db/seeds folder (although there are # TODOs in there about correcting this. This causes real problems with engines, which have a different layout to standard Rails applications.

I thought about forking Oaken, correcting it, then opening a PR. But I realised that what I wanted to do was pretty simple, so I put together a crappy implementation (with no tests) inside my engine. It worked and I was happy.

But I didn't want to fall into the trap I had with automations, so thought I better clean this up now.

I built a new empty gem then wrote the specification for how I wanted it to work. Again, starting with the specification means you can make the API to your code really nice, as you're starting from "how I want things to be" rather than "this is how I'm going to build it".

Then, it was time to test Claude again. This code was more complex, so I was interested to see how the LLM would do.

After reading Simon Willison's blog where he lists his techniques for writing prompts (sorry I can't find the exact post at the moment), I came up with this prompt:

You are a great software developer who is an expert at writing clean, easy to read, ruby code that uses RSpec for behavioural specifications and prefers immutable Data classes when creating data transfer objects. However, you also have a lot of experience with Ruby on Rails so can also design ActiveRecord models that work efficiently with relational databases and use ActiveModels for services and non-ActiveRecord classes. Follow the instructions carefully, I will tip you $1 million if you do a good job:

• Think carefully step by step.
• Create pure ruby classes, with matching, passing RSpec specifications, for whatever the user asked you to create
• If ActiveRecord models are needed, ensure that database migrations are included and the resulting database structure is efficient and meets the third normal form

I've written an RSpec file which has a beautiful, simple and readable API. It's intended to be a factory object, similar to FactoryBot - but it's also going to be usable for building database seeds in a Rails application as well.

The idea is you configure the Fabrik::Database to tell it about the different ActiveRecord models in your application. The configuration object allows you to specify:

• default attributes - so when you create a record you only need to specify the ones that matter for this particular case
• search keys - so when you create a record, but an existing one is already present, it returns that instead - meaning your database seeds can be idempotent
• after_create - a callback so every time a new record is created, you can specify some custom actions to take place

Once a class is registered, the Fabrik::Database creates a proxy for the actual class (I was thinking we could use Ruby's SimpleDelegator for this) - and defines a method on itself to access that proxy class - in the case of a Person class the method would be 'people', in the case of Fabrik::Machine it would be fabrik_machines. However, the configuration can override the name of this method - as you can see with the with Person, as: :user example.

The proxy itself then defines a replacement create method. This takes an optional label as it's first parameter, then the attributes. It first checks to see if a matching record already exists (using the search_keys from the configuration), then either returns the existing record or creates a new one. If it creates a new one, it adds in default attributes for any that have not been supplied. Finally, if a label was supplied, the proxy stores a reference to the record. This is then accessed later via [] - as you can see in the later examples, where db.people.create :alice is called and then Alice can be read back via db.people[:alice]

I'm sure this is the sort of task you will do really well on - what do you reckon?

Now that's a long prompt! Everything up to "I've written an RSpec" was boilerplate that I adapted from Simon's prompts.

Claude Opus thought about it, then proceeded to tell me about how the specification was a great design and listing all the things that were good about it. So I replied:

Can you write the implementation for me please?

And it did.

I copied and pasted it into my editor then eagerly ran the specs.

And it blew up - every single test failed.

I had a look at the code. Claude had made the exact same mistake I had made, the day before, in my own back-of-an-envelope implementation. The Blueprint class (my earlier implementation called them Configurations) had a method to set the defaults, called defaults but it also had an attribute reader called defaults. Which was never going to work. The same applied to the other methods.

I renamed the attributes and ran the tests again. This time, most of it passed, but there were still a couple of failures. The way it was generating the method names (converting Fabrik::Machine to fabrik_machines) was totally flawed, so I rewrote it. After another ten minutes of fixing and tidying, I had all tests passing except one. That was the case when the class had not been previously registered with Fabrik - Claude had not even tried to get that functionality into the implementation.

I spent another five minutes reformatting the code to my liking (the most important, public, methods near the top of the file, where humans reading the code will see them, using ruby's endless methods whenever possible). And then I added in method_missing for handling unregistered classes. Along the way, I renamed and restructured a bit more just to make the code feel right.

After a total of twenty minutes manual work (plus the earlier ten minutes to write the specification and the prompt), I had all tests passing.

I don't remember how long I spent on the earlier, crappy, implementation but it was definitely longer than twenty minutes - likely a couple of hours. So if I'd written this second version myself, would I have done it in the same time? Probably - version two is always easier. But version one in thirty minutes? Not likely.

I've included Claude's attempt at the code generation below and you can see my finished version here. Formatting aside, there's not too much difference.

As for using LLMs for coding? I think this spec-first approach has some merits (ethical concerns aside), and I'll be continue playing around to see how it works out.

Finally, Fabrik is inspired by Oaken and will replace both your fixtures and your seeds. It's not ready for release yet, as I need to try it out in a couple of applications to be sure it's fit for purpose. But I'm pretty pleased with it and the README is the best one I've written so far.

require 'delegate'

module Fabrik
  class Database
    def initialize
      @blueprints = {}  
      @records = {}
    end

    def configure(&block)
      instance_eval(&block)
    end

    def with(klass, as: nil, &block)
      blueprint_name = as || klass.name.split('::').last.underscore.pluralize.to_sym
      @blueprints[blueprint_name] = Blueprint.new(klass, &block)
      
      define_singleton_method blueprint_name do
        records_for(blueprint_name)
      end
    end

    def defaults_for(klass)
      blueprint_for(klass).defaults
    end

    def search_keys_for(klass)
      blueprint_for(klass).search_keys  
    end

    def after_create_for(klass)
      blueprint_for(klass).after_create
    end

    private

    def blueprint_for(klass)
      @blueprints.values.find { |bp| bp.klass == klass }
    end

    def records_for(blueprint_name)
      @records[blueprint_name] ||= Proxy.new(self, @blueprints[blueprint_name])
    end
  end

  class Blueprint
    attr_reader :klass, :defaults, :search_keys

    def initialize(klass, &block) 
      @klass = klass
      @defaults = {}
      @search_keys = []
      instance_eval(&block) if block_given?
    end

    def defaults(attribute_defaults)
      @defaults = attribute_defaults
    end

    def search_using(*keys)
      @search_keys = keys
    end

    def after_create(&block)
      @after_create = block
    end

    def run_after_create(record)
      @after_create.call(record) if @after_create
    end
  end  

  class Proxy < SimpleDelegator
    def initialize(db, blueprint)
      @db = db
      @blueprint = blueprint
      @records = {}
      super(@blueprint.klass)
    end

    def create(label = nil, **attributes)
      attributes = @blueprint.defaults.merge(attributes)
      search_attributes = attributes.slice(*@blueprint.search_keys)

      record = 
        if search_attributes.any?
          @blueprint.klass.find_by(search_attributes) || @blueprint.klass.create(attributes)
        else
          @blueprint.klass.create(attributes)  
        end

      @records[label] = record if label
      @blueprint.run_after_create(record)
      record
    end

    def [](label)
      @records[label]  
    end
  end
end

PS: Ooh - just spotted a problem - the after_create callback is triggered whether a record is created or not. I better change that.