Operations - documenting complex business logic as easy to read ruby code

First of all - a quick question: am I stupid? The idea behind Operations seems so simple and so natural to me, I'm really surprised that no-one seems to have done it before. So either I'm a total genius who has just invented something amazing, or an utter idiot, for coming up with something will turn out to be useless.

Of course, if it has been done before I'd love to hear about it and compare notes.

The background

Nearly all of the systems I build are for businesses. Historically small businesses, often tiny businesses, but, nowadays, some of our customers are big companies with large revenues. My day job, Collabor8Online, is a software-as-a-service business and our main application is a multi-tenanted Ruby on Rails monolith. I describe it as "github for the construction industry" - at its heart it handles version control for architects' drawings and then offers a load of construction-specific project management functionality around that.

Because our clients are big businesses, they all have their own ways of working. And as the system is multi-tenanted, that means we have to put configuration options and feature flags in, generally at the Account level.

In turn, this means that nearly every piece of functionality is not a straightforward process:

Ideally, we would receive an HTTP request and send it to the correct controller. The controller then grabs a few models out of the database and either assembles them into a view, or calls some methods on those models and does a redirect.

In practice, we receive an HTTP request and send it to the correct controller. We then need to examine the configuration of the relevant account, then make a decision as to what happens next. Permissions are fine-grained and determined at run-time (as each account has its own set of roles). Simply calling a method on a model is not enough, because what it's supposed to do depends upon a series of inter-related configuration options.

Recently, a client paid us for a new feature which alters the way that documents are downloaded. In effect, we had to intercept the download request, manipulate the document (in a way that's specific to the account, client and user performing the download) and then decide whether it's a direct download or something that will be shown in our "inline viewer".

I knew that downloads were already quite messy - there are several different places where different types of download can take place, and there were already a few configuration options affecting them.

So I spent almost a whole day tracing through the code, looking at all the variations. And I came up with this flowchart.

All those steps, just to download a single document (never mind downloading multiple ones at once).

Tracing through the code took me hours. Because all the logic was scattered all over the place. In controllers, in models, in "commands" (an earlier attempt at solving this same problem, although I didn't realise it at the time) and in background jobs. It was horrible.

What I wanted was some way of representing that flowchart, as ruby code, so when I (or any other developer) needs to understand how the download process works, they have one place to look.

Sketches

Which led me to Operations. This is a Rails Engine that tries to represent the flow of logic for a business operation.

I'm not a computer scientist and I don't really think in engineering terms. I use ruby because I can write code that looks beautiful and reads like english. But, in Computer Science terms, I guess the flowchart is basically a state machine of some kind. Or maybe it's a graph. Like I say, I'm not a computer scientist. But each node (or is it a state?) is either a decision or an action.

I'm liking "declarative" ways of doing things now. I've been playing with Flutter, where you describe your hierarchy of widgets (user interface components) and then that hierarchy reacts as the data it is linked to changes. ActiveRecord is essentially declarative too - you declare the invariants (validations) and relationships (associations) of an individual model, then it reacts (via callbacks) as it comes in to contact with actual data.

Ruby meta-programming is an excellent way of building declarative interfaces. Unlike dart, the language beneath flutter, which is terrible at meta-programming. Declaring your widgets becomes an impenetrable mess of nested brackets and anonymous functions.

Anyway, I opened BBEdit, created a file called granular_tasks.rb on my desktop and started writing out how I would like this flowchart to work. I drew out a whole load of classes this is one of them.

class Collabor8::SendDocumentToInlineViewer < Tasks::Job
  attribute :project_member, class_name: "ProjectMember", required: true
  attribute :folder, class_name: "Folder", required: true
  attribute :document, class_name: "Document", required: true
  attribute :use_dynamic_document_generator, type: :boolean, default: false
	
  starts_with :is_autocad_file? 
  ends_with :url 
	
  decision :is_autocad_file? do 
    document.is_autocad_file? ? perform(Collabor8::SendDocumentToAutocadViewer) : next_step(:is_document_pdf?)
  end 
	
  decision :is_document_pdf? do 
    document.is_pdf? ? next_step(:record_activity) : next_step(:is_document_dynamically_generated?)
  end 
	
  step :is_document_dynamically_generated? do 
    (document.is_dynamic? && use_dynamic_document_generator?) ? next_step(:generate_dynamic_document) : perform(Collabor8::PrepareDocumentForDownload)
  end
	
  step :generate_dynamic_document do 
    perform Collabor8::MergeDocumentData, filename: document.filename.to_s do
      on_success do |results|
        self.document = results[:document]
        next_step :has_cached_pdf? 
      end
    end
  end
  
  decision :has_cached_pdf do 
    document.has_cached_pdf_for?(project_member.user) ? next_step(:get_cached_pdf) : next_step(:generate_pdf)
  end
	
  step :generate_pdf do 
    self.document = document.generate_pdf_for(project_member.user)
    next_step :record_activity 
  end 
  
  step :get_cached_pdf do 
    self.document = document.cached_pdf_for(project_member.user)
  end
  
  step :record_activity
    perform Collabor8::RecordActivity activity: "document_viewed", document: document, filename: filename, user: project_member.user do 
      on_success { |results| completed url: document.contents.url }
    end
  end
end

The key things:

• we have decisions - depending upon the outcome of the decision, we choose where we're going to go next
• we have steps - the bits that do things
• we have next steps - where we go to next
• the class inherits from Tasks::Job - because it is supposed to represent some business process.

I wasn't happy with it, so a bit later I came up with this:

class Collabor8::DownloadDocument < Tasks::DecisionTree 
  inputs :project_member, :document, :use_inline_viewer, :use_dynamic_document_generator
  outputs :filename, :url
  starts_with :authorise_downloads
  
  decision :authorise_downloads do 
    if_result_is true, :use_inline_viewer
    if_result_is false, :authorisation_failure
  end
  
  decision :use_inline_viewer do 
    if_result_is true, :prepare_for_inline_viewer 
    if_result_is false, :prepare_for_direct_download 
  end
  
  def authorise_downloads
    user = project_member.user
    user.can?(:download, document) && !project_member.has_breached_download_limit? 
  end 
  
  def prepare_for_inline_viewer 
    perform Collabor8::PrepareForInlineViewer do 
      on_success do |inline_viewer|
        complete! url: inline_viewer[:url], filename: inline_viewer[:filename]
      end
    end
  end
  
  def prepare_for_direct_download 
    perform Collabor8::PrepareForDirectDownload do 
      on_success do |download|
        complete! url: download[:url], filename: download[:filename]
      end
    end
  end
end

Now the base class is Tasks::DecisionTree because it's supposed to be about making a decision. We've simplified the attributes into inputs, the decisions are cleaner and easier to read, and the steps have gone, replaced by methods.

But, unlike the decisions, the methods are not easy to understand at a glance. You can't see what they do and where they end up. The previous next_step seemed like a good idea.

Plus, I still wasn't happy with the names - it wasn't a decision tree, as it did things. But it wasn't a job, as it is supposed to make decisions. Command didn't fit for the same reason. Business logic? Business rule? Algorithm? Process?

And then I hit upon Operation - vague enough to represent both decisions and jobs whilst still fitting in with the idea of a business process.

Implementation

So I built myself a Rails Engine, fired up RSpec and started building a framework that could implement the classes I'd sketched out. As ever, I started with the README, using a simplified version of the Collabor8Online download example shown above. Once I'd written out the start of the README, I copied the example code into the spec/examples folder and wrote some specifications around it. Of course, these all failed. So I then fleshed those out, as separate "unit specs", in the spec/models folder.

I knew there was lots of stuff I wanted this code to do - background tasks (sometimes we have to send documents to external services to be converted, which can take a while) and building a version that does not rely on ActiveRecord or ActiveJob, so it can be used outside of Rails projects. But, I started with the basics decision handlers, action handlers and result handlers (because we need a way to say this operation has finished).

Once I'd built that very simple first version, I added it to Collabor8Online's bundle, and started implementing the real, complicated, downloads flowchart. As code hits the real world, design decisions get tested and the functionality that's actually important comes to the fore.

One decision that worked really well was divorcing the handlers from their container task. This makes handlers completely stateless, which in turn means they are very easy to test. If you've got an operation with 20+ states and 10+ decisions, coming up with all the combinations of data for an end-to-end test is going to take forever. But testing each individual handler in isolation makes it easy. Even better, because the handlers are stateless, writing the TestDataCarrier, which intercepts the important actions that handlers can perform so your expections/assertions can test them, was trivial.

One decision that didn't work out so well was the idea that operations did not need to carry their own data around with them in the database. I was hoping to only store the results of an operation in the database, but I soon realised that having the input parameters available, after the event, is going to be extremely valuable for diagnosing failures.

Using it

So now, figure out what your business process looks like. Draw it out as a flowchart and show it to your boss. Bosses like flowcharts. Then translate that flowchart into clean and simple ruby code.

Below is the actual current version of the PrepareDownload operation from Collabor8Online (although it does not represent the full flowchart yet). A few application-specific notes - a revision is a version of a document, but a revision may have variations. In this case, if the dynamic document generator is used, a revision is transformed with user and account-specific data, then stored as a variation of that revision. So we need to decide if we want to return the original revision or the user-specific variation. In addition, revisions may be sym-linked into other folders or documents, so we can pass in an optional folder or document to the operation, to make sure the correct references are kept.

Note that the actions in the operation are very very short - they delegate the actual work to other objects. That's because the point of this class is to act as live documentation for how the operation works.

class Document::Revision::PrepareDownload < Operations::Task
  broadcasts_refreshes
  inputs :project_member, :revision
  optional :folder, :document
  starts_with :use_filename_scrambler?
  
  decision :use_filename_scrambler? do
    inputs :project_member
    condition { project_member.scramble_downloaded_filenames? }
  
    if_true :generate_scrambled_filename
    if_false :use_original_filename
  end
  
  action :generate_scrambled_filename do
    inputs :revision
  
    self.filename = "#{task.random_name}.#{revision.file_extension}"
    go_to :use_dynamic_document_generator?
  end
  
  action :use_original_filename do
    inputs :revision
  
    self.filename = revision.name_with_extension
    go_to :use_dynamic_document_generator?
  end
  
  decision :use_dynamic_document_generator? do
    inputs :project_member
    condition { project_member.uses_dynamic_document_generator? && project_member.has_dynamic_document_generator_field_mapping? }
  
    if_true :generate_dynamic_document
    if_false :use_revision
  end
  
  action :generate_dynamic_document do
    inputs :project_member, :revision
  
    self.variation = call(Document::Revision::GenerateDynamicDocumentTask, project_member: project_member, revision: revision)[:variation]
    go_to :use_variation
  end
  
  action :use_revision do
    inputs :revision
    
    self.file_to_download = revision
    go_to :record_download
  end
  
  action :use_variation do
    inputs :variation
  
    self.file_to_download = variation
    go_to :record_download
  end
  
  action :record_download do
    inputs :revision, :project_member
    optional :document, :folder
  
    self.document ||= revision.document
    self.folder ||= document.folder
    folder.record_download_by(project_member.user, revisions: [revision], completed: true)
    go_to :return_document_details
  end
  
  result :return_document_details do |result|
    inputs :file_to_download, :filename
  
    result.filename = filename
    result.file_to_download = file_to_download
  end
  
  def random_name = [Faker::Lorem.word, Time.now.to_i.to_s, Faker::Lorem.word].join("-")
  
end