Building a Rete algorithm based Rule Engine in Golang

Disclaimer: Generated by Me!

Introduction

The motivation of this personal initiative was this post from Anthropic: Building a C Compiler. Nicholas Carlini of Anthropic wrote this post before we saw even more significant initiatives such as solving Navier-Stokes and the OpenAI breakout. To me, this was the first example of using swarms of agents to complete a task, rather than single process model SDLCs, and also a task I’d imagined would not be one independently achievable by AI.

One fact from the C-Compiler post stuck out for me, that the cost of the compiler was $20,000. This is now low compared to the Navier-Stokes investment, but significantly out of range for a mere mortal such as myself. My question was this: How much and what impact could I get with a budget more like $20 per month!

An OPS5 interpreter is significantly lighter to implement than a C-Compiler, so I chose that to investigate the impact of AI on Enterprise Systems development.

OPS5 is a Lisp based interpreted language in which it is possible to define working memory elements to represent facts, production rules that may produce other facts, and a conflict resolution system that can provide an order when several rules are being satisfied that may execute at once. Facts are represented as matched facts within an Alpha network and joined with nodes in a Beta network to determine what state to reach next in the rule evaluation.

OPS5 Recognize-Act Cycle Figure 1: A Rule Engine Abstract - A rule engine may be stateless, or stateful depending on how the memories are managed

Rules Engines in the enterprise have been generally passed over due to their complexity, however, you should notice that there has been significant interest. One of the reference papers about Business Rules was written with EF Codd. There are a number of example systems we’ve seen such as Drools and Jess, however, none in a central role providing a true logical middle tier. Rule engines provide major unblocker for managing the complexity of an enterprise, you may have an interest in Rule Engines as a part of your workflow if some of the following resonate:

  • most enterprise systems are executing relatively simple logic - variable defaults, enrichment from lookup table, application of a formula. The complexity we see in enterprise systems typically comes down to logic being spread over multiple disjoint systems either resulting in the inability to query across domain boundaries or problems where information is propagated across domain boundaries without consideration of the new rules. Rule engines provide a logic tier that unifies disjoint systems.
  • when applications extract views of data, the view does not contain the business rules nor the underlying relationships of the system that delivered the view. A view complemented with rules fully describes the relationships of cells within the resultset. Rule engines enable a connection between a view and a ruleset which may be passed between processes.
  • using (nondeterministic) AI is not feasible in the majority of realtime and even batch and stream processing requirements. This is because the amount of time taken to solve a problem is not known at the point of execution. OPS5 can evaluate a rule problem in a deterministic time.
  • Consider the following simple architecture: AI manages Working Memory elements within a Rule Engine. Production Rules constrain the elements, and provide a report if an attempt is made to perform a bad move. Here is a very simple Governance Framework! A Rule Engine provides rules for both human and machine operators that cannot be circumvented.
  • being able to separate join logic from DAO logic enables architecture separation for certain problem domains, some significant ones being true bitemporality, others being schema transformations such as denormalization or normalization.
  • The lineage problem in enterprises is difficult to comprehensively solve. When Business Rules are applied to ingress datasets, ideally we would like to know what version of the business rules was used to create that datapoint, and what the decision logic was. This typically is lost, and has to be painstakingly recreated when required. We can snapshot a full rule-engine such that we can associate transformantions with the exact business logic at the time decisions were made.
  • logic in enterprise systems sometimes exists when implemented through trigger mechanism in a database. This logic can be very difficult to unwind due to the complexity of foreign key constraints. OPS5 systems enable very good control in joins, and join failures. In fact, a relational schema can be completely externalized from a datastore if required.

Implementation

The project can be located here: https://github.com/graemenewlands/ops5. Building the project took roughly three weeks and held the following phases:

  • Tool acquisition
  • Specification materials
  • Initial Implementation
  • Specification and Documentation
  • Testing
  • Doing something cool to prove it works …

Tool Acquisition

The tooling used for generating the ops5 module in Golang was Gemini 3.8 available via a Pro account. When obtaining the Pro license, there were some nice things bundled in, such as $10/month free GCP credits. The Pro license enabled me to use Gemini 3.8 for about three hours per evening before hitting a maximum usage limit. I found that at the limit, it was not possible to downgrade to a lower model and replenish token supply. There were certain activities that would significantly consume tokens, reading and processing a large codebase was one, running servers and committing and pushing code was not one.

I found that using antigravity in conjunction with vscode was my preferred modus operandi. With antigravity, the commands I used most were /usage, /btw, /fork and /grill-me, which were learnt in that order over the course of three weeks (the Google AI offering was new to me at the start of the project). VS Code was used for reviewing changes to the code base.

Specification Materials

The Rete algorithm has long been a fascination for me and I have acquired some good materials along the way:

Initial Implementation

There have been Rete algorithm and expression evaluation implementations for Golang before, on GitHub these are: go-rules and goexpr. Neither is an OPS5 implementation, which is what I was looking for. I decided to go from scratch, using a detailed description of the rete algorithm along with positive and negative matching rules. It appeared as though Gemini was already quite familiar with the rules problem, as it was already able to provide an MEA and LEX conflict resolution strategy.

The first commit, done on evening one, contained an implementation of an alpha and beta network for Rete, Dynamic Memory management, LEX and MEA conflict resolution and basic functional testing. Impressive stuff.

After this, the next week of evenings was spent working on detail. OPS5 instructions like cbind, bind, a CLI, support for vectors, debugging instructions, print, tabto, crlf, multivalued objects, literalize, visualization of the rete network, (modify)ing working memory elements, openfile… There was a lot missing. None of this was suggested by the agent, but was supplied by prompt.

Hello Alice Fig 2: Running the interactive cli for the ops5 engine via the console

The second week was spent:

  • building out features for the OPS5 interpreter. One was compute to allow efficient aggregations over working memory elements.
  • implementing efficiency gains for the engine.

OPS5 Working Memory Visualization Figure 3: Visualizing the Initial State of integration test tests/integration/2_5_2.ops5

Specification and Documentation

There is a fine line between documentation and specification. It was decided early on to maintain a keywords document, which includes a high level description of every command available to ops5. There are details however, that are not available here, for instance, how is file_io managed, how does the engine manage the BetaNetwork, how does conflict resolution work, the details of which are important to maintain compatibility between systems. Each significant detail has a specification page within the docs, and the purpose here is as much for allowing the agent to regression test its assumptions as for allowing us to understand how an OPS5 interpreter works.

Testing

There are three distinct tests being performed for every build of the OPS5 interpreter:

  1. Functional testing:

    • of the parser. These assert that the Lisp syntax is processed as expected.
    • of the engine. These are written as unit tests and evaluate the result of engine and working memory states

  2. Integration testing:

    • there are two large integration test samples in which we assert a solution is found.

  3. Performance testing:

    • we benchmark the time taken to solve known ops5 problems on a given hardware configuration.
    • we assert that the benchmark does not degrade for known problems with every build.

Doing something cool to prove it works …

I’d like to invite you to look at: Conway’s Game of Life (OPS5 WASM)

The Game of Life is a good demonstration for a rules engine for the following reasons:

  • simple rules result in complex behaviors
  • we can see that rule engines may not result in a steady state solution, sometimes rules may not halt
  • we can compare our rule engine against other Game of Life implementations
  • we get a sense for how efficient it is
  • we can interact with it by setting some grids on and off against the known Conway life structures

This runs the go module in a Wasm file in your browser.

Summary

While OpenAI and Anthropic are publishing breakthroughs in science and engineering, making meaningful and significant advances in personal projects is definitely feasible. When working with AI, it is necessary to continuously be updating the documentation and specification of your product with a natural language definition of how it works, its inputs and outputs. In addition, making continuous assertions about performance became necessary, as it was found that some changes did not necessarily make improvements from a performance perspective.