In the vast majority of enterprises that adopt business rules, the aim is to maintain a rule repository over a long period, evolving it to meet the rapidly evolving needs of a business. It is surprising, then, that although much is written about how to analyse, capture and implement business rules, much less has been published about how to maintain them over time. How can you extend the life of your business rules against the ravages of time?

Introduction

It is a sad fact that, often, once a business rule repository is established – once rules are mined, a fact model built, rules authored, reviewed, tested and released – the job is often considered done. The business rule experts are ‘rolled off’ the project onto pastures new and a maintenance team is established to sustain and evolve the rule repository thereafter. Frequently, maintenance teams have less experience than the original project members because it is assumed that to maintain something requires less skill than to create it initially. As anyone experienced in software maintenance will tell you, this is wrong and it frequently leads to degraded and even ‘lost’ rule repositories over time.

Although it’s always good to start with a well-built rule repository, it is just as important to maintain it carefully. A bad start can be saved by good maintenance, but poor maintenance is always terminal for the repository concerned – no matter how good it was at the outset. The skills that make a good rule-steward are poorly understood.

In this article, we to explore what specific dangers a rule repository faces over the medium and long term and how to combat them. How can we avoid these issues and extend the life of a rule repository?

The Issues

Threats to long term use of business rules in a repository all have one source: the entropy of change – whether it is change in the rules or the underlying fact model itself, or change in the business environment around it. Over time this can cause the following problems:

• ‘Repository Bloat’ (Explicit Complexity) – so many rules are added to the repository over time that its original structure is compromised and it becomes hard to find or understand rules; this may be exacerbated by accidental duplication and dead rules. The central structural concepts and patterns that promote consistency within the repository are undermined by addition of rules that don’t conform or that use other patterns—this makes the rule set hard to understand and therefore to modify. Interestingly, it is not the number of rules itself which is the problem, but the number of concepts they rely on: 10,000 rules conforming to the same pattern is less complex than a 1,000 each with a unique pattern.
• Implicit Complexity – the rules depend on each other, or some hidden backdrop of implemented code, in a myriad of ways which are either poorly documented or not articulated at all. This makes rules hard to understand in isolation and difficult to change safely without unintended side-effects or failures.
• Cryptic or ‘lost’ rules – rules are not changed or even viewed for so long that people forget their purpose and are afraid to modify them. This term also applies to rules that are built from a non-business vocabulary (‘IT verbalisation’) and that are, as a consequence, difficult to understand and short-lived.
• Paradigm Shift – the business context (or even motivation) of a rule set alters over time so significantly that the repository is no longer aligned to the original business purpose and therefore does not have the appropriate degrees of freedom. Ultimately this means the rules in the repository will be subjected to multiple changes for which they were never designed or that some rules become irrelevant.

Fighting Repository Bloat (Explicit Complexity)

The larger a repository is and the more concepts and rule patterns (a rule patterns is a general style or shape of a rule, its conditions and its conclusions – multiple rules can share the same pattern) it has to support, the harder it can be to understand it. Large repositories with many different patterns can make it difficult to locate an existing rule or correctly place a new rule. Rules can be obscured and even, in extreme cases, lost by this approach.

Furthermore, once rules are lost, it is easy for one rule author to add a rule to fulfil a functional already addressed by an existing (‘lost’) rule – worsening the bloat of the repository with duplicates. All of these impediments impact the ease of comprehension and agility of the rule repository: the key reasons for developing it in the first instance.

Another goal of a rules repository is to visualize business logic effectively and unambiguously – to inform everyone in an organisation about how business decisions are reached. If the size and conceptual clarity of the repository are not actively managed, it can become markedly less effective as a means of visualising business policy. To prevent this:

• Avoid excessive growth – When changes are needed, don’t be tempted just to add another set of rules without a complete understanding of what the existing rules already do. Try to exploit reuse as often as you can and look for ways in which existing rules can be used to meet the new requirement. If you must add a set of rules wholesale without assessing how it could be integrated, for example to meet a deadline, ensure the assessment is done later.
• Encourage Consistency – establish rule patterns (and rule templates to support them) as ‘cookie cutters’ to address common rules. Design your rule patterns to cover as many common variations as possible in one generalisation without obscuring them. By this means the same, single pattern can support a large set of related rule forms, not just one. This can avoid the need for many subtly different, but overlapping rule types. Always have a firm rationale for introducing a new rule that does not comply with an existing pattern.
• Cull unused rules – monitor the execution frequency of rules to ensure that ‘dead’ rules, those that have not been executed in a significant time, are investigated. If these are genuinely redundant (and this fact has to be verified with the business), remove them.
• Economise on Idioms – Frequently consider refactoring your rules during quite periods. Periodically analyse your rules to see if a new abstraction can replace a large number of rules with a much smaller one, or a new pattern can combine many existing rules into one abstraction. Place an upper limit of the number of concepts (terms, fact types and rules) that your repository addresses. If it grows beyond this…
• Subpartition – if a subset of rules is emerging, which is itself cohesive and easy to separate from the other rules, consider placing it in a rule repository of its own. Split only if the question ‘in which repository should a given rule be related?’ can be easily answered for each rule.
• Design for ‘searchability’ – ensure that the repository rule hierarchy is aligned by those properties of a rule that are most well understood when placing a new rule or locating an existing rule. For example if the best understood properties of a rule are tax jurisdiction, income class and household type – ensure that these are the keys used to structure the repository hierarchy. This will help to avoid accidental duplication of rules and assist users in finding what they need to know. If your repository allows, place the rules in multiple hierarchies according to the most likely search keys (e.g. tax type, governance status, how long ago it was last changed).

Managing Implicit Complexity

The rules in a repository have two dependencies:

• on their context, the underlying business and technical backdrop against which they are implemented and their integration with this backdrop. This determines: the terms and fact types that are available to build rules, the rule vocabulary, that point in the business process when rules are used, what information they are passed and what conclusions are passed back from them and, most importantly, the boundary between rules/decisions in the repository and those processes and decisions controlled by other means ‘in the background’
• on each other – one rule may have a consequence that influences another, we say that rules have inferential dependencies.

Both dependencies are an entirely natural, indeed necessary, means of rules to collaborate to reach a compound conclusion and to contribute towards an ensemble behaviour – but these dependencies must be visible, controlled and explicit. When they are implicit, or uncontrolled, rule behaviour may be hard to understand and modify safely.

Implicit Dependence on Context

This complexity is most dangerous over time: the developers of a rule set are, initially, intimately familiar with the context because they developed it! It will seem patently obvious to them when a repository is first delivered. However, if the context is designed well, it will change only occasionally. Often, it will only be partially visible to business rule maintainers. As a consequence, understanding of this context can be lost over time. Once lost, it will be hard for business users to understand the ensemble behaviour of rules and how to change it as needed.
To avoid this:

• Document – ensure the vocabulary, context and rule patterns are documented within your rule repository or on a wiki or some other form of ‘active’ documentation that all stakeholders can access and which is regularly updated. It helps if the source of this documentation is as integrated with the rule repository as possible. The best solution is if the documentation is directly generated from fields maintained in the rule repository, so that a change to rules automatically incurs a change to the documentation.
• Standardize – make the context and vocabulary adhere to a business standard (e.g. ACORD for insurance, ISO20022 for finance, etc) that has an independent, well publicized, explicit definition that will endure. Ensure that technical, ephemeral and other ‘business foreign’ terminology is kept out of the vocabulary.
• Simplify – make the context as simple as you can: avoid synonyms, avoid mixing rule and non-rule decision making in one process workflow, follow the business process as closely as possible, avoid arbitrary sequencing of rules and where you must have them ensure the sequence is obvious and required by mutual dependencies (see below).

Implicit Inter-Rule Dependence

Often a ruleset has an established firing sequence, defined by a clear ‘chain’ of inferential dependency in which the conclusions reached by one rule are used as conditions in others. This is entirely normal. Indeed rules can be divided into two groups: structural rules that derive new information from existing facts (e.g., determining the credit worthiness of a client, their loyalty score or some other classification of existing, or derived data) and behavioural rules that take action (e.g. declining a loan, determining that an underwriter should be involved to price an insurance application, etc). Normally, within a repository, behavioural rules are dependent on structural rules which themselves depend on other structural rules or input data itself.

This structure of dependency is entirely healthy. Where these dependencies become problematic is if they are too numerous, hidden or inconsistent. Hidden dependencies occur when conclusions reached by one rule have a hidden side effect that alters the behaviour of others. For example a rule that determines an applicant’s current debt burden might also have an invisible side effect of modifying their credit rating, influencing the behaviour of rules that depend on credit rating even though, on inspection, the two appear unrelated. This is undesirable because it makes the rules’ ensemble behaviour difficult to predict and it means that when rules are changed the results are often unexpected – we say that the ruleset is ‘brittle’. Such fragility makes rule authors apprehensive about changing rules which eliminates the key value proposition of the rules approach. Too many dependencies or dependencies that are poorly normalized (see the decision model for an effective means of normalizing these relationships) can also result in a ruleset that is confusing and difficult to understand.

To avoid this:

• Separate structural and behavioural rules into separate parts of the repository and make the distinction clear to rule authors.
• Make all inferential dependencies completely explicit, do not have any implicit independencies or side-effects.
• Ensure that each rule has only one explicit conclusion, this may have multiple attributes and properties – but there should only be one conclusion fact type per rule.
• Use as few inferential dependencies as you can to solve the problem.
• Using a rule capture and representation technique (like the decision model, TDM) which insures that independencies are clearly stated, structurally sound and fully normalized.
• Ensure that all of the above points are checked during the routine, governance mandated, rule review process.
• Develop a fully automated, regression testing environment for the rule set. This should allow rule authors to see the entire impact of a proposed rule change before they commit to it. This step requires a significant investment: a set of input data that exercises all of the key boundary conditions of the rules needs to be built and maintained. But the payback is enormous: nothing short of the confidence to make changes quickly and know that they will only have the desired impact and no collateral damage. We call this safe agility.

Avoiding ‘Lost’ Rules

Believe it or not, rules can become lost, even in smaller repositories. There are two means of losing rules, or in extremis, an entire repository:

• Loss by Obscurity – the rules are expressed either using IT terminology, or in terms of the implementation specifics, rather than using widely accepted business terms. These IT terms obfuscate the business meaning of rules, over complicating them and reducing the likelihood of successful ownership and maintenance of a rule set by the business. Also, IT terms are frequently less enduring and portable than business terminology – decreasing the likelihood that they will survive over time. In a merger between two companies (in the same area of business), business orientation of the rules (as opposed to expressing them in IT terms) could well mean the difference being reusing existing rule repositories, with some modifications, versus having to re-write them.
• Loss by Lack of Familiarity – the rules change so slowly that they are very rarely visited by rule authors. Over long time spans this can cause the rule to be forgotten and, unless the logic of the rule is trivial, it will effectively be incomprehensible and therefore hard to change safely. Such rules are sometimes called ‘magic’ rules because, like some arcane spell, their meaning is lost and they become an object of fear, rather than something that can be controlled and modified with confidence.

To combat this:

• Always use only approved business terms, from business SMEs and practitioners, to represent your rules.
• Periodically review these to ensure they are up to date.
• Have the rules regularly reviewed by the business rule authors and the larger community for ease of comprehension. Make this review part of the governance process for approving new and changed rules.
• Know the likely change frequency of rules: rarely changing rules must be designed to be trivially easy to understand and to stand-alone. Implementation of static rules (i.e., rules that you know will never change) should be avoided.

Paradigm Shift (Maintaining Business Relevance)

Like any other piece of software, a rule repository is either changing or it’s dying. There is no middle ground. Constant change is to be expected; indeed supporting this is one of the key value propositions of rules – if your business rules are not changing why are you using rules? But what happens when there is a sea change in business motivation that changes or even undermines the very rationale of the rule repository?

The two issues are: that the change can happen slowly and imperceptibly, over a long period of time – so it is not recognised – and that, even if it is recognised, cost pressures forbid properly restructuring and refactoring the rule repository for its new purpose. Rules are agile; surely we can change them to fit the new purpose?

A rule repository is like a house: it is an assembly formed of a chain of interdependent and mutually supportive structures, from the foundations to the roof. It is reasonable to extend a house by, say, adding a garage, a new room or perhaps an attic conversion. However, effective extensions are typically small in comparison with the house itself and have the same overall motivation – in this case to serve as a family residence.

No one would dream of extending a house into a 20-storey block of flats or repurposing it has an opera house over a period of months, whilst still serving residents at each stage of its development. Yet this is often what we expect of rule repositories and indeed, other software structures. Managers, keen to cut costs, are often reluctant to pay for refactoring (the equivalent of stripping the original house down to its foundations and rebuilding it with the new purpose in mind) and instead are tempted to introduce layer after layer of extensions and redesigns, one atop the other, until the entire edifice is like a house of cards: each different but overlapping abstraction stacked precariously on the previous one in an ungainly, rickety and dangerous structure.

To combat this:

• Structure the Rational for Your Rules  – Determine the business goals, tactics and policies your rules should address (Building Business Solutions has more details about methods to do this). A goal is a formal business objective (e.g., increase profitability) and a tactic is a means to achieve this (e.g., reduce costs of non-automated work). A tactic is supported by business policies that are directly implemented in rules and have their effectiveness measured.
• Document Rationale – Document the business rationale of the rule repository as a series of business tactics and policies that support them. State it as clearly and as succinctly as you can within the rule repository. Express each policy as a named, SMART objective, with named KPIs for measurement and the business tactics it supports. Following our example above, an example policy is ‘be generate fewer referrals for manual processing.’ A KPI to measure this might be ‘percentage referral frequency’ and a target for this KPI might be ‘reduce the referral frequency from 12% to 4.5% by Q4 2014’. For each rule, or subset of rules, document the policies and tactics it contributes to and the KPIs by which it should be measured – if these change later, the impact to rules can be assessed more readily.
• Keep It Up to Date – Periodically re-assess the business tactics to ensure they are still relevant, appropriately prioritized and aligned to current business goals. If there is a market resurgence, profitability might best be supported by exploiting new markets, rather than be reducing costs.
• Keep It Optimised – Also check that the measurement KPI of the policy’s achievement is still effective. If the cost of non-automated work starts to vary strongly on a case by case basis – reducing the frequency of referral may not be the best way to measure our success. Measurement of the proportion of time spent on non-automated processes, or their costs, might be more effective. Ideally, measure the KPIs in practice to see how effective the rules are in supporting them.
• Re-evaluation – Periodically re-evaluate rules against these tactics and policies, especially if they have changed recently. Are they as effective as they used to be? Has their priority changed?
• Partition – When new rules are required that address different tactics and policies, consider putting them in a different repository or a separate part of the existing repository.

Conclusion

Like any software asset, a rule repository is under the assault of requirements change and entropy from the moment it is first released. The hard work of maintaining a released repository requires just as much planning, effort and expertise as creating it in the first place, albeit effort and expertise of a different kind. After investing significant time and money in the initial creation of a repository, failing to maintain it would seem a strange and unnecessary dereliction – a wasted investment that quickly loses its value in the ways we have described.

Contrary to intuition, there is much proactive work to do after the initial creation of a repository. This requires a dedicated and experienced resource. It is not tenable to allow a repository to ‘survive’ with occasional attentions only by rule authors and reviewers – a dedicated repository steward is required. We urge all owners of non-trivial rule repositories to consider: ‘who is looking after your rules?’

Useful? Please share: