Introduction

In all but the simplest cases, our experience shows that machine learning (predictive analytics) greatly benefits from decision management to deliver true business value, improve interpretability and mitigate risk. In turn, the days of static business decisions based only on simple decision table based business rules are numbered. Decision management needs machine learning to meet the needs of data driven businesses – to deliver decisions that are capable of statistical inference and are reactive to evolving business conditions: decisions that learn. In short machine learning and decision management form a powerful partnership.

In this article I discuss how decision management can be used as a unifying framework to integrate machine learning (ML) models and business rules (often expressed as decision tables) into decisions that learn. I show why this partnership is beneficial to both ML and decision modelling. Using BPMN these learning decisions can then be integrated into business processes yielding a powerful, coherent platform for automating dynamic business operations. We assert that the framework of decision management is essential for any complete, accountable application of machine learning to automated business processes.

‘‘the days of static business decisions based only on simple decision table based business rules are numbered’’

About Machine Learning

ML has been variously referred to as predictive, proscriptive or descriptive analytics or even AI – I eschew these marketing terms here in favour of machine learning because these algorithms are machines capable of learning and providing actionable insight in a business context.

Machine learning can be supervised or unsupervised. By supervised machine learning, I refer to a range of techniques and algorithms that use an important subset of data attributes (features) to predict vital facts (labels) about new data, given exposure to labelled old data. These include classifiers where the label is one of a number of discrete values (e.g., whether a transaction if fraudulent or not) or regressors for which the label is a continuous value (e.g., a person’s salary). Unsupervised Machine Learning does not use labels. It includes algorithms that draw actionable insight from the distribution of data, often in high dimensions (e.g., clustering, outlier detection), and that make assertions (e.g., predictions, classifications) about new data instances given this distribution.

ML is also frequently partitioned into deep and shallow learning. Deep learning uses neural networks, typically well suited to image or audio classification, that offer automatic feature detection at the cost of additional compute resources. Whereas shallow learning uses a range of alternative linear and non-linear techniques which are often more efficient and yield good results for tabular data, but which require features be explicitly identified in advance by domain experts. This is independent of supervised or unsupervised ML.

I believe business use of machine learning, supervised or unsupervised, deep or shallow, will become increasingly pervasive as a means of making smarter, real-time operational decisions and enabling decisions that learn.

Machine Learning: The Missing Pieces

Machine learning is very widely applicable. Various industries have used it for tasks as diverse as predicting new uses for established drugs, forecasting credit default, detecting fraud in real time, forecasting customer churn and determining market sentiment from twitter and newspaper articles. Despite some very promising results by some companies, many find their first applications of machine learning to be disappointing. It is often marred by:

• Rush to implementation, with resulting poor understanding, shared vision and satisfaction of the real business need. Machine learning is fun and often ML projects are started with very unclear business goals
• Poor data, poor understanding of data, Jumping to building models before a complete understanding of the data and a clean dataset and set of features have been acquired
• Wasted Business Expertise, use integration of machine learning with the existing knowledge of a company’s human subject matter experts (SMEs) with the frequent result that ML models conclude what the experts already know
• Ethical and regulatory concerns (e.g., GDPR), what safeguards are needed for models with outcomes that can impact people’s lives?
• Unpleasant surprises, discontinuities in normally well behaved models can yield surprisingly bad results under some circumstances. Some neural network models, for example, can be tricked into producing farcical results with small changes to the input data (such as the failure of facial recognition systems when the subject wears a small badge)
• Poor interpretability, machine learning models can yield accurate results. However, they are often unable to explain the rationale of their outcomes or the specific data on which they relied. Likewise, machine learning models cannot show what changes in the case would yield a different result
• Badly handled model drift, machine learning models can perform well initially, but their prediction performance declines over time. This is often due to covariate or prior drift, a natural change in the relationship between variables or their distributions over time
• Unintentional bias resulting from biased training data or even a biased model or attribute selection
• Poor integration with business processes so that the outcomes of machine learning can be actioned safely
• Conflicted goals, An attempt to use a single machine learning algorithm to solve many problems at once which yields a sub-optimal model (for any one of them)
• No quantified benefit, an inability to take a business context into account, to connect and contribute to business key performance indicators
• Incomplete, Inability to capture higher level human criteria such as compassion, ethics, business goals and common sense

How do we address these problems?

How Decision Management Helps

Decision management is a framework for expressing, maintaining and executing business decisions and supporting their integration into a business process to produce actionable insights, either under the auspices of a human operator or automatically. Decision management can combine business rules (including decision tables), machine learning models and invocations of external services into a single model (expressed in a decision model using a notation called DMN), that:

• Is open and transparent. Can be understood and maintained by business subject matter experts (not just developers or data scientists)
• Supports process automation. Can be fully integrated to an automated business process in BPMN and support directly executable models with quantified business performance indicators
• Strengthens controls. Clearly expresses the data dependencies, business objectives and performance indicators of all operations

Using this framework, we can address the above issues with machine learning.

Focuses on Business Benefit

Decision management focuses on the business need and benefit. All elements of decision making (machine learning or otherwise) are driven by a strong understanding of these factors. Every decision is explicitly associated with business objectives and key performance indicators (KPIs). Example business objectives include attaining some regulatory standard, lowering costs or retaining more customers. KPIs are more specific, providing a specific metric goal and timescale. Example KPIs include: ‘the total negative margin should not exceed 5% of the value of the portfolio’, ‘the offer rejection rate should fall at least 1% per quarter’ or ‘the number of cases requiring manual oversight should remain below 3%’. Many decision management stacks allow these business goals to be measured and tracked.

By associating learning decisions with strong business goals in this way we can avoid nebulous ML projects. Furthermore, the outcomes of machine learning can be viewed as insights that can be actioned for business benefit. Decisions that use them can be directly integrated with a company’s business process, improving business performance and accountability.

Provides Strong Data Provenance and Ethical Compliance

Decision management fosters complete understanding of all input data, business rules and machine learning models required to make a decision and the dependencies between them. This is expressed in a highly explicit and visible way. This is invaluable when applying ML.

All machine learning models are explicitly associated with specific attributes of input data called features. Decision management provides a clear chain of custody between each input data attribute and business decisions and outcomes that rely on it. This improves the rigour of the application of machine learning and makes decision models an ideal vehicle for regulatory and ethical controls such as the European GDPR and the American CCA. Decision management can support techniques such as Active Adversarial Impact Mitigation (AAIM) which determines how effectively your model can predict protected attributes such as gender or race. If it can, this is an indication that some of your features are proxies for these attributes and will result in a biased model. This works if you selected the features manually or automatically (e.g., using a neural network).

In this way decision models can be used to avoid the use of protected attributes in machine learning, either directly or as inferred or proxy attributes. Decision models can even be used to detect and address the bias introduced by these accidental dependencies.

Augments Machine Learning Models with Expert Knowledge

Inexperienced data scientists often make the assumption that machine learning models are the best solution for every business decision. All that is needed is a sufficiently large data set of previous observations and their associated decision outcomes (labels) and a supervised model to train and optimize through cross-validation. Thereafter one can replace the decision-making process with the trained model. Surely by this approach we can learn to automate any business decision using ML alone? This approach is often very poor because:

• In some cases it is not feasible because previous observations may not be available and, even if they are, the efficacy of the outcomes are unknown and may be biased.
• ‘Learning’ business expertise, which is already on-hand (in the heads of human experts), is inefficient. It requires a lot of time and computer resources. If there are complex but deterministic elements of the decision making which are well understood and do need to be relearned, these can be better represented as a network of decision tables created by human experts. Even if these are not flawless, they can be improved with ML, rather than learning everything from scratch.
• Some logic, such as compliance, is volatile, complex and well-defined by nature. It is unwise to make this the focus of a machine learning model otherwise frequent (and expensive) retraining will be required.
• Decision table networks created by human experts have much higher interpretability than machine learning models that, despite their accuracy, are often unable to clarify their rationale. Many aspects of automated decision making are under increasing pressure to explain their outcomes. For example, regulations demand decisions demonstrate In other words that they exhibit no bias between ethnic groups, genders or other protected criteria. In some cases, they are not allowed to use the protected criteria in any way. This is more readily achieved, and crucially more easily demonstrated, using static business rules.
• It is sometimes not possible to train a machine learning model to have the same recall and precision of a decision model constructed by subject matter experts because the training data can be noisy and because the training process itself is not perfect.
• Decision outcomes are often the results of several independent decisions, combining them into a single optimized machine learning model will increase its complexity thereby compromising its accuracy and its interpretability. A better solution is to embed a set of narrowly focussed, tightly defined machine learning models into a decision that can augment them with traditional business rules as needed. An example of this is a credit award decision which may consist of a machine learning model to determine likelihood of default and a network of supporting decision tables to provide compliance support and parity checking.

In short we should use a combination of ML and traditional decision making as dictated by requirements and use decision modelling as a means of integrating the two approaches.

Improves Safety and Performance of Machine Learning

Machine learning models predictive performance can be monitored from decision models and any drift or discontinuity can be contained and addressed by the overall logic of the decision improving the robustness of the outcome.

Using decision management, ML models can be combined with business knowledge from subject matter experts expressed as rules such as decision tables. These rules can augment the models with real-world business expertise, enhancing their predictive accuracy and reducing their training time. Furthermore, these rules can be used to constrain the machine learning component in accordance with required controls. Of course, there are other means of establishing these controls, but only decision management allows machine learning models and controls to be combined in ways that are transparent to non-technical business experts.

Decision management can also be used to track on-line machine learning models, those which learn and constantly re-train as they process production data.

Enhances Power and Interpretability of Machine Learning

Sets of different ML models can be combined, in a decision model, into an ensemble (or set of alternate specialists) which collaborate to improve overall business performance. This facilitates integration of existing models with different zones of applicability.

Decision models are highly transparent, and these techniques can be used to improve the interpretability of inscrutable machine learning models, making it possible for business subject matter experts to understand them holistically and to understand specifically how they generated each outcome in production. This approach is more applicable to shallow ML working on tabular data than deep learning on image data. However even here techniques like exemplars, attention and bounding boxes can be used to justify the outcome of neural networks. This facilitates an equal partnership between data scientists and business SMEs in the design and governance of machine learning models.

Conclusion

Although originally designed to manage the expression, governance and execution of business decisions based solely on decision tables, decision management is rapidly becoming the best means of integrating machine learning (especially on-line machine learning) into a robust, end-to-end business process. ML and decision management mutually strengthen one another:

• ML provides decision models with be ability to supplement decision making with effective behaviours learned from data
• decision management brings the benefit of improved alignment with business goals and greater interpretability.