Explainable Anomaly Detection with RuleFit: An Intuitive Information

your anomaly detection outcomes to your stakeholders, the instant subsequent query is at all times “why?”.

In apply, merely flagging an anomaly isn’t sufficient. Understanding what went improper is crucial to figuring out one of the best subsequent motion.

But, most machine learning-based anomaly detection strategies cease at producing an anomaly rating. They’re black-box in nature, which makes it painful to make sense of their outputs-why does this pattern have the next anomaly rating than its neighbors?

To sort out this explainability problem, you will have possible already resorted to common eXplainable AI (XAI) strategies. Maybe you’re calculating function significance to establish which variables are driving the abnormality, or you’re operating counterfactual evaluation to see how shut a case was to regular.

These are helpful, however what for those who may do extra? What for those who can derive a set of interpretable IF-THEN guidelines that characterize the recognized anomalies?

That is precisely what the RuleFit algorithm [1] guarantees.

On this submit, we’ll discover how the RuleFit algorithm works intuitively, how it may be utilized to clarify detected anomalies, and stroll via a concrete case examine.

1. How Does It Work?

Earlier than diving into the technical particulars, let’s first make clear what we goal to have after making use of the algorithm: We wish to have a set of IF-THEN guidelines that quantitatively characterize the irregular samples, in addition to the significance of these guidelines.

To get there, we have to reply two questions:

(1) How can we generate significant IF-THEN circumstances from the information?

(2) How can we calculate the rule significance rating to find out which of them truly matter?

The RuleFit algorithm addresses these questions by splitting the work into two complementary components, the “Rule” and the “Match”.

1.1 The “Rule” in RuleFit

In RuleFit, a rule appears to be like like this:

IF x1 < 10 AND x2 > 5 THEN 1 ELSE 0

Would this construction look a bit extra acquainted if we visualize it like this:

Sure, it’s a resolution tree! The rule right here is simply traversing one particular path via the tree, from the foundation node to the leaf node.

In RuleFit, the rule era course of closely depends on constructing resolution timber, which predict the goal consequence given the enter options. As soon as the tree is constructed, any path from the foundation to a node in a tree will be transformed to a call rule, as we’ve simply seen within the instance above.

To make sure the foundations are numerous, RuleFit doesn’t simply match one resolution tree. As an alternative, it leverages tree ensemble algorithms (e.g., random forest, Gradient Boosting timber, and so on.) to generate many various resolution timber.

Additionally, the depths of these timber are, normally, totally different. This brings the advantages of producing guidelines with variable lengths, additional enhancing the variety.

Right here, we should always be aware that though the ensemble timber are constructed with predicting the goal consequence in thoughts, the RuleFit algorithm does not likely care concerning the finish prediction outcomes. It merely makes use of this tree-building train because the car to extract significant, quantitative guidelines.

Successfully, which means we are going to discard the expected worth in every node and solely hold the circumstances that lead us to a node. These circumstances produce the foundations we care about.

Okay, we will now wrap up the primary processing step within the RuleFit algorithm: the rule constructing. The end result of this step is a pool of candidate guidelines that would doubtlessly clarify the precise knowledge habits.

However out of all these guidelines, which of them truly deserve our consideration?

Nicely, that is the place the second step of RuleFit is available in. We “match” to rank.

1.2 The “Match” in RuleFit

Basically, RuleFit uncovers a very powerful guidelines by way of function choice.

First, RuleFit treats every rule as a brand new binary function, that’s, if the rule is glad for a particular pattern, it will get a worth of 1 for this binary function; in any other case, its worth is 0.

Then, RuleFit performs sparse linear regression with Lasso by utilizing all of the “uncooked” options from the unique dataset, in addition to the newly engineered binary options derived from the foundations, to foretell the goal consequence. This manner, every function (uncooked options + binary rule options) will get a coefficient.

One key attribute of Lasso is that its loss operate forces the coefficients of these unimportant options to be precisely zero. This successfully means these unimportant options are faraway from the mannequin.

In consequence, by merely inspecting which binary rule options survived the Lasso evaluation, we might instantly know which guidelines are necessary when it comes to getting correct predictions of the goal consequence. As well as, by trying on the coefficient magnitudes related to the rule options, we might be capable to rank the significance of the foundations.

1.3 Recap

Now we have simply lined the important idea behind the RuleFit algorithm. To summarize, we will view this method as a two-step answer for offering explainability:

(1) It first extracts the foundations by coaching an ensemble of resolution timber. That’s the “Rule” half.

(2) It then cleverly converts these guidelines into binary options and performs normal function choice by utilizing sparse linear regression (Lasso). That’s the “Match” half.

Lastly, the surviving guidelines with non-zero coefficients are necessary ones which can be value our consideration.

At this level, you will have seen that “predicting goal consequence” pops up at each the “Rule” and “Match” steps. If we’re coping with a regression or classification downside, it’s simply comprehensible that the “goal consequence” is the numerical worth or the label we wish to predict, and the foundations will be interpreted as patterns that drive the prediction.

However what about anomaly detection, which is basically an unsupervised job? How can we apply RuleFit there?

2. Anomaly Clarification with RuleFit

2.1 Software Sample

To start with, we have to rework the unsupervised explainability downside right into a supervised one. Right here’s how.

As soon as we’ve our anomaly detection outcomes (doesn’t matter which algorithm we used), we will create binary labels, i.e., 1 for an recognized anomaly and 0 for a standard knowledge level, as our “goal consequence.” This manner, we’ve precisely what RuleFit wants: the uncooked options, and the goal consequence to foretell.

Then, the RuleFit can work its magic to generate a pool of candidate guidelines and match a sparse linear regression mannequin to retain solely the necessary guidelines. The coefficients of the ensuing mannequin would then point out how a lot every rule contributes to the log-odds of an occasion being categorized as an anomaly. To place it one other method, they inform us which rule mixtures most strongly push a pattern towards being labeled as anomalous.

Notice you could, in idea, additionally use the anomaly rating (produced by the first anomaly detection mannequin) because the “goal consequence”. This can change the applying of RuleFit from a classification setting to a regression setting.

Each approaches are legitimate, however they reply barely totally different questions: With the binary label classification setting, the RuleFit uncovers “What makes one thing an anomaly?“; With the anomaly rating regression setting, the RuleFit uncovers “What drives the severity of an anomaly?“.

In apply, the foundations generated by each approaches will in all probability be very comparable. Nonetheless, utilizing a binary anomaly label because the goal for a RuleFit is extra generally used for explaining detected anomalies. It’s easy when it comes to interpretation and direct applicability to creating enterprise guidelines for flagging future anomalies.

2.2 Case Examine

Let’s stroll via a concrete instance to see how RuleFit works in motion. Right here, we’ll create an anomaly detection situation utilizing the Iris dataset [2] (licensed CC BY 4.0), the place every pattern consists of 4 options (sepal_length, sepal_width, petal_length, petal_width) and is labeled as one of many following three classes: Setosa, Versicolor, and Virginica.

Step 1: Knowledge Setup

First, we’ll use all Setosa samples (50) and all Versicolor samples (50) because the “regular” samples. For the “irregular” samples, we’ll use a subset of Virginica samples (10).

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, confusion_matrix
np.random.seed(42)

# Load the Iris dataset
iris = load_iris()
X = pd.DataFrame(iris.knowledge, columns=iris.feature_names)
y_true = iris.goal

# Get regular samples (Setosa + Versicolor)
normal_mask = (y_true == 0) | (y_true == 1)
X_normal_all = X[normal_mask].copy()

# Get Virginica samples
virginica_mask = (y_true == 2)
X_virginica = X[virginica_mask].copy()

# Randomly choose 10
anomaly_indices = np.random.selection(len(X_virginica), measurement=10, substitute=False)
X_anomalies = X_virginica.iloc[anomaly_indices].copy()

To make the situation extra lifelike, we create a separate coaching set and check set. The prepare set incorporates pure “regular” samples, whereas the check set consists of randomly sampled 20 “regular” samples and 10 “irregular” samples.

train_indices = np.random.selection(len(X_normal_all), measurement=80, substitute=False)
test_indices = np.setdiff1d(np.arange(len(X_normal_all)), train_indices)

X_train = X_normal_all.iloc[train_indices].copy()
X_normal_test = X_normal_all.iloc[test_indices].copy()

# Create check set (20 regular + 10 anomalous)
X_test = pd.concat([X_normal_test, X_anomalies], ignore_index=True)
y_test_true = np.concatenate([
    np.zeros(len(X_normal_test)),   
    np.ones(len(X_anomalies))       
])

Step 2: Anomaly Detection

Subsequent, we carry out anomaly detection. Right here, we faux we don’t know the precise labels. On this case examine, we apply Native Outlier Issue (LOF) because the anomaly detection algorithm, which locates anomalies by measuring how remoted a knowledge level is in comparison with the density of its native neighbors. In fact, you can even attempt different anomaly detection algorithms, comparable to Gaussian Combination Fashions (GMM), Okay-Nearest Neighbors (KNN), and Autoencoders, amongst others. Nevertheless, needless to say the intention right here is barely to get the detection outcomes, our principal focus is the anomaly clarification in step 3.

Particularly, we’ll use the pyOD library to coach the mannequin and make inferences:

# Set up the pyOD library
#!pip set up pyod

from pyod.fashions.lof import LOF

# Standardize options
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.rework(X_test)

# Native Outlier Issue
lof = LOF(n_neighbors=3)
lof.match(X_train_scaled)

train_scores = lof.decision_function(X_train_scaled)
test_scores = lof.decision_function(X_test_scaled)
threshold = np.percentile(train_scores_lof, 99)
y_pred = (test_scores > threshold).astype(int)

Discover that we’ve used the 99% quantile of the anomaly scores obtained on the coaching set as the edge. For particular person check samples, if its anomaly rating is greater than the edge, this pattern will probably be labeled as “anomaly”. In any other case, the pattern is taken into account “regular”.

At this stage, we will shortly examine the detection efficiency with:

classification_report(y_test_true, y_pred, target_names=['Normal', 'Anomaly'])

Not tremendous nice outcomes. Out of 10 true anomalies, solely 5 of them are caught. Nevertheless, the excellent news is that LOF didn’t produce any false positives. You may additional enhance the efficiency by tuning the LOF mannequin hyperparameters, adjusting the edge, and even contemplating ensemble studying methods. However remember: our purpose right here is to not get one of the best detection accuracy. As an alternative, we goal to see if RuleFit can correctly generate guidelines to clarify the anomalies detected by the LOF mannequin.

Step 3: Anomaly Clarification

Now we’re attending to the core subject. To use RuleFit, let’s first set up the library from imodels, which is a sklearn-compatible, Interpretable ML bundle for concise, clear, and correct predictive modeling:

pip set up imodels

On this case, we are going to contemplate a binary label classification setting, the place the irregular samples (within the check set) flagged by the LOF mannequin are labeled as 1, and different un-flagged regular samples (additionally within the check set) are labeled as 0. Notice that we’re labeling primarily based on LOF’s detection outcomes, not the precise floor fact, which we faux we don’t know.

To provoke the RuleFit mannequin:

from imodels import RuleFitClassifier

rf = RuleFitClassifier(                 
        max_rules = 30,           
        lin_standardise=True,           
        include_linear=True,           
        random_state = 42
)

We are able to then proceed with becoming the RuleFit mannequin:

rf.match(
    X_test, 
    y_pred, 
    feature_names=X_test.columns
)

In apply, it’s often a great apply to do a fast sanity examine to judge how nicely the RuleFit mannequin’s predictions align with the anomaly labels decided by the LOF algorithm:

from sklearn.metrics import accuracy_score, roc_auc_score

y_label = rf.predict(X_test)               
y_prob  = rf.predict_proba(X_test)[:, 1]   

print("accuracy:", accuracy_score(y_pred, y_label))
print("roc-auc:", roc_auc_score (y_pred, y_prob))

For our case, we see that each printouts are 1. This confirms that the RuleFit mannequin has efficiently realized the patterns that LOF used to establish anomalies. In your personal issues, for those who observe values a lot decrease than 1, you would want to fine-tune your RuleFit hyperparameters.

Now let’s look at the foundations:

guidelines = rf._get_rules()
guidelines = guidelines[rules.coef != 0]                         
guidelines = guidelines[~rules.type.str.contains('linear')]      
guidelines['abs_coef'] = guidelines['coef'].abs()
guidelines = guidelines.sort_values('significance', ascending=False)

The RuleFit algorithm returns a complete of 24 guidelines. A snapshot is proven beneath:

Let’s first make clear the that means of the outcomes columns:

• The “rule” column and the “abs_coef” column are self-explanatory.
• The “sort” column has two distinctive values: “linear” and “rule”. The “linear” denotes the unique enter options, whereas “rule” denotes the “IF-THEN” circumstances generated from resolution timber.
• The “coef” column represents the coefficients produced by the Lasso regression evaluation. A constructive worth signifies that if the rule applies, the log-odds of being categorized because the irregular class will increase. A bigger magnitude signifies a stronger affect of that rule on the prediction.
• The “assist” column information the fraction of knowledge samples the place the rule applies.
• The “significance” column is calculated as absolutely the worth of the coefficient multiplied by the usual deviation of the binary (0 or 1) values that the rule takes on. So why this calculation? As we’ve simply mentioned, a bigger absolute coefficient means a stronger direct impression on the log-odds. That’s clear. For the usual deviation time period, it successfully measures the “discriminative energy” of the foundations. For instance, if a rule is sort of at all times TRUE (very small normal deviation), it doesn’t cut up your knowledge successfully. The identical holds if the rule is sort of at all times FALSE. In different phrases, the rule can not clarify a lot of the variation within the goal variable. Due to this fact, the significance rating combines each the power of the rule’s impression (coefficient magnitude) and the way nicely it discriminates between totally different samples (normal deviation).

For our particular case, we see just one high-impact rule (Rule #24):

If a flower’s petal is longer than 5.45 cm and wider than 2 cm, the percentages that LOF classifies it as “anomalous” enhance 85-fold. (Notice that exp(4.448999) ~= 85)

Guidelines #26 and #27 are nested inside Rule #24. That is frequent in apply, as RuleFit typically produces “households” of comparable guidelines as a result of they arrive from neighbouring tree splits. Due to this fact, the one rule that really issues for characterizing the LOF-identified anomalies is Rule #24.

Additionally, we see that the assist for Rule #24 is 0.1667 (5/30). This successfully signifies that all 5 LOF-identified anomalies will be defined by this rule. We are able to see that extra clearly within the determine beneath:

There you might have it: the rule to explain the recognized anomalies!

3. Conclusion

On this weblog submit, we explored the RuleFit algorithm as a strong answer for explainable anomaly detection. We mentioned:

• The way it works: A two-step method the place resolution timber are first fitted to derive significant guidelines, adopted by a sparse linear regression to rank the rule significance.
• The best way to apply to anomaly clarification: Use the detection outcomes because the pseudo labels and use them because the “goal consequence” for the RuleFit mannequin.

With RuleFit in your modeling toolkit, the following time stakeholders ask “Why is that this anomaly?”, you’ll have concrete IF-THEN guidelines that they’ll perceive and act upon.

Reference

[1] Jerome H. Friedman, Bogdan E. Popescu, Predictive studying by way of rule ensembles, arXiv, 2008.

[2] Fisher, R. A., Iris [Data set]. UCI Machine Studying Repository, 1936.