Customize test outputs using output templates

Run individual tests and tests for experimentation and when exploring and building output templates to create custom results.

The ValidMind Developer Framework provides a suite of tests to help you evaluate the performance of your machine learning models. The out-of-the-box results are designed to be informative and easy to understand, but you may want to customize the look and feel of the results to better suit your needs. This might include things like removing or adding columns from the results, changing the formatting or structure of a table, or adding entirely new tables to the results. Output templates allow you to do all of these things and more.

Output templates currently can create and cutomize tables. They are written in HTML and use the Jinja2 templating language.

As part of the notebook, you will build on the simple quickstart_customer_churn notebook and learn how to:

This interactive notebook uses the Bank Customer Churn Prediction sample dataset from Kaggle to train a simple classification model.

Contents

About ValidMind

ValidMind is a platform for managing model risk, including risk associated with AI and statistical models.

You use the ValidMind Developer Framework to automate documentation and validation tests, and then use the ValidMind AI Risk Platform UI to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.

Before you begin

This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language.

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

New to ValidMind?

If you haven’t already seen our Get started with the ValidMind Developer Framework, we recommend you explore the available resources for developers at some point. There, you can learn more about documenting models, find code samples, or read our developer reference.

For access to all features available in this notebook, create a free ValidMind account.

Signing up is FREE — Sign up now

Key Concepts

  • Output Templates: Customizable HTML templates that define the look and feel of the results produced by the ValidMind tests. They are written in HTML and use the Jinja2 templating language.
  • Jinja2 Templating Language: A powerful templating language for Python that allows you to embed expressions and control structures in your HTML templates.
  • Customizing Tables: Output templates allow you to customize the look and feel of the tables produced by the ValidMind tests. This includes things like adding or removing columns, changing the formatting or structure of the table, and adding entirely new tables to the results.
  • Documentation Templates: Documentation templates are covered in the quickstart notebook and are the base for all model documentation. They are written in YAML and define the entire structure and content of a model’s documentation. Output templates are not part of the documentation template, but they are defined in and shared via a field in the documentation template.

How documentation template work with output templates

Below is a section of the standard Binary Classification Documentation Template that comes pre-installed with the ValidMind Developer Framework.

The top-level model_evaluation section contains a list of sub sections which contain content blocks. These blocks can be either editable text blocks or test-driven blocks where the content_id identifies the test within the Developer Framework whose results will be displayed in that block. Now the key thing here is that each of these tests produces a specific output that is not directly editable from the ValidMind platform. This is where output templates come in, both figuraively and literally. They can be added as an optional field in the content block and will use the raw test output data in an HTML template to produce a custom table that can be displayed in the documentation.

Example documentation template

- id: model_development
  title: Model Development
  index_only: true
  sections:
    - id: model_training
      title: Model Training
      guidelines:
        - Describe the model training process, including the algorithm used, any
          hyperparameters or settings, and the optimization techniques employed
          to minimize the loss function or maximize the objective function.
        - ... (additional guidelines)
      contents:
        - content_type: metric
          content_id: validmind.model_validation.ModelMetadata
        - ... (additional content blocks)
      parent_section: model_development
    - id: model_evaluation
      title: Model Evaluation
      guidelines:
        - Describe the process used to evaluate the model's performance on a
          test or validation dataset that was not used during training, to
          assess its generalizability and robustness.
        - ... (additional guidelines)
      contents:
        - content_type: metric
          content_id: validmind.model_validation.sklearn.ConfusionMatrix
        - content_type: metric
          content_id: validmind.model_validation.sklearn.ClassifierPerformance
        - ... (additional content blocks)
      parent_section: model_development

In the above example, the validmind.model_validation.sklearn.ClassifierPerformance produces two tables like this:

Alt text

But with output templates, you can customize the look and feel of the output to produce a much simpler/clearer version like this:

Alt text

How this is accomplished is with the following output template:

- content_type: metric
  content_id: validmind.model_validation.sklearn.ClassifierPerformance:with_template
  output_template: |
    <table>
        <thead>
            <tr>
                <th>Accuracy</th>
                <th>Precision</th>
                <th>Recall</th>
                <th>F1 Score</th>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td>{{ value["accuracy"] }}</td>
                <td>{{ value["weighted avg"]["precision"] }}</td>
                <td>{{ value["weighted avg"]["recall"] }}</td>
                <td>{{ value["weighted avg"]["f1-score"] }}</td>
            </tr>
        </tbody>
    </table>

As you can see, the output template is a simple HTML table that uses the Jinja2 templating language to embed expressions that reference the raw test output data. The { value["accuracy"] } expression, for example, references the accuracy key in the raw test output data. This is how you can customize the look and feel of the results produced by the ValidMind tests.

Now that you understand the basics of output templates, the following sections will guide you through the process of creating and using them in your code.

Install the client library

The client library provides Python support for the ValidMind Developer Framework. To install it:

%pip install -q validmind

Initialize the client library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the client library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet:

  1. In a browser, log into the Platform UI.

  2. In the left sidebar, navigate to Model Inventory and click + Register new model.

  3. Enter the model details and click Continue. (Need more help?)

    For example, to register a model for use with this notebook, select:

    • Documentation template: Binary classification
    • Use case: Marketing/Sales - Attrition/Churn Management

    You can fill in other options according to your preference.

  4. Go to Getting Started and click Copy snippet to clipboard.

Next, replace this placeholder with your own code snippet:

# Replace with your code snippet

import validmind as vm

vm.init(
    api_host="https://api.prod.validmind.ai/api/v1/tracking",
    api_key="...",
    api_secret="...",
    project="...",
)

Initialize the Python environment

Next, let’s import the necessary libraries and set up your Python environment for data analysis:

import xgboost as xgb

%matplotlib inline

Load the sample dataset

The sample dataset used here is provided by the ValidMind library. To be able to use it, you need to import the dataset and load it into a pandas DataFrame, a two-dimensional tabular data structure that makes use of rows and columns:

# Import the sample dataset from the library

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Document the model

As part of documenting the model with the ValidMind Developer Framework, you need to preprocess the raw dataset, initialize some training and test datasets, initialize a model object you can use for testing, and then run the full suite of tests.

Prepocess the raw dataset

Preprocessing performs a number of operations to get ready for the subsequent steps:

  • Preprocess the data: Splits the DataFrame (df) into multiple datasets (train_df, validation_df, and test_df) using demo_dataset.preprocess to simplify preprocessing.
  • Separate features and targets: Drops the target column to create feature sets (x_train, x_val) and target sets (y_train, y_val).
  • Initialize XGBoost classifier: Creates an XGBClassifier object with early stopping rounds set to 10.
  • Set evaluation metrics: Specifies metrics for model evaluation as “error,” “logloss,” and “auc.”
  • Fit the model: Trains the model on x_train and y_train using the validation set (x_val, y_val). Verbose output is disabled.
train_df, validation_df, test_df = demo_dataset.preprocess(raw_df)

x_train = train_df.drop(demo_dataset.target_column, axis=1)
y_train = train_df[demo_dataset.target_column]
x_val = validation_df.drop(demo_dataset.target_column, axis=1)
y_val = validation_df[demo_dataset.target_column]

model = xgb.XGBClassifier(early_stopping_rounds=10)
model.set_params(
    eval_metric=["error", "logloss", "auc"],
)
model.fit(
    x_train,
    y_train,
    eval_set=[(x_val, y_val)],
    verbose=False,
)

Initialize the ValidMind datasets

Before you can run tests, you must first initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module.

This function takes a number of arguments:

  • dataset — the raw dataset that you want to provide as input to tests
  • input_id - a unique identifier that allows tracking what inputs are used when running each individual test
  • target_column — a required argument if tests require access to true values. This is the name of the target column in the dataset
  • class_labels — an optional value to map predicted classes to class labels

With all datasets ready, you can now initialize the raw, training and test datasets (raw_df, train_df and test_df) created earlier into their own dataset objects using vm.init_dataset():

import validmind as vm

vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column=demo_dataset.target_column,
    class_labels=demo_dataset.class_labels,
)

vm_train_ds = vm.init_dataset(
    dataset=train_df, input_id="train_dataset", target_column=demo_dataset.target_column
)

vm_test_ds = vm.init_dataset(
    dataset=test_df, input_id="test_dataset", target_column=demo_dataset.target_column
)

Initialize a model object

Additionally, you need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data. You simply intialize this model object with vm.init_model():

vm_model = vm.init_model(
    model,
    input_id="model",
)

Assign predictions to the datasets

We can now use the assign_predictions() method from the Dataset object to link existing predictions to any model. If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

Run individual tests and customize the results

Instead of running the full suite of tests, you can run individual tests. This is useful for experimentation and when exploring and building output templates to create custom results. Lets go ahead and run a single test, ClassifierInSamplePerformance, and see how we can create fully customized results from the output using output templates.

from validmind.tests import run_test

First, let’s run the test as normal and see the standard output:

result = run_test(
    test_id="validmind.model_validation.sklearn.ClassifierPerformance",
    inputs={
        "dataset": vm_train_ds,
        "model": vm_model,
    },
)

Let’s also take a look at the result object that is returned when running the test and see how we can grab the raw metric value from it to start developing our output template:

import json

print("In Sample Performance Raw Value:")
print(json.dumps(result.metric.value, indent=2))

This is the raw value object that will get passed into the output template and accessible just with the value variable name. Now let’s go ahead and create a simple output template like in the example and then see how we can test it directly against the result object:

output_template = """
<table>
    <thead>
        <tr>
            <th>Accuracy</th>
            <th>Precision</th>
            <th>Recall</th>
            <th>F1 Score</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>{{ value["accuracy"] }}</td>
            <td>{{ value["weighted avg"]["precision"] }}</td>
            <td>{{ value["weighted avg"]["recall"] | number }}</td>
            <td>{{ value["weighted avg"]["f1-score"] | number }}</td>
        </tr>
    </tbody>
</table>
"""
# specifically notice how the values from the result are being accessed inside the template
# also notice that we can use filters to format the values e.g. `| number` to format the number to 4 decimal places
# we can immediately re-render while trying different output templates
result.render(output_template=output_template)

And, there you go, you have successfully created and used a custom output template. Try making some changes to the template html and see how it affects the output. You can also add more complex logic and control structures to the template using the Jinja2 templating language here.

Now that you have a working output template, it can also be passed right into the run_test function to produce the same results as before:

result = run_test(
    test_id="validmind.model_validation.sklearn.ClassifierPerformance",
    inputs={
        "dataset": vm_train_ds,
        "model": vm_model,
    },
    output_template=output_template,
)

Awesome! So you have seen how to create and use output templates when running individual tests. In a real-world scenario though, you would want to add the output template to the documentation template so that it can live there as a permanent customization. This is what we will cover next.

Run the full suite of tests with output templates

Now that you’ve seen how to run an individual test and customize the output, let’s see how you can apply that concept to model documentation by adding the output template to the documentation template and running the full suite of tests.

Add the output template to the documentation template

First, go to your project in the ValidMind UI and go to Settings > Templates. Find the Binary Classification Template that is used by the Customer Churn project. Click on the Edit button to bring up the template editor. Then, add the following content block below the existing validmind.model_validation.sklearn.ClassifierPerformance test:

- content_type: metric
  content_id: validmind.model_validation.sklearn.ClassifierPerformance:with_template
  output_template: |
    <table>
        <thead>
            <tr>
                <th>Accuracy</th>
                <th>Precision</th>
                <th>Recall</th>
                <th>F1 Score</th>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td>{{ value["accuracy"] }}</td>
                <td>{{ value["weighted avg"]["precision"] }}</td>
                <td>{{ value["weighted avg"]["recall"] }}</td>
                <td>{{ value["weighted avg"]["f1-score"] }}</td>
            </tr>
        </tbody>
    </table>

This will add a second version of the ClassifierPerformance test so we can compare the standard output with the custom output.

Run the full suite of tests

Now that you’ve added the output template to the documentation template, you can run the following code cells to initialize the client which retrieves the template. Then you can run the full suite of tests to see the custom output in the documentation and on the ValidMind UI.

full_suite = vm.run_documentation_tests(
    section=["model_development"],
    inputs={
        "dataset": vm_test_ds,
        "datasets": (vm_train_ds, vm_test_ds),
        "model": vm_model,
    },
)

Next steps

You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind platform to work with your model documentation.

Work with your model documentation

  1. From the Model Inventory in the ValidMind Platform UI, go to the model you registered earlier.

  2. Click and expand the Model Development section.

What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it’s ready. Learn more …

Discover more learning resources

We offer many interactive notebooks to help you document models:

Or, visit our documentation to learn more about ValidMind.