Run tests and test suites

Published

September 4, 2024

ValidMind provides many built-in tests and test suites, which help you produce documentation during stages of the model development lifecycle where you need to validate that your work satisfies MRM (model risk management) requirements.

model documentation
A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses.

Within the realm of model risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.

validation report
A formal document produced after a model validation process, outlining the findings, assessments, and recommendations related to a specific model’s performance, appropriateness, and limitations. Provides a comprehensive review of the model’s conceptual framework, data sources and integrity, calibration methods, and performance outcomes.

Within model risk management, the validation report is crucial for ensuring transparency, demonstrating regulatory compliance, and offering actionable insights for model refinement or adjustments.

template, documentation template
Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.

ValidMind templates come with pre-defined sections, similar to test placeholders, including boilerplates and spaces designated for documentation and test results. When rendered, produces a document that model developers can use for model validation.

test
A function contained in the developer framework, designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind platform to generate the model documentation according to the template that is associated with the documentation.

Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.

metrics, custom metrics
Metrics are a subset of tests that do not have thresholds. Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with ValidMind to be used in the platform.

In the context of ValidMind’s Jupyter Notebooks, metrics and tests can be thought of as interchangeable concepts.

inputs
Objects to be evaluated and documented in the developer framework. They can be any of the following:
  • model: A single model that has been initialized in ValidMind with vm.init_model(). See the Model Documentation or the for more information.
  • dataset: Single dataset that has been initialized in ValidMind with vm.init_dataset(). See the Dataset Documentation for more information.
  • models: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom tests.
  • datasets: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. See this example for more information.
parameters
Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.
outputs
Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.
test suite
A collection of tests which are run together to generate model documentation end-to-end for specific use cases.

For example, the classifier_full_suite test suite runs tests from the tabular_dataset and classifier test suites to fully document the data and model sections for binary classification model use cases.

Getting started

Start by running a pre-made test, then modify it, and finally create your own test:

Explore tests and test suites

Next, find available tests and test suites using the developer framework or the interactive test sandbox:

Intermediate

Building on previous sections, add your own test provider, set up datasets, run tests on individual sections in your model documentation, and more:

Advanced

Need more? Try some of the advanced features provided by the developer framework:

When do I use tests and test suites?

While you have the flexibility to decide when to use which ValidMind tests, we have identified a few typical scenarios with their own characteristics and needs:

Dataset testing

To document and validate your dataset:

Model testing

To document and validate your model:

  • For binary classification models: use the classifier test suite.
  • For time series models: use the timeseries test suite.

End-to-end testing

To document a binary classification model and the relevant dataset end-to-end:

Use the classifier_full_suite test suite.

Can I use my own tests?

Absolutely! ValidMind supports custom tests that you develop yourself or that are provided by third-party test libraries, also referred to as test providers. We provide instructions with code examples that you can adapt:

Reference

ValidMind Developer Framework Reference