Glossary

Published

September 4, 2024

This glossary of terms provides short definitions for technical terms you find commonly used in our product documentation grouped by terms related to:

ValidMind
Models and model risk management
Model documentation
Validation reports
Ongoing monitoring
Developer tools
Artificial intelligence

ValidMind

ValidMind AI risk platform

These two features are intertwined and work in tandem to help streamline your model lifecycle.

ValidMind Developer Framework (developer framework): An open-source¹ suite of documentation tools and test suites designed to document models, test models for weaknesses, and identify overfit areas. Enables automating the generation of model documentation by uploading documentation and test results to the ValidMind AI risk platform.
ValidMind Platform UI (platform UI): A hosted multi-tenant architecture² that includes the ValidMind cloud-based web interface, APIs, databases, documentation and validation engine, and various internal services.

ValidMind core features

client library, Python client library: Enables the interaction of your development environment with ValidMind as part of the ValidMind Developer Framework.
documentation automation: A core benefit of ValidMind that allows for the automatic creation of model documentation using predefined templates and test suites.
model inventory: A feature of the ValidMind platform where you can track, manage, and oversee the lifecycle of models. Covers the full model lifecycle, including customizable approval workflows for different user roles, status and activity tracking, and periodic revalidation.
template, documentation template: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.

ValidMind templates come with pre-defined sections, similar to test placeholders, including boilerplates and spaces designated for documentation and test results. When rendered, produces a document that model developers can use for model validation.

test: A function contained in the developer framework, designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind platform to generate the model documentation according to the template that is associated with the documentation.

Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.

test suite: A collection of tests which are run together to generate model documentation end-to-end for specific use cases.

For example, the classifier_full_suite test suite runs tests from the tabular_dataset and classifier test suites to fully document the data and model sections for binary classification model use cases.

Models and model risk management

Models

model: SR 11-7³ defines a model as a “quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.”
model development: An iterative process in which many models are derived, tested, and built upon until a model fitting the desired criteria is achieved.
model documentation: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses.

Within the realm of model risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.

model inventory⁴: A systematic and organized record of all quantitative and qualitative models used within an organization. This inventory facilitates oversight, tracking, and assessment by listing each model’s purpose, characteristics, owners, validation status, and associated risks.
model lifecycle: Subset of stages defining the lifecycle of a model; encompasses all steps for operating, governing, and maintaining a model until it is decommissioned (model development, model validation, model approval, model implementation, retirement).
model risk: The potential for financial loss, incorrect decisions, or unintended consequences resulting from errors or inaccuracies in AI or machine learning models. Model risk typically arises from incorrect or inappropriate use of models, inaccurate assumptions, or limitations in data quality.

For example, consequences of unmitigated model risk can include adverse outcomes such as financial loss, damage to reputation, and regulatory penalties.

model risk management (MRM): A structured approach to identifying, assessing, mitigating, and monitoring risks arising from the use of quantitative and qualitative models within an organization. Ensures that models are developed, validated, and used appropriately, with robust controls in place. Encompasses practices such as maintaining a model inventory, conducting periodic validations, and ensuring proper documentation.
model vendor: A company that develops, documents, and sells models to financial institutions.
vendor model: A model created by an external source, such as a model vendor.

Model risk management

1st line of defense: Business unit(s) responsible for model development, initial validation, and implementation during the model lifecycle. As the 1st line of defense, model developers must document and test models to ensure that they are accurate, robust, and fit for purpose.
2nd line of defense: An independent oversight function that provides a governance framework for the model lifecycle. As the 2nd line of defense, model validators must independently validate and challenge models created by model developers to ensure that model risk management principles are followed.
3rd line of defense: Typically an internal audit function responsible for providing an independent and comprehensive review of the risk management processes and controls that the first two lines have implemented.
model developer: Responsible for the design, implementation, and maintenance of models to ensure they are fit-for-purpose, accurate, and aligned with business requirements. As subject matter experts, they collaborate with model validators and other business units, ensuring the models are conceptually sound and robust.
model governance: A framework of policies, procedures, and standards established to oversee the lifecycle of models within an organization. Ensures that models are developed, validated, implemented, and retired in a controlled and consistent manner, promoting accountability, transparency, and adherence to regulatory requirements.
model implementation: A collaborative effort among model developers and model owners. Model implementation includes a formalized implementation plan and associated procedures, a review of results, and a record of model change procedures.
model owner: Responsible for coordinating model development, model implementation, ongoing model monitoring and maintaining the model’s administration, such as model documentation and model risk reporting.
model user: Those who rely on the model’s outputs to inform business decisions.
model validation: A systematic process to evaluate and verify that a model is performing as intended, accurately represents the phenomena it is designed to capture, and is appropriate for its specified purpose. This assessment encompasses a review of the model’s conceptual soundness, data integrity, calibration, and performance outcomes, as well as testing against out-of-sample datasets.

Within model risk management, model validation ensures that potential risks associated with model errors, misuse, or misunderstanding are identified and mitigated.

model validator: Responsible for conducting independent assessments of models to ensure their accuracy, reliability, and appropriateness for intended purposes. The role involves evaluating a model’s conceptual soundness, data integrity, calibration methods, and overall performance, typically using out-of-sample datasets.

Model validators identify potential risks and weaknesses, ensuring that models within an organization meet established standards and regulatory requirements, and provide recommendations to model developers for improvements or modifications.

three lines of defense: A structured approach to model risk management, consisting of three independent functions:

The first line consists of business units responsible for model development, validation, and implementation. They ensure that models are accurate, robust, and fit for purpose.
The second line is an independent model risk oversight function that provides a governance framework and guidance for model risk management.
The third line is the internal or external audit function, which assesses the robustness of model risk management practices and controls.

validation report: A formal document produced after a model validation process, outlining the findings, assessments, and recommendations related to a specific model’s performance, appropriateness, and limitations. Provides a comprehensive review of the model’s conceptual framework, data sources and integrity, calibration methods, and performance outcomes.

Within model risk management, the validation report is crucial for ensuring transparency, demonstrating regulatory compliance, and offering actionable insights for model refinement or adjustments.

Model documentation

Each section of your model documentation should address critical aspects of the model’s lifecycle, from conceptualization and data preparation through development and ongoing management. This comprehensive documentation approach is essential for ensuring the model’s reliability, relevance, and compliance with business and regulatory standards.

conceptual soundness: Establishes the foundation of the model, covering the model overview, intended use and business use case, regulatory requirements, model limitations, and the rationale behind the model selection. It emphasizes the model’s purpose, scope, and constraints, which are crucial for stakeholders to understand the model’s applicability and limitations.
data preparation: Details the data description, including dataset summary, data quality tests, descriptive statistics, correlations and interactions, and feature selection and engineering. It provides transparency into the data used for model training, ensuring that the model is built on a solid and relevant dataset.
model development: Discusses the model training, evaluation, explainability, interpretability, and diagnosis, including model weak spots, overfit regions, and robustness. This section is vital for understanding how the model was developed, how it performs, and its areas of strength and weakness.
monitoring and governance: Focuses on the model’s ongoing monitoring plan, implementation, and governance plan. It outlines strategies for maintaining the model’s performance over time and ensuring that it remains compliant with regulatory requirements and ethical standards.

Validation reports

A validation report is a comprehensive review that evaluates a model’s accuracy, performance, and suitability for its intended purpose. It encompasses the process of model risk assessment, identifying areas of potential error or risk within the model’s components, such as data inputs and algorithms. The report follows established validation guidelines to ensure consistency and adherence to internal and regulatory standards.

actions: Recommended steps or measures to address findings from model validation or risk assessments.
evidence: Material provided by the developer and reviewed by the validator, such as model documentation, source code, datasets, monitoring reports or previous validation reports.
findings: Observations or issues identified during model validation, including any deviations from expected performance or standards.
review: entails a comprehensive evaluation process covering four key aspects of model documentation to ensure thoroughness, compliance, and reliability:

Conceptual soundness: Examines the foundational elements of the model, including its overview, intended business use, regulatory requirements, and limitations. Ensures that the model’s purpose, scope, and constraints are well-defined and understood by stakeholders.
Data preparation: Assesses the quality and preparation of the data used for model training. Includes a detailed look at dataset summaries, data quality tests, descriptive statistics, correlations, interactions, and feature engineering. The aim is to verify that the model is built on a robust and relevant dataset.
Model development: Focuses on the model’s development process, including training, evaluation, explainability, interpretability, and diagnosis. Highlights the model’s performance, identifies strengths and weaknesses, and ensures that any potential issues such as overfitting are addressed. Evaluates the assumptions made and examines the qualitative information and judgments to ensure they are conducted appropriately and systematically.
Monitoring and governance: Evaluates the ongoing strategies for monitoring the model’s performance and ensuring compliance with regulatory and ethical standards. Involves checking the implementation of the monitoring plan and governance strategies to maintain the model’s efficacy over time. Covers reporting outputs to ensure transparency and accuracy in the model’s documented results.

These elements collectively ensure that the model documentation is thorough, transparent, and meets all necessary standards and regulatory requirements.

model risk assessment: The process of identifying and evaluating risks associated with the use and potential errors in a financial model.
model risk areas: Specific components or aspects of a model where risk might be present, such as data inputs, algorithms, or implementation.
validation guidelines: Established standards or procedures for conducting thorough and consistent model validations, usually aligned with principles within specific models or AI risk frameworks.

Ongoing monitoring

backtesting: Comparing a model’s predictions against actual outcomes to verify its predictive power and reliability.
compliance and regulatory adherence: Ensuring that the model continues to meet evolving regulatory requirements and standards.
model drift: Changes in data patterns, input distributions, or model behavior that may indicate a degradation in model performance over time.
model performance: The measure of a model’s accuracy, stability, and robustness in achieving its intended outcomes, which is regularly evaluated through monitoring after deployment to ensure ongoing reliability.
ongoing monitoring: A periodic report assessing the model’s performance and compliance over time, ensuring it remains valid under changing conditions.
recalibrating models: The process of adjusting a model to account for detected drift or changes in the underlying data or environment.
reporting and governance: The documentation of monitoring findings and communication to stakeholders to support decision-making and maintain transparency.

Developer tools

decorator, Python decorator: A design pattern in Python⁵ that allows a user to add new functionality to an existing object without modifying its structure.

Decorators are a simpler way for users to run their own code as a ValidMind test.

inputs: Objects to be evaluated and documented in the developer framework. They can be any of the following:

model: A single model that has been initialized in ValidMind with vm.init_model(). See the Model Documentation or the for more information.
dataset: Single dataset that has been initialized in ValidMind with vm.init_dataset(). See the Dataset Documentation for more information.
models: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom tests.
datasets: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. See this example for more information.

outputs: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.
parameters: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.
pip: A package manager for Python, used to install and manage software packages written in the Python programming language.

ValidMind uses the pip command to install the Python client library that is part of the ValidMind Developer Framework so that model developers can make use of its features.

JupyterHub: A multi-user server provides a platform for users to interactively work with data science and scientific computing tools in a collaborative environment.

ValidMind uses JupyterHub to share live code, how-to instructions, and visualizations via notebooks as part of our getting started experience for new users.

Jupyter Notebook: Allows users to create and share documents containing live code, data visualizations, and narrative text. Supports various programming languages, most notably Python, and is widely used for data analysis, machine learning, scientific research, and educational purposes.

ValidMind uses notebooks to share sample code and how-to instructions with users that you can adapt to your own use case.

metrics, custom metrics: Metrics are a subset of tests that do not have thresholds. Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with ValidMind to be used in the platform.

In the context of ValidMind’s Jupyter Notebooks, metrics and tests can be thought of as interchangeable concepts.

GitHub: A cloud-based platform that provides hosting for software development and version control using Git. GitHub⁶ offers collaboration tools such as bug tracking, feature requests, task management, and continuous integration pipelines.

ValidMind uses GitHub to share [pen-source software⁷ with you.

⁷ GitHub: validmind

Artificial intelligence

Refer to IBM’s series on artificial intelligence for more in-depth resources.

artificial intelligence (AI): Artificial intelligence is a broad term used to classify machines that mimic human intelligence and human cognitive functions like problem-solving and learning.
deep-learning: A subset of machine learning that uses multi-layered neural networks (deep neural networks) to simulate the complex decision-making power of the human brain.
generative AI (GenAI): Generative AI refers to deep-learning models that can generate high-quality text, images, and other content based on the data they were trained on.
large language model (LLM): Advanced types of artificial intelligence models designed to understand, generate, and interact with human language at a sophisticated level, such as ChatGPT.⁸
machine learning: Machine learning is a subset of artificial intelligence that allows for optimization. It helps make predictions that minimize the errors that arise from merely guessing.
traditional statistical models: Mathematical frameworks used to analyze and make inferences from data. These models are foundational in statistics and serve to explain relationships, predict outcomes, and guide decision-making across various fields, such as economics, biology, engineering, and social sciences.