Test descriptions

Published

September 4, 2024

This topic describes the tests that are available as part of the ValidMind Developer Framework, grouped by type of validation or monitoring test.

Try the test sandbox (BETA)

Explore our interactive sandbox to see what tests are available in the ValidMind Developer Framework.

ACFandPACFPlot
Analyzes time series data using Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots to reveal trends and correlations.
ADF
Assesses the stationarity of a time series dataset using the Augmented Dickey-Fuller (ADF) test.
ANOVAOneWayTable
Applies one-way ANOVA (Analysis of Variance) to identify statistically significant numerical features in the dataset.
AutoAR
Automatically identifies the optimal Autoregressive (AR) order for a time series using BIC and AIC criteria.
AutoMA
Automatically selects the optimal Moving Average (MA) order for each variable in a time series dataset based on minimal BIC and AIC values.
AutoSeasonality
Automatically identifies and quantifies optimal seasonality in time series data to improve forecasting model performance.
AutoStationarity
Automates Augmented Dickey-Fuller test to assess stationarity across multiple time series in a DataFrame.
BivariateFeaturesBarPlots
Generates visual bar plots to analyze the relationship between paired features within categorical data in the model.
BivariateHistograms
Generates bivariate histograms for paired features, aiding in visual inspection of categorical variables distributions and correlations.
BivariateScatterPlots
Generates bivariate scatterplots to visually inspect relationships between pairs of predictor variables in machine learning classification tasks.
ChiSquaredFeaturesTable
Executes Chi-Squared test for each categorical feature against a target column to assess significant association.
ClassImbalance
Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.
CommonWords
Identifies and visualizes the 40 most frequent non-stopwords in a specified text column within a dataset.
DFGLSArch
Executes Dickey-Fuller GLS metric to determine order of integration and check stationarity in time series data.
DatasetDescription
Provides comprehensive analysis and statistical summaries of each field in a machine learning model’s dataset.
DatasetSplit
Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML model.
DescriptiveStatistics
Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model’s dataset.
Duplicates
Tests dataset for duplicate entries, ensuring model reliability via data quality verification.
EngleGrangerCoint
Validates co-integration in pairs of time series data using the Engle-Granger test and classifies them as Cointegrated’ or ‘Not cointegrated’.
FeatureTargetCorrelationPlot
Visualizes the correlation between input features and model’s target output in a color-coded horizontal bar plot.
Hashtags
Assesses hashtag frequency in a text column, highlighting usage trends and potential dataset bias or spam.
HeatmapFeatureCorrelations
Creates a heatmap to visually represent correlation patterns between pairs of numerical features in a dataset.
HighCardinality
Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.
HighPearsonCorrelation
Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.
IQROutliersBarPlot
Visualizes outlier distribution across percentiles in numerical data using Interquartile Range (IQR) method.
IQROutliersTable
Determines and summarizes outliers in numerical features using Interquartile Range method.
IsolationForestOutliers
Detects outliers in a dataset using the Isolation Forest algorithm and visualizes results through scatter plots.
KPSS
Executes KPSS unit root test to validate stationarity of time-series data in machine learning model.
LaggedCorrelationHeatmap
Assesses and visualizes correlation between target variable and lagged independent variables in a time-series dataset.
LanguageDetection
Detects the language of each text entry in a dataset and visualizes the distribution of languages as a histogram.
Mentions
Calculates and visualizes frequencies of ‘@’ prefixed mentions in a text-based dataset for NLP model analysis.
MissingValues
Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold.
MissingValuesBarPlot
Creates a bar plot showcasing the percentage of missing values in each column of the dataset with risk categorization based on a user-defined threshold.
MissingValuesRisk
Assesses and quantifies the risk related to missing values in a dataset used for training an ML model.
PearsonCorrelationMatrix
Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.
PhillipsPerronArch
Executes Phillips-Perron test to assess the stationarity of time series data in each ML model feature.
PolarityAndSubjectivity
Analyzes the polarity and subjectivity of text data within a dataset.
Punctuations
Analyzes and visualizes the frequency distribution of punctuation usage in a given text dataset.
RollingStatsPlot
This test evaluates the stationarity of time series data by plotting its rolling mean and standard deviation.
ScatterPlot
Creates a scatter plot matrix to visually analyze feature relationships, patterns, and outliers in a dataset.
SeasonalDecompose
Decomposes dataset features into observed, trend, seasonal, and residual components to identify patterns and validate dataset.
Sentiment
Analyzes the sentiment of text data within a dataset using the VADER sentiment analysis tool.
Skewness
Evaluates the skewness of numerical data in a machine learning model and checks if it falls below a set maximum threshold.
SpreadPlot
Visualizes the spread relationship between pairs of time-series variables in a dataset, thereby aiding in identification of potential correlations.
StopWords
Evaluates and visualizes the frequency of English stop words in a text dataset against a defined threshold.
TabularCategoricalBarPlots
Generates and visualizes bar plots for each category in categorical features to evaluate dataset’s composition.
TabularDateTimeHistograms
Generates histograms to provide graphical insight into the distribution of time intervals in model’s datetime data.
TabularDescriptionTables
Summarizes key descriptive statistics for numerical, categorical, and datetime variables in a dataset.
TabularNumericalHistograms
Generates histograms for each numerical feature in a dataset to provide visual insights into data distribution and detect potential issues.
TargetRateBarPlots
Generates bar plots visualizing the default rates of categorical features for a classification machine learning model.
TextDescription
Performs comprehensive textual analysis on a dataset using NLTK, evaluating various parameters and generating visualizations.
TimeSeriesDescription
Generates a detailed analysis for the provided time series dataset.
TimeSeriesDescriptiveStatistics
Generates a detailed table of descriptive statistics for the provided time series dataset.
TimeSeriesFrequency
Evaluates consistency of time series data frequency and generates a frequency plot.
TimeSeriesHistogram
Visualizes distribution of time-series data using histograms and Kernel Density Estimation (KDE) lines.
TimeSeriesLinePlot
Generates and analyses time-series data through line plots revealing trends, patterns, anomalies over time.
TimeSeriesMissingValues
Validates time-series data quality by confirming the count of missing values is below a certain threshold.
TimeSeriesOutliers
Identifies and visualizes outliers in time-series data using z-score method.
TooManyZeroValues
Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold percentage.
Toxicity
Analyzes the toxicity of text data within a dataset using a pre-trained toxicity model.
UniqueRows
Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.
WOEBinPlots
Generates visualizations of Weight of Evidence (WoE) and Information Value (IV) for understanding predictive power of categorical variables in a data set.
WOEBinTable
Calculates and assesses the Weight of Evidence (WoE) and Information Value (IV) of each feature in a ML model.
ZivotAndrewsArch
Evaluates the order of integration and stationarity of time series data using Zivot-Andrews unit root test.
No matching items
AdjustedMutualInformation
Evaluates clustering model performance by measuring mutual information between true and predicted labels, adjusting for chance.
AdjustedRandIndex
Measures the similarity between two data clusters using the Adjusted Rand Index (ARI) metric in clustering machine learning models.
AnswerCorrectness
Evaluates the correctness of answers in a dataset with respect to the provided ground truths and visualizes the results in a histogram.
AnswerRelevance
Assesses how pertinent the generated answer is to the given prompt.
AnswerSimilarity
Calculates the semantic similarity between generated answers and ground truths
AspectCritique
Evaluates generations against the following aspects: harmfulness, maliciousness, coherence, correctness, and conciseness.
AutoARIMA
Evaluates ARIMA models for time-series forecasting, ranking them using Bayesian and Akaike Information Criteria.
BertScore
Evaluates the quality of machine-generated text using BERTScore metrics and visualizes the results through histograms and bar charts, alongside compiling a comprehensive table of descriptive statistics for each BERTScore metric.
BleuScore
Evaluates the quality of machine-generated text using BLEU metrics and visualizes the results through histograms and bar charts, alongside compiling a comprehensive table of descriptive statistics for BLEU scores.
BoxPierce
Detects autocorrelation in time-series data through the Box-Pierce test to validate model performance.
ClassifierPerformance
Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.
ClusterCosineSimilarity
Measures the intra-cluster similarity of a clustering model using cosine similarity.
ClusterDistribution
Assesses the distribution of text embeddings across clusters produced by a model using KMeans clustering.
ClusterPerformance
Evaluates and compares a clustering model’s performance on training and testing datasets using multiple defined metrics.
ClusterPerformanceMetrics
Evaluates the performance of clustering machine learning models using multiple established metrics.
ClusterSizeDistribution
Compares and visualizes the distribution of cluster sizes in model predictions and actual data for assessing clustering model performance.
CompletenessScore
Evaluates a clustering model’s capacity to categorize instances from a single class into the same cluster.
ConfusionMatrix
Evaluates and visually represents the classification ML model’s predictive performance using a Confusion Matrix heatmap.
ContextEntityRecall
Evaluates the context entity recall for dataset entries and visualizes the results.
ContextPrecision
Context Precision is a metric that evaluates whether all of the ground-truth relevant items present in the contexts are ranked higher or not. Ideally all the relevant chunks must appear at the top ranks. This metric is computed using the question, g…
ContextRecall
Context recall measures the extent to which the retrieved context aligns with the annotated answer, treated as the ground truth. It is computed based on the ground truth and the retrieved context, and the values range between 0 and 1, with higher…
ContextualRecall
Evaluates a Natural Language Generation model’s ability to generate contextually relevant and factually correct text, visualizing the results through histograms and bar charts, alongside compiling a comprehensive table of descriptive statistics for…
CosineSimilarityComparison
Computes pairwise cosine similarities between model embeddings and visualizes the results through bar charts, alongside compiling a comprehensive table of descriptive statistics for each model pair.
CosineSimilarityDistribution
Assesses the similarity between predicted text embeddings from a model using a Cosine Similarity distribution histogram.
CosineSimilarityHeatmap
Generates an interactive heatmap to visualize the cosine similarities among embeddings derived from a given model.
CumulativePredictionProbabilities
Visualizes cumulative probabilities of positive and negative classes for both training and testing in logistic regression models.
DescriptiveAnalytics
Evaluates statistical properties of text embeddings in an ML model via mean, median, and standard deviation histograms.
DurbinWatsonTest
Assesses autocorrelation in time series data features using the Durbin-Watson statistic.
EmbeddingsVisualization2D
Visualizes 2D representation of text embeddings generated by a model using t-SNE technique.
EuclideanDistanceComparison
Computes pairwise Euclidean distances between model embeddings and visualizes the results through bar charts, alongside compiling a comprehensive table of descriptive statistics for each model pair.
EuclideanDistanceHeatmap
Generates an interactive heatmap to visualize the Euclidean distances among embeddings derived from a given model.
Faithfulness
Evaluates the faithfulness of the generated answers with respect to retrieved contexts.
FeatureImportanceComparison
Compare feature importance scores for each model and generate a summary table with the top important features.
FeaturesAUC
Evaluates the discriminatory power of each individual feature within a binary classification model by calculating the Area Under the Curve (AUC) for each feature separately.
FowlkesMallowsScore
Evaluates the similarity between predicted and actual cluster assignments in a model using the Fowlkes-Mallows score.
GINITable
Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets.
HomogeneityScore
Assesses clustering homogeneity by comparing true and predicted labels, scoring from 0 (heterogeneous) to 1 (homogeneous).
HyperParametersTuning
Exerts exhaustive grid search to identify optimal hyperparameters for the model, improving performance.
JarqueBera
Assesses normality of dataset features in an ML model using the Jarque-Bera test.
KMeansClustersOptimization
Optimizes the number of clusters in K-means models using Elbow and Silhouette methods.
KolmogorovSmirnov
Executes a feature-wise Kolmogorov-Smirnov test to evaluate alignment with normal distribution in datasets.
LJungBox
Assesses autocorrelations in dataset features by performing a Ljung-Box test on each feature.
Lilliefors
Assesses the normality of feature distributions in an ML model’s training dataset using the Lilliefors test.
MeteorScore
Computes and visualizes the METEOR score for each text generation instance, assessing translation quality.
MinimumAccuracy
Checks if the model’s prediction accuracy meets or surpasses a specified threshold.
MinimumF1Score
Evaluates if the model’s F1 score on the validation set meets a predefined minimum threshold.
MinimumROCAUCScore
Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.
ModelMetadata
Extracts and summarizes critical metadata from a machine learning model instance for comprehensive analysis.
ModelMetadataComparison
Compare metadata of different models and generate a summary table with the results.
ModelPredictionResiduals
Plot the residuals and histograms for each model, and generate a summary table with the Kolmogorov-Smirnov normality test results.
ModelsPerformanceComparison
Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy, precision, recall, and F1 score.
OverfitDiagnosis
Identify overfit regions in a model’s predictions.
PCAComponentsPairwisePlots
Generates scatter plots for pairwise combinations of principal component analysis (PCA) components of model embeddings.
PermutationFeatureImportance
Assesses the significance of each feature in a model by evaluating the impact on model performance when feature values are randomly rearranged.
PopulationStabilityIndex
Evaluates the Population Stability Index (PSI) to quantify the stability of an ML model’s predictions across different datasets.
PrecisionRecallCurve
Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.
PredictionProbabilitiesHistogram
Generates and visualizes histograms of the Probability of Default predictions for both positive and negative classes in training and testing datasets.
ROCCurve
Evaluates binary classification model performance by generating and plotting the Receiver Operating Characteristic (ROC) curve and calculating the Area Under Curve (AUC) score.
RegardScore
Computes and visualizes the regard score for each text instance, assessing sentiment and potential biases.
RegressionCoeffsPlot
Visualizes regression coefficients with 95% confidence intervals to assess predictor variables’ impact on response variable.
RegressionErrors
Purpose: This metric is used to measure the performance of a regression model. It gauges the model’s accuracy by computing several error metrics such as Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean…
RegressionErrorsComparison
Compare regression error metrics for each model and generate a summary table with the results.
RegressionFeatureSignificance
Assesses and visualizes the statistical significance of features in a set of regression models.
RegressionModelForecastPlot
Generates plots to visually compare the forecasted outcomes of one or more regression models against actual observed values over a specified date range.
RegressionModelForecastPlotLevels
Compares and visualizes forecasted and actual values of regression models on both raw and transformed datasets.
RegressionModelSensitivityPlot
Tests the sensitivity of a regression model to variations in independent variables by applying shocks and visualizing the effects.
RegressionModelSummary
Evaluates regression model performance using metrics including R-Squared, Adjusted R-Squared, MSE, and RMSE.
RegressionModelsCoeffs
Compares feature importance by evaluating and contrasting coefficients of different regression models.
RegressionModelsPerformanceComparison
Compares and evaluates the performance of multiple regression models using five different metrics: MAE, MSE, RMSE, MAPE, and MBD.
RegressionPermutationFeatureImportance
Assesses the significance of each feature in a model by evaluating the impact on model performance when feature values are randomly rearranged. Specifically designed for use with statsmodels, this metric offers insight into the importance of…
RegressionR2Square
Purpose: The purpose of the RegressionR2Square Metric test is to measure the overall goodness-of-fit of a regression model. Specifically, this Python-based test evaluates the R-squared (R2) and Adjusted R-squared (Adj R2) scores: two statistical…
RegressionR2SquareComparison
Compare R-Squared and Adjusted R-Squared values for each model and generate a summary table with the results.
RegressionResidualsPlot
Evaluates regression model performance using residual distribution and actual vs. predicted plots.
RobustnessDiagnosis
Evaluate the robustness of a machine learning model to noise
RougeScore
Evaluates the quality of machine-generated text using ROUGE metrics and visualizes the results through histograms and bar charts, alongside compiling a comprehensive table of descriptive statistics for each ROUGE metric.
RunsTest
Executes Runs Test on ML model to detect non-random patterns in output data sequence.
SHAPGlobalImportance
Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.
ScorecardHistogram
Creates histograms of credit scores, from both default and non-default instances, generated by a credit-risk model.
ShapiroWilk
Evaluates feature-wise normality of training data using the Shapiro-Wilk test.
SilhouettePlot
Calculates and visualizes Silhouette Score, assessing degree of data point suitability to its cluster in ML models.
StabilityAnalysis
Base class for embeddings stability analysis tests
StabilityAnalysisKeyword
Evaluate robustness of embeddings models to keyword swaps on the test dataset
StabilityAnalysisRandomNoise
Evaluate robustness of embeddings models to random noise introduced by using a probability parameter to choose random locations in the text to apply random perturbations. These perturbations include:
StabilityAnalysisSynonyms
Evaluates the stability of text embeddings models when words in test data are replaced by their synonyms randomly.
StabilityAnalysisTranslation
Evaluate robustness of embeddings models to noise introduced by translating the original text to another language and back.
TSNEComponentsPairwisePlots
Plots individual scatter plots for pairwise combinations of t-SNE components of embeddings.
TimeSeriesPredictionWithCI
Plot actual vs predicted values for a time series with confidence intervals and compute breaches.
TimeSeriesPredictionsPlot
Plot actual vs predicted values for time series data and generate a visual comparison for each model.
TimeSeriesR2SquareBySegments
Plot R-Squared values for each model over specified time segments and generate a bar chart with the results.
TokenDisparity
Evaluates the token disparity between reference and generated texts, visualizing the results through histograms and bar charts, alongside compiling a comprehensive table of descriptive statistics for token counts.
ToxicityScore
Computes and visualizes the toxicity score for input text, true text, and predicted text, assessing content quality and potential risk.
TrainingTestDegradation
Tests if model performance degradation between training and test datasets exceeds a predefined threshold.
VMeasure
Evaluates homogeneity and completeness of a clustering model using the V Measure Score.
WeakspotsDiagnosis
Identifies and visualizes weak spots in a machine learning model’s performance across various sections of the feature space.
No matching items