arTIco

Learning Expert Ratings for Digital Twin Validation

A research ML framework for classifying expert assessments from multichannel crash-simulation signals and tabular context.

Industrial AI / Time-Series Machine Learning / Model Validation

A research ML framework for learning expert assessments from multichannel crash-simulation signals to support digital-twin validation.

01 // MULTICHANNEL SIGNALSInput Signal TensorsCrash & simulation sensor time series

Sig A Sig B Sig C

→

02 // PREPROCESSINGScaling & AlignmentResampling & feature integration

Tabular Context Features

Mass·Velocity·Angle

03 // PIPELINESTime-Series ClassifiersCandidate ML pipeline wrappers

1D CNNLSTM-FCNInceptionTimeRocket

→

04 // EVALUATIONEngineering EvaluationDomain-weighted scoring & 95% CI

■□□□■□□□■

95% Bootstrap CI

Simplified architecture using synthetic signals. No project dataset or internal experiment result is shown.

Overview

Digital-twin validation in crash simulation commonly relies on correlation criteria and comparisons between individual measured and simulated signals. These methods are useful, but a limited set of signals may not capture the full physical behaviour of a complex system.

arTIco explored whether machine-learning models could learn patterns from broader multichannel signal data and expert assessments. The goal was not to replace engineering validation, but to investigate a more objective, reproducible, and scalable way to support it.

The problem

Expert assessments contain engineering context that fixed correlation factors may not fully capture. However, expert labels can also be subjective, difficult to reproduce, and expensive to apply across large multichannel datasets.

The technical challenge was therefore broader than training a classifier.

Why this was difficult

Multichannel signal integration

Combine multichannel time-series data with tabular context.

Model family comparison

Compare different model families through a consistent interface.

Unseen sample evaluation

Evaluate predictions on samples unseen during training.

Engineering relevance

Distinguish minor rating differences from engineering-relevant mistakes.

Experiment reproducibility

Preserve enough experiment context to reproduce and compare results.

My contribution

I contributed to the modular Python framework used to transform multichannel signal and tabular data into reproducible classification experiments.

My work covered selected parts of the data-to-evaluation workflow, including reusable pipeline interfaces, cross-validated model evaluation, domain-weighted scoring, experiment configuration, and hyperparameter studies.

Framework contribution

Contributed to reusable interfaces separating data preparation, model implementation, evaluation, and experimentation.

Evaluation contribution

Implemented or extended evaluation logic for unseen-fold predictions, engineering-specific scoring, and uncertainty reporting.

Experiment contribution

Supported configurable experiments and systematic comparison of candidate model and preprocessing configurations.

System workflow

01Prepare signal data
Load multichannel time-series data, additional tabular features, and expert-rating targets.
02Standardize inputs
Apply reusable preprocessing for scaling, resampling, and feature integration.
03Configure candidate pipelines
Run different classifier and preprocessing configurations through a common pipeline interface.
04Evaluate on unseen folds
Generate predictions using stratified cross-validation so each evaluation sample is predicted by a model that did not train on it.
05Measure engineering relevance
Compare generic classification metrics with a domain-weighted score that assigns different penalties to different rating errors.
06Track and compare experiments
Store parameters, results, logs, environment information, and tuning studies for reproducible comparison.

Technical decisions

Problem: Training accuracy or a single hold-out result could give a misleading view of performance, especially when rating classes are imbalanced.
Decision: Use stratified five-fold cross-validation and assemble the predictions made on each unseen fold.
Result: Every sample contributes to the final test evaluation without being evaluated by a model that trained on that sample.

Problem: A single score can hide how strongly the result depends on the evaluated samples.
Decision: Bootstrap the combined unseen predictions and report a median together with a confidence range.
Result: Model comparisons include an estimate of result stability instead of relying only on one reported value.

Problem: Not every classification error has the same engineering meaning. Predicting a neighbouring rating class is different from confusing ratings at opposite ends of the scale.
Decision: Use a custom score derived from a normalized confusion matrix and a domain-defined error-weight matrix.
Result: The evaluation reflects the severity of rating mistakes rather than counting every incorrect class equally.

Problem: Model results are difficult to compare when preprocessing, package versions, and hyperparameters are not recorded consistently.
Decision: Store experiment parameters, input references, output locations, timestamps, and Python package versions in a standardized experiment structure.
Result: Previous experiments can be inspected and compared without relying on undocumented notebook state.

Evaluation rigor

Methodology Note: The parameters below represent evaluation framework setup settings (cross-validation folds, bootstrap resample count, confidence intervals), not model performance or accuracy figures.

Evaluation setup ensuring no model evaluates samples it trained on5-foldStratified cross-validation setup

Resampling parameter for empirical confidence interval calculation1,000Bootstrap resample count

Statistical confidence level for evaluated metric distributions95%Bootstrap confidence range

Custom weighted rating error matrix penalty calculationDomain-weightedEngineering evaluation metric

Automated hyperparameter optimization frameworkOptunaConfigurable hyperparameter search framework

Methodology note: The parameters above describe evaluation framework setup settings (5-fold CV, 1,000 bootstrap resamples, 95% CI bounds), not model performance or accuracy figures.

What the framework enabled

Consistent model comparison

Different model and preprocessing approaches could be evaluated through the same data, pipeline, and scoring interfaces.

Domain-aware evaluation

Prediction errors were assessed according to their engineering significance rather than generic accuracy alone.

Uncertainty-aware reporting

Bootstrap confidence ranges helped distinguish stable improvements from sample-dependent variation.

Reproducible experiments

Parameters, results, logs, and environment metadata were stored as part of each experiment.

Research context

arTIco was a public research project investigating AI-supported validation of digital twins. The public framework and workshop material document the project motivation, software structure, and approach to combining multivariate time-series data with expert knowledge.

Lessons & takeaways

A generic classification score is not always sufficient for engineering decision support.
Evaluation design can be as important as model architecture when expert-labelled data is limited.
Reusable experiment interfaces make it easier to compare model families without rebuilding the workflow for each approach.
Expert knowledge can be represented not only in labels, but also in the way prediction errors are weighted and interpreted.

Links

Next project05Broken Signal DetectionIndustrial AI / Binary Classification