Skip to content

Latest commit

 

History

History
197 lines (144 loc) · 10.7 KB

README.md

File metadata and controls

197 lines (144 loc) · 10.7 KB
Raw

CleanIMP: A comprehensive Framework to Evaluate the Impact of Imputation on Downstream Tasks

CleanIMP is a unified framework designed to extensively evaluate the downstream effects of 10 advanced and five basic imputation algorithms for time series data. It evaluates two downstream tasks: classification and forecasting using 89 datasets, 27 downstream techniques, and various contamination scenarios. Technical details can be found in our paper: Does Cleaning Time Series Really Matter? An Evaluation of the Impact of Imputation on Downstream Tasks (under review at PVLDB'25) .

Prerequisites | Build | Configuration | Execution | Analysis | Contributors |

Prerequisites

  • Ubuntu 22 or Ubuntu 24 (including Ubuntu derivatives, e.g., Xubuntu) or the same distribution under WSL.
  • Clone this repository

Build

  • Build the Testing Framework using the installation script located in the root folder
    $ sh setup.sh

Benchmark Configuration

  • Datasets: The datasets will be downloaded by executing the setup script. They are task-dependent and can be found in WorkDir/_RawDataStorage in resp. folders UniClass or Forecasting.

  • Missing Patterns: Scenarios control the patterns of contamination performed by the benchmark. The available options are listed in the table below.

Scenario Task Description
miss_percNN Classification contaminate NN% of all time series and vary the size of the missing block from 10% to 80% of the length of the series; NN ∈ {10, 20, ... 50}
mc_NN Classification vary the number of contaminated series from 10% to 100%, each affected time series has a missing block of NN% of the length of the series; NN ∈ {10, 20, ... 50}
miss_perc_rev Forecasting contaminate a single time series and vary the size of the missing block from 10% to 80% of the length of the series
mc_rev Forecasting vary the number of contaminated series from 10% to 100%, each affected time series has a missing block of 10% of the length of the series
  • Imputation Algorithms: The list of imputation algorithms and their parameters is provided below. The parameters can be updated in the config file by modifying algorithm:p00 where p is the name of the parameter and 00 is the value. For example, IMM with the neighborhood size 5 is IIM:n5.
Algorithms param default param. descr. range
CDRec k 3 truncation [1, 10]
SVDImp k 3 truncation [1, 10]
SoftImp k 3 truncation [1, 10]
STMVL n/a
DynaMMo k 3 hidden var. [1, 10]
IIM n 3 neighbors [1, 100]
GROUSE k 3 truncation [1, 10]
SVT n/a
ROSL k hidden var. [1, 10]
TRMF k 3 truncation [1, 10]
-------- -------- -------- -------- --------
MeanImp n/a
ZeroImp n/a
1NNImp n/a
LinearImp n/a
knnimp n 3 neighbors [1, 100]
  • Downstream Algorithms: The list of per-task downstream algorithms is provided below (see sktime and darts for documentation).
Classification Forecasting
arsenal sf-arima*
stc fbprophet
weasel hw-add
cboss hw-mul
tsf bats
catch22 darts-nbeats
svc sf-ets
knn ltsf*
itde croston
shapedtw darts-lstm*
signature darts-deepar*
tsfresh darts-transformer*
cif
proxstump
  • Reference: This parameter controls whether the algorithms are ran on uncontaminated data during the evaluation (downstream) experiment. Available options are listed in the table below.
Reference Description
Both (default) Downstream algorithms are executed on contaminated and uncontaminated data
NoReference Downstream algorithms are executed on contaminated data only
ReferenceOnly Downstream algorithms are executed on uncontaminated data only
ReferenceReplace Downstream algorithms are executed on uncontaminated data, only overwriting the existing results
  • Notes:
    • Downstream algorithms marked with * are already parallelized. If they are included in the experiment - parallelization has to be disabled on the level on the benchmark by setting the parameter ParallelizeDownstream to False.
    • Set the values in the config parameters PerformContamination and PerformEvaluation to True to enable a specific type of experiment. The contamination results (upstream) are required to run evaluation experiments (downstream).
    • Standard benchmark behavior is to overwrite existing results in case of overlap with cached results for contaminated data and to not overwrite the results for uncontaminated data.

Analysis

  • Once the experiment is executed, the analysis routines can be invoked. The general pattern for the command is
    $ dotnet run config_file.cfg analysis {job_type}:{metric1},{metric2},{metric3}[:aggregation_level]
  • Job type: The table below gives the list of available options with the descriptions for job types and lists the prerequisites to run them (which experiments have to already been executed). Jobs are task-independent.
Job type Requirements Description
simpledump None Displays basic information about the dataset
datachar None Displays advanced information (features) about the dataset
upstream Contamination Displays the upstream experiment results
downstream Evaluation Displays the downstream experiment results
bydata Evaluation Displays the downstream experiment results with a different aggregation mechanism
reference Evaluation* Displays the evaluation results of the downstream algorithms on uncontaminated data; * - No experiments on contaminated data are required to run this job, see Reference parameter
  • Metrics: A list of metrics to be used in the evaluation. Metrics are task-dependent. Note: forecasting metrics need to have forecasting horizon appended to them. Supported values are: (12, 24, 48, 60). E.g. for smape metric with the horizon of 24 - the argument is smape24.
Upstream Description
rmse Root Mean Squared Error
mae Mean Absolute Error
pearson Pearson Correlation Coefficient
spearman Spearman Correlation Coefficient
MI-k{N} Kraskov Mutual Information. {N} - is the neighborhood size. Supported values are 3 and 5.
Classification Description
accuracy Accuracy
precision Precision
recall Recall
f1 F1-Measure (F-Score)
mcc Matthew's Correlation Coefficient (also known as Phi coefficient)
Forecasting Description
smape Symmetric Mean Absolute Percentage Error
rmse Root Mean Squared Error
pearson Pearson Correlation Coefficient
  • Aggregation level: The parameter is optional and only applicable to jobs upstream, downstream and bydata. Aggregation levels are task-independent.
Level Description
all (default) Shows a single value per contamination pattern by averaging all contamination levels
none Shows all levels of contamination as defined by a contamination pattern
low Shows a single value per contamination pattern by averaging the lower half of contaminations levels
high Shows a single value per contamination pattern by averaging the upper half of contaminations levels
- Other more technical aggregations options are available too, refere to the source file Testing/TestRoutinesAnalysis.cs for more information

Execution

  • To produce a curated set of results, run the following command (takes ~ 5 days on a server-grade CPU):
    $ sh full_results.sh

The output will be stored in the Results/ folder, which will be created in the root folder.

Execution (fine-grained)

  • This section gives some examples on how to produce different analysis results.

  • The existing config files for classification config_uniclass_custom.cfg and config_forecast_custom.cfg create new customized experiment runs. Those files contain lists of available options so they can be tweaked to produce a desired experiment. Their default configuration is geared towards a smaller experiment that can be completed in a reasonable timespan.

  • To produce the classification and resp. forecasting experiment runs, run the following commands:

    $ cd TestFramework/
    $ dotnet run ../configs/config_uniclass_custom.cfg
    $ dotnet run ../configs/config_forecast_custom.cfg
  • Running the configuration file will execute the experiment specified there and cache the upstream/downstream result data.
  • To produce the analysis of the runs, a parametrized analysis argument needs to be specified after the name of the config file. For example, the following command produces the analysis for the classification run using rmse as an upstream metric:
    $ dotnet run ../configs/config_uniclass_custom.cfg analysis upstream:rmse
  • To produce results for multiple metrics, or (in this example) for forecasting task the same metrics (smape) but different forecasting horizions (12-36), and aggregate them by dataset one can run the following command:
   $ dotnet run ../configs/config_forecast_custom.cfg analysis downstream:smape12,smape24,smape36:bydata

Contributors