Training Materials on Cross-Validation

Cross-validation is not a one-size-fits-all technique — it should always be adapted to the structure and nature of your data.

Why Cross-Validation Matters

The main goal of cross-validation is to obtain a reliable estimate of a model’s generalization performance, i.e., how well it will perform on unseen data.

A single train/test split is often not enough because it can lead to high variance in performance estimates. Depending on how the data is split, the model may appear artificially strong or weak.

To address this issue, we use resampling strategies, which repeatedly split the data in different ways and average the results to get a more stable evaluation.

Key Insight

The way you split your data directly influences the perceived performance of your model.

A poorly chosen validation strategy can make a model look better (or worse) than it truly is in practice, leading to misleading conclusions.

library(rsample)
library(dplyr)


Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

library(arrow)


Attaching package: 'arrow'

The following object is masked from 'package:utils':

    timestamp

data_cv0 <-  read_parquet("data/data_synth_cv0.parquet", as_data_frame = TRUE)

str(data_cv0)

tibble [3,496 × 42] (S3: tbl_df/tbl/data.frame)
 $ ANO         : num [1:3496] 2026 2026 2026 2026 2026 ...
 $ DD_CC_FF_COD: chr [1:3496] "140120" "010312" "050807" "310307" ...
 $ N2_COD_1    : chr [1:3496] "16" "16" "16" "30" ...
 $ ENUT        : chr [1:3496] "2" "2" "2" "7" ...
 $ L_NUT       : chr [1:3496] "Centro" "Centro" "Centro" "Madeira" ...
 $ NPS         : num [1:3496] 3 1 2 1 7 9 10 10 14 12 ...
 $ INQ         : chr [1:3496] "1" "1" "1" "1" ...
 $ L_EPS       : chr [1:3496] "1 a 19" "1 a 19" "1 a 19" "1 a 19" ...
 $ VVN         : num [1:3496] 247262 49222 49404 32239 267866 ...
 $ EVN         : chr [1:3496] "2" "1" "1" "1" ...
 $ CAE_BC      : chr [1:3496] "41200" "43340" "43330" "43390" ...
 $ ECAE        : chr [1:3496] "4120" "4334" "4333" "4339" ...
 $ ESTRATO     : chr [1:3496] "41200.B.2" "43340.B.1" "43330.B.1" "43390.B.1" ...
 $ AMH         : num [1:3496] 171 18 15 18 218 180 114 12 182 114 ...
 $ UNI         : num [1:3496] 14287 477 466 516 218 ...
 $ IMP         : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ I187701     : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ I20540      : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ I20550      : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ QF185001    : num [1:3496] 108056 52986 54000 1 278644 ...
 $ QF185002    : num [1:3496] 0 0 0 0 105000 ...
 $ QF185003    : num [1:3496] 108056 52986 54000 0 173644 ...
 $ QF185004    : num [1:3496] 0 0 0 1 0 ...
 $ QF185101    : num [1:3496] 108056 52986 54000 0 278644 ...
 $ QF185501    : num [1:3496] 0 0 0 1 0 ...
 $ QF186301    : num [1:3496] 0 0 0 0 0 ...
 $ QF186302    : num [1:3496] 0 0 0 0 0 ...
 $ QF186303    : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ QF186304    : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ QF186401    : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ QF187001    : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ QF187301    : num [1:3496] 0 0 0 0 0 ...
 $ QF187401    : num [1:3496] 0 0 0 0 0 ...
 $ QF187701    : num [1:3496] 108056 52986 54000 1 278644 ...
 $ QF187702    : num [1:3496] 0 0 0 0 105000 ...
 $ QF187703    : num [1:3496] 108056 52986 54000 0 173644 ...
 $ QF187704    : num [1:3496] 0 0 0 1 0 ...
 $ QF20540     : num [1:3496] 0 0 0 0 0 0 0 0 0 0 ...
 $ QF20550     : num [1:3496] 108056 3255 1 1 10604 ...
 $ source      : chr [1:3496] "B" "B" "B" "B" ...
 $ ID          : num [1:3496] 1 2 3 4 5 6 7 8 9 10 ...
 $ unit        : int [1:3496] 2 2 2 2 2 2 2 2 2 2 ...

Construction Enterprises Survey Dataset (Synthetic File Description)

Overview

This dataset is a synthetic file designed to simulate responses from the Survey on Construction Enterprises. The survey collects key information about the structure and performance of enterprises within the construction sector.

The resulting data are used to define stratification variables, which form the basis for estimation procedures that extend beyond the sampled units. The reliability of these estimation methods depends critically on the completeness and quality of all variables involved in the stratification process.

However, the dataset reflects real-world challenges such as non-response and partial non-response, which affect a subset of key variables. These issues introduce challenges related to data quality, accuracy, and overall reliability. To address this, the use case in WP9 focuses on predicting the variables required for stratification.

Key Variables

Time and Identifiers

ANO: Reference year of the observation
ID: Anonymous identifier of the enterprise
UNIT: Anonymous identifier of the enterprise unit (used when an enterprise has multiple units)

Enterprise Characteristics

Variables ranging from DD_CC_FF_COD to UNI describe structural and operational characteristics of each enterprise.

Imputation Indicator

IMP: Indicates the origin of the data value:
- 0 → Real (observed) response
- 1 or 2 → Imputed values based on year n-1 or n-2
- 3 → Other types of imputation

Integrated Administrative Variables

Variables prefixed with I* represent integrated administrative data sources.

Survey Variables

Variables prefixed with QF* correspond to survey questions.

Target Variable

QF185001: The target variable of interest in the WP9 use case

Notes

This dataset is specifically designed to support methodological development in handling missing data and improving prediction of stratification variables. It reflects both observed and imputed data conditions commonly encountered in large-scale enterprise surveys.