One of the most important practices for Machine Learning projects is to strictly separate data, code (incl. model architecture, training code, API etc.) and the compute environment.
Enforcing such a separation enable to:
This separation is the first of the “three levels of ML software” (data, model, code) described by ml-ops.org, and its absence is one of the main sources of the hidden technical debt of ML systems (Sculley et al. 2015). In the official statistics context, it also directly supports the FAIR principles: data should be findable, accessible, interoperable and reusable (Wilkinson et al. 2016).
In that spirit, data should absolutely lie in a stable storage - preferably cloud-based, far from the messy environment of code and compute. If your code or your computer crashes, your data should be safe.
Beyond where data is stored, how it is stored matters just as much. For text classification pipelines, columnar formats such as Parquet are generally recommended over raw CSV or JSON files. Parquet is compressed, schema-aware, and optimized for analytical workloads, which means faster reads, lower storage costs, and more predictable behavior across tools. In practice, this allows you to load only the columns you need (e.g. text and labels), enforce consistent data types, and efficiently handle large datasets during training, validation, and monitoring. Parquet is also natively supported by most modern data processing frameworks (Spark, Pandas, Polars, DuckDB), making it a robust and interoperable choice for production-grade ML pipelines (The Apache Software Foundation, n.d.).
Any preprocessing step should be clearly documented, with a fully reproducible script.
Data errors are the most frequent cause of silent failures in ML pipelines: a renamed column, a new encoding, a label set that changed. The model keeps training and predicting, but on the wrong data. A good practice is therefore to validate the data automatically before every training run, against an explicit schema (expected columns, types, allowed values for codes and categorical variables, share of missing or empty texts, text length ranges) (Breck et al. 2019; Schelter et al. 2018). The same checks can be reused at inference time on the incoming data (see the Monitoring chapter).
For automatic coding, a few checks are especially useful:
Finally, each dataset should come with a short documentation sheet describing its origin, collection process, time span, version of the classification, known biases and labelling procedure. Datasheets for Datasets (Gebru et al. 2021) provide a ready-to-use template. In NSIs, much of this information already exists in survey metadata; it just has to be attached to the training data.
At Insee, we extensively use cloud-based S3 data storage solution, based on the open-source MinIO framework - be it on the SSP Cloud (public Onyxia instance for collaborative, non-sensitive use cases) or LS3 (the internal Onyxia instance for secured, offline projects).
Access your data from the storage is then very easy, from any compute environment (think of it as a Google Drive share link).
For instance in Python:
# Connecting to the storage via a filesystem
fs = S3FileSystem(
client_kwargs={"endpoint_url": f"https://{os.environ['AWS_S3_ENDPOINT']}"},
key=os.environ["AWS_ACCESS_KEY_ID"],
secret=os.environ["AWS_SECRET_ACCESS_KEY"],
)
# Loading a dataframe is very easy !
df_train = pd.read_parquet("df_train.parquet", filesystem=fs)
# Saving too
df_train.to_parquet("df_train.parquet", filesystem=fs)In order to ensure that the data is stored and used efficiently we make use of the Hadoop Distributed File System (HDFS) and parquet for data partitioning. HDFS is especially made for handling a large amount of data. For programming and data processing, we use Cloudera AI Workbench with PySpark, which allows us to efficiently work on the data. We store our data in the Parquet format, which is ideal for big data and in addition, to make it easier for users to handle and cross-check the data, we use Hue (Hadoop User Experience), an open-source SQL-based cloud editor. For rights management, we use Ranger, which provides a big variety of access control to ensure data security.
The data cleaning in our project is quite straightforward, since the text entries contain short texts (mostly keywords) instead of long ones. First, data augmentation is performed by adding new text entries (e.g. text like “groceries” or “beverages”) to the dataset, adding multiple newly generated text values to each household to enrich the data. Adding a variety of new textual entries helps the model to generelize better. Secondly, we clean the data by removing punctation and handling missing values.
Training data is stored in a centralized network drive that serves as the stable storage layer for the project. By separating data storage from the local development and compute environments, the dataset remains accessible, backed up, and protected against accidental loss caused by local machine failures or code issues. Subject matter experts contribute new labeled data quarterly, typically consisting of 300–500 additional entries per update cycle.
Just as code (see chapter 2), a good practice is to version the dataset in order to:
Dedicated tools exist for this, such as DVC (Iterative, n.d.) or the dataset tracking of MLflow (Zaharia et al. 2018). A simple convention can also work: immutable, dated files in object storage (e.g. train_2025-Q3.parquet) plus a latest pointer, as long as the exact version is logged with each training run. If the dataset is re-split at each version, store the observation IDs of each split as well (see the Statistics Austria example in the Model chapter).
We use the Datasets component of the framework MLFLow to seamlessly version datasets and attach each model to its training set.
import mlflow
import pandas as pd
train_df = pd.read_parquet("s3://my-bucket/train.parquet")
# Wrap the DataFrame as an MLflow Dataset, pointing to its source
dataset = mlflow.data.from_pandas(
train_df,
source="s3://my-bucket/train.parquet",
name="complaints-train",
targets="label",
)
with mlflow.start_run():
# Log the dataset — MLflow records its hash, schema, and source URI
mlflow.log_input(dataset, context="training")
# ... train model, log metrics, log model artifact ...Any run logged this way will display its linked dataset in the MLflow UI, making it straightforward to answer “which data was this model trained on?”.