Data

One of the most important practices for Machine Learning projects is to strictly separate data, code (incl. model architecture, training code, API etc.) and the compute environment.

Enforcing such a separation enable to:

Data storage

In that spirit, data should absolutely lie in a stable storage - preferably cloud-based, far from the messy environment of code and compute. If your code or your computer crashes, your data should be safe.

Beyond where data is stored, how it is stored matters just as much. For text classification pipelines, columnar formats such as Parquet are generally recommended over raw CSV or JSON files. Parquet is compressed, schema-aware, and optimized for analytical workloads, which means faster reads, lower storage costs, and more predictable behavior across tools. In practice, this allows you to load only the columns you need (e.g. text and labels), enforce consistent data types, and efficiently handle large datasets during training, validation, and monitoring. Parquet is also natively supported by most modern data processing frameworks (Spark, Pandas, Polars, DuckDB), making it a robust and interoperable choice for production-grade ML pipelines.

Any preprocessing step should be clearly documented, with a fully reproducible script.

Insee: S3-based storage

At Insee, we extensively use cloud-based S3 data storage solution, based on the open-source MinIO framework - be it on the SSP Cloud (public Onyxia instance for collaborative, non-sensitive use cases) or LS3 (the internal Onyxia instance for secured, offline projects).

Access your data from the storage is then very easy, from any compute environment (think of it as a Google Drive share link).

For instance in Python:

Code 
# Connecting to the storage via a filesystem
fs = S3FileSystem(
        client_kwargs={"endpoint_url": f"https://{os.environ['AWS_S3_ENDPOINT']}"},
        key=os.environ["AWS_ACCESS_KEY_ID"],
        secret=os.environ["AWS_SECRET_ACCESS_KEY"],
    )

# Loading a dataframe is very easy !
df_train = pd.read_parquet("df_train.parquet", filesystem=fs)

# Saving too
df_train.to_parquet("df_train.parquet", filesystem=fs)
German Federal Statistical Office: Hadoop Distributed File System, Parquet format and minimal preprocessing pipeline

In order to ensure that the data is stored and used efficiently we make use of the Hadoop Distributed File System (HDFS) and parquet for data partitioning. HDFS is especially made for handling a large amount of data. For programming and data processing, we use Cloudera AI Workbench with PySpark, which allows us to efficiently work on the data. We store our data in the Parquet format, which is ideal for big data and in addition, to make it easier for users to handle and cross-check the data, we use Hue (Hadoop User Experience), an open-source SQL-based cloud editor. For rights management, we use Ranger, which provides a big variety of access control to ensure data security.

The data cleaning in our project is quite straightforward, since the text entries contain short texts (mostly keywords) instead of long ones. First, data augmentation is performed by adding new text entries (e.g. text like “groceries” or “beverages”) to the dataset, adding multiple newly generated text values to each household to enrich the data. Adding a variety of new textual entries helps the model to generelize better. Secondly, we clean the data by removing punctation and handling missing values.

Statistics Austria: Automated pre-processing for new data

Training data is stored in a centralized network drive that serves as the stable storage layer for the project. By separating data storage from the local development and compute environments, the dataset remains accessible, backed up, and protected against accidental loss caused by local machine failures or code issues. Subject matter experts contribute new labeled data quarterly, typically consisting of 300–500 additional entries per update cycle.

Data versioning

Just as code (see chapter 2), a good practice is to version the dataset in order to:

  1. For any trained model, know on which data it has been trained (a model is attached to its training data)
  2. Always a latest dataset with a stable access link for easy training
Insee: MLFlow Datasets

We use the Datasets component of the framework MLFLow to seamlessly version datasets and attach each model to its training set.

import mlflow
import pandas as pd

train_df = pd.read_parquet("s3://my-bucket/train.parquet")

# Wrap the DataFrame as an MLflow Dataset, pointing to its source
dataset = mlflow.data.from_pandas(
    train_df,
    source="s3://my-bucket/train.parquet",
    name="complaints-train",
    targets="label",
)

with mlflow.start_run():
    # Log the dataset — MLflow records its hash, schema, and source URI
    mlflow.log_input(dataset, context="training")

    # ... train model, log metrics, log model artifact ...

Any run logged this way will display its linked dataset in the MLflow UI, making it straightforward to answer “which data was this model trained on?”.

TipFurther reading