Rendered from the project README. The pipeline ran offline at build time. Contoso Coffee Co. is fictional, not a client.

DataMill

DataMill is an offline batch ETL demo for Contoso Coffee Co., a fictional company. Every store, receipt, customer, and inventory count is generated by fixtures/generate.py with seed 2026. There are no real clients, reviews, or revenue claims.

The pipeline reads dirty POS CSV files, an inventory workbook, a product catalog, and a paged CRM payload. It lands the bytes, validates them, quarantines bad rows, loads a star schema, runs data-quality checks, and publishes three marts. SQLite is the default warehouse. DATABASE_URL can point at PostgreSQL

The demo run does not use that variable.

What a run does

extract → land → validate → quarantine → transform → load → dq → publish → manifest

POS dates and amounts follow the store locale. Money is stored as integer cents.

sold_at is interpreted in the store time zone and stored as UTC.

A file whose columns do not match the canonical POS layout is quarantined as

SCHEMA_DRIFT. Other stores in the same run still load. That is why 2026-09-17 is WARN and still publishes marts.

Row-level rejects (BAD_DATE, BAD_AMOUNT, MISSING_STORE, UNKNOWN_SKU,

DUPLICATE_ROW) do not by themselves warn the run.

ENCODING_FIXED is a notice. Latin-1 is used only when UTF-8 fails.
dim_product is SCD2 (valid_from, valid_to, is_current). A same-day

rerun does not open another version.

Facts upsert on their natural keys. Watermarks and file hashes move the load

forward. --full-refresh rebuilds the warehouse.

Blocking data-quality failures roll back and skip publish. Warnings publish.
Customer email is reduced to its domain. customer_key is SHA-256 of the

normalized address. A production system should use a keyed HMAC and should read the key from an environment variable. This demo does not.

Layout

datamill/ package, including python -m datamill and python -m datamill.stub_api
config/ store locales and DQ rule levels
fixtures/ generated sources (committed)
orchestration/ example systemd, cron, and Airflow files (not installed)
scripts/record_demo.py runs September incrementally, reruns one day, then

full-refreshes a second warehouse

scripts/build_showcase.py writes site/datamill/ from results/summary.json
results/ committed artifacts from the demo run
warehouse/ local SQLite, landing, and logs (gitignored)
docs/ ERD, pipeline drawing, and screenshots

The static site has no directory named data and does not call a CDN. Page numbers are written by the showcase generator from results/summary.json.

Setup

The pinned stack installs on CPython 3.9 / Windows (pandas==2.2.3, numpy==2.0.2, pyarrow==17.0.0, greenlet==3.1.1, and the other pins in requirements.txt). psycopg[binary] is an optional extra and is not required for the SQLite demo.

This machine's Python 3.9 does not ship venv / ensurepip. The existing .venv was created with virtualenv. If you recreate it, copy python.exe to pythonw.exe next to the base interpreter first. If virtualenv leaves .venv/Scripts/python39.zip as a symlink, delete that symlink. Leaving it makes sys.base_prefix point at Scripts and breaks the standard library. Do not commit .venv.

.venv\Scripts\python.exe -m pip install -r requirements.txt -r requirements-dev.txt
      .venv\Scripts\python.exe fixtures\generate.py

Commands

.venv\Scripts\python.exe -m datamill run --date 2026-09-03
      .venv\Scripts\python.exe -m datamill run --from 2026-09-01 --to 2026-09-30
      .venv\Scripts\python.exe -m datamill run --from 2026-09-01 --to 2026-09-30 --full-refresh
      .venv\Scripts\python.exe -m datamill run --date 2026-09-03 --dry-run
      .venv\Scripts\python.exe -m datamill dq
      .venv\Scripts\python.exe -m datamill report --month 2026-09

record_demo.py starts the CRM stub on 127.0.0.1:18899 for the build and shuts it down before exit. The stub is not a deployed service. Runtime code does not call a paid API and does not read an API key.

.venv\Scripts\python.exe scripts\record_demo.py
      .venv\Scripts\python.exe scripts\build_showcase.py
      .venv\Scripts\python.exe -m pytest -q --cov=datamill
      .venv\Scripts\python.exe -m ruff check

docker-compose.yml is an example (postgres:16 plus a one-shot container). Do not execute it as part of the portfolio build. The Airflow module is parsed with ast only.

Demo checks

Among the 30 daily runs, only 2026-09-17 is WARN, because

pos/07/2026-09-17.csv is SCHEMA_DRIFT.

A same-day rerun adds 0 rows.
Full-refresh marts match the incremental marts.
The demo finishes in under 120 seconds and stays under 500 MB peak RSS.