|
1 | | -# Finance Data Pipelines — Pro (with Real Extractor) |
| 1 | +# Finance Data Pipelines |
2 | 2 |
|
3 | | -A professional, interview-ready Python ETL template for **financial data** (stocks/indices/crypto). |
4 | | -Includes **real data extraction via CoinGecko** (no API key), CLI, config, tests, CI, and Docker. |
| 3 | +Finance Data Pipelines is a small, correctness-first time-series ETL and SQLite loading |
| 4 | +project. The current release provides a deterministic offline generator, strict validation, |
| 5 | +an independent logical-state oracle, and **Strategy A: atomic full replacement**. |
5 | 6 |
|
6 | | -## Quickstart |
| 7 | +The repository is a baseline for a later controlled systems study. It does not yet contain |
| 8 | +Strategies B/C or performance results, and it is not presented as production-ready, |
| 9 | +high-performance, scalable, or research-grade software. |
7 | 10 |
|
8 | | -### 1) Setup (Windows) |
9 | | -```powershell |
10 | | -py -m venv .venv |
11 | | -.venv\Scripts\activate |
12 | | -pip install -r requirements.txt |
13 | | -pip install -r requirements-dev.txt |
14 | | -copy .env.sample .env |
15 | | -python -m fdp.cli run-all |
| 11 | +## Requirements |
| 12 | + |
| 13 | +- Python 3.11 or newer |
| 14 | +- SQLite supplied by Python |
| 15 | +- No API key or network data source |
| 16 | + |
| 17 | +## Installation |
| 18 | + |
| 19 | +```bash |
| 20 | +python3 -m venv .venv |
| 21 | +source .venv/bin/activate |
| 22 | +python -m pip install --upgrade pip |
| 23 | +python -m pip install . |
| 24 | +fdp --help |
16 | 25 | ``` |
17 | 26 |
|
18 | | -### 1) Setup (macOS/Linux) |
| 27 | +For development: |
| 28 | + |
| 29 | +```bash |
| 30 | +python -m pip install -e '.[dev]' |
| 31 | +``` |
| 32 | + |
| 33 | +`pyproject.toml` is the authoritative dependency definition. `requirements*.txt` are |
| 34 | +compatibility wrappers only. |
| 35 | + |
| 36 | +## Deterministic offline quickstart |
| 37 | + |
| 38 | +Choose the output directory explicitly. All generated Parquet, JSON, SQLite, WAL, and SHM |
| 39 | +files remain under that directory unless `--db-path` explicitly selects another location. |
| 40 | + |
19 | 41 | ```bash |
20 | | -python3 -m venv .venv && source .venv/bin/activate |
21 | | -pip install -r requirements.txt -r requirements-dev.txt |
22 | | -cp .env.sample .env |
23 | | -python -m fdp.cli run-all |
| 42 | +fdp run-all \ |
| 43 | + --source synthetic \ |
| 44 | + --seed 20270916 \ |
| 45 | + --rows 1000 \ |
| 46 | + --output-dir /tmp/fdp-run |
24 | 47 | ``` |
25 | 48 |
|
26 | | -### 2) Switch data source |
27 | | -Edit `config.yaml`: |
28 | | -```yaml |
29 | | -source: coingecko # options: coingecko | synthetic |
30 | | -coingecko: |
31 | | - coin_id: bitcoin |
32 | | - vs_currency: usd |
33 | | - days: 30 |
34 | | - |
35 | | -table: |
36 | | - name: prices |
| 49 | +The command performs two identical atomic full replacements so exact-rerun idempotency is |
| 50 | +verified. It prints correctness fields as JSON and writes: |
| 51 | + |
| 52 | +```text |
| 53 | +/tmp/fdp-run/ |
| 54 | + raw/prices_raw.parquet |
| 55 | + normalized/prices_normalized.parquet |
| 56 | + dataset_manifest.json |
| 57 | + correctness.json |
| 58 | + warehouse.db |
37 | 59 | ``` |
38 | | -- `coingecko` uses the public API (no key). |
39 | | -- `synthetic` generates demo data (offline; always works). |
40 | 60 |
|
41 | | -### 3) Outputs |
42 | | -- `data/raw/prices_raw.parquet` |
43 | | -- `data/clean/prices_clean.parquet` |
44 | | -- `warehouse.db` (SQLite) → table: `prices` |
| 61 | +No throughput, latency, or comparative benchmark conclusion is produced. |
| 62 | + |
| 63 | +## Optional configuration file |
| 64 | + |
| 65 | +CLI values override a YAML file. Only `source`, `seed`, and `rows` are accepted: |
45 | 66 |
|
46 | | -## CLI Examples |
47 | 67 | ```bash |
48 | | -python -m fdp.cli extract --source coingecko --coin-id bitcoin --vs usd --days 30 |
49 | | -python -m fdp.cli extract --source synthetic --days 60 |
50 | | -python -m fdp.cli transform |
51 | | -python -m fdp.cli load --table prices |
52 | | -python -m fdp.cli run-all |
| 68 | +fdp run-all --config config.yaml --output-dir /tmp/fdp-run |
53 | 69 | ``` |
54 | 70 |
|
55 | | -## Project Layout |
| 71 | +The package never searches the source checkout for configuration. It does not use `.env`, so |
| 72 | +no `.env.example` is required. |
| 73 | + |
| 74 | +## Logical model |
| 75 | + |
| 76 | +The logical key is `(series_id, event_ts_utc)` and exact event identity is |
| 77 | +`(series_id, event_ts_utc, revision)`. Timestamps are UTC Unix seconds aligned to one minute. |
| 78 | +Prices and optional volume use fixed-point integers with scale `10^-6`. |
| 79 | + |
| 80 | +Duplicate/update rules: |
| 81 | + |
| 82 | +- an exact duplicate event is a safe no-op; |
| 83 | +- different payloads for the same key and revision reject the entire snapshot; |
| 84 | +- the highest revision is current; |
| 85 | +- a lower revision cannot regress the current state; |
| 86 | +- an empty replacement is rejected by default. |
| 87 | + |
| 88 | +The SQLite current-state table has a composite primary key, required checks, and an |
| 89 | +`event_ts_utc` index. Loads use WAL, `synchronous=FULL`, foreign keys, a 5-second busy timeout, |
| 90 | +and one explicit transaction. The staging table is validated against the independent oracle |
| 91 | +before publication. A pre-commit failure rolls back to the prior committed table. |
| 92 | + |
| 93 | +## Tests and local CI-equivalent checks |
| 94 | + |
| 95 | +```bash |
| 96 | +ruff check . |
| 97 | +ruff format --check . |
| 98 | +pytest -q |
| 99 | + |
| 100 | +tmp_dir="$(mktemp -d)" |
| 101 | +fdp run-all --source synthetic --seed 20270916 --rows 1000 \ |
| 102 | + --output-dir "$tmp_dir/output" |
56 | 103 | ``` |
57 | | -src/fdp/ |
58 | | - cli.py # click-based CLI (with flags) |
59 | | - config.py # Pydantic settings + YAML config |
60 | | - extract.py # real extractor (CoinGecko) + synthetic fallback |
61 | | - transform.py # cleaning + returns |
62 | | - load.py # SQLAlchemy load into SQLite |
63 | | - utils/ |
64 | | - io.py |
65 | | - logging.py |
66 | | -tests/ |
67 | | - test_flow.py |
68 | | - test_extract_synthetic.py |
69 | | - test_extract_coingecko_stub.py |
70 | | -.github/workflows/python-ci.yml |
71 | | -.pre-commit-config.yaml |
72 | | -pyproject.toml |
73 | | -requirements.txt |
74 | | -requirements-dev.txt |
75 | | -Makefile |
76 | | -Dockerfile |
77 | | -.env.sample |
78 | | -config.yaml |
79 | | -LICENSE |
80 | | -README.md |
81 | | -docs/DATA_SOURCES.md |
| 104 | + |
| 105 | +Tests use temporary directories and make no live network calls. |
| 106 | + |
| 107 | +## Docker |
| 108 | + |
| 109 | +The image installs the package, runs as a non-root user, and defaults to the deterministic |
| 110 | +offline 1,000-row flow: |
| 111 | + |
| 112 | +```bash |
| 113 | +docker build -t finance-data-pipelines:phase3d . |
| 114 | +docker run --rm -v "$PWD/docker-output:/work/output" \ |
| 115 | + finance-data-pipelines:phase3d |
82 | 116 | ``` |
83 | 117 |
|
84 | | -> Tip: Replace the CoinGecko extractor with your preferred exchange/stock API if needed, |
85 | | -and keep credentials in `.env` (never commit secrets). |
| 118 | +Docker support is part of the baseline, but a particular release should be called verified |
| 119 | +only when build and run commands have actually executed in that release environment. |
| 120 | + |
| 121 | +## Current maturity and limitations |
| 122 | + |
| 123 | +- Strategy A only; Strategies B/C belong to the next phase. |
| 124 | +- Single-process, single-writer SQLite baseline. |
| 125 | +- Synthetic correctness fixture only; no market-behavior claims. |
| 126 | +- No concurrency or distributed-system evaluation. |
| 127 | +- No performance measurements or rankings. |
| 128 | +- Deterministic exception injection tests transaction rollback; they are not a claim that |
| 129 | + every OS/power-loss mode has been tested. |
| 130 | + |
| 131 | +## Project layout |
| 132 | + |
| 133 | +```text |
| 134 | +src/fdp/ |
| 135 | + cli.py installed Click interface |
| 136 | + pipeline.py ordinary Python orchestration |
| 137 | + extract.py deterministic synthetic generator |
| 138 | + transform.py strict normalization |
| 139 | + validation.py loader-side validation and revision resolution |
| 140 | + oracle.py independent current-state oracle |
| 141 | + load.py atomic SQLite full replacement |
| 142 | + encoding.py canonical binary checksum encoding |
| 143 | + manifest.py deterministic JSON manifests |
| 144 | + parquet_io.py frozen Parquet schema |
| 145 | +tests/ isolated correctness and CLI tests |
| 146 | +.github/workflows/ci.yml |
| 147 | +``` |
0 commit comments