Data & training

Building the dataset

The dataset is built offline from public history, never at scoring time. The goal is a leakage-controlled table of repository snapshots where every feature is measured before the observation date and every label is measured after it. See Where data comes from for the sources feeding this pipeline.

The pipeline, step by step

1
Generate a repository seed
A seed is a list of GitHub repositories to learn from, sampled across active, dormant, and archived strata (with a star floor, a required license, and no forks). The strata are only for sampling — they are never used as labels.
```
npm run ml:seed:foundation
```
2
Download filtered GH Archive history
For every repo in the seed, hourly GH Archive event files are streamed and filtered down to that seed's events, with coverage manifests. Only days with all 24 hourly files count as covered — sparse hours would make “no activity” indistinguishable from “missing data”.
```
.\scripts\download-gharchive-seed-filtered-parallel.ps1 -Workers 4 \
  -SeedPath .\tmp\training-foundation\foundation-seed.csv \
  -OutDir .\tmp\gharchive-foundation -KeepRaw -Start 2021-01-01
```
3
Build labeled snapshots
Each snapshot is one repository observed at a date t. Observation-time features are computed only from events at or before t; the 12-month label is computed only from the window (t, t+12 months]. Rows whose future window isn't fully covered by the archive stay unlabeled.
```
npm run ml:dataset:foundation
```

How the 12-month label is defined

The exported label is label_inactive_12m, the inverse of a maintained_12m variable. A repository counts as maintained in the horizon if it is not archived/deleted by t+12 months and at least two of four future-window activity checks pass (commits, releases, merged PRs, and issue activity). Completeness is gated on a dataset-wide archive coverage horizon, so a repository that genuinely goes quiet is labeled inactive rather than being dropped as “incomplete”.

What the build produces

snapshots.json — the labeled training rows (features + label + provenance).
repository-feature-cache.json — per-repository full-history features used later for live full-history scoring.

With the dataset in place, continue to Train it yourself.