github

Definition

The dominant host for Git repositories, and the source of two things this field depends on: public code used as training data, and real issue-and-pull-request histories used to build evaluations. Because a merged fix arrives with the tests that verify it, GitHub supplies ready-made ground truth for coding agents — which is also why contamination between training data and benchmarks is a standing worry.

Explainers using this term

• Paper Explained: Repo-To-Skill — Distilling GitHub Repositories Into Skills an AI Can UseRepo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
• A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually MeasureSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Evaluating Agents — How Benchmarks and Harnesses Are BuiltSWE-bench: Can Language Models Resolve Real-World GitHub Issues?