GitHub Models - AI Model Hubs Tool

Overview

GitHub Models is an official suite of developer tools from GitHub for cataloging, testing, evaluating, and managing AI models alongside application code. According to GitHub's documentation (https://docs.github.com/en/github-models), the offering centers on a repository-native model catalog, structured prompt management, and quantitative evaluation capabilities so teams can iterate on models and prompts using the same workflows and review processes used for code. The platform emphasizes reproducibility by keeping model metadata, prompt templates, and evaluation artifacts under version control. GitHub Models is designed to fit into repository CI/CD and governance—for example, teams can run reproducible evaluation jobs in Actions, compare models using recorded metrics, and manage model access and provenance in a central catalog. This helps move model work from ad-hoc experimentation to auditable, team-scaled development. While GitHub provides the docs and integrations, many operational details (rate limits, enterprise pricing tiers) are documented in the official GitHub documentation and may vary by account and organization.

Key Features

• Repository-native model catalog for storing model metadata and provenance
• Prompt template management versioned alongside code and model artifacts
• Quantitative evaluation workflows that run in GitHub Actions
• Comparative model metrics and evaluation artifacts stored in the repo
• Integration with existing GitHub access controls, PR reviews, and audit trails

Code Examples

Python

import base64
import json
import requests

# Example: commit a model metadata file to a repository (illustrative)
GITHUB_TOKEN = "YOUR_TOKEN"
owner = "my-org"
repo = "my-repo"
path = ".github/models/my-model-card.md"
message = "Add model card for my-model"
content = "# My model\nModel metadata and evaluation summaries."
encoded = base64.b64encode(content.encode()).decode()

url = f"https://api.github.com/repos/{owner}/{repo}/contents/{path}"
headers = {"Authorization": f"token {GITHUB_TOKEN}", "Accept": "application/vnd.github+json"}
payload = {"message": message, "content": encoded}
resp = requests.put(url, headers=headers, data=json.dumps(payload))
resp.raise_for_status()
print("Committed model card:", resp.json().get("content", {}).get("path"))

# Example: trigger a GitHub Actions workflow to run model evaluation
workflow_dispatch_url = f"https://api.github.com/repos/{owner}/{repo}/actions/workflows/evaluate-model.yml/dispatches"
dispatch_payload = {"ref": "main", "inputs": {"model_path": path}}
resp2 = requests.post(workflow_dispatch_url, headers=headers, data=json.dumps(dispatch_payload))
if resp2.status_code == 204:
    print("Evaluation workflow dispatched")
else:
    print("Dispatch response:", resp2.status_code, resp2.text)

# Note: These examples use general GitHub REST APIs; consult GitHub Models docs for model-specific guidance.

Curl

## Commit a file (model card) to a repo using the REST API (illustrative)
GITHUB_TOKEN="YOUR_TOKEN"
OWNER="my-org"
REPO="my-repo"
PATH=".github/models/my-model-card.md"
CONTENT_B64=$(echo -n "# My model\nModel metadata and evaluation summaries." | base64)

curl -X PUT \
  -H "Authorization: token $GITHUB_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/repos/$OWNER/$REPO/contents/$PATH \
  -d "{\"message\": \"Add model card\", \"content\": \"$CONTENT_B64\"}"

## Trigger a workflow dispatch to run an evaluation (illustrative)
WORKFLOW_FILE="evaluate-model.yml"

curl -X POST \
  -H "Authorization: token $GITHUB_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/repos/$OWNER/$REPO/actions/workflows/$WORKFLOW_FILE/dispatches \
  -d '{"ref":"main","inputs":{"model_path":".github/models/my-model-card.md"}}'

# Replace placeholders with actual values. See GitHub REST API docs for details.

Javascript

const { Octokit } = require("@octokit/rest");

// Illustrative example: create/update a model card file and dispatch evaluation workflow
const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
const owner = "my-org";
const repo = "my-repo";
const path = ".github/models/my-model-card.md";

async function run() {
  // create/update a file (model card)
  await octokit.rest.repos.createOrUpdateFileContents({
    owner,
    repo,
    path,
    message: "Add model card",
    content: Buffer.from("# My model\nModel metadata and evaluation summaries.").toString("base64"),
  });

  // dispatch a workflow to evaluate the model
  await octokit.rest.actions.createWorkflowDispatch({
    owner,
    repo,
    workflow_id: "evaluate-model.yml",
    ref: "main",
    inputs: { model_path: path },
  });

  console.log("Model card committed and evaluation dispatched");
}

run().catch(err => { console.error(err); process.exit(1); });

// These examples use generic GitHub APIs to show repository-based model workflows.

API Overview

• Authentication: OAuth
• Base URL: https://api.github.com

Key Information

• Category: Model Hubs
• Type: AI Model Hubs Tool