gitea-infrastructure/lessons-learned.md

# Lessons Learned

## Session: 2026-02-18 -- Gitea Agent Skill Implementation

### Technical Discoveries

1. **Gitea 1.22.6 has no project board REST API**
   - `/api/v1/user/projects` returns 404
   - The existing SKILL.md had endpoints that do not exist
   - Project board API is tracked in Gitea issue #14299, targeted for 1.26+
   - **Mitigation:** Use milestones as project containers -- they have built-in progress tracking

2. **Scoped label exclusivity is UI-only in Gitea 1.22.6**
   - Labels with `exclusive: true` display mutual exclusivity in the web UI
   - The API `POST /repos/{owner}/{repo}/issues/{index}/labels` only appends -- it does NOT remove conflicting scoped labels
   - **Fix:** Use a swap pattern: GET current labels, filter out old scope, add new label ID, PUT full set atomically
   - This was the single biggest "gotcha" -- the documentation implies API enforcement that does not exist

3. **POST /issues/{index}/assignees does not exist in Gitea 1.22.6**
   - Returns 404
   - **Fix:** Use `PATCH /repos/{owner}/{repo}/issues/{index}` with `{"assignees": ["username"]}`
   - This is a general issue edit endpoint, not a dedicated assignee endpoint

4. **Issue dependency format is IssueMeta, not simple ID**
   - Wrong: `{"depends_on_id": 2}` -- returns "repository does not exist [id: 0]"
   - Correct: `{"owner": "terraphim", "repo": "agent-tasks", "index": 2}`
   - Must check the Swagger spec for exact body format rather than guessing

### Debugging Approaches That Worked

1. **Test against live API first, then write code**
   - Verified all 7 open questions against the live Gitea instance before writing any helper functions
   - This caught the project board API gap before writing fake endpoints

2. **Integration tests with real cleanup**
   - Each test run creates a timestamped repo (`skill-test-{timestamp}`), runs all tests, then deletes it
   - Trap on EXIT ensures cleanup even on failure
   - No mocks -- tests prove the actual API behavior

3. **Incremental testing (run, fail, fix, rerun)**
   - First test run: 23/26 pass, 3 fail -- immediately identified the label/assignee issues
   - Fixed the swap pattern and PATCH approach, second run: 26/26 pass

### Pitfalls to Avoid

1. **Do not trust Gitea documentation for API behavior** -- always verify against the live instance. The swagger spec is the source of truth, but even that can be misleading for edge cases like scoped label exclusivity.

2. **Do not assume POST endpoints exist for every resource subpath** -- Gitea's API is inconsistent. Some resources (labels) have POST, others (assignees) only work via PATCH on the parent resource.

3. **Do not rely on UBS scanner for infrastructure projects** -- UBS found no scannable language files (expected for a project with only shell scripts, YAML, and markdown). Pre-commit hooks are more useful here.

4. **Pre-commit python version** -- The system may not have the Python version specified in `.pre-commit-config.yaml`. Changed from `python3.9` to `python3.12` to match the system. Also needed `python3.12-venv` apt package for virtualenv support.

### Best Practices Discovered

1. **Disciplined development phases work well for API integration**
   - Phase 1 (Research): Discovered the project board API gap before design
   - Phase 2 (Design): Created the label swap architecture before implementation
   - Phase 3 (Implementation): Tests caught 3 API bugs on first run

2. **Default repository pattern** -- Using env vars with defaults (`GITEA_OWNER=${GITEA_OWNER:-terraphim}`) lets agents work out of the box while remaining configurable.

3. **JSON output from all helpers** -- Every shell function outputs valid JSON, making it trivial for agents to parse results with jq.

4. **Idempotent setup scripts** -- `setup-labels.sh` checks for existing resources before creating, making it safe to run repeatedly.

## Session: 2026-02-18 -- Articles and Gitea Publishing

### Technical Discoveries

1. **GitHub Actions billing blocks automated workflows silently**
   - The `sync-to-gitea.yml` workflow was failing with: "The job was not started because recent account payments have failed or your spending limit needs to be increased"
   - This was not visible unless you explicitly checked `gh run view <id>` -- the push itself succeeds, only the triggered workflow fails
   - **Workaround:** Push directly to Gitea remote using 1Password token: `git push https://oauth2:${GITEA_TOKEN}@git.terraphim.cloud/...`

2. **Gitea org repos require explicit creation before push**
   - `git push` to a non-existent org repo returns "Push to create is not enabled for organizations" (403)
   - Must create the repo first via API: `POST /api/v1/orgs/{org}/repos` with `auto_init: false`
   - Personal repos may allow push-to-create, but org repos do not by default

3. **Parallel agent execution works well for article writing**
   - Launched `technical-cto` and `technical-writer` agents simultaneously
   - Each produced a distinct article style without interference
   - CTO agent: 402 lines, bold narrative voice
   - Technical writer agent: 1289 lines, comprehensive reference with appendices
   - Both independently verified code snippets against the codebase

### Debugging Approaches That Worked

1. **Check workflow run details, not just push status**
   - `git push` succeeding does not mean the triggered workflow succeeded
   - Always verify with `gh run list --workflow=<name>` after push

2. **Direct remote push as fallback for CI failures**
   - When GitHub Actions is unavailable, adding a gitea remote and pushing with `op` token works immediately
   - No need to wait for CI billing resolution

### Pitfalls to Avoid

1. **Do not assume GitHub Actions workflows are running** -- billing issues cause silent failures. The push succeeds but triggered workflows silently fail. Check `gh run list` periodically.

2. **Do not push to a Gitea org repo without creating it first** -- unlike personal namespaces, organization repos require explicit API creation before accepting pushes.

### Best Practices Discovered

1. **Dual-agent article writing** -- Using two agents with different voices (CTO narrative + technical reference) produces complementary content. The CTO article draws readers in; the technical article provides the implementation detail.

2. **Verify all code in articles against live systems** -- Both agents independently verified their code snippets against the actual codebase files. This caught zero errors because the source material (SKILL.md, test scripts) was already tested, but the verification step builds confidence.

3. **Superseded: Add gitea remote for direct push** -- This was the session 2 approach. Session 3 replaced it with Gitea's native pull mirror (see below).

## Session: 2026-02-18 -- Replace GitHub Actions with Gitea Pull Mirror

### Technical Discoveries

1. **Gitea native pull mirror is the right approach**
   - Gitea can poll GitHub and pull new commits automatically (configurable interval, minimum 10m)
   - No GitHub Actions, no tokens in GitHub, no billing dependency
   - Set up via `POST /api/v1/repos/migrate` with `"mirror": true`
   - The Gitea repo becomes read-only; all pushes go to GitHub only

2. **Existing repos cannot be converted to pull mirrors**
   - There is no API to add `mirror: true` to an existing repo
   - Must delete the repo and recreate via `POST /repos/migrate`
   - This is documented in Gitea's official docs

3. **The `service` field in migrate API matters for private repos**
   - `service: "git"` tries `git clone` which fails with "terminal prompts disabled" for private repos
   - `service: "github"` uses the GitHub API and handles auth correctly
   - Always use `service: "github"` with `auth_token` for private GitHub repos

4. **Failed migrations leave broken empty repos**
   - If the async clone fails (bad token, network), Gitea creates the repo entry anyway (`mirror: true, empty: true`)
   - The `POST /mirror-sync` endpoint returns "Repository is not a mirror" on these broken repos
   - Must delete and recreate; cannot repair in place
   - The migrate API returns empty JSON `{}` (null fields) even on success -- check repo status separately

5. **The `github.personal.token` in 1Password is expired**
   - `op://TerraphimPlatform/github.personal.token/token` returns a token rejected by GitHub (HTTP 401)
   - `gh auth token` provides a working OAuth token as alternative
   - Action item: update the 1Password item with a fresh GitHub PAT

6. **Race condition between delete and create on Gitea**
   - `DELETE /repos/{owner}/{repo}` returns 204 immediately
   - But the repo name may not be freed yet -- `POST /repos/migrate` returns 409
   - **Fix:** Always `sleep 10` between delete and create, then verify 404 before proceeding

### Pitfalls to Avoid

1. **Do not use GitHub Actions for Gitea sync** -- it adds unnecessary dependency on GitHub billing, requires storing secrets in GitHub, and consumes Actions minutes. Use Gitea's native pull mirror instead.

2. **Do not use `service: "git"` for private GitHub repos** -- it cannot authenticate. Use `service: "github"` with `auth_token`.

3. **Do not trust the migrate API response body** -- it returns null/empty JSON even on success. Always check repo status via `GET /repos/{owner}/{repo}` after waiting for the async clone.

4. **Do not assume deleted repos are immediately gone** -- always wait and verify with a GET returning 404 before recreating.

### Best Practices Discovered

1. **Simplest mirroring wins** -- Gitea pull mirror is one API call to set up, zero maintenance, and replaces two GitHub Actions workflows plus a monitoring check.

2. **Use `gh auth token` as GitHub token source** -- more reliable than maintaining a separate PAT in 1Password. The `gh` CLI handles token refresh automatically.