gitea-infrastructure/lessons-learned.md

Lessons Learned

Session: 2026-02-18 -- Gitea Agent Skill Implementation

Technical Discoveries

1. Gitea 1.22.6 has no project board REST API

   • /api/v1/user/projects returns 404
   • The existing SKILL.md had endpoints that do not exist
   • Project board API is tracked in Gitea issue #14299, targeted for 1.26+
   • Mitigation: Use milestones as project containers -- they have built-in progress tracking

2. Scoped label exclusivity is UI-only in Gitea 1.22.6

   • Labels with exclusive: true display mutual exclusivity in the web UI
   • The API POST /repos/{owner}/{repo}/issues/{index}/labels only appends -- it does NOT remove conflicting scoped labels
   • Fix: Use a swap pattern: GET current labels, filter out old scope, add new label ID, PUT full set atomically
   • This was the single biggest "gotcha" -- the documentation implies API enforcement that does not exist

3. POST /issues/{index}/assignees does not exist in Gitea 1.22.6

   • Returns 404
   • Fix: Use PATCH /repos/{owner}/{repo}/issues/{index} with {"assignees": ["username"]}
   • This is a general issue edit endpoint, not a dedicated assignee endpoint

4. Issue dependency format is IssueMeta, not simple ID

   • Wrong: {"depends_on_id": 2} -- returns "repository does not exist [id: 0]"
   • Correct: {"owner": "terraphim", "repo": "agent-tasks", "index": 2}
   • Must check the Swagger spec for exact body format rather than guessing

Debugging Approaches That Worked

1. Test against live API first, then write code

   • Verified all 7 open questions against the live Gitea instance before writing any helper functions
   • This caught the project board API gap before writing fake endpoints

2. Integration tests with real cleanup

   • Each test run creates a timestamped repo (skill-test-{timestamp}), runs all tests, then deletes it
   • Trap on EXIT ensures cleanup even on failure
   • No mocks -- tests prove the actual API behavior

3. Incremental testing (run, fail, fix, rerun)

   • First test run: 23/26 pass, 3 fail -- immediately identified the label/assignee issues
   • Fixed the swap pattern and PATCH approach, second run: 26/26 pass

Pitfalls to Avoid

1. Do not trust Gitea documentation for API behavior -- always verify against the live instance. The swagger spec is the source of truth, but even that can be misleading for edge cases like scoped label exclusivity.

2. Do not assume POST endpoints exist for every resource subpath -- Gitea's API is inconsistent. Some resources (labels) have POST, others (assignees) only work via PATCH on the parent resource.

3. Do not rely on UBS scanner for infrastructure projects -- UBS found no scannable language files (expected for a project with only shell scripts, YAML, and markdown). Pre-commit hooks are more useful here.

4. Pre-commit python version -- The system may not have the Python version specified in .pre-commit-config.yaml. Changed from python3.9 to python3.12 to match the system. Also needed python3.12-venv apt package for virtualenv support.

Best Practices Discovered

1. Disciplined development phases work well for API integration

   • Phase 1 (Research): Discovered the project board API gap before design
   • Phase 2 (Design): Created the label swap architecture before implementation
   • Phase 3 (Implementation): Tests caught 3 API bugs on first run

2. Default repository pattern -- Using env vars with defaults (GITEA_OWNER=${GITEA_OWNER:-terraphim}) lets agents work out of the box while remaining configurable.

3. JSON output from all helpers -- Every shell function outputs valid JSON, making it trivial for agents to parse results with jq.

4. Idempotent setup scripts -- setup-labels.sh checks for existing resources before creating, making it safe to run repeatedly.

Session: 2026-02-18 -- Articles and Gitea Publishing

Technical Discoveries

1. GitHub Actions billing blocks automated workflows silently

   • The sync-to-gitea.yml workflow was failing with: "The job was not started because recent account payments have failed or your spending limit needs to be increased"
   • This was not visible unless you explicitly checked gh run view <id> -- the push itself succeeds, only the triggered workflow fails
   • Workaround: Push directly to Gitea remote using 1Password token: git push https://oauth2:${GITEA_TOKEN}@git.terraphim.cloud/...

2. Gitea org repos require explicit creation before push

   • git push to a non-existent org repo returns "Push to create is not enabled for organizations" (403)
   • Must create the repo first via API: POST /api/v1/orgs/{org}/repos with auto_init: false
   • Personal repos may allow push-to-create, but org repos do not by default

3. Parallel agent execution works well for article writing

   • Launched technical-cto and technical-writer agents simultaneously
   • Each produced a distinct article style without interference
   • CTO agent: 402 lines, bold narrative voice
   • Technical writer agent: 1289 lines, comprehensive reference with appendices
   • Both independently verified code snippets against the codebase

Debugging Approaches That Worked

1. Check workflow run details, not just push status

   • git push succeeding does not mean the triggered workflow succeeded
   • Always verify with gh run list --workflow=<name> after push

2. Direct remote push as fallback for CI failures

   • When GitHub Actions is unavailable, adding a gitea remote and pushing with op token works immediately
   • No need to wait for CI billing resolution

Pitfalls to Avoid

1. Do not assume GitHub Actions workflows are running -- billing issues cause silent failures. The push succeeds but triggered workflows silently fail. Check gh run list periodically.

2. Do not push to a Gitea org repo without creating it first -- unlike personal namespaces, organization repos require explicit API creation before accepting pushes.

Best Practices Discovered

1. Dual-agent article writing -- Using two agents with different voices (CTO narrative + technical reference) produces complementary content. The CTO article draws readers in; the technical article provides the implementation detail.

2. Verify all code in articles against live systems -- Both agents independently verified their code snippets against the actual codebase files. This caught zero errors because the source material (SKILL.md, test scripts) was already tested, but the verification step builds confidence.

3. Superseded: Add gitea remote for direct push -- This was the session 2 approach. Session 3 replaced it with Gitea's native pull mirror (see below).

Session: 2026-02-18 -- Replace GitHub Actions with Gitea Pull Mirror

Technical Discoveries

1. Gitea native pull mirror is the right approach

   • Gitea can poll GitHub and pull new commits automatically (configurable interval, minimum 10m)
   • No GitHub Actions, no tokens in GitHub, no billing dependency
   • Set up via POST /api/v1/repos/migrate with "mirror": true
   • The Gitea repo becomes read-only; all pushes go to GitHub only

2. Existing repos cannot be converted to pull mirrors

   • There is no API to add mirror: true to an existing repo
   • Must delete the repo and recreate via POST /repos/migrate
   • This is documented in Gitea's official docs

3. The service field in migrate API matters for private repos

   • service: "git" tries git clone which fails with "terminal prompts disabled" for private repos
   • service: "github" uses the GitHub API and handles auth correctly
   • Always use service: "github" with auth_token for private GitHub repos

4. Failed migrations leave broken empty repos

   • If the async clone fails (bad token, network), Gitea creates the repo entry anyway (mirror: true, empty: true)
   • The POST /mirror-sync endpoint returns "Repository is not a mirror" on these broken repos
   • Must delete and recreate; cannot repair in place
   • The migrate API returns empty JSON {} (null fields) even on success -- check repo status separately

5. The github.personal.token in 1Password is expired

   • op://TerraphimPlatform/github.personal.token/token returns a token rejected by GitHub (HTTP 401)
   • gh auth token provides a working OAuth token as alternative
   • Action item: update the 1Password item with a fresh GitHub PAT

6. Race condition between delete and create on Gitea

   • DELETE /repos/{owner}/{repo} returns 204 immediately
   • But the repo name may not be freed yet -- POST /repos/migrate returns 409
   • Fix: Always sleep 10 between delete and create, then verify 404 before proceeding

Pitfalls to Avoid

1. Do not use GitHub Actions for Gitea sync -- it adds unnecessary dependency on GitHub billing, requires storing secrets in GitHub, and consumes Actions minutes. Use Gitea's native pull mirror instead.

2. Do not use service: "git" for private GitHub repos -- it cannot authenticate. Use service: "github" with auth_token.

3. Do not trust the migrate API response body -- it returns null/empty JSON even on success. Always check repo status via GET /repos/{owner}/{repo} after waiting for the async clone.

4. Do not assume deleted repos are immediately gone -- always wait and verify with a GET returning 404 before recreating.

Best Practices Discovered

1. Simplest mirroring wins -- Gitea pull mirror is one API call to set up, zero maintenance, and replaces two GitHub Actions workflows plus a monitoring check.

2. Use gh auth token as GitHub token source -- more reliable than maintaining a separate PAT in 1Password. The gh CLI handles token refresh automatically.

Session: 2026-02-18 -- 1Password Service Account Limitations

Technical Discoveries

1. 1Password service accounts are read-only by default
   • op_zesticai_non_prod.sh sources a SERVICE_ACCOUNT (type visible via op whoami)
   • Service accounts can op read but op item edit fails with "Couldn't update the item"
   • No specific error about permissions -- just a generic failure message
   • Write access must be granted explicitly in 1Password admin settings per vault

Pitfalls to Avoid

1. Do not assume op item edit works with service accounts -- test write access before building automation that depends on it. The error message gives no indication that it's a permissions issue.

Best Practices Discovered

1. Track infrastructure tasks in a central repo -- Created issue in AlexMikhalev/cto-executive-system to track the token update. This prevents losing track of manual steps that couldn't be automated.