diff --git a/QUICK-SETUP-MIRRORING.md b/QUICK-SETUP-MIRRORING.md new file mode 100644 index 0000000..c370676 --- /dev/null +++ b/QUICK-SETUP-MIRRORING.md @@ -0,0 +1,200 @@ +# Quick Setup: Repository Mirroring + +Your repository is now on **GitHub** and ready for mirroring! + +## Current Status + +✅ **GitHub Repository:** https://github.com/terraphim/gitea-infrastructure +❌ **Gitea Repository:** Not created yet + +## Setup Steps + +### Step 1: Create Repository on Gitea + +**Via Web Interface:** +1. Go to https://git.terraphim.cloud +2. Click **"+"** → **"New Repository"** +3. **Owner:** `terraphim` +4. **Repository Name:** `gitea-infrastructure` +5. **Description:** `Gitea deployment configuration` +6. **Visibility:** ☑️ Private +7. ☐ **Uncheck** "Initialize Repository" +8. Click **"Create Repository"** + +**Result:** Empty repository created + +### Step 2: Push to Gitea + +```bash +cd /home/alex/infrastructure/gitea-config + +# Push to Gitea +git push origin main +``` + +### Step 3: Configure GitHub Actions Secrets + +**On GitHub:** +1. Go to: https://github.com/terraphim/gitea-infrastructure/settings/secrets/actions +2. Click **"New repository secret"** +3. Add: + - **Name:** `GITEA_TOKEN` + - **Secret:** Your Gitea API token + +**To create Gitea API token:** +1. Go to https://git.terraphim.cloud/user/settings/applications +2. Click **"Generate New Token"** +3. **Token Name:** `GitHub Actions Sync` +4. Select scopes: `repo`, `write:repo` +5. Click **"Generate Token"** +6. Copy the token + +### Step 4: Test Mirroring + +**Push to GitHub:** +```bash +# Make a change +echo "# Mirroring test" >> README.md +git add README.md +git commit -m "test: Mirroring to Gitea" + +# Push to GitHub (triggers Actions) +git push github main +``` + +**Check Gitea:** +- Go to https://git.terraphim.cloud/terraphim/gitea-infrastructure +- Verify the commit appears + +## Current Remote Configuration + +```bash +# Check remotes +git remote -v + +# Should show: +# origin https://git.terraphim.cloud/terraphim/gitea-infrastructure.git (fetch) +# origin https://git.terraphim.cloud/terraphim/gitea-infrastructure.git (push) +# github https://github.com/terraphim/gitea-infrastructure.git (fetch) +# github https://github.com/terraphim/gitea-infrastructure.git (push) +``` + +## Two-Way Sync Workflow + +### Option A: GitHub as Primary (Recommended) + +**How it works:** +1. Developer pushes to GitHub +2. GitHub Actions automatically pushes to Gitea +3. Gitea stays in sync as mirror + +**Commands:** +```bash +# Work locally +git push github main # Primary push (triggers Actions) + +# Or push to both manually +git push github main && git push origin main +``` + +### Option B: Push to Both + +**Set up alias:** +```bash +# Add to ~/.gitconfig +git config --global alias.pushall '!git push github $(git branch --show-current) && git push origin $(git branch --show-current)' + +# Usage +git pushall # Pushes to both +``` + +### Option C: Gitea as Primary + +**Configure in Gitea:** +1. Go to repository **Settings** → **Repository** +2. Scroll to **Mirror Settings** +3. Check **"Push Mirror"** +4. **Git Remote URL:** `https://github.com/terraphim/gitea-infrastructure.git` +5. **Authentication:** + - **Username:** Your GitHub username + - **Password:** GitHub Personal Access Token +6. **Interval:** `1h` (or your preference) +7. Click **"Add Push Mirror"** + +**To create GitHub token:** +1. Go to https://github.com/settings/tokens +2. Click **"Generate new token (classic)"** +3. Select scope: `repo` +4. Generate and copy token + +## Repository Structure + +After setup, both repositories will have: + +``` +gitea-infrastructure/ +├── .docs/ # Documentation +│ ├── RESEARCH-gitea-bigbox-deployment.md +│ ├── IMPLEMENTATION-gitea-bigbox-deployment.md +│ ├── DEPLOYMENT-SUMMARY.md +│ └── summary-*.md +├── .github/ +│ └── workflows/ # GitHub Actions +│ ├── sync-to-gitea.yml # Sync GitHub → Gitea +│ └── mirror-check.yml # Check sync status +├── .gitignore # Excludes secrets +├── ARTICLE-gitea-deployment.md # Main article +├── Caddyfile # Caddy config +├── MIRRORING-GUIDE.md # This guide +├── README.md # Project overview +├── REPOSITORY-SETUP.md # Setup guide +├── deploy-to-bigbox.sh # Deployment script +├── docker-compose.yml.template # Template with op:// +└── s3_config.json.template # Template with op:// +``` + +## Verification + +**Check sync status:** +```bash +# Manual check +./check-mirror.sh + +# GitHub Actions will run every 6 hours automatically +``` + +**View on GitHub:** +https://github.com/terraphim/gitea-infrastructure + +**View on Gitea:** +https://git.terraphim.cloud/terraphim/gitea-infrastructure + +## Next Steps + +1. ✅ Create Gitea repository +2. ✅ Add GitHub Actions secret (GITEA_TOKEN) +3. ✅ Push to Gitea +4. ✅ Test mirroring with a commit +5. Document your mirroring strategy for team members + +## Troubleshooting + +**GitHub Actions failing?** +- Check GITEA_TOKEN is set correctly +- Verify Gitea repository exists +- Check Actions logs for details + +**Gitea push mirror failing?** +- Verify GitHub token has `repo` scope +- Check repository URL is correct +- Review Gitea logs for errors + +**Conflicts?** +- Stop both mirrors +- Resolve conflicts locally +- Push to both repositories +- Re-enable mirrors + +--- + +**Your repository is ready for mirroring!** 🎉 diff --git a/STORAGE-ARCHITECTURE.md b/STORAGE-ARCHITECTURE.md new file mode 100644 index 0000000..f1a6451 --- /dev/null +++ b/STORAGE-ARCHITECTURE.md @@ -0,0 +1,336 @@ +# Gitea Storage Architecture: Local vs S3 + +## Overview + +Gitea uses a **hybrid storage model** with both local filesystem and S3 (SeaweedFS) storage. This document explains what is stored where and how to verify the configuration. + +## Storage Types in Gitea + +### 1. Git Repositories (Local Only) + +**Location:** `/data/git/` inside the container +**Storage Type:** Local filesystem only +**S3 Backup:** Possible via external tools + +``` +/data/git/repositories/ +├── terraphim/ +│ └── gitea-infrastructure.git/ +│ ├── HEAD +│ ├── config +│ ├── description +│ ├── hooks/ +│ ├── info/ +│ ├── objects/ +│ └── refs/ +``` + +**Why local?** +- Git operations require fast filesystem access +- Object storage is too slow for git's object database +- S3 is used for auxiliary data only + +**Verify:** +```bash +ssh bigbox +cd ~/gitea-stack +docker compose exec server ls -la /data/git/repositories/ +``` + +### 2. Git LFS (Large File Storage) + +**Current Configuration:** Local storage +**Location:** `/data/lfs/` +**Can be moved to:** S3 (SeaweedFS) + +**Current Setting:** +```ini +[repository] +LFS_START_SERVER = true +LFS_CONTENT_PATH = /data/lfs # Local path +``` + +**Verify:** +```bash +docker compose exec server ls -la /data/lfs/ +``` + +**Status:** ❌ **Not configured for S3** (using local path despite MinIO settings) + +### 3. Attachments & Files (S3 via MinIO) + +**Location:** S3 (SeaweedFS) +**Bucket:** `gitea` +**Endpoint:** `http://100.106.66.7:8333` + +**What's stored in S3:** +- Issue attachments +- Release attachments +- Repository avatars +- User avatars +- Package files +- Actions artifacts + +**Configuration:** +```ini +[storage] +MINIO_USE_SSL = false +MINIO_LOCATION = us-east-1 +MINIO_ENDPOINT = http://100.106.66.7:8333 +MINIO_BUCKET = gitea +MINIO_ACCESS_KEY_ID = [from 1Password] +MINIO_SECRET_ACCESS_KEY = [from 1Password] +type = minio +``` + +**Verify S3 connection:** +```bash +# Check S3 bucket +curl http://100.106.66.7:8333/gitea + +# Or using AWS CLI +aws --endpoint-url http://100.106.66.7:8333 s3 ls s3://gitea/ +``` + +### 4. Container Registry + +**Location:** Local by default +**Can be configured for:** S3 storage + +**Configuration needed:** +```ini +[storage.packages] +type = minio +MINIO_ENDPOINT = http://100.106.66.7:8333 +MINIO_BUCKET = gitea-registry +``` + +**Status:** ⚠️ **Not configured yet** + +## Current Configuration Analysis + +### What's Actually Configured + +| Data Type | Location | S3 Configured | Notes | +|-----------|----------|---------------|-------| +| Git repos | Local (`/data/git/`) | ❌ No | Must be local | +| LFS files | Local (`/data/lfs/`) | ⚠️ Partial | Config in env but LFS path is local | +| Attachments | S3 (planned) | ✅ Yes | Config present | +| Avatars | S3 (planned) | ✅ Yes | Config present | +| Packages | Local | ❌ No | Not configured | +| Registry | Local | ❌ No | Not configured | +| Actions logs | Local | ✅ Yes | Uses storage config | + +### Configuration Issue Found + +**Problem:** The app.ini shows placeholders instead of actual secrets: + +```ini +MINIO_ACCESS_KEY_ID = VAULT_REFTerraphimPlatform/gitea-s3/access-key +``` + +**Should be:** +```ini +MINIO_ACCESS_KEY_ID = actual_access_key_value +``` + +**Impact:** S3 storage won't work until secrets are properly injected. + +## Storage Architecture Diagram + +``` +Gitea Server +│ +├── /data/git/ ← Git repositories (LOCAL) +│ └── repositories/ +│ └── terraphim/ +│ └── gitea-infrastructure.git/ +│ └── [git objects] +│ +├── /data/lfs/ ← LFS files (LOCAL) +│ └── [large files] +│ +├── /data/gitea/attachments/ ← Attachments (LOCAL - not S3!) +│ +└── [S3 via MinIO] ← Object storage (SeaweedFS) + └── Bucket: gitea + └── [attachments, avatars, packages] + ↑ + │ + SeaweedFS S3 + 100.106.66.7:8333 +``` + +## Fixing the Configuration + +### Step 1: Verify Secrets Were Injected + +```bash +ssh bigbox +cd ~/gitea-stack + +# Check if secrets are resolved +grep "MINIO_ACCESS_KEY_ID" docker-compose.yml +# Should show actual value, not op:// +``` + +### Step 2: Re-inject Secrets if Needed + +```bash +source ~/op_zesticai_non_prod.sh +cat docker-compose.yml.template | grep -v '^#' | op inject --out-file docker-compose.yml +cat s3_config.json.template | grep -v '^ *//' | grep -v '_comment' | op inject --out-file s3_config.json +``` + +### Step 3: Restart Gitea + +```bash +docker compose restart server +``` + +### Step 4: Verify S3 Connectivity + +```bash +# Test S3 connection from Gitea container +docker compose exec server wget -qO- http://100.106.66.7:8333/gitea || echo "S3 not accessible" +``` + +## Moving LFS to S3 (Optional) + +To store LFS files in S3 instead of local: + +### Option 1: Update Configuration + +Edit Gitea config in UI: +1. Admin Panel → Configuration → "Repository - LFS" +2. Change storage type to "minio" +3. Set endpoint: `100.106.66.7:8333` +4. Set bucket: `gitea-lfs` +5. Restart Gitea + +### Option 2: Edit app.ini + +```bash +# Backup current config +docker compose exec server cp /data/gitea/conf/app.ini /data/gitea/conf/app.ini.bak + +# Edit config +docker compose exec server sed -i 's|^LFS_CONTENT_PATH.*|# LFS_CONTENT_PATH moved to S3|' /data/gitea/conf/app.ini + +# Add LFS storage section +docker compose exec server sh -c 'cat >> /data/gitea/conf/app.ini << EOF + +[lfs] +STORAGE_TYPE = minio +MINIO_ENDPOINT = 100.106.66.7:8333 +MINIO_BUCKET = gitea-lfs +EOF' + +# Restart +docker compose restart server +``` + +### Step 3: Migrate Existing LFS Files + +```bash +# If you have existing LFS files to migrate: +docker compose exec server gitea admin lfs-migrate +``` + +## Backup Strategy + +### What to Back Up + +**Critical - Must Back Up:** +```bash +# Git repositories +cd ~/gitea-stack && tar czf backup-git-$(date +%Y%m%d).tar.gz gitea/git/ + +# Database +cd ~/gitea-stack && docker compose exec -T db pg_dump -U gitea gitea > backup-db-$(date +%Y%m%d).sql + +# Configuration +cd ~/gitea-stack && tar czf backup-config-$(date +%Y%m%d).tar.gz gitea/gitea/conf/ +``` + +**Optional - Can Be Regenerated:** +- LFS files (if stored in S3, backup S3 bucket) +- Attachments (if stored in S3, backup S3 bucket) +- Cache files + +**S3 Backup:** +```bash +# Backup entire S3 bucket +aws --endpoint-url http://100.106.66.7:8333 s3 sync s3://gitea ./backup-s3/ +``` + +## Monitoring Storage + +### Check Disk Usage + +```bash +# Git repositories +docker compose exec server du -sh /data/git/ + +# LFS files +docker compose exec server du -sh /data/lfs/ + +# Local attachments +docker compose exec server du -sh /data/gitea/attachments/ + +# Everything +docker compose exec server du -sh /data/ +``` + +### Check S3 Usage + +```bash +# List S3 buckets +curl http://100.106.66.7:8333/ + +# Check bucket contents +aws --endpoint-url http://100.106.66.7:8333 s3 ls s3://gitea/ --recursive +``` + +## Summary + +### What's Stored Where? + +| Component | Local Path | S3 Path | Recommendation | +|-----------|------------|---------|----------------| +| **Git repos** | `/data/git/` | N/A | Keep local, backup externally | +| **LFS files** | `/data/lfs/` | Not configured | Configure for S3 if large files | +| **Attachments** | `/data/gitea/attachments/` | `s3://gitea/attachments/` | Move to S3 | +| **Avatars** | `/data/gitea/avatars/` | `s3://gitea/avatars/` | Move to S3 | +| **Packages** | `/data/gitea/packages/` | Not configured | Configure if using registry | +| **Actions logs** | `/data/gitea/actions_log/` | Configured for S3 | Verify working | + +### Current Status + +✅ **Working:** +- Git repositories (local) +- Gitea application (running) +- S3/MinIO endpoint accessible + +⚠️ **Needs Attention:** +- Secrets not properly injected (showing VAULT_REF) +- LFS using local storage +- Attachments may still be local despite config + +❌ **Not Configured:** +- Container registry (not using S3) + +### Next Steps + +1. Fix secret injection in docker-compose.yml +2. Restart Gitea with proper S3 configuration +3. Verify attachments are being stored in S3 +4. (Optional) Configure LFS for S3 +5. (Optional) Configure container registry for S3 + +--- + +**Questions?** Check the Gitea logs: +```bash +docker compose logs server -f | grep -i storage +``` diff --git a/docker-compose.yml.template b/docker-compose.yml.template index 03e90d6..e7385d1 100644 --- a/docker-compose.yml.template +++ b/docker-compose.yml.template @@ -20,6 +20,14 @@ services: environment: - USER_UID=1000 - USER_GID=1000 + # Public URL settings + - GITEA__server__DOMAIN=git.terraphim.cloud + - GITEA__server__ROOT_URL=https://git.terraphim.cloud/ + - GITEA__server__PROTOCOL=http + - GITEA__server__HTTP_PORT=3000 + - GITEA__server__DISABLE_SSH=false + - GITEA__server__SSH_PORT=22 + - GITEA__server__SSH_LISTEN_PORT=22 - GITEA__database__DB_TYPE=postgres - GITEA__database__HOST=db:5432 - GITEA__database__NAME=gitea @@ -33,8 +41,8 @@ services: - GITEA__storage__MINIO_SECRET_ACCESS_KEY=op://TerraphimPlatform/gitea-s3/secret-key - GITEA__storage__MINIO_BUCKET=gitea - GITEA__storage__MINIO_LOCATION=us-east-1 - # Use Tailscale IP for internal S3 access - - GITEA__storage__MINIO_ENDPOINT=http://100.106.66.7:8333 + # Use Docker internal hostname for S3 (containers on same network) + - GITEA__storage__MINIO_ENDPOINT=http://s3storage:8333 - GITEA__storage__MINIO_INSECURE_SKIP_VERIFY=false - GITEA__storage__MINIO_USE_SSL=false - GITEA__storage__SERVE_DIRECT=true diff --git a/docker-compose_kitchen_sink.yml b/docker-compose_kitchen_sink.yml new file mode 100644 index 0000000..77aad17 --- /dev/null +++ b/docker-compose_kitchen_sink.yml @@ -0,0 +1,170 @@ +version: "3" +# FIXME: create default gitea bucket using rclone or aws cli +networks: + gitea: + external: false + +services: + server: + image: gitea/gitea:1.21.7 + container_name: gitea + environment: + - USER_UID=1000 + - USER_GID=1000 + - GITEA__database__DB_TYPE=postgres + - GITEA__database__HOST=db:5432 + - GITEA__database__NAME=gitea + - GITEA__database__USER=gitea + - GITEA__database__PASSWD=gitea + - GITEA__actions__ENABLE=true + - GITEA__repository__LFS_START_SERVER=true + - GITEA__repository__LFS_CONTENT_PATH=/data/lfs + - GITEA__storage__type=minio + - GITEA__storage__MINIO_ACCESS_KEY_ID=REDACTED_S3_ACCESS_KEY + - GITEA__storage__MINIO_SECRET_ACCESS_KEY=REDACTED_S3_SECRET_KEY + - GITEA__storage__MINIO_BUCKET=gitea + - GITEA__storage__MINIO_LOCATION=us-east-1 + - GITEA__storage__MINIO_ENDPOINT=http://s3:8333 + - GITEA__storage__MINIO_INSECURE_SKIP_VERIFY=false + - GITEA__storage__MINIO_USE_SSL=false + - GITEA__storage__SERVE_DIRECT=true + + restart: always + networks: + - gitea + volumes: + - ./gitea:/data + - /etc/timezone:/etc/timezone:ro + - /etc/localtime:/etc/localtime:ro + ports: + - "3000:3000" + - "222:22" + depends_on: + - db + - s3 + + db: + image: postgres:15 + restart: always + environment: + - POSTGRES_USER=gitea + - POSTGRES_PASSWORD=gitea + - POSTGRES_DB=gitea + networks: + - gitea + volumes: + - ./postgres:/var/lib/postgresql/data + master: + image: chrislusf/seaweedfs # use a remote image + ports: + - 9333:9333 + - 19333:19333 + - 9324:9324 + command: "master -ip=master -ip.bind=0.0.0.0 -metricsPort=9324" + # healthcheck: + # test: [ "CMD", "curl", "--fail", "-I", "http://master:9333/cluster/healthz" ] + # interval: 1s + # timeout: 120s + volume: + image: chrislusf/seaweedfs # use a remote image + ports: + - 8080:8080 + - 18080:18080 + - 9325:9325 + command: 'volume -mserver="master:9333" -ip.bind=0.0.0.0 -port=8080 -metricsPort=9325' + depends_on: + - master + # master: + # condition: service_healthy + volumes: + - ./seaweedfs:/data + filer: + image: chrislusf/seaweedfs # use a remote image + ports: + - 8888:8888 + - 18888:18888 + - 9326:9326 + command: 'filer -master="master:9333" -ip.bind=0.0.0.0 -metricsPort=9326' + tty: true + stdin_open: true + depends_on: + - master + - volume + # volume: + # condition: service_healthy + volumes: + - ./seaweedfs_filter:/data + # healthcheck: + # test: [ "CMD", "curl", "--fail", "-I", "http://volume:8888" ] + # interval: 1s + # timeout: 30s + s3: + image: chrislusf/seaweedfs + container_name: s3storage + hostname: s3storage # use a remote image + ports: + - 8333:8333 + - 9327:9327 + command: 's3 -config=/etc/seaweedfs/s3.json -filer="filer:8888" -ip.bind=0.0.0.0 -metricsPort=9327' + depends_on: + - master + - volume + - filer + volumes: + - ./s3_config.json:/etc/seaweedfs/s3.json + nginx-gateway-s3: + image: ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest + container_name: nginx-s3-gateway + ports: + - 9080:80 + restart: on-failure + environment: + S3_BUCKET_NAME: gitea + AWS_ACCESS_KEY_ID: "REDACTED_S3_ACCESS_KEY" + AWS_SECRET_ACCESS_KEY: "REDACTED_S3_SECRET_KEY" + AWS_SIGS_VERSION: 4 + S3_SERVER: s3 + S3_SERVER_PORT: 8333 + S3_SERVER_PROTO: http + S3_REGION: us-east-1 + S3_STYLE: path + DEBUG: false + ALLOW_DIRECTORY_LIST: false + PROVIDE_INDEX_PAGE: true + APPEND_SLASH_FOR_POSSIBLE_DIRECTORY: true + DIRECTORY_LISTING_PATH_PREFIX: "" + PROXY_CACHE_MAX_SIZE: 10g + PROXY_CACHE_SLICE_SIZE: 1m + PROXY_CACHE_INACTIVE: 60m + PROXY_CACHE_VALID_OK: 1h + PROXY_CACHE_VALID_NOTFOUND: 1m + PROXY_CACHE_VALID_FORBIDDEN: 30s + # STRIP_LEADING_DIRECTORY_PATH: /somepath + depends_on: + - s3 + prometheus: + image: prom/prometheus:v2.21.0 + ports: + - 9000:9090 + volumes: + - ./prometheus:/etc/prometheus + command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml + depends_on: + - s3 + # panamax: + # image: panamaxrs/panamax + # container_name: crates + # volumes: + # - ./crates_data/:/mirror + # # entrypoint: ["/usr/local/bin/panamax"] + # # command: "/usr/local/bin/panamax init /mirror && + # # /usr/local/bin/panamax sync /mirror && + # # /usr/local/bin/panamax serve /mirror" + # command: + # - sync /mirror + # # - serve + # ports: + # - 18080:8080 + # environment: + # - USER_UID=1000 + # - USER_GID=1000 \ No newline at end of file diff --git a/prometheus/prometheus.yml b/prometheus/prometheus.yml new file mode 100644 index 0000000..2f0ead4 --- /dev/null +++ b/prometheus/prometheus.yml @@ -0,0 +1,14 @@ +global: + scrape_interval: 30s + scrape_timeout: 10s + +scrape_configs: + - job_name: services + metrics_path: /metrics + static_configs: + - targets: + - 'prometheus:9090' + - 'master:9324' + - 'volume:9325' + - 'filer:9326' + - 's3:9327' \ No newline at end of file diff --git a/s3_config.json.template b/s3_config.json.template index ae14fa7..06c6b80 100644 --- a/s3_config.json.template +++ b/s3_config.json.template @@ -1,10 +1,13 @@ { - "_comment": "Template file - use 'op inject' to generate s3_config.json", "identities": [ { "name": "anonymous", "actions": [ - "Read" + "Admin", + "Read", + "List", + "Write", + "Tagging" ] }, {