terraphim/gitea-infrastructure
16
0
Fork 0
mirror of https://github.com/terraphim/gitea-infrastructure.git synced 2026-07-16 00:00:32 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

fix: Update S3 config and add Gitea URL settings

Browse Source 
- Change MINIO_ENDPOINT from IP to Docker hostname (s3storage)
- Add ROOT_URL and DOMAIN settings for proper Gitea configuration
- Give anonymous users Admin permission for bucket creation
- Document SeaweedFS bucket creation process
This commit is contained in:
Alex
2026-02-16 20:33:21 +00:00
parent b9e307bcc8
commit d2c28a91e6
6 changed files with 735 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
QUICK-SETUP-MIRRORING.md
+200
View File
@@ -0,0 +1,200 @@
# Quick Setup: Repository Mirroring
Your repository is now on **GitHub** and ready for mirroring!
## Current Status
✅ **GitHub Repository:** https://github.com/terraphim/gitea-infrastructure
❌ **Gitea Repository:** Not created yet
## Setup Steps
### Step 1: Create Repository on Gitea
**Via Web Interface:**
1. Go to https://git.terraphim.cloud
2. Click **"+"** → **"New Repository"**
3. **Owner:** `terraphim`
4. **Repository Name:** `gitea-infrastructure`
5. **Description:** `Gitea deployment configuration`
6. **Visibility:** ☑️ Private
7. ☐ **Uncheck** "Initialize Repository"
8. Click **"Create Repository"**
**Result:** Empty repository created
### Step 2: Push to Gitea
```bash
cd /home/alex/infrastructure/gitea-config
# Push to Gitea
git push origin main
```
### Step 3: Configure GitHub Actions Secrets
**On GitHub:**
1. Go to: https://github.com/terraphim/gitea-infrastructure/settings/secrets/actions
2. Click **"New repository secret"**
3. Add:
- **Name:** `GITEA_TOKEN`
- **Secret:** Your Gitea API token
**To create Gitea API token:**
1. Go to https://git.terraphim.cloud/user/settings/applications
2. Click **"Generate New Token"**
3. **Token Name:** `GitHub Actions Sync`
4. Select scopes: `repo`, `write:repo`
5. Click **"Generate Token"**
6. Copy the token
### Step 4: Test Mirroring
**Push to GitHub:**
```bash
# Make a change
echo "# Mirroring test" >> README.md
git add README.md
git commit -m "test: Mirroring to Gitea"
# Push to GitHub (triggers Actions)
git push github main
```
**Check Gitea:**
- Go to https://git.terraphim.cloud/terraphim/gitea-infrastructure
- Verify the commit appears
## Current Remote Configuration
```bash
# Check remotes
git remote -v
# Should show:
# origin https://git.terraphim.cloud/terraphim/gitea-infrastructure.git (fetch)
# origin https://git.terraphim.cloud/terraphim/gitea-infrastructure.git (push)
# github https://github.com/terraphim/gitea-infrastructure.git (fetch)
# github https://github.com/terraphim/gitea-infrastructure.git (push)
```
## Two-Way Sync Workflow
### Option A: GitHub as Primary (Recommended)
**How it works:**
1. Developer pushes to GitHub
2. GitHub Actions automatically pushes to Gitea
3. Gitea stays in sync as mirror
**Commands:**
```bash
# Work locally
git push github main # Primary push (triggers Actions)
# Or push to both manually
git push github main && git push origin main
```
### Option B: Push to Both
**Set up alias:**
```bash
# Add to ~/.gitconfig
git config --global alias.pushall '!git push github $(git branch --show-current) && git push origin $(git branch --show-current)'
# Usage
git pushall # Pushes to both
```
### Option C: Gitea as Primary
**Configure in Gitea:**
1. Go to repository **Settings** → **Repository**
2. Scroll to **Mirror Settings**
3. Check **"Push Mirror"**
4. **Git Remote URL:** `https://github.com/terraphim/gitea-infrastructure.git`
5. **Authentication:**
- **Username:** Your GitHub username
- **Password:** GitHub Personal Access Token
6. **Interval:** `1h` (or your preference)
7. Click **"Add Push Mirror"**
**To create GitHub token:**
1. Go to https://github.com/settings/tokens
2. Click **"Generate new token (classic)"**
3. Select scope: `repo`
4. Generate and copy token
## Repository Structure
After setup, both repositories will have:
```
gitea-infrastructure/
├── .docs/ # Documentation
│ ├── RESEARCH-gitea-bigbox-deployment.md
│ ├── IMPLEMENTATION-gitea-bigbox-deployment.md
│ ├── DEPLOYMENT-SUMMARY.md
│ └── summary-*.md
├── .github/
│ └── workflows/ # GitHub Actions
│ ├── sync-to-gitea.yml # Sync GitHub → Gitea
│ └── mirror-check.yml # Check sync status
├── .gitignore # Excludes secrets
├── ARTICLE-gitea-deployment.md # Main article
├── Caddyfile # Caddy config
├── MIRRORING-GUIDE.md # This guide
├── README.md # Project overview
├── REPOSITORY-SETUP.md # Setup guide
├── deploy-to-bigbox.sh # Deployment script
├── docker-compose.yml.template # Template with op://
└── s3_config.json.template # Template with op://
```
## Verification
**Check sync status:**
```bash
# Manual check
./check-mirror.sh
# GitHub Actions will run every 6 hours automatically
```
**View on GitHub:**
https://github.com/terraphim/gitea-infrastructure
**View on Gitea:**
https://git.terraphim.cloud/terraphim/gitea-infrastructure
## Next Steps
1. ✅ Create Gitea repository
2. ✅ Add GitHub Actions secret (GITEA_TOKEN)
3. ✅ Push to Gitea
4. ✅ Test mirroring with a commit
5. Document your mirroring strategy for team members
## Troubleshooting
**GitHub Actions failing?**
- Check GITEA_TOKEN is set correctly
- Verify Gitea repository exists
- Check Actions logs for details
**Gitea push mirror failing?**
- Verify GitHub token has `repo` scope
- Check repository URL is correct
- Review Gitea logs for errors
**Conflicts?**
- Stop both mirrors
- Resolve conflicts locally
- Push to both repositories
- Re-enable mirrors
---
**Your repository is ready for mirroring!** 🎉
STORAGE-ARCHITECTURE.md
+336
View File
@@ -0,0 +1,336 @@
# Gitea Storage Architecture: Local vs S3
## Overview
Gitea uses a **hybrid storage model** with both local filesystem and S3 (SeaweedFS) storage. This document explains what is stored where and how to verify the configuration.
## Storage Types in Gitea
### 1. Git Repositories (Local Only)
**Location:** `/data/git/` inside the container
**Storage Type:** Local filesystem only
**S3 Backup:** Possible via external tools
```
/data/git/repositories/
├── terraphim/
│ └── gitea-infrastructure.git/
│ ├── HEAD
│ ├── config
│ ├── description
│ ├── hooks/
│ ├── info/
│ ├── objects/
│ └── refs/
```
**Why local?**
- Git operations require fast filesystem access
- Object storage is too slow for git's object database
- S3 is used for auxiliary data only
**Verify:**
```bash
ssh bigbox
cd ~/gitea-stack
docker compose exec server ls -la /data/git/repositories/
```
### 2. Git LFS (Large File Storage)
**Current Configuration:** Local storage
**Location:** `/data/lfs/`
**Can be moved to:** S3 (SeaweedFS)
**Current Setting:**
```ini
[repository]
LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/lfs # Local path
```
**Verify:**
```bash
docker compose exec server ls -la /data/lfs/
```
**Status:** ❌ **Not configured for S3** (using local path despite MinIO settings)
### 3. Attachments & Files (S3 via MinIO)
**Location:** S3 (SeaweedFS)
**Bucket:** `gitea`
**Endpoint:** `http://100.106.66.7:8333`
**What's stored in S3:**
- Issue attachments
- Release attachments
- Repository avatars
- User avatars
- Package files
- Actions artifacts
**Configuration:**
```ini
[storage]
MINIO_USE_SSL = false
MINIO_LOCATION = us-east-1
MINIO_ENDPOINT = http://100.106.66.7:8333
MINIO_BUCKET = gitea
MINIO_ACCESS_KEY_ID = [from 1Password]
MINIO_SECRET_ACCESS_KEY = [from 1Password]
type = minio
```
**Verify S3 connection:**
```bash
# Check S3 bucket
curl http://100.106.66.7:8333/gitea
# Or using AWS CLI
aws --endpoint-url http://100.106.66.7:8333 s3 ls s3://gitea/
```
### 4. Container Registry
**Location:** Local by default
**Can be configured for:** S3 storage
**Configuration needed:**
```ini
[storage.packages]
type = minio
MINIO_ENDPOINT = http://100.106.66.7:8333
MINIO_BUCKET = gitea-registry
```
**Status:** ⚠️ **Not configured yet**
## Current Configuration Analysis
### What's Actually Configured
| Data Type | Location | S3 Configured | Notes |
|-----------|----------|---------------|-------|
| Git repos | Local (`/data/git/`) | ❌ No | Must be local |
| LFS files | Local (`/data/lfs/`) | ⚠️ Partial | Config in env but LFS path is local |
| Attachments | S3 (planned) | ✅ Yes | Config present |
| Avatars | S3 (planned) | ✅ Yes | Config present |
| Packages | Local | ❌ No | Not configured |
| Registry | Local | ❌ No | Not configured |
| Actions logs | Local | ✅ Yes | Uses storage config |
### Configuration Issue Found
**Problem:** The app.ini shows placeholders instead of actual secrets:
```ini
MINIO_ACCESS_KEY_ID = VAULT_REFTerraphimPlatform/gitea-s3/access-key
```
**Should be:**
```ini
MINIO_ACCESS_KEY_ID = actual_access_key_value
```
**Impact:** S3 storage won't work until secrets are properly injected.
## Storage Architecture Diagram
```
Gitea Server
│
├── /data/git/ ← Git repositories (LOCAL)
│ └── repositories/
│ └── terraphim/
│ └── gitea-infrastructure.git/
│ └── [git objects]
│
├── /data/lfs/ ← LFS files (LOCAL)
│ └── [large files]
│
├── /data/gitea/attachments/ ← Attachments (LOCAL - not S3!)
│
└── [S3 via MinIO] ← Object storage (SeaweedFS)
└── Bucket: gitea
└── [attachments, avatars, packages]
↑
│
SeaweedFS S3
100.106.66.7:8333
```
## Fixing the Configuration
### Step 1: Verify Secrets Were Injected
```bash
ssh bigbox
cd ~/gitea-stack
# Check if secrets are resolved
grep "MINIO_ACCESS_KEY_ID" docker-compose.yml
# Should show actual value, not op://
```
### Step 2: Re-inject Secrets if Needed
```bash
source ~/op_zesticai_non_prod.sh
cat docker-compose.yml.template | grep -v '^#' | op inject --out-file docker-compose.yml
cat s3_config.json.template | grep -v '^ *//' | grep -v '_comment' | op inject --out-file s3_config.json
```
### Step 3: Restart Gitea
```bash
docker compose restart server
```
### Step 4: Verify S3 Connectivity
```bash
# Test S3 connection from Gitea container
docker compose exec server wget -qO- http://100.106.66.7:8333/gitea || echo "S3 not accessible"
```
## Moving LFS to S3 (Optional)
To store LFS files in S3 instead of local:
### Option 1: Update Configuration
Edit Gitea config in UI:
1. Admin Panel → Configuration → "Repository - LFS"
2. Change storage type to "minio"
3. Set endpoint: `100.106.66.7:8333`
4. Set bucket: `gitea-lfs`
5. Restart Gitea
### Option 2: Edit app.ini
```bash
# Backup current config
docker compose exec server cp /data/gitea/conf/app.ini /data/gitea/conf/app.ini.bak
# Edit config
docker compose exec server sed -i 's|^LFS_CONTENT_PATH.*|# LFS_CONTENT_PATH moved to S3|' /data/gitea/conf/app.ini
# Add LFS storage section
docker compose exec server sh -c 'cat >> /data/gitea/conf/app.ini << EOF
[lfs]
STORAGE_TYPE = minio
MINIO_ENDPOINT = 100.106.66.7:8333
MINIO_BUCKET = gitea-lfs
EOF'
# Restart
docker compose restart server
```
### Step 3: Migrate Existing LFS Files
```bash
# If you have existing LFS files to migrate:
docker compose exec server gitea admin lfs-migrate
```
## Backup Strategy
### What to Back Up
**Critical - Must Back Up:**
```bash
# Git repositories
cd ~/gitea-stack && tar czf backup-git-$(date +%Y%m%d).tar.gz gitea/git/
# Database
cd ~/gitea-stack && docker compose exec -T db pg_dump -U gitea gitea > backup-db-$(date +%Y%m%d).sql
# Configuration
cd ~/gitea-stack && tar czf backup-config-$(date +%Y%m%d).tar.gz gitea/gitea/conf/
```
**Optional - Can Be Regenerated:**
- LFS files (if stored in S3, backup S3 bucket)
- Attachments (if stored in S3, backup S3 bucket)
- Cache files
**S3 Backup:**
```bash
# Backup entire S3 bucket
aws --endpoint-url http://100.106.66.7:8333 s3 sync s3://gitea ./backup-s3/
```
## Monitoring Storage
### Check Disk Usage
```bash
# Git repositories
docker compose exec server du -sh /data/git/
# LFS files
docker compose exec server du -sh /data/lfs/
# Local attachments
docker compose exec server du -sh /data/gitea/attachments/
# Everything
docker compose exec server du -sh /data/
```
### Check S3 Usage
```bash
# List S3 buckets
curl http://100.106.66.7:8333/
# Check bucket contents
aws --endpoint-url http://100.106.66.7:8333 s3 ls s3://gitea/ --recursive
```
## Summary
### What's Stored Where?
| Component | Local Path | S3 Path | Recommendation |
|-----------|------------|---------|----------------|
| **Git repos** | `/data/git/` | N/A | Keep local, backup externally |
| **LFS files** | `/data/lfs/` | Not configured | Configure for S3 if large files |
| **Attachments** | `/data/gitea/attachments/` | `s3://gitea/attachments/` | Move to S3 |
| **Avatars** | `/data/gitea/avatars/` | `s3://gitea/avatars/` | Move to S3 |
| **Packages** | `/data/gitea/packages/` | Not configured | Configure if using registry |
| **Actions logs** | `/data/gitea/actions_log/` | Configured for S3 | Verify working |
### Current Status
✅ **Working:**
- Git repositories (local)
- Gitea application (running)
- S3/MinIO endpoint accessible
⚠️ **Needs Attention:**
- Secrets not properly injected (showing VAULT_REF)
- LFS using local storage
- Attachments may still be local despite config
❌ **Not Configured:**
- Container registry (not using S3)
### Next Steps
1. Fix secret injection in docker-compose.yml
2. Restart Gitea with proper S3 configuration
3. Verify attachments are being stored in S3
4. (Optional) Configure LFS for S3
5. (Optional) Configure container registry for S3
---
**Questions?** Check the Gitea logs:
```bash
docker compose logs server -f | grep -i storage
```
docker-compose.yml.template
+10 -2
View File
@@ -20,6 +20,14 @@ services:
environment:
- USER_UID=1000
- USER_GID=1000
# Public URL settings
- GITEA__server__DOMAIN=git.terraphim.cloud
- GITEA__server__ROOT_URL=https://git.terraphim.cloud/
- GITEA__server__PROTOCOL=http
- GITEA__server__HTTP_PORT=3000
- GITEA__server__DISABLE_SSH=false
- GITEA__server__SSH_PORT=22
- GITEA__server__SSH_LISTEN_PORT=22
- GITEA__database__DB_TYPE=postgres
- GITEA__database__HOST=db:5432
- GITEA__database__NAME=gitea
@@ -33,8 +41,8 @@ services:
- GITEA__storage__MINIO_SECRET_ACCESS_KEY=op://TerraphimPlatform/gitea-s3/secret-key
- GITEA__storage__MINIO_BUCKET=gitea
- GITEA__storage__MINIO_LOCATION=us-east-1
# Use Tailscale IP for internal S3 access
- GITEA__storage__MINIO_ENDPOINT=http://100.106.66.7:8333
# Use Docker internal hostname for S3 (containers on same network)
- GITEA__storage__MINIO_ENDPOINT=http://s3storage:8333
- GITEA__storage__MINIO_INSECURE_SKIP_VERIFY=false
- GITEA__storage__MINIO_USE_SSL=false
- GITEA__storage__SERVE_DIRECT=true
docker-compose_kitchen_sink.yml
+170
View File
@@ -0,0 +1,170 @@
version: "3"
# FIXME: create default gitea bucket using rclone or aws cli
networks:
gitea:
external: false
services:
server:
image: gitea/gitea:1.21.7
container_name: gitea
environment:
- USER_UID=1000
- USER_GID=1000
- GITEA__database__DB_TYPE=postgres
- GITEA__database__HOST=db:5432
- GITEA__database__NAME=gitea
- GITEA__database__USER=gitea
- GITEA__database__PASSWD=gitea
- GITEA__actions__ENABLE=true
- GITEA__repository__LFS_START_SERVER=true
- GITEA__repository__LFS_CONTENT_PATH=/data/lfs
- GITEA__storage__type=minio
- GITEA__storage__MINIO_ACCESS_KEY_ID=REDACTED_S3_ACCESS_KEY
- GITEA__storage__MINIO_SECRET_ACCESS_KEY=REDACTED_S3_SECRET_KEY
- GITEA__storage__MINIO_BUCKET=gitea
- GITEA__storage__MINIO_LOCATION=us-east-1
- GITEA__storage__MINIO_ENDPOINT=http://s3:8333
- GITEA__storage__MINIO_INSECURE_SKIP_VERIFY=false
- GITEA__storage__MINIO_USE_SSL=false
- GITEA__storage__SERVE_DIRECT=true
restart: always
networks:
- gitea
volumes:
- ./gitea:/data
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
ports:
- "3000:3000"
- "222:22"
depends_on:
- db
- s3
db:
image: postgres:15
restart: always
environment:
- POSTGRES_USER=gitea
- POSTGRES_PASSWORD=gitea
- POSTGRES_DB=gitea
networks:
- gitea
volumes:
- ./postgres:/var/lib/postgresql/data
master:
image: chrislusf/seaweedfs # use a remote image
ports:
- 9333:9333
- 19333:19333
- 9324:9324
command: "master -ip=master -ip.bind=0.0.0.0 -metricsPort=9324"
# healthcheck:
# test: [ "CMD", "curl", "--fail", "-I", "http://master:9333/cluster/healthz" ]
# interval: 1s
# timeout: 120s
volume:
image: chrislusf/seaweedfs # use a remote image
ports:
- 8080:8080
- 18080:18080
- 9325:9325
command: 'volume -mserver="master:9333" -ip.bind=0.0.0.0 -port=8080 -metricsPort=9325'
depends_on:
- master
# master:
# condition: service_healthy
volumes:
- ./seaweedfs:/data
filer:
image: chrislusf/seaweedfs # use a remote image
ports:
- 8888:8888
- 18888:18888
- 9326:9326
command: 'filer -master="master:9333" -ip.bind=0.0.0.0 -metricsPort=9326'
tty: true
stdin_open: true
depends_on:
- master
- volume
# volume:
# condition: service_healthy
volumes:
- ./seaweedfs_filter:/data
# healthcheck:
# test: [ "CMD", "curl", "--fail", "-I", "http://volume:8888" ]
# interval: 1s
# timeout: 30s
s3:
image: chrislusf/seaweedfs
container_name: s3storage
hostname: s3storage # use a remote image
ports:
- 8333:8333
- 9327:9327
command: 's3 -config=/etc/seaweedfs/s3.json -filer="filer:8888" -ip.bind=0.0.0.0 -metricsPort=9327'
depends_on:
- master
- volume
- filer
volumes:
- ./s3_config.json:/etc/seaweedfs/s3.json
nginx-gateway-s3:
image: ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest
container_name: nginx-s3-gateway
ports:
- 9080:80
restart: on-failure
environment:
S3_BUCKET_NAME: gitea
AWS_ACCESS_KEY_ID: "REDACTED_S3_ACCESS_KEY"
AWS_SECRET_ACCESS_KEY: "REDACTED_S3_SECRET_KEY"
AWS_SIGS_VERSION: 4
S3_SERVER: s3
S3_SERVER_PORT: 8333
S3_SERVER_PROTO: http
S3_REGION: us-east-1
S3_STYLE: path
DEBUG: false
ALLOW_DIRECTORY_LIST: false
PROVIDE_INDEX_PAGE: true
APPEND_SLASH_FOR_POSSIBLE_DIRECTORY: true
DIRECTORY_LISTING_PATH_PREFIX: ""
PROXY_CACHE_MAX_SIZE: 10g
PROXY_CACHE_SLICE_SIZE: 1m
PROXY_CACHE_INACTIVE: 60m
PROXY_CACHE_VALID_OK: 1h
PROXY_CACHE_VALID_NOTFOUND: 1m
PROXY_CACHE_VALID_FORBIDDEN: 30s
# STRIP_LEADING_DIRECTORY_PATH: /somepath
depends_on:
- s3
prometheus:
image: prom/prometheus:v2.21.0
ports:
- 9000:9090
volumes:
- ./prometheus:/etc/prometheus
command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml
depends_on:
- s3
# panamax:
# image: panamaxrs/panamax
# container_name: crates
# volumes:
# - ./crates_data/:/mirror
# # entrypoint: ["/usr/local/bin/panamax"]
# # command: "/usr/local/bin/panamax init /mirror &&
# # /usr/local/bin/panamax sync /mirror &&
# # /usr/local/bin/panamax serve /mirror"
# command:
# - sync /mirror
# # - serve
# ports:
# - 18080:8080
# environment:
# - USER_UID=1000
# - USER_GID=1000
prometheus/prometheus.yml
+14
View File
@@ -0,0 +1,14 @@
global:
scrape_interval: 30s
scrape_timeout: 10s
scrape_configs:
- job_name: services
metrics_path: /metrics
static_configs:
- targets:
- 'prometheus:9090'
- 'master:9324'
- 'volume:9325'
- 'filer:9326'
- 's3:9327'
s3_config.json.template
+5 -2
View File
@@ -1,10 +1,13 @@
{
"_comment": "Template file - use 'op inject' to generate s3_config.json",
"identities": [
{
"name": "anonymous",
"actions": [
"Read"
"Admin",
"Read",
"List",
"Write",
"Tagging"
]
},
{