CI/CD Integration 🟡 BETA

General Bots uses Forgejo (ALM) as Git server with Forgejo Runner for CI/CD. The runner lives in a separate container (alm-ci) and builds are triggered by pushing to the ALM repository.

Architecture

Deploy flow: Push to ALM → Runner picks up job → cargo build → tar+gzip binary → SSH to system container → extract to /opt/gbo/bin/botserver → restart via systemctl

Status Codes

Database Queries

All queries run against the PROD-ALM database:

sudo incus exec tables -- psql -h localhost -U postgres -d PROD-ALM

List Recent Runs

SELECT id, title, workflow_id, status,
       to_timestamp(created) AS created_at
FROM action_run
ORDER BY id DESC LIMIT 10;

Get Jobs for a Run

SELECT id, name, status, task_id
FROM action_run_job
WHERE run_id = <RUN_ID>;

Get Step-Level Status

SELECT name, status, log_index, log_length
FROM action_task_step
WHERE task_id = <TASK_ID>
ORDER BY index;

Check Runner Token

SELECT * FROM action_runner_token;

List Registered Runners

SELECT * FROM action_runner;

Reset a Stuck Run (status 6)

UPDATE action_task SET status = 0 WHERE id = <ID>;
UPDATE action_run_job SET status = 0 WHERE run_id = <RUN_ID>;
UPDATE action_run SET status = 0 WHERE id = <RUN_ID>;

Reading Build Logs

Build logs are stored as zstd-compressed files in the alm container. The database tracks the filename.

Step-by-Step

# 1. Get log filename from database
sudo incus exec tables -- psql -h localhost -U postgres -d PROD-ALM \
  -c "SELECT log_filename FROM action_task WHERE id = <TASK_ID>;"

# 2. Pull compressed log from alm container
sudo incus file pull alm/opt/gbo/data/data/actions_log/<LOG_FILENAME> /tmp/ci-log.log.zst

# 3. Decompress and read
zstd -d /tmp/ci-log.log.zst -o /tmp/ci-log.log
cat /tmp/ci-log.log

One-Liner: Read Latest Failed Run

TASK_ID=$(sudo incus exec tables -- psql -h localhost -U postgres -d PROD-ALM -t -c \
  "SELECT at.id FROM action_task at JOIN action_run_job arj ON at.job_id = arj.id \
   JOIN action_run ar ON arj.run_id = ar.id \
   WHERE ar.status = 2 ORDER BY at.id DESC LIMIT 1;" | tr -d ' ')
LOG_FILE=$(sudo incus exec tables -- psql -h localhost -U postgres -d PROD-ALM -t -c \
  "SELECT log_filename FROM action_task WHERE id = $TASK_ID;" | tr -d ' ')
sudo incus file pull "alm/opt/gbo/data/data/actions_log/$LOG_FILE" /tmp/ci-log.log.zst
zstd -d /tmp/ci-log.log.zst -o /tmp/ci-log.log 2>/dev/null && cat /tmp/ci-log.log

Real-Time Monitoring

# Tail runner logs (live but ephemeral)
sudo incus exec alm-ci -- tail -f /opt/gbo/logs/forgejo-runner.log

# Watch for new runs
sudo incus exec tables -- psql -h localhost -U postgres -d PROD-ALM \
  -c "SELECT id, title, workflow_id, status FROM action_run ORDER BY id DESC LIMIT 5;"

# Check runner logs for build activity
sudo incus exec alm-ci -- tail -f /opt/gbo/logs/forgejo-runner.log | grep -E "Clone|Build|Deploy|Success|Failure"

Build Timing

Verify Deployment

# Check binary timestamp
sudo incus exec system -- stat -c '%y' /opt/gbo/bin/botserver

# Check running version
sudo incus exec system -- /opt/gbo/bin/botserver --version

# Check systemd status
sudo incus exec system -- systemctl status botserver --no-pager

# Health endpoint
curl -sf https://<system-domain>/api/health && echo "OK" || echo "FAILED"

Runner Configuration

• Binary: /opt/gbo/bin/forgejo-runner
• Config: /opt/gbo/bin/config.yaml
• Systemd: /etc/systemd/system/alm-ci-runner.service
• User: gbuser (uid 1000)
• Workspace: /opt/gbo/data/
• SSH deploy key: /home/gbuser/.ssh/id_ed25519
• Cache: Native Cargo incremental (target/ persistence)
• Cargo cache: /home/gbuser/.cargo/
• Rustup: /home/gbuser/.rustup/

Register New Runner

forgejo-runner register \
  --instance http://<alm-ip>:4747 \
  --token <TOKEN> \
  --name gbo \
  --labels ubuntu-latest:docker://node:20-bookworm \
  --no-interactive

Token from action_runner_token table in PROD-ALM database.

Restart Runner

sudo incus exec alm-ci -- pkill -9 forgejo
sleep 2
sudo incus exec alm-ci -- bash -c 'cd /opt/gbo/bin && nohup ./forgejo-runner daemon --config config.yaml >> /opt/gbo/logs/forgejo-runner.log 2>&1 &'

Troubleshooting

Deploy Workflow

# 1. Push submodules first
cd botserver && git push alm main && git push origin main
cd ../botui && git push alm main && git push origin main
cd ../botlib && git push alm main && git push origin main

# 2. Push main repo
cd .. && git add botserver botui botlib
git commit -m "Update submodules: <description>"
git push alm main && git push origin main

# 3. Wait for CI (~2-6 min)
# Monitor via runner logs or database queries

# 4. Verify deployment
sudo incus exec system -- stat -c '%y' /opt/gbo/bin/botserver
sudo incus exec system -- systemctl status botserver --no-pager
curl -sf https://<system-domain>/api/health