Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 33f967e42e | |||
| 4076cf67b7 | |||
| 80a358e8da | |||
| 46bef04a7d | |||
| 6bf4ed8314 | |||
| 501953c636 |
@@ -1,57 +0,0 @@
|
||||
# tinqs/ci — Roadmap
|
||||
|
||||
## Done (2026-05-22)
|
||||
|
||||
- [x] Create `tinqs/ci` repo on tinqs.com
|
||||
- [x] Composite actions: checkout, setup-go, setup-node, setup-aws
|
||||
- [x] Wire all 6 studio workflows to `tinqs/ci/*@v1`
|
||||
- [x] Alpine + Debian compatibility in all actions
|
||||
- [x] Runner image Dockerfiles: base, go, node, docker, deploy, godot
|
||||
- [x] Architecture doc: `tinqs/internal/architecture/ci-runner-architecture.md`
|
||||
|
||||
## Next: Runner Images
|
||||
|
||||
Build and push purpose-built images to ECR. Each workflow type gets a lean image.
|
||||
|
||||
| Image | ECR name | Contents | Priority |
|
||||
|-------|----------|----------|----------|
|
||||
| base | tinqs-runner-base | Alpine + git + AWS CLI + SSH | High |
|
||||
| go | tinqs-runner-go | base + Go 1.26 | High |
|
||||
| node | tinqs-runner-node | base + Node 22 + pnpm | High |
|
||||
| docker | tinqs-runner-docker | dind + Go + AWS CLI | High |
|
||||
| deploy | tinqs-runner-deploy | base only | Medium |
|
||||
| godot | tinqs-runner-godot | base + headless Godot 4.6 | Low (future) |
|
||||
|
||||
**Blocker:** No Docker on Forge, Lightsail unreachable. Options:
|
||||
1. AWS CodeBuild project to build images from this repo
|
||||
2. SSH to Lightsail when back online
|
||||
3. Self-build: add a workflow here that builds images on the current runner
|
||||
|
||||
## Next: Label Routing
|
||||
|
||||
Once images exist:
|
||||
1. Update ECS task definitions — one per image
|
||||
2. Register runners with matching labels: `go`, `node`, `docker`, `deploy`
|
||||
3. Update studio workflows: `runs-on: host` → specific labels
|
||||
4. Lambda dispatcher (future): webhook → Fargate task with correct image
|
||||
|
||||
## Future: Lambda Dispatch + Cancel
|
||||
|
||||
Lambda code already exists in `tinqs-ltd/docs/lambda/` (ci-dispatch + ci-exec).
|
||||
Move to `tinqs/studio/deploy/lambda/`, deploy with SAM.
|
||||
|
||||
- DynamoDB table for run tracking (`tinqs-ci-runs`)
|
||||
- Cancel via `ecs:StopTask` on webhook or timeout
|
||||
- EventBridge cron for stale task cleanup
|
||||
- Deploy-only jobs run directly in Lambda (no Fargate needed)
|
||||
|
||||
See: `tinqs/internal/architecture/ci-runner-architecture.md`
|
||||
|
||||
## Future: More Actions
|
||||
|
||||
| Action | Purpose |
|
||||
|--------|---------|
|
||||
| `tinqs/ci/setup-python` | Python + pip/uv for ML pipelines |
|
||||
| `tinqs/ci/deploy-ecs` | ECS update-service wrapper with wait |
|
||||
| `tinqs/ci/deploy-s3` | S3 sync + CloudFront invalidation |
|
||||
| `tinqs/ci/notify` | Post build status to Lobster GChat |
|
||||
@@ -1,57 +1,44 @@
|
||||
# tinqs/ci
|
||||
|
||||
Tinqs CI toolchain — composite Gitea Actions and purpose-built runner images.
|
||||
CI toolchain for Tinqs Studio — composite Gitea Actions and a Lambda dispatcher that orchestrates ephemeral EC2 Spot runners.
|
||||
|
||||
## Actions
|
||||
> ⚠️ **This repo must stay public.** `act_runner` (go-git) clones action repos without auth; every other tinqs repo is private. If this repo goes private, every `uses: tinqs/ci/...` step breaks.
|
||||
|
||||
| Action | What it does |
|
||||
|--------|-------------|
|
||||
| `tinqs/ci/checkout@v1` | Clone a repo from tinqs.com |
|
||||
| `tinqs/ci/setup-go@v1` | Install Go (skips if pre-baked) |
|
||||
| `tinqs/ci/setup-node@v1` | Install Node.js + pnpm (skips if pre-baked) |
|
||||
| `tinqs/ci/setup-aws@v1` | Install AWS CLI + optional ECR login |
|
||||
```
|
||||
Push → Gitea webhook → Lambda (tinqs-ci-dispatch) → EC2 Spot → act_runner → job → self-terminate
|
||||
```
|
||||
|
||||
## Using the actions
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
- uses: tinqs/ci/checkout@v1
|
||||
with:
|
||||
sparse: 'cmd/tstudio'
|
||||
- uses: tinqs/ci/setup-go@v1
|
||||
- uses: tinqs/ci/setup-aws@v1
|
||||
with:
|
||||
ecr-login: 'true'
|
||||
ecr-repo: '149751500842.dkr.ecr.eu-west-1.amazonaws.com/tinqs-git'
|
||||
```
|
||||
|
||||
## Runner Images
|
||||
| Action | What it does |
|
||||
|--------|-------------|
|
||||
| `tinqs/ci/checkout@v1` | Clone a repo from tinqs.com (sparse, depth, token auth) |
|
||||
| `tinqs/ci/setup-go@v1` | Install Go (skips if pre-baked in AMI) |
|
||||
| `tinqs/ci/setup-node@v1` | Install Node.js + pnpm (skips if pre-baked) |
|
||||
| `tinqs/ci/setup-aws@v1` | Install AWS CLI + optional ECR login |
|
||||
|
||||
Each workflow type gets a lean, purpose-built runner image. No fat "everything" image.
|
||||
## Layout
|
||||
|
||||
| Image | `runs-on` | Contents | Used by |
|
||||
|-------|-----------|----------|---------|
|
||||
| **base** | — | Alpine + git + AWS CLI + SSH | Foundation for all others |
|
||||
| **go** | `go` | base + Go 1.26 | build-tstudio, deploy-proxy |
|
||||
| **node** | `node` | base + Node 22 + pnpm | deploy-docs |
|
||||
| **docker** | `docker` | docker:dind + Go + AWS CLI | build (platform), deploy-bot |
|
||||
| **deploy** | `deploy` | base only (lightest) | deploy-arikigame, release |
|
||||
| **godot** | `godot` | base + headless Godot 4.6 + export templates | ariki-game builds (future) |
|
||||
- `checkout/`, `setup-go/`, `setup-node/`, `setup-aws/` — composite actions
|
||||
- `orchestrator/dispatch/` — the dispatcher Lambda (`main.go`)
|
||||
- `images/` — runner image Dockerfiles
|
||||
|
||||
### Build all images
|
||||
## 📖 Full docs → [`wiki/`](wiki/README.md)
|
||||
|
||||
```bash
|
||||
cd images
|
||||
./build-all.sh v1
|
||||
```
|
||||
The team wiki lives in **[`wiki/`](wiki/README.md)** (plain markdown, rendered by Gitea):
|
||||
|
||||
Images push to ECR as `tinqs-runner-{name}:v1`.
|
||||
|
||||
### How routing works
|
||||
|
||||
The Lambda dispatcher reads `runs-on:` from the workflow and starts a Fargate task with the matching image:
|
||||
|
||||
```
|
||||
runs-on: go → tinqs-runner-go (Go builds)
|
||||
runs-on: node → tinqs-runner-node (frontend builds)
|
||||
runs-on: docker → tinqs-runner-docker (Docker image builds)
|
||||
runs-on: deploy → tinqs-runner-deploy (S3/ECS deploys)
|
||||
runs-on: godot → tinqs-runner-godot (game exports)
|
||||
runs-on: host → legacy runner (forge-runner, temporary)
|
||||
```
|
||||
- [Architecture](wiki/Architecture.md) — design, dispatcher, labels, runner lifecycle
|
||||
- [DevOps Reference](wiki/DevOps-Reference.md) — AWS resources, webhook flow, Spot lifecycle, env vars, cost
|
||||
- [Operations](wiki/Operations.md) — deploy the dispatcher, template deploy, rotate tokens, AMI, monitoring, incidents
|
||||
- [Roadmap](wiki/Roadmap.md) — done / next
|
||||
|
||||
+24
-3
@@ -1,11 +1,11 @@
|
||||
# tinqs/ci/checkout — Tinqs Studio CI
|
||||
# Clones a repo from tinqs.com (self-hosted Gitea) using plain git.
|
||||
# Supports depth control, branch/tag selection, and token auth for private repos.
|
||||
# Supports depth control, branch/tag selection, token auth, and sparse checkout.
|
||||
# Composite action — runs directly on the host, no Node.js runtime needed.
|
||||
# Author: Ozan + Claude Code — 2026-05-22
|
||||
|
||||
name: 'Tinqs Checkout'
|
||||
description: 'Clone a Gitea repository (replaces actions/checkout)'
|
||||
description: 'Clone a Gitea repository'
|
||||
|
||||
inputs:
|
||||
repository:
|
||||
@@ -23,6 +23,9 @@ inputs:
|
||||
token:
|
||||
description: 'Gitea access token (for private repos)'
|
||||
default: ''
|
||||
sparse:
|
||||
description: 'Space-separated paths for sparse checkout (empty = full clone)'
|
||||
default: ''
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
@@ -33,6 +36,7 @@ runs:
|
||||
DEPTH="${{ inputs.depth }}"
|
||||
TARGET="${{ inputs.path }}"
|
||||
TOKEN="${{ inputs.token }}"
|
||||
SPARSE="${{ inputs.sparse }}"
|
||||
|
||||
if [ "$DEPTH" = "0" ]; then
|
||||
DEPTH_FLAG=""
|
||||
@@ -46,11 +50,28 @@ runs:
|
||||
URL="https://tinqs.com/${REPO}.git"
|
||||
fi
|
||||
|
||||
if [ -n "$SPARSE" ]; then
|
||||
# Sparse checkout — only fetch specified paths
|
||||
mkdir -p "$TARGET" && cd "$TARGET"
|
||||
git init
|
||||
git remote add origin "$URL"
|
||||
git config core.sparseCheckout true
|
||||
for P in $SPARSE; do
|
||||
echo "$P" >> .git/info/sparse-checkout
|
||||
done
|
||||
# Also always include go.mod/go.sum for Go builds
|
||||
echo "go.mod" >> .git/info/sparse-checkout
|
||||
echo "go.sum" >> .git/info/sparse-checkout
|
||||
git fetch $DEPTH_FLAG origin "$REF"
|
||||
git checkout FETCH_HEAD
|
||||
echo "Sparse checkout ${REPO}@${REF} ($(echo $SPARSE | wc -w) paths)"
|
||||
else
|
||||
# Full clone
|
||||
if [ "$TARGET" = "." ]; then
|
||||
git clone $DEPTH_FLAG --branch "$REF" "$URL" .
|
||||
else
|
||||
git clone $DEPTH_FLAG --branch "$REF" "$URL" "$TARGET"
|
||||
fi
|
||||
|
||||
echo "Checked out ${REPO}@${REF}"
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
@@ -1,45 +0,0 @@
|
||||
.PHONY: build build-dispatch build-exec tidy test-local deploy deploy-guided clean
|
||||
|
||||
REGION ?= eu-west-1
|
||||
|
||||
build: build-dispatch build-exec
|
||||
|
||||
build-dispatch:
|
||||
cd dispatch && GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap .
|
||||
cd dispatch && zip -j function.zip bootstrap && rm bootstrap
|
||||
|
||||
build-exec:
|
||||
cd exec && GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap .
|
||||
cd exec && zip -j function.zip bootstrap && rm bootstrap
|
||||
|
||||
tidy:
|
||||
cd dispatch && go mod tidy
|
||||
cd exec && go mod tidy
|
||||
|
||||
test-local:
|
||||
cd deploy && sam local invoke DispatchFunction \
|
||||
--template-file template.yaml \
|
||||
--event ../test-event.json \
|
||||
--region $(REGION)
|
||||
|
||||
deploy:
|
||||
cd deploy && sam build --template-file template.yaml && \
|
||||
sam deploy \
|
||||
--template-file .aws-sam/build/template.yaml \
|
||||
--stack-name tinqs-ci \
|
||||
--region $(REGION) \
|
||||
--capabilities CAPABILITY_IAM \
|
||||
--parameter-overrides \
|
||||
GiteaToken=$${GITEA_TOKEN} \
|
||||
Subnets=$${SUBNETS} \
|
||||
SecurityGroup=$${SECURITY_GROUP} \
|
||||
--resolve-s3 \
|
||||
--no-confirm-changeset
|
||||
|
||||
deploy-guided:
|
||||
cd deploy && sam build --template-file template.yaml && \
|
||||
sam deploy --guided --template-file .aws-sam/build/template.yaml --stack-name tinqs-ci
|
||||
|
||||
clean:
|
||||
rm -f dispatch/bootstrap dispatch/function.zip
|
||||
rm -f exec/bootstrap exec/function.zip
|
||||
@@ -1,56 +0,0 @@
|
||||
# tinqs/ci orchestrator
|
||||
|
||||
Lambda-based CI dispatcher for Tinqs Studio. Receives Gitea webhooks and routes jobs to the right execution environment.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Gitea push webhook
|
||||
│
|
||||
▼
|
||||
API Gateway POST /webhook
|
||||
│
|
||||
▼
|
||||
ci-dispatch Lambda
|
||||
│
|
||||
├── runs-on: go/node/docker/godot
|
||||
│ → Start Fargate task with matching image
|
||||
│ → Track in DynamoDB for cancel
|
||||
│
|
||||
├── runs-on: deploy
|
||||
│ → Invoke ci-exec Lambda directly
|
||||
│
|
||||
└── runs-on: host
|
||||
→ Skip (handled by registered runner)
|
||||
```
|
||||
|
||||
## Deploy
|
||||
|
||||
Requires: AWS SAM CLI, AWS credentials, Gitea token.
|
||||
|
||||
```bash
|
||||
# First time (interactive)
|
||||
GITEA_TOKEN=xxx make deploy-guided
|
||||
|
||||
# Subsequent deploys
|
||||
GITEA_TOKEN=xxx SUBNETS=subnet-abc,subnet-def SECURITY_GROUP=sg-xxx make deploy
|
||||
```
|
||||
|
||||
After deploy, configure the webhook URL as a Gitea **system webhook**:
|
||||
- URL: `https://<api-id>.execute-api.eu-west-1.amazonaws.com/prod/webhook`
|
||||
- Method: POST
|
||||
- Content type: application/json
|
||||
- Events: Push, Workflow Job (for cancel)
|
||||
|
||||
## Cancel support
|
||||
|
||||
When a user cancels a job in the Gitea UI, the `workflow_job` webhook fires with `action: cancelled`. The dispatcher looks up the Fargate task ARN in DynamoDB and calls `ecs:StopTask`.
|
||||
|
||||
A cleanup cron (every 5 min) also kills Fargate tasks that have been running longer than 30 minutes.
|
||||
|
||||
## Local testing
|
||||
|
||||
```bash
|
||||
make build
|
||||
make test-local # requires SAM CLI + Docker
|
||||
```
|
||||
@@ -1,177 +0,0 @@
|
||||
AWSTemplateFormatVersion: '2010-09-09'
|
||||
Transform: AWS::Serverless-2016-10-31
|
||||
Description: Tinqs CI Orchestrator — Lambda dispatch + EC2 Spot runners
|
||||
|
||||
Parameters:
|
||||
GiteaURL:
|
||||
Type: String
|
||||
Default: https://tinqs.com
|
||||
GiteaToken:
|
||||
Type: String
|
||||
NoEcho: true
|
||||
RunnerAMI:
|
||||
Type: AWS::EC2::Image::Id
|
||||
Description: Pre-baked AMI with Go, Node, Docker, AWS CLI, act_runner
|
||||
Subnet:
|
||||
Type: AWS::EC2::Subnet::Id
|
||||
Description: Public subnet for spot instances
|
||||
SecurityGroup:
|
||||
Type: AWS::EC2::SecurityGroup::Id
|
||||
Description: Security group for spot instances
|
||||
InstanceProfileArn:
|
||||
Type: String
|
||||
Description: IAM instance profile ARN for runners
|
||||
|
||||
Globals:
|
||||
Function:
|
||||
Runtime: provided.al2023
|
||||
Architectures: [x86_64]
|
||||
Timeout: 30
|
||||
MemorySize: 128
|
||||
|
||||
Resources:
|
||||
|
||||
# --- API Gateway ---
|
||||
WebhookApi:
|
||||
Type: AWS::Serverless::Api
|
||||
Properties:
|
||||
Name: tinqs-ci-webhook
|
||||
StageName: prod
|
||||
|
||||
# --- DynamoDB ---
|
||||
RunsTable:
|
||||
Type: AWS::DynamoDB::Table
|
||||
Properties:
|
||||
TableName: tinqs-ci-runs
|
||||
BillingMode: PAY_PER_REQUEST
|
||||
AttributeDefinitions:
|
||||
- AttributeName: repo
|
||||
AttributeType: S
|
||||
- AttributeName: run_id
|
||||
AttributeType: S
|
||||
KeySchema:
|
||||
- AttributeName: repo
|
||||
KeyType: HASH
|
||||
- AttributeName: run_id
|
||||
KeyType: RANGE
|
||||
TimeToLiveSpecification:
|
||||
AttributeName: ttl
|
||||
Enabled: true
|
||||
|
||||
# --- Dispatcher Lambda ---
|
||||
DispatchFunction:
|
||||
Type: AWS::Serverless::Function
|
||||
Properties:
|
||||
FunctionName: tinqs-ci-dispatch
|
||||
Handler: bootstrap
|
||||
CodeUri: ../dispatch/
|
||||
Description: Receives webhook, starts Spot instances or invokes Lambda executor
|
||||
Timeout: 60
|
||||
MemorySize: 256
|
||||
Environment:
|
||||
Variables:
|
||||
GITEA_URL: !Ref GiteaURL
|
||||
GITEA_TOKEN: !Ref GiteaToken
|
||||
EXECUTOR_FUNCTION_NAME: !Ref ExecFunction
|
||||
RUNNER_AMI: !Ref RunnerAMI
|
||||
SUBNET: !Ref Subnet
|
||||
SECURITY_GROUP: !Ref SecurityGroup
|
||||
DDB_TABLE: !Ref RunsTable
|
||||
INSTANCE_PROFILE: !Ref InstanceProfileArn
|
||||
Policies:
|
||||
- LambdaInvokePolicy:
|
||||
FunctionName: !Ref ExecFunction
|
||||
- DynamoDBCrudPolicy:
|
||||
TableName: !Ref RunsTable
|
||||
- Version: '2012-10-17'
|
||||
Statement:
|
||||
- Effect: Allow
|
||||
Action:
|
||||
- ec2:RunInstances
|
||||
- ec2:TerminateInstances
|
||||
- ec2:DescribeInstances
|
||||
- ec2:CreateTags
|
||||
Resource: '*'
|
||||
- Effect: Allow
|
||||
Action: iam:PassRole
|
||||
Resource: !Ref InstanceProfileArn
|
||||
Events:
|
||||
Webhook:
|
||||
Type: Api
|
||||
Properties:
|
||||
RestApiId: !Ref WebhookApi
|
||||
Path: /webhook
|
||||
Method: POST
|
||||
|
||||
# --- Executor Lambda (deploy-only jobs) ---
|
||||
ExecFunction:
|
||||
Type: AWS::Serverless::Function
|
||||
Properties:
|
||||
FunctionName: tinqs-ci-exec
|
||||
Handler: bootstrap
|
||||
CodeUri: ../exec/
|
||||
Description: Executes deploy-only workflow steps directly in Lambda
|
||||
Timeout: 900
|
||||
MemorySize: 2048
|
||||
EphemeralStorage:
|
||||
Size: 5120
|
||||
Environment:
|
||||
Variables:
|
||||
GITEA_URL: !Ref GiteaURL
|
||||
GITEA_TOKEN: !Ref GiteaToken
|
||||
Policies:
|
||||
- S3CrudPolicy:
|
||||
BucketName: tinqs-cli-releases
|
||||
- S3CrudPolicy:
|
||||
BucketName: arikigame.com
|
||||
- S3CrudPolicy:
|
||||
BucketName: docs.tinqs.com
|
||||
- Version: '2012-10-17'
|
||||
Statement:
|
||||
- Effect: Allow
|
||||
Action:
|
||||
- ecs:UpdateService
|
||||
- ecs:DescribeServices
|
||||
Resource: !Sub 'arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:service/tinqs-git/*'
|
||||
- Effect: Allow
|
||||
Action:
|
||||
- cloudfront:CreateInvalidation
|
||||
Resource: '*'
|
||||
|
||||
# --- Cleanup Cron (every 5 min) ---
|
||||
CleanupRule:
|
||||
Type: AWS::Events::Rule
|
||||
Properties:
|
||||
Name: tinqs-ci-cleanup
|
||||
ScheduleExpression: 'rate(5 minutes)'
|
||||
State: ENABLED
|
||||
Targets:
|
||||
- Id: cleanup
|
||||
Arn: !GetAtt DispatchFunction.Arn
|
||||
Input: '{"body":"{\"action\":\"cleanup\"}","headers":{}}'
|
||||
|
||||
CleanupPermission:
|
||||
Type: AWS::Lambda::Permission
|
||||
Properties:
|
||||
FunctionName: !Ref DispatchFunction
|
||||
Action: lambda:InvokeFunction
|
||||
Principal: events.amazonaws.com
|
||||
SourceArn: !GetAtt CleanupRule.Arn
|
||||
|
||||
# --- Log Group ---
|
||||
CILogGroup:
|
||||
Type: AWS::Logs::LogGroup
|
||||
Properties:
|
||||
LogGroupName: /tinqs/ci
|
||||
RetentionInDays: 14
|
||||
|
||||
Outputs:
|
||||
WebhookURL:
|
||||
Description: Configure as Gitea system webhook
|
||||
Value: !Sub 'https://${WebhookApi}.execute-api.${AWS::Region}.amazonaws.com/prod/webhook'
|
||||
DispatchArn:
|
||||
Value: !GetAtt DispatchFunction.Arn
|
||||
ExecArn:
|
||||
Value: !GetAtt ExecFunction.Arn
|
||||
RunsTable:
|
||||
Value: !Ref RunsTable
|
||||
@@ -0,0 +1,3 @@
|
||||
/bootstrap
|
||||
/function.zip
|
||||
/dispatch.exe
|
||||
@@ -4,7 +4,7 @@ go 1.23
|
||||
|
||||
require (
|
||||
github.com/aws/aws-lambda-go v1.47.0
|
||||
github.com/aws/aws-sdk-go-v2 v1.32.5
|
||||
github.com/aws/aws-sdk-go-v2 v1.32.6
|
||||
github.com/aws/aws-sdk-go-v2/config v1.28.0
|
||||
github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.15.12
|
||||
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.38.0
|
||||
@@ -12,3 +12,21 @@ require (
|
||||
github.com/aws/aws-sdk-go-v2/service/lambda v1.69.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.17.41 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.17 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.25 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.25 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/dynamodbstreams v1.24.2 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.1 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.6 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.6 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.24.2 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.2 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.32.2 // indirect
|
||||
github.com/aws/smithy-go v1.22.1 // indirect
|
||||
github.com/jmespath/go-jmespath v0.4.0 // indirect
|
||||
)
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
github.com/aws/aws-lambda-go v1.47.0 h1:0H8s0vumYx/YKs4sE7YM0ktwL2eWse+kfopsRI1sXVI=
|
||||
github.com/aws/aws-lambda-go v1.47.0/go.mod h1:dpMpZgvWx5vuQJfBt0zqBha60q7Dd7RfgJv23DymV8A=
|
||||
github.com/aws/aws-sdk-go-v2 v1.32.6 h1:7BokKRgRPuGmKkFMhEg/jSul+tB9VvXhcViILtfG8b4=
|
||||
github.com/aws/aws-sdk-go-v2 v1.32.6/go.mod h1:P5WJBrYqqbWVaOxgH0X/FYYD47/nooaPOZPlQdmiN2U=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.7 h1:lL7IfaFzngfx0ZwUGOZdsFFnQ5uLvR0hWqqhyE7Q9M8=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.7/go.mod h1:QraP0UcVlQJsmHfioCrveWOC1nbiWUl3ej08h4mXWoc=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.28.0 h1:FosVYWcqEtWNxHn8gB/Vs6jOlNwSoyOCA/g/sxyySOQ=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.28.0/go.mod h1:pYhbtvg1siOOg8h5an77rXle9tVG8T+BWLWAo7cOukc=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.17.41 h1:7gXo+Axmp+R4Z+AK8YFQO0ZV3L0gizGINCOWxSLY9W8=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.17.41/go.mod h1:u4Eb8d3394YLubphT4jLEwN1rLNq2wFOlT6OuxFwPzU=
|
||||
github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.15.12 h1:zYf8E8zaqolHA5nQ+VmX2r3wc4K6xw5i6xKvvMjZBL0=
|
||||
github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.15.12/go.mod h1:vYGIVLASk19Gb0FGwAcwES+qQF/aekD7m2G/X6mBOdQ=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.17 h1:TMH3f/SCAWdNtXXVPPu5D6wrr4G5hI1rAxbcocKfC7Q=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.17/go.mod h1:1ZRXLdTpzdJb9fwTMXiLipENRxkGMTn1sfKexGllQCw=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.25 h1:s/fF4+yDQDoElYhfIVvSNyeCydfbuTKzhxSXDXCPasU=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.25/go.mod h1:IgPfDv5jqFIzQSNbUEMoitNooSMXjRSDkhXv8jiROvU=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.25 h1:ZntTCl5EsYnhN/IygQEUugpdwbhdkom9uHcbCftiGgA=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.25/go.mod h1:DBdPrgeocww+CSl1C8cEV8PN1mHMBhuCDLpXezyvWkE=
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 h1:VaRN3TlFdd6KxX1x3ILT5ynH6HvKgqdiXoTxAF4HQcQ=
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1/go.mod h1:FbtygfRFze9usAadmnGJNc8KsP346kEe+y2/oyhGAGc=
|
||||
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.38.0 h1:isKhHsjpQR3CypQJ4G1g8QWx7zNpiC/xKw1zjgJYVno=
|
||||
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.38.0/go.mod h1:xDvUyIkwBwNtVZJdHEwAuhFly3mezwdEWkbJ5oNYwIw=
|
||||
github.com/aws/aws-sdk-go-v2/service/dynamodbstreams v1.24.2 h1:E7Tuo0ipWpBl0f3uThz8cZsuyD5H8jLCnbtbKR4YL2s=
|
||||
github.com/aws/aws-sdk-go-v2/service/dynamodbstreams v1.24.2/go.mod h1:txOfweuNPBLhHodsV+C2lvPPRTommVTWbts9SZV6Myc=
|
||||
github.com/aws/aws-sdk-go-v2/service/ec2 v1.198.0 h1:ivPJXmGlzAjgy0jLO9naExUWE8IM8lLRcRKLPBEx6Q0=
|
||||
github.com/aws/aws-sdk-go-v2/service/ec2 v1.198.0/go.mod h1:00zqVNJFK6UASrTnuvjJHJuaqUdkVz5tW8Ip+VhzuNg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.1 h1:iXtILhvDxB6kPvEXgsDhGaZCSC6LQET5ZHSdJozeI0Y=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.1/go.mod h1:9nu0fVANtYiAePIBh2/pFUSwtJ402hLnp854CNoDOeE=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.6 h1:nbmKXZzXPJn41CcD4HsHsGWqvKjLKz9kWu6XxvLmf1s=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.6/go.mod h1:SJhcisfKfAawsdNQoZMBEjg+vyN2lH6rO6fP+T94z5Y=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.6 h1:50+XsN70RS7dwJ2CkVNXzj7U2L1HKP8nqTd3XWEXBN4=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.6/go.mod h1:WqgLmwY7so32kG01zD8CPTJWVWM+TzJoOVHwTg4aPug=
|
||||
github.com/aws/aws-sdk-go-v2/service/lambda v1.69.0 h1:BXt75frE/FYtAmEDBJRBa2HexOw+oAZWZl6QknZEFgg=
|
||||
github.com/aws/aws-sdk-go-v2/service/lambda v1.69.0/go.mod h1:guz2K3x4FKSdDaoeB+TPVgJNU9oj2gftbp5cR8ela1A=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.24.2 h1:bSYXVyUzoTHoKalBmwaZxs97HU9DWWI3ehHSAMa7xOk=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.24.2/go.mod h1:skMqY7JElusiOUjMJMOv1jJsP7YUg7DrhgqZZWuzu1U=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.2 h1:AhmO1fHINP9vFYUE0LHzCWg/LfUWUF+zFPEcY9QXb7o=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.2/go.mod h1:o8aQygT2+MVP0NaV6kbdE1YnnIM8RRVQzoeUH45GOdI=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.32.2 h1:CiS7i0+FUe+/YY1GvIBLLrR/XNGZ4CtM1Ll0XavNuVo=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.32.2/go.mod h1:HtaiBI8CjYoNVde8arShXb94UbQQi9L4EMr6D+xGBwo=
|
||||
github.com/aws/smithy-go v1.22.1 h1:/HPHZQ0g7f4eUeK6HKglFz8uwVfZKgoI25rb/J+dnro=
|
||||
github.com/aws/smithy-go v1.22.1/go.mod h1:irrKGvNn1InZwb2d7fkIRNucdfwR8R+Ts3wxYa/cJHg=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
|
||||
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
|
||||
github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
|
||||
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.7.2 h1:4jaiDzPyXQvSd7D0EjG45355tLlV3VOECpq10pLC+8s=
|
||||
github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
|
||||
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -31,7 +31,6 @@ import (
|
||||
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
|
||||
"github.com/aws/aws-sdk-go-v2/service/ec2"
|
||||
ec2types "github.com/aws/aws-sdk-go-v2/service/ec2/types"
|
||||
awslambda "github.com/aws/aws-sdk-go-v2/service/lambda"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
@@ -100,38 +99,36 @@ type spotConfig struct {
|
||||
}
|
||||
|
||||
var labelToSpot = map[string]spotConfig{
|
||||
"go": {InstanceType: "t3.small", MaxPrice: "0.008"},
|
||||
"node": {InstanceType: "t3.small", MaxPrice: "0.008"},
|
||||
"docker": {InstanceType: "t3.medium", MaxPrice: "0.016"},
|
||||
"deploy": {InstanceType: "t3.micro", MaxPrice: "0.004"},
|
||||
"godot": {InstanceType: "t3.medium", MaxPrice: "0.016"},
|
||||
"go": {InstanceType: "t3.small", MaxPrice: "0.02"},
|
||||
"node": {InstanceType: "t3.medium", MaxPrice: "0.04"},
|
||||
"docker": {InstanceType: "t3.medium", MaxPrice: "0.04"},
|
||||
"deploy": {InstanceType: "t3.micro", MaxPrice: "0.01"},
|
||||
"godot": {InstanceType: "t3.medium", MaxPrice: "0.04"},
|
||||
}
|
||||
|
||||
// --- Config from env ---
|
||||
|
||||
type cfg struct {
|
||||
GiteaURL string
|
||||
GiteaToken string
|
||||
ExecFnName string
|
||||
GiteaToken string // API token (for fetching workflows, setting commit status)
|
||||
RunnerToken string // Runner registration token (for act_runner register)
|
||||
AMI string // pre-baked AMI with Go, Node, Docker, AWS CLI, act_runner
|
||||
Subnet string
|
||||
SecurityGroup string
|
||||
DDBTable string
|
||||
InstanceProfile string
|
||||
Region string
|
||||
}
|
||||
|
||||
func loadCfg() cfg {
|
||||
return cfg{
|
||||
GiteaURL: os.Getenv("GITEA_URL"),
|
||||
GiteaToken: os.Getenv("GITEA_TOKEN"),
|
||||
ExecFnName: os.Getenv("EXECUTOR_FUNCTION_NAME"),
|
||||
RunnerToken: os.Getenv("RUNNER_TOKEN"),
|
||||
AMI: os.Getenv("RUNNER_AMI"),
|
||||
Subnet: os.Getenv("SUBNET"),
|
||||
SecurityGroup: os.Getenv("SECURITY_GROUP"),
|
||||
DDBTable: os.Getenv("DDB_TABLE"),
|
||||
InstanceProfile: os.Getenv("INSTANCE_PROFILE"), // IAM instance profile ARN
|
||||
Region: os.Getenv("AWS_REGION"),
|
||||
InstanceProfile: os.Getenv("INSTANCE_PROFILE"),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -144,31 +141,61 @@ set -euo pipefail
|
||||
exec > /var/log/tinqs-ci.log 2>&1
|
||||
|
||||
echo "=== Tinqs CI Runner: %s ==="
|
||||
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
|
||||
REGION=%s
|
||||
|
||||
# Self-termination trap — kill instance on exit (success or failure)
|
||||
cleanup() {
|
||||
echo "=== Job done, terminating $INSTANCE_ID ==="
|
||||
aws ec2 terminate-instances --instance-ids "$INSTANCE_ID" --region "$REGION" || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
# Get instance metadata (IMDSv2)
|
||||
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
|
||||
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
|
||||
REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/region)
|
||||
|
||||
# act_runner is pre-installed in the AMI at /usr/local/bin/act_runner
|
||||
cd /tmp
|
||||
# Self-termination: shutdown triggers terminate (InstanceInitiatedShutdownBehavior=terminate)
|
||||
# No IAM needed — kernel handles it
|
||||
trap 'echo "=== Job done, shutting down ===" && shutdown -h now' EXIT
|
||||
|
||||
export PATH=$PATH:/usr/local/go/bin:/usr/local/bin
|
||||
export HOME=/root
|
||||
|
||||
# Git auth for private repo clones (checkout action uses git binary which reads this)
|
||||
# Note: act_runner's internal action resolution uses go-git, which does NOT read
|
||||
# gitconfig — so tinqs/ci must stay public for act_runner to clone actions.
|
||||
GITEA_TOKEN="%s"
|
||||
git config --global url."https://token:${GITEA_TOKEN}@tinqs.com/".insteadOf "https://tinqs.com/"
|
||||
|
||||
# Create proper working directory for act_runner
|
||||
mkdir -p /opt/runner && cd /opt/runner
|
||||
|
||||
# Register as ephemeral runner (picks one job, then exits)
|
||||
act_runner register --no-interactive \
|
||||
--instance %s \
|
||||
--token %s \
|
||||
--name %s \
|
||||
--labels "%s:host"
|
||||
--labels "%s:host" \
|
||||
--ephemeral
|
||||
|
||||
# Run — blocks until the job completes, then exits (ephemeral mode)
|
||||
act_runner daemon
|
||||
# Configure runner
|
||||
cat > .runner.yaml << 'RUNCFG'
|
||||
log:
|
||||
level: info
|
||||
runner:
|
||||
capacity: 1
|
||||
timeout: 30m
|
||||
envs:
|
||||
HOME: /root
|
||||
AWS_DEFAULT_REGION: eu-west-1
|
||||
PATH: /usr/local/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
|
||||
host:
|
||||
workdir_parent: /opt/runner/work
|
||||
RUNCFG
|
||||
|
||||
mkdir -p /opt/runner/work
|
||||
|
||||
# Run — blocks until the job completes, then exits (ephemeral = one job only)
|
||||
act_runner daemon --config .runner.yaml
|
||||
|
||||
# Safety: if daemon exits without triggering trap (shouldn't happen), force shutdown
|
||||
shutdown -h now
|
||||
|
||||
echo "=== Runner exited, cleanup will terminate instance ==="
|
||||
`, runnerName, c.Region, c.GiteaURL, c.GiteaToken, runnerName, label)
|
||||
`, runnerName, c.GiteaToken, c.GiteaURL, c.RunnerToken, runnerName, label)
|
||||
|
||||
return base64.StdEncoding.EncodeToString([]byte(script))
|
||||
}
|
||||
@@ -236,15 +263,14 @@ func handler(ctx context.Context, request events.APIGatewayProxyRequest) (events
|
||||
runID := fmt.Sprintf("%s-%s-%d", push.After[:7], name, time.Now().UnixMilli())
|
||||
|
||||
switch label {
|
||||
case "deploy":
|
||||
if err := invokeLambdaExec(ctx, c, push, name, content); err != nil {
|
||||
log.Printf("Failed to invoke executor for %s: %v", name, err)
|
||||
continue
|
||||
}
|
||||
case "host":
|
||||
log.Printf("Skipping %s (runs-on: host — registered runner)", name)
|
||||
continue
|
||||
default:
|
||||
// All non-host jobs (go/node/docker/deploy/godot) go through Spot.
|
||||
// The old `deploy` → Lambda-executor path was retired when
|
||||
// tinqs-ci-exec was deleted (26 May 2026); deploy now launches
|
||||
// a t3.micro Spot runner labelled `deploy` like any other job.
|
||||
instanceID, err := startSpotRunner(ctx, c, label, runID)
|
||||
if err != nil {
|
||||
log.Printf("Failed to start spot for %s: %v", name, err)
|
||||
@@ -276,11 +302,23 @@ func startSpotRunner(ctx context.Context, c cfg, label, runID string) (string, e
|
||||
return "", fmt.Errorf("unknown label: %s", label)
|
||||
}
|
||||
|
||||
runnerName := fmt.Sprintf("spot-%s-%s", label, runID[:12])
|
||||
// runID = "<sha7>-<workflow>-<ms>" already encodes everything unique.
|
||||
// The old runID[:12] truncation collapsed every deploy of a commit to the
|
||||
// same name (e.g. deploy-arikigame + release on one push), confusing task
|
||||
// routing and same-commit reruns. Use the full runID, sanitised for the
|
||||
// runner-name charset (no dots/slashes from workflow filenames).
|
||||
safeID := strings.NewReplacer(".", "-", "/", "-").Replace(runID)
|
||||
runnerName := fmt.Sprintf("spot-%s-%s", label, safeID)
|
||||
userData := userDataScript(c, label, runnerName)
|
||||
|
||||
client := ec2.NewFromConfig(awsCfg)
|
||||
|
||||
// Docker builds need more disk (monorepo + layers)
|
||||
diskSize := int32(20)
|
||||
if label == "docker" {
|
||||
diskSize = 40
|
||||
}
|
||||
|
||||
out, err := client.RunInstances(ctx, &ec2.RunInstancesInput{
|
||||
ImageId: aws.String(c.AMI),
|
||||
InstanceType: ec2types.InstanceType(spot.InstanceType),
|
||||
@@ -288,6 +326,16 @@ func startSpotRunner(ctx context.Context, c cfg, label, runID string) (string, e
|
||||
MaxCount: aws.Int32(1),
|
||||
UserData: aws.String(userData),
|
||||
|
||||
// Root volume — bigger for Docker builds
|
||||
BlockDeviceMappings: []ec2types.BlockDeviceMapping{{
|
||||
DeviceName: aws.String("/dev/xvda"),
|
||||
Ebs: &ec2types.EbsBlockDevice{
|
||||
VolumeSize: aws.Int32(diskSize),
|
||||
VolumeType: ec2types.VolumeTypeGp3,
|
||||
DeleteOnTermination: aws.Bool(true),
|
||||
},
|
||||
}},
|
||||
|
||||
// Spot request
|
||||
InstanceMarketOptions: &ec2types.InstanceMarketOptionsRequest{
|
||||
MarketType: ec2types.MarketTypeSpot,
|
||||
@@ -339,39 +387,6 @@ func startSpotRunner(ctx context.Context, c cfg, label, runID string) (string, e
|
||||
return instanceID, nil
|
||||
}
|
||||
|
||||
// --- Lambda executor (for deploy-only jobs) ---
|
||||
|
||||
func invokeLambdaExec(ctx context.Context, c cfg, push GiteaPushEvent, workflow, content string) error {
|
||||
if c.ExecFnName == "" {
|
||||
log.Printf("[DRY RUN] Would invoke executor for %s", workflow)
|
||||
return nil
|
||||
}
|
||||
|
||||
awsCfg, err := config.LoadDefaultConfig(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
payload, _ := json.Marshal(map[string]interface{}{
|
||||
"repo": push.Repository,
|
||||
"ref": push.Ref,
|
||||
"commit_sha": push.After,
|
||||
"pusher": push.Pusher.Login,
|
||||
"workflow_name": workflow,
|
||||
"workflow_yaml": content,
|
||||
"gitea_url": c.GiteaURL,
|
||||
"gitea_token": c.GiteaToken,
|
||||
})
|
||||
|
||||
client := awslambda.NewFromConfig(awsCfg)
|
||||
_, err = client.Invoke(ctx, &awslambda.InvokeInput{
|
||||
FunctionName: aws.String(c.ExecFnName),
|
||||
InvocationType: "Event",
|
||||
Payload: payload,
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
// --- Cancel handler ---
|
||||
|
||||
func handleCancel(ctx context.Context, c cfg, body string) (events.APIGatewayProxyResponse, error) {
|
||||
|
||||
@@ -1,8 +0,0 @@
|
||||
module tinqs.com/tinqs/ci/orchestrator/exec
|
||||
|
||||
go 1.23
|
||||
|
||||
require (
|
||||
github.com/aws/aws-lambda-go v1.47.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
@@ -1,233 +0,0 @@
|
||||
// tinqs/ci orchestrator — executor Lambda
|
||||
//
|
||||
// Handles deploy-only jobs directly in Lambda (no Fargate needed).
|
||||
// Parses workflow YAML, runs shell steps, reports commit status to Gitea.
|
||||
// 15 min timeout, 2 GB memory, 5 GB /tmp.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-lambda-go/lambda"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
type ExecPayload struct {
|
||||
Repo GiteaRepo `json:"repo"`
|
||||
Ref string `json:"ref"`
|
||||
CommitSHA string `json:"commit_sha"`
|
||||
Pusher string `json:"pusher"`
|
||||
Workflow string `json:"workflow_name"`
|
||||
WorkflowYAML string `json:"workflow_yaml"`
|
||||
GiteaURL string `json:"gitea_url"`
|
||||
GiteaToken string `json:"gitea_token"`
|
||||
}
|
||||
|
||||
type GiteaRepo struct {
|
||||
FullName string `json:"full_name"`
|
||||
CloneURL string `json:"clone_url"`
|
||||
}
|
||||
|
||||
type Workflow struct {
|
||||
Name string `yaml:"name"`
|
||||
Jobs map[string]WorkflowJob `yaml:"jobs"`
|
||||
}
|
||||
|
||||
type WorkflowJob struct {
|
||||
Steps []WorkflowStep `yaml:"steps"`
|
||||
Env map[string]string `yaml:"env"`
|
||||
}
|
||||
|
||||
type WorkflowStep struct {
|
||||
Name string `yaml:"name"`
|
||||
Run string `yaml:"run"`
|
||||
Uses string `yaml:"uses"`
|
||||
Env map[string]string `yaml:"env"`
|
||||
}
|
||||
|
||||
type StepResult struct {
|
||||
Name string `json:"name"`
|
||||
Status string `json:"status"`
|
||||
Duration string `json:"duration"`
|
||||
}
|
||||
|
||||
type JobResult struct {
|
||||
Workflow string `json:"workflow"`
|
||||
Status string `json:"status"`
|
||||
Steps []StepResult `json:"steps"`
|
||||
Duration string `json:"duration"`
|
||||
}
|
||||
|
||||
func handler(ctx context.Context, payload ExecPayload) (JobResult, error) {
|
||||
sha := payload.CommitSHA
|
||||
if len(sha) > 7 {
|
||||
sha = sha[:7]
|
||||
}
|
||||
log.Printf("Executor: repo=%s workflow=%s sha=%s", payload.Repo.FullName, payload.Workflow, sha)
|
||||
|
||||
start := time.Now()
|
||||
|
||||
var wf Workflow
|
||||
if err := yaml.Unmarshal([]byte(payload.WorkflowYAML), &wf); err != nil {
|
||||
return JobResult{Workflow: payload.Workflow, Status: "failure"}, err
|
||||
}
|
||||
|
||||
setCommitStatus(payload, "pending", "CI running...")
|
||||
|
||||
overallStatus := "success"
|
||||
var steps []StepResult
|
||||
|
||||
for _, job := range wf.Jobs {
|
||||
// Build environment
|
||||
env := os.Environ()
|
||||
env = append(env,
|
||||
"CI=true",
|
||||
fmt.Sprintf("GITHUB_REPOSITORY=%s", payload.Repo.FullName),
|
||||
fmt.Sprintf("GITHUB_REF=%s", payload.Ref),
|
||||
fmt.Sprintf("GITHUB_SHA=%s", payload.CommitSHA),
|
||||
)
|
||||
for k, v := range job.Env {
|
||||
env = append(env, fmt.Sprintf("%s=%s", k, v))
|
||||
}
|
||||
|
||||
workdir := "/tmp/workspace"
|
||||
os.MkdirAll(workdir, 0755)
|
||||
os.RemoveAll(workdir)
|
||||
os.MkdirAll(workdir, 0755)
|
||||
|
||||
for i, step := range job.Steps {
|
||||
stepStart := time.Now()
|
||||
name := step.Name
|
||||
if name == "" {
|
||||
name = fmt.Sprintf("Step %d", i+1)
|
||||
}
|
||||
|
||||
// Handle checkout action
|
||||
if step.Uses != "" && contains(step.Uses, "checkout") {
|
||||
cloneURL := payload.Repo.CloneURL
|
||||
if payload.GiteaToken != "" {
|
||||
cloneURL = fmt.Sprintf("https://token:%s@%s",
|
||||
payload.GiteaToken, cloneURL[8:]) // strip https://
|
||||
}
|
||||
branch := payload.Ref
|
||||
if len(branch) > 11 && branch[:11] == "refs/heads/" {
|
||||
branch = branch[11:]
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, "git", "clone", "--depth=1", "--branch", branch, cloneURL, workdir)
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
log.Printf("Checkout failed: %s", string(out))
|
||||
steps = append(steps, StepResult{Name: name, Status: "failure", Duration: since(stepStart)})
|
||||
overallStatus = "failure"
|
||||
break
|
||||
}
|
||||
steps = append(steps, StepResult{Name: name, Status: "success", Duration: since(stepStart)})
|
||||
continue
|
||||
}
|
||||
|
||||
// Skip unknown actions
|
||||
if step.Uses != "" {
|
||||
steps = append(steps, StepResult{Name: name, Status: "skipped", Duration: since(stepStart)})
|
||||
continue
|
||||
}
|
||||
|
||||
if step.Run == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
// Run shell step
|
||||
stepEnv := make([]string, len(env))
|
||||
copy(stepEnv, env)
|
||||
for k, v := range step.Env {
|
||||
stepEnv = append(stepEnv, fmt.Sprintf("%s=%s", k, v))
|
||||
}
|
||||
|
||||
cmd := exec.CommandContext(ctx, "bash", "-c", step.Run)
|
||||
cmd.Dir = workdir
|
||||
cmd.Env = stepEnv
|
||||
|
||||
var output bytes.Buffer
|
||||
cmd.Stdout = io.MultiWriter(&output, os.Stdout)
|
||||
cmd.Stderr = io.MultiWriter(&output, os.Stderr)
|
||||
|
||||
if err := cmd.Run(); err != nil {
|
||||
log.Printf("Step %d FAILED: %v", i+1, err)
|
||||
steps = append(steps, StepResult{Name: name, Status: "failure", Duration: since(stepStart)})
|
||||
overallStatus = "failure"
|
||||
break
|
||||
}
|
||||
|
||||
steps = append(steps, StepResult{Name: name, Status: "success", Duration: since(stepStart)})
|
||||
}
|
||||
|
||||
if overallStatus == "failure" {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
desc := fmt.Sprintf("CI %s in %s", overallStatus, time.Since(start).Round(time.Second))
|
||||
setCommitStatus(payload, overallStatus, desc)
|
||||
|
||||
return JobResult{
|
||||
Workflow: payload.Workflow,
|
||||
Status: overallStatus,
|
||||
Steps: steps,
|
||||
Duration: time.Since(start).Round(time.Second).String(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func setCommitStatus(payload ExecPayload, state, description string) {
|
||||
if payload.GiteaURL == "" || payload.GiteaToken == "" {
|
||||
return
|
||||
}
|
||||
|
||||
url := fmt.Sprintf("%s/api/v1/repos/%s/statuses/%s",
|
||||
payload.GiteaURL, payload.Repo.FullName, payload.CommitSHA)
|
||||
|
||||
body, _ := json.Marshal(map[string]string{
|
||||
"state": state,
|
||||
"description": description,
|
||||
"context": fmt.Sprintf("ci/%s", payload.Workflow),
|
||||
})
|
||||
|
||||
req, _ := http.NewRequest("POST", url, bytes.NewReader(body))
|
||||
req.Header.Set("Authorization", "token "+payload.GiteaToken)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
resp, err := (&http.Client{Timeout: 10 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
log.Printf("Failed to set commit status: %v", err)
|
||||
return
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
|
||||
func contains(s, sub string) bool {
|
||||
return len(s) >= len(sub) && (s == sub || len(s) > 0 && containsStr(s, sub))
|
||||
}
|
||||
|
||||
func containsStr(s, sub string) bool {
|
||||
for i := 0; i <= len(s)-len(sub); i++ {
|
||||
if s[i:i+len(sub)] == sub {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func since(t time.Time) string {
|
||||
return time.Since(t).Round(time.Millisecond).String()
|
||||
}
|
||||
|
||||
func main() {
|
||||
lambda.Start(handler)
|
||||
}
|
||||
@@ -1,9 +0,0 @@
|
||||
{
|
||||
"body": "{\"ref\":\"refs/heads/main\",\"before\":\"0000000\",\"after\":\"abc1234567890\",\"repository\":{\"id\":1,\"name\":\"studio\",\"full_name\":\"tinqs/studio\",\"clone_url\":\"https://tinqs.com/tinqs/studio.git\",\"html_url\":\"https://tinqs.com/tinqs/studio\"},\"pusher\":{\"login\":\"ozan\",\"email\":\"ozan@tinqs.com\"},\"commits\":[{\"id\":\"abc1234567890\",\"message\":\"test push\",\"added\":[\"cmd/tstudio/main.go\"],\"removed\":[],\"modified\":[]}]}",
|
||||
"headers": {
|
||||
"x-gitea-event": "push",
|
||||
"content-type": "application/json"
|
||||
},
|
||||
"httpMethod": "POST",
|
||||
"path": "/webhook"
|
||||
}
|
||||
@@ -1,8 +1,9 @@
|
||||
# tinqs/ci/setup-aws — Tinqs Studio CI
|
||||
# Installs AWS CLI and optionally logs into ECR.
|
||||
# Detects Alpine (musl) vs Debian (glibc) and uses the right install method.
|
||||
# Credentials come from the runner environment (IAM task role on Fargate, instance
|
||||
# profile on EC2/Lightsail) — no explicit key configuration needed.
|
||||
# Detects Alpine (apk/pip), Debian (apt-get), and Amazon Linux/RHEL (dnf).
|
||||
# Skips install if AWS CLI is already present (pre-baked AMI).
|
||||
# Credentials come from the runner environment (IAM instance profile on EC2,
|
||||
# task role on Fargate) — no explicit key configuration needed.
|
||||
# Composite action — runs directly on the host.
|
||||
# Author: Ozan + Claude Code — 2026-05-22
|
||||
|
||||
@@ -35,9 +36,8 @@ runs:
|
||||
# Alpine — use pip (AWS CLI v2 binary needs glibc)
|
||||
apk add --no-cache python3 py3-pip
|
||||
pip3 install --break-system-packages awscli 2>/dev/null || pip3 install awscli
|
||||
elif command -v apt-get &>/dev/null; then
|
||||
# Debian/Ubuntu — use official installer
|
||||
apt-get update && apt-get install -y unzip curl
|
||||
elif command -v dnf &>/dev/null || command -v apt-get &>/dev/null; then
|
||||
# Amazon Linux / Debian — official installer (glibc available)
|
||||
curl -fsSL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o /tmp/awscli.zip
|
||||
unzip -q /tmp/awscli.zip -d /tmp/aws-install
|
||||
/tmp/aws-install/aws/install
|
||||
|
||||
+14
-3
@@ -1,7 +1,7 @@
|
||||
# tinqs/ci/setup-node — Tinqs Studio CI
|
||||
# Installs Node.js and optionally pnpm.
|
||||
# Detects Alpine vs Debian and uses the right package manager.
|
||||
# Skips install if the correct major version is already present (pre-baked runner image).
|
||||
# Detects Alpine (apk), Debian (apt-get), and Amazon Linux/RHEL (dnf).
|
||||
# Skips install if the correct major version is already present (pre-baked AMI).
|
||||
# Composite action — runs directly on the host.
|
||||
# Author: Ozan + Claude Code — 2026-05-22
|
||||
|
||||
@@ -29,17 +29,28 @@ runs:
|
||||
if [ "$CURRENT" = "$NODE_VERSION" ]; then
|
||||
echo "Node $NODE_VERSION already installed"
|
||||
node --version
|
||||
if [ "$INSTALL_PNPM" = "true" ] && command -v pnpm &>/dev/null; then
|
||||
if [ "$INSTALL_PNPM" = "true" ]; then
|
||||
if command -v pnpm &>/dev/null; then
|
||||
pnpm --version
|
||||
exit 0
|
||||
else
|
||||
npm install -g pnpm
|
||||
pnpm --version
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "Installing Node.js $NODE_VERSION..."
|
||||
if command -v apk &>/dev/null; then
|
||||
# Alpine
|
||||
apk add --no-cache nodejs npm
|
||||
elif command -v dnf &>/dev/null; then
|
||||
# Amazon Linux / RHEL / Fedora
|
||||
curl -fsSL "https://rpm.nodesource.com/setup_${NODE_VERSION}.x" | bash -
|
||||
dnf install -y nodejs
|
||||
elif command -v apt-get &>/dev/null; then
|
||||
# Debian/Ubuntu
|
||||
curl -fsSL "https://deb.nodesource.com/setup_${NODE_VERSION}.x" | bash -
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# Architecture
|
||||
|
||||
[← Home](README.md) · [DevOps Reference](DevOps-Reference.md) · [Operations](Operations.md) · [Roadmap](Roadmap.md)
|
||||
|
||||
```
|
||||
Push → Gitea webhook → Lambda (tinqs-ci-dispatch) → EC2 Spot → act_runner → job → self-terminate
|
||||
```
|
||||
|
||||
Runners are **ephemeral**: one Spot instance per job, self-terminating on completion. Private-repo clones are authenticated via `git config url.insteadOf` injected in the runner user-data.
|
||||
|
||||
## Key design decisions
|
||||
|
||||
- **Ephemeral Spot instances** (not Fargate, not persistent runners) — cheapest, cleanest, no state to manage.
|
||||
- **`--ephemeral` on `act_runner register`** — the runner exits after one job, triggering `shutdown -h now` → the instance terminates. Without this, runners pile up as zombies (see the 25 May 2026 incident in [Operations](Operations.md)).
|
||||
- **No local action cache** — `act_runner` uses go-git internally, which ignores `~/.gitconfig`. The `url.insteadOf` trick only works for the `git` binary (used by the `checkout` action), so action repos are cloned fresh each run. This is why `tinqs/ci` must stay public.
|
||||
- **`tinqs.com`** — Gitea's `ROOT_URL` is `tinqs.com`. The old `git.tinqs.com` subdomain is retired.
|
||||
|
||||
## Composite actions
|
||||
|
||||
Bash-only composite actions (no Node.js runtime). Resolve via `@v1` (the main branch).
|
||||
|
||||
| Action | What it does |
|
||||
|--------|-------------|
|
||||
| `tinqs/ci/checkout@v1` | Clone a repo from tinqs.com (sparse checkout, depth control, token auth) |
|
||||
| `tinqs/ci/setup-go@v1` | Install Go (skips if pre-baked in the AMI) |
|
||||
| `tinqs/ci/setup-node@v1` | Install Node.js + pnpm (skips if pre-baked) |
|
||||
| `tinqs/ci/setup-aws@v1` | Install AWS CLI + optional ECR login |
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
- uses: tinqs/ci/checkout@v1
|
||||
with:
|
||||
sparse: 'cmd/tstudio'
|
||||
- uses: tinqs/ci/setup-go@v1
|
||||
- uses: tinqs/ci/setup-aws@v1
|
||||
with:
|
||||
ecr-login: 'true'
|
||||
```
|
||||
|
||||
## Dispatcher (Lambda)
|
||||
|
||||
`orchestrator/dispatch/main.go` receives Gitea push webhooks, fetches `.gitea/workflows/*.yml` via the Gitea API, evaluates triggers (branch + path filters), reads each matched workflow's `runs-on` label, and launches a Spot instance with that label. Run state is tracked in DynamoDB.
|
||||
|
||||
Routing by label (`labelToSpot` map in `main.go`):
|
||||
|
||||
| Label | Instance | Use |
|
||||
|-------|----------|-----|
|
||||
| `go` | t3.small | Go builds (tstudio, proxy, docgen) |
|
||||
| `docker` | t3.medium | Docker image builds (platform, bot) |
|
||||
| `deploy` | t3.micro | S3 sync, CloudFront invalidation, SSM template deploy |
|
||||
| `node` | t3.medium | Frontend builds |
|
||||
| `godot` | t3.medium | Game exports (future) |
|
||||
|
||||
`runs-on: host` is skipped by the dispatcher (it's for a standing registered runner, not Spot).
|
||||
|
||||
> **Fixed 2026-06-07:** `deploy`-labelled jobs used to route to a separate executor Lambda (`tinqs-ci-exec`) that was deleted 26 May, so they silently hit a `[DRY RUN] Would invoke executor` no-op and never ran. They now fall through to the normal Spot path like every other label. A second bug — runner names derived from `runID[:12]` collided across same-commit deploys — was also fixed (names now use the full sanitised runID).
|
||||
|
||||
## Runner lifecycle (user-data)
|
||||
|
||||
```
|
||||
boot → git auth config (url.insteadOf with GITEA_TOKEN)
|
||||
→ act_runner register --ephemeral --labels <label>:host
|
||||
→ act_runner daemon (blocks until job completes)
|
||||
→ EXIT trap → shutdown -h now → instance terminates
|
||||
```
|
||||
|
||||
## Runner images
|
||||
|
||||
Dockerfiles in `images/` — lean, purpose-built. Push to ECR with `images/build-all.sh v1`.
|
||||
|
||||
| Image | Contents |
|
||||
|-------|----------|
|
||||
| `base` | Alpine + git + AWS CLI + SSH |
|
||||
| `go` | base + Go |
|
||||
| `node` | base + Node + pnpm |
|
||||
| `docker` | docker:dind + Go + AWS CLI |
|
||||
| `deploy` | base only (lightest) |
|
||||
| `godot` | base + headless Godot |
|
||||
|
||||
> Note: the live Spot runners boot from a **pre-baked AMI** (`RUNNER_AMI`, with Go/Node/Docker/act_runner installed), not these container images. The images exist for purpose-built runner variants; the AMI is the fast path.
|
||||
@@ -0,0 +1,109 @@
|
||||
# DevOps Reference
|
||||
|
||||
[← Home](README.md) · [Architecture](Architecture.md) · [Operations](Operations.md) · [Roadmap](Roadmap.md)
|
||||
|
||||
## AWS resources (eu-west-1)
|
||||
|
||||
| Resource | Name/ID | Purpose |
|
||||
|----------|---------|---------|
|
||||
| Lambda | `tinqs-ci-dispatch` | Webhook handler + Spot launcher |
|
||||
| DynamoDB | `tinqs-ci-runs` | Run tracking (repo, run_id, instance_id, status) |
|
||||
| AMI | `tinqs-ci-runner-v2` (ami-00a129385002e4de9) | Pre-baked runner (Go, Node, Docker, act_runner) |
|
||||
| Security Group | sg-030bf74b43d3faac7 | Runner SG (outbound HTTPS) |
|
||||
| Subnet | subnet-04b5aeec9bfc4ec2c | Default VPC subnet |
|
||||
| Instance Profile | `tinqs-ci-runner` → role `tinqs-git-task` | Runner IAM role (S3, ECR, SSM) |
|
||||
| CloudWatch | /aws/lambda/tinqs-ci-dispatch | Dispatcher logs |
|
||||
| API Gateway | `q4ohxovfr8…/webhook` | Receives the per-repo Gitea push webhook |
|
||||
|
||||
### Platform host (NOT CI — context)
|
||||
|
||||
| Resource | Name/ID | Purpose |
|
||||
|----------|---------|---------|
|
||||
| EC2 | `tinqs-prod-gitea` (i-0d085288f467083e0, t3.medium) | Runs tinqs.com as a single `docker` Gitea container |
|
||||
| ALB | `tinqs-git` | Fronts the platform |
|
||||
| ECR | `tinqs-git:latest` | Platform image (built by `build.yml` → CodeBuild) |
|
||||
| RDS | `tinqs-prod` (PostgreSQL) | Platform DB |
|
||||
|
||||
The platform mounts host `/data`; `GITEA_CUSTOM=/data/gitea`, so **custom templates live at `/data/gitea/templates/`**. Template-only changes deploy here via SSM — see [Operations](Operations.md).
|
||||
|
||||
### Retired resources
|
||||
|
||||
| Resource | When / why |
|
||||
|----------|------------|
|
||||
| ECS Cluster `tinqs-git` | Deleted **2026-06-05** — platform moved to the `tinqs-prod-gitea` EC2 box |
|
||||
| EFS `tinqs-git-repos` | Retired in the 2026-06-05 EC2 migration (repos now on instance `/data`) |
|
||||
| Lambda `tinqs-ci-exec` | Deleted **26 May 2026** — never ran a build; deploy jobs go through Spot now |
|
||||
| CloudWatch `/aws/lambda/tinqs-ci-exec`, `/ecs/tinqs-runner` | Log groups for the above / the Fargate era |
|
||||
| Fargate runner service | Scaled to 0 then removed |
|
||||
|
||||
## Webhook flow
|
||||
|
||||
```
|
||||
Gitea (tinqs.com)
|
||||
└─ per-repo webhook on push
|
||||
└─ POST https://<api-gw>/webhook
|
||||
└─ Lambda tinqs-ci-dispatch
|
||||
├─ Fetch .gitea/workflows/*.yml via Gitea API
|
||||
├─ Evaluate triggers (branch + path filters)
|
||||
├─ For each matched workflow:
|
||||
│ ├─ Read runs-on label
|
||||
│ └─ RunInstances (Spot, ephemeral) [host → skipped]
|
||||
└─ Track in DynamoDB
|
||||
```
|
||||
|
||||
## Spot instance lifecycle
|
||||
|
||||
```
|
||||
1. Lambda calls RunInstances (Spot, InstanceInitiatedShutdownBehavior=terminate)
|
||||
2. User-data runs:
|
||||
a. Configure git auth (url.insteadOf with GITEA_TOKEN)
|
||||
b. act_runner register --ephemeral --labels <label>:host
|
||||
c. act_runner daemon (blocks until job completes)
|
||||
d. EXIT trap fires → shutdown -h now → instance terminates
|
||||
3. DynamoDB record: running → completed (or timeout after 30 min cleanup)
|
||||
```
|
||||
|
||||
Offline runners listed in Gitea admin → Actions → Runners are **normal** — they're spent ephemeral registrations, not a fault.
|
||||
|
||||
## Cleanup cron
|
||||
|
||||
The dispatcher Lambda also handles cleanup when invoked with an empty body or `{"action":"cleanup"}`. Triggered by EventBridge every 5 minutes.
|
||||
|
||||
- Scans DynamoDB for runs older than 30 min with `status=running`
|
||||
- Terminates matching EC2 instances
|
||||
- Sweeps for orphan instances (tagged `tinqs-ci`, running > 30 min)
|
||||
|
||||
## Lambda env vars
|
||||
|
||||
Configured in the AWS console, not in code:
|
||||
|
||||
| Var | Purpose |
|
||||
|-----|---------|
|
||||
| `GITEA_URL` | `https://tinqs.com` |
|
||||
| `GITEA_TOKEN` | API token — fetches workflows AND provides runner git auth |
|
||||
| `RUNNER_TOKEN` | act_runner registration token (Gitea admin → Runners) |
|
||||
| `RUNNER_AMI` | Pre-baked AMI ID |
|
||||
| `SUBNET` | VPC subnet for Spot instances |
|
||||
| `SECURITY_GROUP` | SG allowing outbound HTTPS |
|
||||
| `DDB_TABLE` | DynamoDB run-tracking table (`tinqs-ci-runs`) |
|
||||
| `INSTANCE_PROFILE` | IAM instance profile for runners |
|
||||
|
||||
## Runner IAM role (`tinqs-git-task`)
|
||||
|
||||
Inline policies of note:
|
||||
|
||||
- `tinqs-ci-s3` — R/W on `tinqs-cli-releases`, `arikigame-com-website`, `docs.tinqs.com` *(corrected 2026-06-07: was the non-existent `arikigame.com`, which broke the arikigame deploy)*
|
||||
- `tinqs-git-s3` — R/W on `tinqs-git-lfs`, `tinqs-git-preview`
|
||||
- `tinqs-ci-deploy` — ECR push, CloudFront `CreateInvalidation`, (legacy ECS update)
|
||||
- `tinqs-ci-ssm-deploy` — `ec2:DescribeInstances` + `ssm:SendCommand` **scoped to the `tinqs-prod-gitea` instance** (added 2026-06-07 for template deploys)
|
||||
- `ssm-exec` — Session Manager channels · `ec2-self-terminate` — terminate own `tinqs-ci`-tagged instance
|
||||
|
||||
## Cost
|
||||
|
||||
| Component | Estimated monthly cost |
|
||||
|-----------|----------------------|
|
||||
| Spot instances (t3.small, ~10 min/build, ~5 builds/day) | ~$1–2 |
|
||||
| Lambda (< 1000 invocations/month) | ~$0 (free tier) |
|
||||
| DynamoDB (< 1 GB, low RCU/WCU) | ~$0 (free tier) |
|
||||
| CloudWatch logs | ~$0.50 |
|
||||
| **Total CI** | **~$2–3/month** |
|
||||
@@ -0,0 +1,105 @@
|
||||
# Operations
|
||||
|
||||
[← Home](README.md) · [Architecture](Architecture.md) · [DevOps Reference](DevOps-Reference.md) · [Roadmap](Roadmap.md)
|
||||
|
||||
## Deploy the dispatcher
|
||||
|
||||
The dispatcher Lambda can't CI itself — deploy manually:
|
||||
|
||||
```bash
|
||||
cd orchestrator/dispatch
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o bootstrap -ldflags "-s -w" .
|
||||
# Windows: powershell -Command "Compress-Archive -Path bootstrap -DestinationPath function.zip -Force"
|
||||
# Mac/Linux: zip -j function.zip bootstrap
|
||||
aws lambda update-function-code --region eu-west-1 \
|
||||
--function-name tinqs-ci-dispatch --zip-file fileb://function.zip
|
||||
# Verify: push a change to cmd/tstudio/ in tinqs/studio and watch the pipeline
|
||||
```
|
||||
|
||||
## Deploy templates to prod (no rebuild)
|
||||
|
||||
Template-only changes don't need a platform rebuild. `tinqs/studio/.gitea/workflows/deploy-templates.yml` (label `deploy`) tars `templates/` → `s3://tinqs-git-lfs/custom-templates.tar.gz`, then over **SSM** tells `tinqs-prod-gitea` to pull + extract into `/data/gitea/templates/` and `docker restart gitea`.
|
||||
|
||||
> Repointed to SSM/EC2 on **2026-06-07**. It previously ran `aws ecs update-service --cluster tinqs-git`, which failed with `ClusterNotFoundException` after the cluster was deleted on 06-05 — that's why the repo Wiki tab and theme CSS never went live. The runner role gained a scoped `ssm:SendCommand` (prod-gitea only).
|
||||
|
||||
Manual one-off (admin creds):
|
||||
|
||||
```bash
|
||||
tar czf /tmp/custom-templates.tar.gz -C templates .
|
||||
aws s3 cp /tmp/custom-templates.tar.gz s3://tinqs-git-lfs/custom-templates.tar.gz --region eu-west-1
|
||||
IID=$(aws ec2 describe-instances --region eu-west-1 \
|
||||
--filters "Name=tag:Name,Values=tinqs-prod-gitea" "Name=instance-state-name,Values=running" \
|
||||
--query "Reservations[0].Instances[0].InstanceId" --output text)
|
||||
aws ssm send-command --region eu-west-1 --instance-ids "$IID" \
|
||||
--document-name AWS-RunShellScript \
|
||||
--parameters 'commands=["aws s3 cp s3://tinqs-git-lfs/custom-templates.tar.gz /tmp/ct.tar.gz --region eu-west-1","tar xzf /tmp/ct.tar.gz -C /data/gitea/templates","docker restart gitea"]'
|
||||
```
|
||||
|
||||
> Note: a template change does **not** bump the platform version string in the footer (that tracks the Go binary build). Unchanged footer ≠ failed deploy.
|
||||
|
||||
## Rotate `GITEA_TOKEN`
|
||||
|
||||
1. Generate a new token in Gitea: Settings → Applications → Generate Token
|
||||
2. `aws lambda update-function-configuration --function-name tinqs-ci-dispatch --environment ...`
|
||||
3. Old token is burned into running instances — they die within 30 min
|
||||
|
||||
## Rotate `RUNNER_TOKEN`
|
||||
|
||||
1. Gitea admin → Actions → Runners → Create new registration token
|
||||
2. Update the Lambda env var
|
||||
3. Running instances keep their existing registration until they die
|
||||
|
||||
## Build a new AMI
|
||||
|
||||
```bash
|
||||
aws ec2 run-instances --image-id ami-00a129385002e4de9 \
|
||||
--instance-type t3.small --key-name <your-key> --region eu-west-1 \
|
||||
--query 'Instances[0].InstanceId'
|
||||
# SSH in, update tools (Go, Node, Docker, act_runner), then:
|
||||
aws ec2 create-image --instance-id <id> --name tinqs-ci-runner-v3
|
||||
aws lambda update-function-configuration --function-name tinqs-ci-dispatch \
|
||||
--environment "Variables={...,RUNNER_AMI=ami-NEW,...}"
|
||||
aws ec2 terminate-instances --instance-id <id>
|
||||
```
|
||||
|
||||
## Add CI to a new repo
|
||||
|
||||
1. Create `.gitea/workflows/<name>.yml` in the repo
|
||||
2. Add a per-repo webhook in Gitea: Settings → Webhooks → Add Webhook
|
||||
- URL: the dispatcher API Gateway URL · Events: Push · Content type: `application/json`
|
||||
3. Push a change matching the workflow trigger
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Zombie check (should be 0 except during active builds)
|
||||
aws ec2 describe-instances --region eu-west-1 \
|
||||
--filters "Name=tag:tinqs-ci,Values=true" "Name=instance-state-name,Values=running" \
|
||||
--query 'Reservations[].Instances[].InstanceId'
|
||||
|
||||
# Dispatcher logs (MSYS_NO_PATHCONV=1 on Windows/Git Bash; or use PowerShell)
|
||||
MSYS_NO_PATHCONV=1 aws logs tail '/aws/lambda/tinqs-ci-dispatch' --region eu-west-1
|
||||
|
||||
# Build/job logs
|
||||
curl -s "https://tinqs.com/api/v1/repos/tinqs/studio/actions/jobs/<JOB_ID>/logs" \
|
||||
-H "Authorization: token <gitea-token>"
|
||||
|
||||
# Stale DynamoDB runs
|
||||
aws dynamodb scan --region eu-west-1 --table-name tinqs-ci-runs \
|
||||
--filter-expression "#s = :r" \
|
||||
--expression-attribute-names '{"#s":"status"}' \
|
||||
--expression-attribute-values '{":r":{"S":"running"}}' --query Count
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
**New composite action:** create `<name>/action.yml` (`using: composite`, `shell: bash`), keep it bash-only, add a `<name>/README.md`, list it in [Architecture](Architecture.md), push to main (resolves via `@v1`).
|
||||
|
||||
**Modify the dispatcher:** edit `orchestrator/dispatch/main.go`, `go build .` to catch errors, deploy manually (above), verify with a push to `tinqs/studio`.
|
||||
|
||||
**New runner label:** add to `labelToSpot` in `main.go`, create `images/<label>/Dockerfile` if needed, build/push (`cd images && ./build-all.sh v1`), deploy the Lambda, add `runs-on: <label>` to the consuming workflow.
|
||||
|
||||
## Incidents
|
||||
|
||||
- **25 May 2026** — 18 zombie runners DDoS-ing Gitea. Root cause: no `--ephemeral` on registration + no git auth after repos went private. Fix: `--ephemeral` + `url.insteadOf` git auth in user-data.
|
||||
- **07 Jun 2026** — all `runs-on: deploy` jobs silently dry-running (dead `tinqs-ci-exec` route) + arikigame IAM bucket mismatch + template deploy pointing at the deleted ECS cluster. All fixed; see [Architecture](Architecture.md) and the template-deploy note above.
|
||||
@@ -0,0 +1,31 @@
|
||||
# tinqs/ci — CI Toolchain
|
||||
|
||||
> **📖 This is the team wiki.** Standard: the in-repo **`wiki/`** folder is the home for team/architecture docs in every repo (distinct from `.agents/` = agent operating context, and `docs/` = public product docs at tinqs.com/docs). Plain markdown, rendered by Gitea — no separate wiki repo, no build. Cross-link with `[Title](Page-Name.md)`.
|
||||
|
||||
**The CI system for Tinqs Studio: composite Gitea Actions + a Lambda dispatcher that launches ephemeral EC2 Spot runners, one per job.** Status baked in — ✅ live · 🔨 built · 📋 planned. Last updated 2026-06-07.
|
||||
|
||||
> ⚠️ **This repo must stay public.** `act_runner` (go-git) clones action repos without auth; every other tinqs repo is private. If `tinqs/ci` goes private, every workflow that does `uses: tinqs/ci/...` breaks.
|
||||
|
||||
```
|
||||
Push → Gitea webhook → Lambda (tinqs-ci-dispatch) → EC2 Spot → act_runner → job → self-terminate
|
||||
```
|
||||
|
||||
## Pages
|
||||
|
||||
- [Architecture](Architecture.md) — design decisions, the dispatcher, runner labels & images, runner lifecycle
|
||||
- [DevOps Reference](DevOps-Reference.md) — AWS resources, webhook flow, Spot lifecycle, cleanup cron, cost, Lambda env vars
|
||||
- [Operations](Operations.md) — deploy the dispatcher, rotate tokens, build an AMI, add CI to a repo, monitoring, incidents
|
||||
- [Roadmap](Roadmap.md) — what's done, what's next
|
||||
|
||||
## Key facts
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Runners** | Ephemeral EC2 Spot, one per job, self-terminate (`--ephemeral` + `shutdown -h now`) |
|
||||
| **Dispatcher** | `tinqs-ci-dispatch` Lambda (`orchestrator/dispatch/main.go`), Go, `provided.al2023` |
|
||||
| **Routing** | Workflow `runs-on` label → Spot instance type (see [Architecture](Architecture.md)) |
|
||||
| **Auth** | `GITEA_TOKEN` injected into runner user-data via `git config url.insteadOf` |
|
||||
| **Region** | eu-west-1 |
|
||||
| **Cost** | ~$2–3/month |
|
||||
|
||||
> **2026-06-05 — platform moved off ECS.** tinqs.com now runs as a single `docker` container on the standalone EC2 box **`tinqs-prod-gitea`** (behind ALB `tinqs-git`, image from ECR `tinqs-git:latest`, state on RDS `tinqs-prod` + local `/data`). The old ECS cluster `tinqs-git` and EFS `tinqs-git-repos` were retired. Any workflow that still referenced ECS (e.g. template deploy) was repointed at the EC2 host via SSM — see [Operations](Operations.md).
|
||||
@@ -0,0 +1,33 @@
|
||||
# Roadmap
|
||||
|
||||
[← Home](README.md) · [Architecture](Architecture.md) · [DevOps Reference](DevOps-Reference.md) · [Operations](Operations.md)
|
||||
|
||||
## Done
|
||||
|
||||
- [x] Composite actions: `checkout`, `setup-go`, `setup-node`, `setup-aws`
|
||||
- [x] Lambda dispatcher with Spot instance routing by `runs-on` label
|
||||
- [x] Ephemeral runners (one job, self-terminate)
|
||||
- [x] Git auth for private repos (`url.insteadOf`)
|
||||
- [x] DynamoDB run tracking + cleanup cron
|
||||
- [x] Runner image Dockerfiles: base, go, node, docker, deploy, godot
|
||||
- [x] Zombie runner incident resolved (25 May 2026)
|
||||
- [x] `deploy`-label jobs routed through Spot (was dead-Lambda dry-run) (07 Jun 2026)
|
||||
- [x] Unique Spot runner names per dispatch (07 Jun 2026)
|
||||
- [x] Template deploy repointed off deleted ECS → EC2 via SSM (07 Jun 2026)
|
||||
|
||||
## Next
|
||||
|
||||
| Priority | Task | Impact |
|
||||
|----------|------|--------|
|
||||
| P1 | Pre-warm Go module + build cache in the AMI | −30s build time |
|
||||
| P1 | Automate AMI build (Packer or script) | Repeatable, no manual SSH |
|
||||
| P2 | Internal DNS for git clones | Faster than public HTTPS |
|
||||
| P2 | CloudWatch agent on the runner AMI | Persistent logs after instance death |
|
||||
| P3 | `tinqs/ci/deploy-s3` action | S3 sync + CloudFront invalidation wrapper |
|
||||
| P3 | `tinqs/ci/deploy-ssm` action | Reusable SSM-to-prod deploy (generalise the template-deploy step) |
|
||||
| P3 | `tinqs/ci/notify` action | Post build status to GChat |
|
||||
|
||||
## Watch / cleanup
|
||||
|
||||
- **Repo size** — `tinqs/studio` now commits the arikigame site assets (~75 MB) as regular files because the CI `checkout` does no `git lfs pull`. If this grows, add `git lfs pull` to the checkout action, then LFS-track `web/arikigame/public/img/**`.
|
||||
- **DEVOPS doc drift** — keep this wiki current when AWS topology changes (the ECS→EC2 move went unnoticed in docs for two days and broke deploys).
|
||||
Reference in New Issue
Block a user