SkyForge

RUNBOOK.md

# SkyForge runbook

Reference steps for the fictional Initech `checkout-api`. This repository does not run `terraform apply`, does not create AWS resources, and does not contain AWS credentials. The portfolio page shows offline test results recorded at build time.

Two routes share `modules/` and `envs/*.tfvars`:

- `single-ec2/` — one Amazon Linux 2023 host, Docker Compose, no SSH port.
- `ecs-fargate/` — two availability zones, an application load balancer, Fargate, and optional Postgres.

`bootstrap/` is a third root used once, to create the remote state bucket and the lock table.

## What you need before a real plan

- Terraform 1.9.8 (1.7 or newer is required for `terraform test` mocks).
- An AWS account you are allowed to change, and a GitHub OIDC role if you use `.github/workflows/terraform.yml`.
- Replace every placeholder in `envs/dev.tfvars` or `envs/prod.tfvars`: `certificate_arn`, `alb_log_bucket`, `container_image`, and `app_secret`. The account id inside the sample certificate ARN is not an account this project uses.
- Pick your own state bucket name. Do not reuse the `example` names if they are already taken.

No AMI id and no account id is hardcoded in the modules. The Amazon Linux 2023 image comes from `data.aws_ami`. Account-scoped ARNs use `data.aws_caller_identity`.

## 1. Remote state, once

```bash
cd bootstrap
terraform init -backend=false
terraform plan \
  -var='bucket_name=YOUR_GLOBALLY_UNIQUE_BUCKET' \
  -var='lock_table_name=YOUR_LOCK_TABLE'
terraform apply \
  -var='bucket_name=YOUR_GLOBALLY_UNIQUE_BUCKET' \
  -var='lock_table_name=YOUR_LOCK_TABLE'
```

Copy `backend.hcl.example` to `backend.hcl`, fill in the bucket and table, and add this block to the route you are deploying:

```hcl
terraform {
  backend "s3" {}
}
```

Then, from that route:

```bash
terraform init -backend-config=../backend.hcl
```

Until you do that, `terraform init -backend=false` keeps state on the machine that ran the command. The offline checks in this repo use `-backend=false` and never read the example file.

## 2. Image

`docker/Dockerfile` is a multi-stage build. The process runs as uid 10001 and has a `/healthz` check. Build and push with `scripts/push_ecr.sh` after the ECR repository exists. The script is not run by the demo.

```bash
export AWS_REGION=us-east-1
export AWS_ACCOUNT_ID=YOUR_ACCOUNT_ID
export ECR_REPO=checkout-api-dev
./scripts/push_ecr.sh
```

Put the printed URI into `container_image` in the tfvars file.

## 3. Plan and apply a route

GitHub Actions runs `terraform plan` only when someone starts the workflow by hand. The workflow assumes `vars.AWS_DEPLOY_ROLE_ARN` with OIDC. It does not apply.

From a workstation, the same plan is:

```bash
terraform -chdir=ecs-fargate init -backend=false
terraform -chdir=ecs-fargate plan -var-file=../envs/dev.tfvars
terraform -chdir=ecs-fargate apply -var-file=../envs/dev.tfvars
```

Use `single-ec2` instead of `ecs-fargate` for the small route. Use `envs/prod.tfvars` for the larger sizes.

The single-EC2 route opens ports 80 and 443 only. Administration is SSM Session Manager:

```bash
aws ssm start-session --target INSTANCE_ID --region us-east-1
```

There is no security group rule for port 22.

On the Fargate route, port 443 forwards to the tasks and port 80 redirects to 443. Set `enable_nat = false` to skip the NAT gateway. Private tasks then use interface endpoints. Set `enable_rds = true` to add Postgres in the private subnets. The master password is stored by RDS in Secrets Manager. The application secret is the SSM parameter `/${project}/${env}/checkout-api/app_secret`.

## 4. Roll back

Fargate keeps a deployment circuit breaker with rollback enabled. A failed deploy returns the service to the previous task definition.

To roll back by hand:

```bash
aws ecs update-service \
  --cluster checkout-api-ecs-dev \
  --service checkout-api-ecs-dev \
  --task-definition checkout-api-ecs-dev:PREVIOUS \
  --region us-east-1
```

`PREVIOUS` is the revision number shown by `aws ecs list-task-definitions --family checkout-api-ecs-dev --sort DESC`.

On the single-EC2 route, Session Manager onto the host and point Compose at the previous image:

```bash
cd /opt/checkout
# edit the image tag in docker-compose.yml, then:
docker compose up -d
```

If the host itself is bad, restore the latest DLM snapshot onto a new volume and attach it, or run `terraform apply` again with the previous tfvars. Snapshots are tagged `Snapshot = daily`.

## 5. Change the instance or task size

Edit `instance_type`, `task_cpu`, and `task_memory` in the tfvars file. Dev and prod in this repo are already different (`t3.small` / 256 CPU versus `t3.medium` / 512 CPU). Fargate CPU and memory must stay a supported pair. The module rejects other pairs.

```bash
terraform -chdir=single-ec2 plan -var-file=../envs/prod.tfvars
terraform -chdir=ecs-fargate plan -var-file=../envs/prod.tfvars
```

Changing the EC2 type replaces the instance. Plan that window. Changing Fargate CPU rolls the service.

## 6. Change the certificate

Request or import a certificate in ACM in the same region as the load balancer. Put the new ARN in `certificate_arn` and apply the Fargate route. The HTTPS listener uses that variable. This stack does not create a certificate.

The single-EC2 route terminates TLS in the nginx container. Port 80 redirects to 443, and nginx proxies to the `api` service on the Compose network. On first boot, if `/opt/checkout/tls/` is empty, `user_data` writes a 30-day self-signed pair so nginx can bind. Replace `fullchain.pem` and `privkey.pem` over Session Manager before real traffic. Do not commit them.

## 7. Logs

Fargate container logs:

```bash
aws logs tail /checkout-api/dev/checkout-api --follow --region us-east-1
```

Load balancer access logs are in the S3 bucket named by `alb_log_bucket`, under the prefix `alb/`. VPC flow logs are in `/checkout-api/dev/vpc-flow`.

Single-EC2 agent logs:

```bash
aws logs tail /checkout-api/dev/ec2 --follow --region us-east-1
```

The host also keeps Compose output. Over Session Manager: `docker compose -f /opt/checkout/docker-compose.yml logs`.

## 8. Destroy

Empty the state bucket and the log bucket first if they still have objects (`force_destroy` is off). Then:

```bash
terraform -chdir=ecs-fargate destroy -var-file=../envs/dev.tfvars
terraform -chdir=single-ec2 destroy -var-file=../envs/dev.tfvars
terraform -chdir=bootstrap destroy \
  -var='bucket_name=YOUR_BUCKET' \
  -var='lock_table_name=YOUR_LOCK_TABLE'
```

Destroy the routes before the state backend. After the backend is gone, local state is the only copy, so do that last.

## 9. What can show up on the bill

Names only. This page does not estimate a price.

Single-EC2 route:

- Amazon EC2
- Amazon EBS (gp3 root volume)
- Amazon EBS snapshots (DLM)
- Elastic IP address
- Amazon CloudWatch Logs
- AWS Key Management Service
- Data transfer

Session Manager itself is not a separate line. The log group it writes to is CloudWatch Logs.

Fargate route:

- Amazon ECS (Fargate)
- Application Load Balancer
- Public IPv4 address (the load balancer)
- Amazon ECR
- NAT Gateway, when `enable_nat` is true
- Amazon VPC interface endpoints, when `enable_nat` is false
- Amazon CloudWatch Logs
- AWS Key Management Service
- Amazon RDS for PostgreSQL, when `enable_rds` is true (instance, storage, backups, Performance Insights)
- AWS Secrets Manager (the RDS-managed master password)
- AWS Systems Manager Parameter Store only if the parameter is advanced tier (the example uses a standard SecureString)
- Data transfer

S3 (state bucket and ALB access logs) and DynamoDB (the lock table) are part of the bootstrap route, not of the application routes.

## 10. Offline checks

From `infra/`, with empty AWS credentials:

```bash
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""
export AWS_EC2_METADATA_DISABLED=true
terraform fmt -check -recursive
terraform -chdir=ecs-fargate init -backend=false
terraform -chdir=ecs-fargate validate
terraform -chdir=single-ec2 init -backend=false
terraform -chdir=single-ec2 validate
terraform init -backend=false
terraform test
tflint --init --config .tflint.hcl
tflint --recursive --config "$PWD/.tflint.hcl"
checkov -d . --framework terraform --download-external-modules false
```

`terraform test` uses `mock_provider` and does not call AWS. Do not point these commands at a real account unless you mean to plan or apply.