From c50f5c5feb7e2b4c43fe3f54a31d853fb828ff04 Mon Sep 17 00:00:00 2001 From: Clawd Date: Tue, 17 Feb 2026 08:11:44 -0800 Subject: Clean up obsolete docs and binaries Removed: - GIT_AND_GOMOD_PLAN.md (old planning) - PLAN_v0.2.0.md (old planning) - PROGRESS.md (v2 progress tracking, complete) - TODO.md (all items fixed) - ship-new binary (added to gitignore) --- .gitignore | 1 + GIT_AND_GOMOD_PLAN.md | 230 -------------------------------- PLAN_v0.2.0.md | 356 -------------------------------------------------- PROGRESS.md | 56 -------- TODO.md | 44 ------- ship-new | Bin 12392993 -> 0 bytes 6 files changed, 1 insertion(+), 686 deletions(-) delete mode 100644 GIT_AND_GOMOD_PLAN.md delete mode 100644 PLAN_v0.2.0.md delete mode 100644 PROGRESS.md delete mode 100644 TODO.md delete mode 100755 ship-new diff --git a/.gitignore b/.gitignore index 67eb35f..a2dd99e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ # Binaries /ship +/ship-new *.exe *.dll *.so diff --git a/GIT_AND_GOMOD_PLAN.md b/GIT_AND_GOMOD_PLAN.md deleted file mode 100644 index 183154f..0000000 --- a/GIT_AND_GOMOD_PLAN.md +++ /dev/null @@ -1,230 +0,0 @@ -# Go Vanity Import + Git Server Architecture - -This document describes a **simple, robust architecture** for hosting Go module source code and enabling `go get` on a **custom domain**, using: - -* A basic Git server (authoritative source) -* Go vanity imports -* Caddy for HTTPS with automatic TLS - -The design prioritizes: - -* Simplicity -* Long-term correctness -* Zero client-side configuration -* Easy future migration to a module proxy if needed - ---- - -## Goals - -* Host all authored Go modules under a single domain (e.g. `yourdomain.com/foo`) -* Allow anyone to run: - - ```bash - go get yourdomain.com/foo - ``` - - with no environment variables or flags -* Retain full control of source code and hosting -* Avoid unnecessary complexity (S3 proxies, CI artifact pipelines) for a single author - ---- - -## High-Level Architecture - -``` -yourdomain.com - ├─ / → Vanity import discovery (?go-get=1) - ├─ /.git → Git repository (HTTPS, read-only) - └─ ssh://git@yourdomain.com/.git → Git writes (SSH) -``` - -Components: - -* **Caddy**: Fronts everything, provides HTTPS automatically -* **Vanity Import Server**: Tiny Go HTTP server that serves `` tags -* **Git Server**: Bare Git repositories served over HTTPS and SSH - ---- - -## How `go get` Works in This Setup - -When a user runs: - -```bash -go get yourdomain.com/foo -``` - -Go performs the following steps: - -1. **Vanity discovery** - - ``` - GET https://yourdomain.com/foo?go-get=1 - ``` - -2. **Vanity server responds** with: - - ```html - - ``` - -3. **Go clones the repository**: - - ``` - git clone https://yourdomain.com/foo.git - ``` - -4. **Go reads `go.mod`**, checks tags (e.g. `v1.2.3`), and builds the module - -No Go module proxy is involved. - ---- - -## Module Requirements - -Each module repository **must**: - -* Be located at `/srv/git/.git` -* Have a `go.mod` file with: - - ```go - module yourdomain.com/ - ``` -* Use semantic version tags: - - ``` - v1.0.0 - v1.2.3 - ``` - -Tags must not be rewritten after publication. - ---- - -## Vanity Import Server - -A single Go HTTP server can serve vanity imports for **all modules**. - -### Responsibilities - -* Respond only to `?go-get=1` requests -* Dynamically map paths to Git repositories -* Serve static HTML with `` - -### Behavior - -For a request to: - -``` -/foo?go-get=1 -``` - -The server returns: - -```html - -``` - -All non-`go-get` requests return `404` (or are handled elsewhere). - ---- - -## Git Server - -### Repository Layout - -``` -/srv/git/ - ├─ foo.git - ├─ bar.git - └─ baz.git -``` - -Repositories are: - -* **Bare** -* Read-only over HTTPS -* Writable only via SSH - -### Write Access - -* SSH only -* Single `git` user -* Access controlled via `authorized_keys` - -Example push: - -```bash -git push git@yourdomain.com:/srv/git/foo.git -``` - ---- - -## Caddy Configuration - -Caddy is used as the front-facing server. - -### Responsibilities - -* Automatic HTTPS (Let’s Encrypt) -* Route `.git` paths to `git-http-backend` -* Route all other paths to the vanity import server - -### Conceptual Routing - -* `/*.git*` → Git HTTP backend -* `/*` → Vanity import server - -This keeps TLS, routing, and process management simple and centralized. - ---- - -## Why This Design Is “Bullet-Proof Simple” - -### Advantages - -* No module proxy implementation required -* No S3, CloudFront, or artifact format rules -* No CI publishing pipeline needed -* Fully compatible with Go tooling -* Easy to debug (if `git clone` works, `go get` works) -* Mirrors Go’s original and still-supported design - -### Trade-offs - -* Git-based fetching is slower than a proxy -* No immutable release guarantees beyond Git tags -* Less suitable for very high traffic - -For a single-author, self-hosted setup, these trade-offs are acceptable. - ---- - -## Future Migration Path (Optional) - -If needed later, this setup can evolve into a module proxy: - -* Keep vanity import paths unchanged -* Switch `` from `git` → `mod` -* Introduce an S3-backed module proxy -* Users do not need to change imports or commands - -This design does not lock you into Git forever. - ---- - -## Summary - -This architecture provides: - -* A single, authoritative domain for all Go modules -* Simple Git-based source hosting -* Zero-config `go get` for users -* Minimal operational complexity -* A clean upgrade path to a full module proxy - -It is the simplest solution that is still fully correct and future-proof for Go modules. - diff --git a/PLAN_v0.2.0.md b/PLAN_v0.2.0.md deleted file mode 100644 index d48e95c..0000000 --- a/PLAN_v0.2.0.md +++ /dev/null @@ -1,356 +0,0 @@ -# Plan: Git-Centric Deployment with Docker Builds and Vanity Imports - -## Context - -Ship currently deploys by SCP'ing pre-built binaries to a VPS. This plan replaces that model entirely: every project starts with a **git remote on the VPS**, pushing triggers an automatic `docker build` and deploy via post-receive hooks. Since all repos live at `/srv/git/`, the same domain serves **Go vanity imports** and **git HTTPS cloning**, making `go get yourdomain.com/foo` work with zero extra setup. - -Using Docker for builds means the VPS only needs Docker installed — no language-specific toolchains. Any language with a Dockerfile works. - -Deployment config (systemd unit, Caddyfile) lives **in the repo** under `.ship/`, making it versioned, customizable, and deployed via `git push` alongside code. Env vars (which may contain secrets) stay managed separately via `ship env`. - -The base domain pulls double duty: `example.com/foo` serves vanity imports, `foo.example.com` runs the app. - -No backward compatibility with the current binary-upload flow — clean slate. - ---- - -## User Flow - -``` -ship host init --host myserver --base-domain example.com # one-time VPS setup -ship init myapp # creates bare repo on VPS + .ship/ locally -git add .ship/ Dockerfile && git commit -m "initial deploy" # commit deploy config -git remote add ship git@myserver:/srv/git/myapp.git -git push ship main # installs configs + docker build + deploy -``` - -To change config (memory limit, Caddy rules, etc.): edit `.ship/service` or `.ship/Caddyfile`, commit, push. - ---- - -## Phase 1: State & Template Rewrite - -### 1A. State — `internal/state/state.go` - -Simplified structs: - -```go -type Host struct { - NextPort int `json:"next_port"` - BaseDomain string `json:"base_domain,omitempty"` - GitSetup bool `json:"git_setup,omitempty"` - Apps map[string]*App `json:"apps"` -} - -type App struct { - Type string `json:"type"` // "app" or "static" - Domain string `json:"domain"` - Port int `json:"port,omitempty"` // only for "app" - Env map[string]string `json:"env,omitempty"` // only for "app" - Repo string `json:"repo"` // e.g., "/srv/git/foo.git" -} -``` - -Removed from App: `Args`, `Files`, `Memory`, `CPU` — these are now in `.ship/service` inside the repo, not in ship's state. - -### 1B. Templates — `internal/templates/templates.go` - -Replace existing templates with: - -1. **`PostReceiveHook(data)`** — app deploy hook (see Hook Detail below) -2. **`PostReceiveHookStatic(data)`** — static site deploy hook -3. **`CodeCaddy(data)`** — base domain Caddy config for vanity imports + git HTTP -4. **`DockerService(data)`** — default systemd unit (generated into `.ship/service` locally) -5. **`AppCaddy(data)`** — default app Caddyfile (generated into `.ship/Caddyfile` locally) -6. **`StaticCaddy(data)`** — default static Caddyfile (generated into `.ship/Caddyfile` locally) - -Remove old: `serviceTemplate`, `appCaddyTemplate`, `staticCaddyTemplate` and their render functions (`SystemdService`, `AppCaddy`, `StaticCaddy`). - ---- - -## Phase 2: Caddy-Native Vanity Imports + Git HTTP - -Caddy handles everything directly. No separate server binary. - -### Vanity imports — Caddy `templates` directive - -HTML template at `/opt/ship/vanity/index.html` on VPS: - -```html - - -{{$path := trimPrefix "/" .Req.URL.Path}} -{{$parts := splitList "/" $path}} -{{$module := first $parts}} - - -go get {{.Host}}/{{$module}} - -``` - -### Git HTTP — fcgiwrap + git-http-backend - -`fcgiwrap` (apt package) bridges Caddy's FastCGI to git's CGI backend. - -### Base domain Caddy config - -Generated by `templates.CodeCaddy()`, written to `/etc/caddy/sites-enabled/ship-code.caddy`: - -```caddyfile -{baseDomain} { - @goget query go-get=1 - handle @goget { - root * /opt/ship/vanity - templates - rewrite * /index.html - file_server - } - - @git path_regexp "^.*/(HEAD|info/refs|objects/info/[^/]+|git-upload-pack|git-receive-pack)$" - handle @git { - reverse_proxy unix//run/fcgiwrap.socket { - transport fastcgi { - env SCRIPT_FILENAME /usr/lib/git-core/git-http-backend - env GIT_PROJECT_ROOT /srv/git - env GIT_HTTP_EXPORT_ALL 1 - env REQUEST_METHOD {method} - env QUERY_STRING {query} - env PATH_INFO {path} - } - } - } - - handle { - respond "not found" 404 - } -} -``` - ---- - -## Phase 3: `host init` Rewrite — `cmd/ship/host/init.go` - -Full setup flow: - -1. **Detect OS** — Ubuntu/Debian only (existing) -2. **Install Caddy** — (existing) -3. **Configure Caddyfile** — (existing) -4. **Install Docker** — Docker's official apt repo, `apt-get install -y docker-ce docker-ce-cli containerd.io` -5. **Install git + fcgiwrap** — `apt-get install -y git fcgiwrap` -6. **Create `git` user** — `useradd -r -m -d /home/git -s $(which git-shell) git`, `usermod -aG docker git` -7. **Copy SSH keys** — admin user's `authorized_keys` → `/home/git/.ssh/authorized_keys` -8. **Create `/srv/git`** — owned by `git:git` -9. **Write sudoers** — `/etc/sudoers.d/ship-git`: passwordless sudo for `systemctl`, `cp` to config dirs, `mkdir`/`chown` -10. **Create directories** — `/etc/ship/env`, `/etc/caddy/sites-enabled`, `/opt/ship/vanity` -11. **Write vanity template** — `/opt/ship/vanity/index.html` -12. **Write base domain Caddy config** — `/etc/caddy/sites-enabled/ship-code.caddy` -13. **Start services** — `systemctl enable --now docker fcgiwrap caddy` -14. **Set `hostState.GitSetup = true`**, save state - ---- - -## Phase 4: `ship init ` — New Command - -**New file:** `cmd/ship/init.go` - -``` -ship init [--static] [--domain custom.example.com] -``` - -Does **both local and remote work**. - -### Remote (on VPS via SSH): - -1. Verify `hostState.GitSetup == true` -2. Create bare repo: `sudo -u git git init --bare /srv/git/{name}.git` -3. For apps: create `/var/lib/{name}/data` and `/var/lib/{name}/src` -4. For static: create `/var/www/{name}` -5. Allocate port (apps only) -6. Write env file `/etc/ship/env/{name}.env` with `PORT={port}` and `DATA_DIR=/data` (apps only) -7. Write post-receive hook to `/srv/git/{name}.git/hooks/post-receive` -8. Save state - -### Local (in current directory): - -9. Create `.ship/` directory -10. Generate `.ship/Caddyfile` (resolved domain, port) -11. For apps: generate `.ship/service` (resolved name, port) -12. Print git remote URL and next steps - ---- - -## Phase 5: `ship deploy ` — Manual Rebuild - -**New file:** `cmd/ship/deploy_cmd.go` - -``` -ship deploy -``` - -SSH in, trigger same steps as post-receive hook. Stream output to terminal. - ---- - -## Phase 6: Rewrite Remaining Commands - -### `root.go` -Remove all deploy flags (`--binary`, `--static`, `--dir`, `--port`, `--args`, `--file`, `--memory`, `--cpu`, `--env`, `--env-file`). Root command shows help. Register: `init`, `deploy`, `list`, `logs`, `status`, `restart`, `remove`, `env`, `host`, `ui`, `version`. - -### `deploy.go` -**Delete entirely.** The `runDeploy`, `deployApp`, `deployStatic`, `updateAppConfig`, `DeployOptions` — all removed. Replaced by `ship init` + `git push` + `ship deploy`. - -### `remove.go` -- `"app"`: stop service, remove systemd unit, `docker rmi {name}:latest`, remove `/var/lib/{name}`, remove `/srv/git/{name}.git`, remove Caddy config, remove env file -- `"static"`: remove `/var/www/{name}`, remove `/srv/git/{name}.git`, remove Caddy config - -### `status.go`, `logs.go`, `restart.go` -Simplify type guard: only `"static"` type is rejected (no service to manage). - -### `list.go` -Show port for `"app"` type. - -### `env/` subcommands -Work the same — manage `/etc/ship/env/{name}.env` on VPS, restart service. - ---- - -## Post-Receive Hook Detail - -### App hook: - -```bash -#!/bin/bash -set -euo pipefail - -REPO=/srv/git/{name}.git -SRC=/var/lib/{name}/src -NAME={name} - -while read oldrev newrev refname; do - branch=$(git rev-parse --symbolic --abbrev-ref "$refname") - [ "$branch" = "main" ] || { echo "Pushed to $branch, skipping deploy."; exit 0; } -done - -echo "==> Checking out code..." -git --work-tree="$SRC" --git-dir="$REPO" checkout -f main -cd "$SRC" - -# Install deployment config from repo -if [ -f .ship/service ]; then - echo "==> Installing systemd unit..." - sudo cp .ship/service /etc/systemd/system/${NAME}.service - sudo systemctl daemon-reload -fi -if [ -f .ship/Caddyfile ]; then - echo "==> Installing Caddy config..." - sudo cp .ship/Caddyfile /etc/caddy/sites-enabled/${NAME}.caddy - sudo systemctl reload caddy -fi - -echo "==> Building Docker image..." -docker build -t ${NAME}:latest . - -echo "==> Restarting service..." -sudo systemctl restart ${NAME} - -echo "==> Deploy complete!" -``` - -### Static site hook: - -```bash -#!/bin/bash -set -euo pipefail - -REPO=/srv/git/{name}.git -WEBROOT=/var/www/{name} -NAME={name} - -while read oldrev newrev refname; do - branch=$(git rev-parse --symbolic --abbrev-ref "$refname") - [ "$branch" = "main" ] || { echo "Pushed to $branch, skipping deploy."; exit 0; } -done - -echo "==> Deploying static site..." -git --work-tree="$WEBROOT" --git-dir="$REPO" checkout -f main - -if [ -f "$WEBROOT/.ship/Caddyfile" ]; then - echo "==> Installing Caddy config..." - sudo cp "$WEBROOT/.ship/Caddyfile" /etc/caddy/sites-enabled/${NAME}.caddy - sudo systemctl reload caddy -fi - -echo "==> Deploy complete!" -``` - ---- - -## VPS File Layout - -``` -/srv/git/ # bare repos (owned by git) -/srv/git/{name}.git/ # per-project bare repo -/srv/git/{name}.git/hooks/post-receive # auto-deploy hook -/home/git/.ssh/authorized_keys # SSH keys for git push -/opt/ship/vanity/index.html # Caddy template for vanity imports -/etc/caddy/sites-enabled/ship-code.caddy # base domain Caddy config -/etc/sudoers.d/ship-git # sudo rules for git user - -# Per app: -/var/lib/{name}/src/ # checked-out source (for docker build) -/var/lib/{name}/data/ # persistent data (mounted as /data in container) -/etc/systemd/system/{name}.service # installed from .ship/service on push -/etc/caddy/sites-enabled/{name}.caddy # installed from .ship/Caddyfile on push -/etc/ship/env/{name}.env # env vars (managed by ship env, not in git) - -# Per static site: -/var/www/{name}/ # checked-out site files -/etc/caddy/sites-enabled/{name}.caddy # installed from .ship/Caddyfile on push -``` - ---- - -## Files Changed / Created - -| File | Action | -|------|--------| -| `internal/state/state.go` | Rewrite — simplified App struct, add `Repo`, `GitSetup` | -| `internal/templates/templates.go` | Rewrite — replace old templates with new ones | -| `cmd/ship/init.go` | **Create** — `ship init ` | -| `cmd/ship/deploy_cmd.go` | **Create** — `ship deploy ` | -| `cmd/ship/host/init.go` | Rewrite — add Docker/git/fcgiwrap/vanity/sudoers | -| `cmd/ship/root.go` | Rewrite — remove deploy flags, register new commands | -| `cmd/ship/deploy.go` | **Delete** | -| `cmd/ship/remove.go` | Rewrite — Docker + git cleanup | -| `cmd/ship/list.go` | Simplify | -| `cmd/ship/status.go` | Simplify | -| `cmd/ship/logs.go` | Simplify | -| `cmd/ship/restart.go` | Simplify | -| `cmd/ship/env/*.go` | Minor updates if needed | - ---- - -## Implementation Order - -1. Phase 1 — state + templates -2. Phase 3 — `host init` rewrite -3. Phase 4 — `ship init` command -4. Phase 5 — `ship deploy` command -5. Phase 6 — rewrite remaining commands (root, remove, list, status, logs, restart, delete deploy.go) - ---- - -## Verification - -1. `ship host init --host myserver --base-domain example.com` — installs Docker, git, fcgiwrap, Caddy, creates git user, writes vanity template + Caddy config -2. `ship init myapp` — creates bare repo on VPS, generates `.ship/service` and `.ship/Caddyfile` locally -3. `git add .ship/ Dockerfile && git commit && git push ship main` — hook installs configs, builds Docker image, app goes live at `https://myapp.example.com` -4. Edit `.ship/service` to add `--memory=512m`, push — service updated -5. `curl -s 'https://example.com/myapp?go-get=1'` — returns `` tag -6. `go get example.com/myapp` — vanity import → HTTPS clone → works -7. `ship list` — shows myapp -8. `ship logs myapp` / `ship status myapp` / `ship restart myapp` — work -9. `ship env set myapp SECRET=foo` — updates env, restarts -10. `ship deploy myapp` — manually rebuilds -11. `ship remove myapp` — cleans up everything -12. `ship init mysite --static` + push — static site deployed diff --git a/PROGRESS.md b/PROGRESS.md deleted file mode 100644 index 6ca83af..0000000 --- a/PROGRESS.md +++ /dev/null @@ -1,56 +0,0 @@ -# Ship v2 Rebuild Progress - -Tracking rebuilding ship for agent-first JSON interface. - -## Status: READY FOR TESTING - -## Completed -- [x] Design docs (SHIP_V2.md, SPEC.md) -- [x] JSON output types and helpers (`internal/output`) -- [x] Auto-detection logic (`internal/detect`) -- [x] Error codes and exit codes -- [x] New CLI structure (`ship [PATH]` as primary) -- [x] Deploy orchestration with placeholder implementations - -## Current Phase: Subcommand Implementations -- [x] Static site deploy (rsync + Caddyfile) -- [x] Docker deploy (build + systemd + Caddy) -- [x] Binary deploy (scp + systemd + Caddy) -- [x] Health check implementation -- [x] TTL support (server-side) -- [x] Port allocation (server-side) - -## Upcoming -- [ ] Testing with real deploys -- [ ] Remove v1 code after validation - -## Wiring -- v2 is now the default interface -- Set `SHIP_V1=1` to use legacy v1 (human-formatted output) - -## Completed Recently -- [x] `ship list` - enumerate all deploys from /etc/ship/ports and /var/www -- [x] `ship status NAME` - show deploy status, port, type, TTL -- [x] `ship logs NAME` - show journalctl logs (or Caddy logs for static) -- [x] `ship remove NAME` - full cleanup of all deploy artifacts - -## Completed Recently -- [x] TTL cleanup timer (server-side systemd timer) -- [x] `ship host init` with JSON output -- [x] Docker + Caddy installation -- [x] Cleanup script for expired TTL deploys - -## Commits -- `5b88935` feat(v2): add output and detect packages -- `8094639` feat(v2): add CLI structure and deploy orchestration -- `1463935` feat(v2): implement deploy flows -- `a7436df` feat(v2): implement ship host init with JSON output -- `d6740a0` feat(v2): implement list, status, logs, remove commands -- `4c20fb3` feat(v2): wire up v2 as default interface - ---- - -## Notes -- Branch: `agent-mode` -- Keep v1 code for reference until v2 is working -- Test with real deploys before merging to main diff --git a/TODO.md b/TODO.md deleted file mode 100644 index 5406485..0000000 --- a/TODO.md +++ /dev/null @@ -1,44 +0,0 @@ -# Ship TODO - -## Open - -### ~~Support custom Caddyfile~~ ✅ FIXED -**Fixed in:** commit b976b14 - -- Caddyfile is only generated on first deploy -- Redeploys preserve existing Caddyfile (manual edits survive) -- SSH in and customize as needed, future deploys won't overwrite - ---- - -### ~~ship list should show custom domains~~ ✅ FIXED -**Fixed in:** commit c1b0bb8 - -- Reads actual domain from Caddyfile instead of assuming subdomain -- Works for both apps and static sites - ---- - -## Fixed - -### ~~Port allocation collision bug~~ ✅ FIXED -**Fixed in:** commit d97bb6f - -- Port allocation now uses atomic flock on `/etc/ship/next_port` -- Prevents race conditions when multiple deploys run concurrently -- Each app still gets its port stored in `/etc/ship/ports/` - -### ~~Docker container port assumption~~ ✅ FIXED -**Fixed in:** commit d97bb6f - -- Added `--container-port` flag (default 80) -- Template now uses `{{.ContainerPort}}` instead of hardcoded 80 -- Supports containers that listen on 8080, 3000, etc. - -### ~~ship host init doesn't create local state.json~~ ✅ FIXED -**Fixed in:** commit 6b2c047 - -- Removed all v1 code (-2837 lines) -- Simplified `internal/state/state.go` to just `default_host` + `base_domain` -- `host init` now creates minimal state.json -- Ports/deploys tracked on server at `/etc/ship/ports/` diff --git a/ship-new b/ship-new deleted file mode 100755 index 28b9780..0000000 Binary files a/ship-new and /dev/null differ -- cgit v1.2.3