Added optional Explorer/Indexer

2026-05-20 15:43:38 -04:00
parent 2cdc4a5f34
commit e41ff31221
27 changed files with 1431 additions and 70 deletions
@@ -37,13 +37,21 @@ through a browser-based terminal.

 | Field | Value |
 | --- | --- |
-| Image | `docker.io/horologger/spaces` |
+| Spaces image | `docker.io/horologger/spaces` |
+| PostgreSQL image | `docker.io/postgres:16.3` |
+| Indexer-Go image | `docker.io/golang:1.23-alpine` |
+| Explorer-UI image | `docker.io/node:20-alpine` |
 | Architectures | `linux/amd64`, `linux/arm64` |
-| Entrypoint | StartOS-managed (image entrypoint is **not** used) |
+| Entrypoint | StartOS-managed (image entrypoints are **not** used directly) |

-The image bundles `spaced`, `space-cli`, `bitcoin-cli`, `gotty`, `node`, `npm`,
-`screen`, and a small shell environment. StartOS ignores the image's
-`docker_entrypoint.sh`; daemons are defined in `startos/main.ts`.
+The Spaces image bundles `spaced`, `space-cli`, `bitcoin-cli`, `gotty`, `node`,
+`npm`, `screen`, and a small shell environment. StartOS ignores the Spaces
+image's `docker_entrypoint.sh`; daemons are defined in `startos/main.ts`. The
+PostgreSQL image is the upstream `postgres:16.3`, launched via its official
+`docker-entrypoint.sh`. The Indexer-Go image is the upstream
+`golang:1.23-alpine`, used both to compile the indexer's `sync` binary +
+`goose` migrator (one-time, at install) and to run the compiled binaries on
+each start.

 ## Volume and Data Layout

@@ -51,53 +59,115 @@ The image bundles `spaced`, `space-cli`, `bitcoin-cli`, `gotty`, `node`, `npm`,
 | --- | --- | --- |
 | `/data` | `main` | Spaces data directory (`SPACED_DATA_DIR`), wallets, indexes, and `store.json` |
 | `/data/mainnet/.cookie` | `main` | Spaced RPC cookie (auto-generated by `spaced` at startup) |
-| `/data/store.json` | `main` | StartOS-managed credentials (web-UI password + bitcoind RPC user/password) |
+| `/data/store.json` | `main` | StartOS-managed credentials (web-UI password, bitcoind RPC user/password, PostgreSQL user/password/database) |
+| `/data/postgres` | `main` | PostgreSQL data directory (`PGDATA`). Owned by uid 999 (`postgres`). The whole `main` volume is backed up, so the database is included. |
+| `/data/explorer-indexer` | `main` | Spaces-protocol explorer-indexer source (`spacesprotocol/explorer-indexer`), fetched from GitHub on first start. Contains `cmd/sync/`, `pkg/`, `sql/schema/`, `go.mod`, etc. |
+| `/data/explorer-indexer/bin/sync` | `main` | Compiled indexer sync binary (`go build ./cmd/sync`). |
+| `/data/explorer-indexer/bin/goose` | `main` | Compiled goose migrator (`go install github.com/pressly/goose/v3/cmd/goose`). Used by `indexer-migrate` to apply `sql/schema/*.sql`. |
+| `/data/explorer-indexer/.installed-sha` | `main` | Marker file recording the pinned commit + build-ID currently extracted. If the build-ID in the package code differs from the marker, the next start wipes and re-fetches. |
+| `/data/explorer-ui` | `main` | SvelteKit explorer source (`randomlogin/explorer`), fetched on first explorer-enable. Contains `src/`, `static/`, `server.js`, `package.json`, etc. |
+| `/data/explorer-ui/build` | `main` | SvelteKit node-adapter build output. `node build` (i.e. `node /data/explorer-ui/build`) runs the production server. |
+| `/data/explorer-ui/node_modules` | `main` | npm dependencies (~300-500 MB). Re-fetched on `Reset Explorer UI State`. |
+| `/data/explorer-ui/.installed-sha` | `main` | Marker file for the pinned explorer commit + build-ID. |

 ## Installation and First-Run Flow

 On the first install, StartOS:

 1. Seeds `store.json.password` with a random 32-char alphanumeric admin password
-   and creates a critical task that points the user at **Show Web UI Credentials**
+   and creates a critical task that points the user at **Show Space-CLI Web UI Credentials**
   so the password can be copied before login.
 2. Seeds `store.json.btcAuth` with a random `spaces:<random>` RPC credential
   pair and creates a critical cross-service task on **Bitcoin** that runs
   bitcoind's `generate-rpc-dependent` action to register the credentials in
   `bitcoin.conf`.
-3. Launches `spaced` as a managed daemon (no `screen`, no shell auto-start).
-4. Launches the `gotty` web terminal once `spaced` is up.
+3. Seeds `store.json.dbAuth` with a random PostgreSQL password (username
+   `postgres`, database `spacesprotocol_explorer`). No task is created — the
+   credentials are only used internally by the embedded daemon.
+4. Runs the `postgres-chown` oneshot to create `/data/postgres` with the right
+   ownership, then launches the PostgreSQL daemon on loopback 5432.
+5. Seeds `store.json.spacedAuth` with a random `spaces:<random>` credential
+   pair. Spaced is configured to require these via `SPACED_RPC_USER` /
+   `SPACED_RPC_PASSWORD` (cookie auth is **not** used in this package), so the
+   gotty terminal's `spaces` alias and the indexer share a single auth path.
+6. Seeds `store.json.enableExplorer = false`. The embedded explorer
+   (PostgreSQL + Go indexer) is **opt-in**.
+7. Launches `spaced` as a managed daemon (no `screen`, no shell auto-start)
+   and the `gotty` web terminal once the bashrc oneshot completes.

-The user is **not** prompted to choose a chain or RPC mode -- this package is
+The user is **not** prompted to choose a chain or RPC mode — this package is
 mainnet-only.

+### Enabling the embedded explorer
+
+Run the **Enable Embedded Explorer** action. That flips
+`store.json.enableExplorer = true` and the service auto-restarts with the full
+daemon graph:
+
+8. Postgres-chown oneshot creates `/data/postgres` with correct ownership.
+9. PostgreSQL daemon starts on loopback `127.0.0.1:5432`.
+10. `indexer-fetch` downloads the pinned `spacesprotocol/explorer-indexer`
+    tarball into `/data/explorer-indexer` and applies two compatibility
+    patches for our older spaced binary (see Limitations).
+11. `indexer-build` runs `CGO_ENABLED=0 go build -o /data/explorer-indexer/bin/sync ./cmd/sync`
+    and `GOBIN=/data/explorer-indexer/bin go install github.com/pressly/goose/v3/cmd/goose@v3.24.3`.
+12. `indexer-cleanup-legacy` checks for and removes leftover TS-indexer schema
+    if present (from earlier package versions).
+13. `indexer-migrate` runs `goose ... up` to apply `sql/schema/*.sql`.
+14. `indexer` daemon (`/data/explorer-indexer/bin/sync`) starts polling
+    bitcoind + spaced and writes blocks, transactions, spaces, rollouts, etc.
+    to PostgreSQL.
+15. `explorer-fetch` downloads the pinned `randomlogin/explorer` tarball into
+    `/data/explorer-ui` (skipped if marker already matches).
+16. `explorer-install` runs `npm install` + `PUBLIC_BTC_NETWORK=mainnet npm run build`
+    to produce `/data/explorer-ui/build/`. Skipped if `build/index.js` already
+    exists. First-time install can take 1-3 minutes for the npm download.
+17. `explorer-ui` daemon (`node /data/explorer-ui/build`) starts on port 3000.
+    The SvelteKit app reads exclusively from PostgreSQL — it does not talk to
+    spaced or bitcoind directly. The UI link will appear in the StartOS
+    dashboard alongside the gotty terminal.
+
+Run **Disable Embedded Explorer** to turn it back off; on-disk data at
+`/data/postgres`, `/data/explorer-indexer`, and `/data/explorer-ui` is
+preserved.
+
 ## Configuration Management

 | StartOS-Managed | Upstream-Managed |
 | --- | --- |
 | Web-UI username / password (`admin` + generated password) | `spaced` runtime tuning via `SPACED_*` env vars in the image |
 | Bitcoin RPC username / password (registered on bitcoind) | Wallet creation, bidding, and registration -- all driven via `space-cli` inside the terminal |
-| Chain selection (locked to `mainnet`) | `space-cli` flags and subcommands |
+| PostgreSQL username / password / database (loopback only) | `space-cli` flags and subcommands |
+| Chain selection (locked to `mainnet`) |  |
 | Spaced data directory and RPC bind |  |

 ## Network Access and Interfaces

 | Interface | Port | Protocol | Exposure | Notes |
 | --- | --- | --- | --- | --- |
-| Web UI (gotty terminal) | 8080 | HTTP | LAN / `.local` / Tor / clearnet (via StartOS) | Basic auth: `admin:<store.password>` |
-| Spaced RPC | 7225 | HTTP JSON-RPC | **Loopback only** | Internal use, cookie-authenticated |
+| Space-CLI Web UI (gotty terminal) | 8080 | HTTP | LAN / `.local` / Tor / clearnet (via StartOS) | Basic auth: `admin:<store.password>` |
+| Explorer Web UI | 3000 | HTTP | LAN / `.local` / Tor / clearnet (via StartOS) | SvelteKit explorer. No built-in auth — relies on StartOS interface exposure controls. Only useful while the embedded explorer is enabled (otherwise nothing is listening). |
+| Spaced RPC | 7225 | HTTP JSON-RPC | **Loopback only** | Internal use, static-cred-authenticated via `SPACED_RPC_USER` / `SPACED_RPC_PASSWORD` from `store.json.spacedAuth` |
+| PostgreSQL | 5432 | Postgres wire protocol | **Loopback only** (`listen_addresses=127.0.0.1`) | Username/password from `store.json.dbAuth`; full `DB_URL` exported into the web terminal |

 ## Actions

 | ID | Name | Visibility | Availability | Purpose |
 | --- | --- | --- | --- | --- |
-| `reset-password` | Reset Web UI Password | Enabled | Any | Regenerates the web-UI password and restarts the terminal daemon |
-| `show-credentials` | Show Web UI Credentials | Hidden | Any | Surfaces the current `admin` username + masked password (launched by the first-install task) |
-| `show-password` | Show Web UI Password | Enabled | Any | Same as `show-credentials` but visible in the actions list, for routine re-display of the current admin credentials |
+| `reset-password` | Reset Space-CLI Web UI Password | Enabled | Any | Regenerates the Space-CLI Web UI password and restarts the terminal daemon |
+| `show-credentials` | Show Space-CLI Web UI Credentials | Hidden | Any | Surfaces the current `admin` username + masked password (launched by the first-install task) |
+| `show-password` | Show Space-CLI Web UI Password | Enabled | Any | Same as `show-credentials` but visible in the actions list, for routine re-display of the current admin credentials |
 | `set-bitcoin-rpc` | Set up Bitcoin RPC | Enabled | Any | Re-invokes bitcoind's `generate-rpc-dependent` with the stored credentials. Safe to call repeatedly. |
 | `sync-status` | Sync Status | Enabled | Only running | Runs `space-cli getserverinfo` inside the daemon container and returns the JSON output |
 | `reset-spaced-state` | Reset Spaced State | Enabled | Any | Deletes `/data/mainnet/` so spaced resyncs its index from spaces' anchor. Preserves `store.json` (passwords + RPC credentials). Use when spaced crash-loops on a stale or corrupt index. |
 | `export-wallet` | Export Wallet | Enabled | Only running | Runs `space-cli exportwallet /data/mainnet/wallets_backup/default.json` and surfaces the resulting JSON as a masked/copyable result. The file is also persisted inside the volume at that path. |
 | `import-wallet` | Import Wallet | Enabled | Only running | Accepts a pasted JSON payload (textarea), writes it to `/data/mainnet/wallets_backup/default.json` (rotating the existing file to `.bakNNN`), rotates `/data/mainnet/wallets/default` to `.bakNNN`, then runs `space-cli importwallet` + `loadwallet`. |
+| `enable-explorer` | Enable Embedded Explorer | Enabled (hidden when already on) | Any | Sets `store.enableExplorer = true` and triggers a service restart so the full daemon graph (postgres + indexer + indexer-sync HC) takes effect. |
+| `disable-explorer` | Disable Embedded Explorer | Enabled (hidden when already off) | Any | Sets `store.enableExplorer = false` and triggers a service restart so the indexer and postgres daemons stop. On-disk data at `/data/postgres` and `/data/explorer-indexer` is preserved. |
+| `show-db-credentials` | Show Database Credentials | Enabled | Any | Surfaces the PostgreSQL username, password, database, and full `DB_URL` (all copyable; secrets masked). Credentials exist regardless of whether the explorer is currently enabled. |
+| `reset-db-state` | Reset Database State | Enabled | Any | Deletes `/data/postgres` so the next start re-initializes an empty database. Warning-gated. `store.json` is preserved. |
+| `reset-indexer-state` | Reset Indexer State | Enabled | Any | Deletes `/data/explorer-indexer` so the next start re-fetches the indexer source and rebuilds the sync + goose binaries. Useful when bumping the pinned commit. PostgreSQL data is preserved. Warning-gated. |
+| `reset-explorer-state` | Reset Explorer UI State | Enabled | Any | Deletes `/data/explorer-ui` so the next start re-fetches the SvelteKit source and rebuilds the bundle. PostgreSQL data is preserved. Warning-gated. |

 ## Backups and Restore

@@ -110,23 +180,57 @@ the same idempotent init logic runs and reuses the existing credentials in

 | ID | Display | Grace period | Behaviour |
 | --- | --- | --- | --- |
+| `postgres` (daemon `ready`) | Database | 60 s | TCP listen on `127.0.0.1:5432` |
 | `spaced` (daemon `ready`) | Spaced RPC | 120 s | TCP listen on `127.0.0.1:7225` |
+| `indexer` (daemon `ready`) | Indexer Process | 60 s | Daemon liveness only — health is reported by `indexer-sync` below |
+| `explorer-ui` (daemon `ready`) | Explorer Web UI | 60 s | TCP listen on `0.0.0.0:3000`. Only present when embedded explorer is enabled. |
 | `web-terminal` (daemon `ready`) | Web Interface | default | TCP listen on `0.0.0.0:8080` |
-| `sync` (standalone) | Spaced Sync | 300 s | Exec `space-cli getserverinfo`; reports `success` when `ready=true`, otherwise `loading` with progress percentage |
+| `sync` (standalone) | Spaced Sync | 30 s | Exec `space-cli --output-format json getserverinfo`; reports `success` when `ready=true && progress=100%`, otherwise `loading` with progress percentage |
+| `indexer-sync` (standalone) | Indexer Sync | 60 s | `psql` the `blocks` table for `MAX(height) WHERE orphan = FALSE AND height >= 0`; cross-check against spaced's current tip via `space-cli getserverinfo`. `success` when within 5 blocks of the tip; otherwise `loading` with explicit lag. |

 ## Limitations and Differences

 1. **Mainnet only.** Testnet, testnet4, signet, and regtest are not exposed.
-2. **The image's `docker_entrypoint.sh` is not used.** Its auto-start of
+2. **The Spaces image's `docker_entrypoint.sh` is not used.** Its auto-start of
   `spaced` inside `screen` would conflict with the StartOS-managed daemon.
-3. **Web UI is a terminal, not a graphical app.** All Spaces operations happen
+3. **Space-CLI Web UI is a terminal, not a graphical app.** All Spaces operations happen
   via `space-cli` (aliased as `spaces` inside the shell).
-4. **Only port 8080 (gotty) is exposed.** Spaced RPC, and any other ports
-   present in the image (e.g. 22253, 3000, 5173, 8081) are not bound.
+4. **Only port 8080 (gotty) is exposed externally.** Spaced RPC (7225) and
+   PostgreSQL (5432) are loopback-only, and any other ports present in the
+   image (e.g. 22253, 3000, 5173, 8081) are not bound.
 5. **Bitcoin Core 31.x is the only supported dependency.** Earlier majors are
   not allowed by the manifest version range.
 6. **The web terminal is independent of spaced.** Gotty stays reachable even
   when `spaced` is crash-looping, so you can always shell in to diagnose.
+7. **The embedded explorer is opt-in.** Fresh installs run in spaces-only
+   mode (spaced + gotty). Run **Enable Embedded Explorer** to start
+   PostgreSQL + the Go indexer; **Disable Embedded Explorer** stops them. The
+   `postgres` and `indexer-go` images are bundled in the .s9pk regardless so
+   toggling is instant, no re-download. Indexed data is preserved across
+   toggles.
+8. **PostgreSQL is embedded, not a separate service.** There is no
+   cross-service dependency on a postgres .s9pk; the database is local to this
+   package, lives at `/data/postgres`, and is intended for the embedded
+   indexer (and any future explorer UI). Connect to it from inside the gotty
+   terminal with `psql $DB_URL` (only meaningful while the explorer is
+   enabled).
+9. **Enabling the explorer requires internet on its first start.** The pinned
+   commit of `spacesprotocol/explorer-indexer` is downloaded from GitHub into
+   `/data/explorer-indexer`, and `go build` + `go install github.com/pressly/goose/v3/cmd/goose`
+   fetch Go modules. After that the indexer runs offline; the only network it
+   needs is bitcoind (already a StartOS dependency) and the local spaced RPC.
+10. **Explorer first-enable latency is ~2-5 minutes.** The Go indexer needs to
+    download modules and compile from scratch the first time you turn it on.
+    Subsequent restarts (and toggle off→on) skip `indexer-fetch` +
+    `indexer-build` because the binaries already exist on the volume. Use the
+    **Reset Indexer State** action to force a rebuild.
+11. **Subspaces pointers data is not indexed.** Our spaced binary predates the
+    pointers feature. The indexer source is patched during `indexer-fetch` to
+    treat `ptrs_root` as optional and to skip the `getptrblockmeta` RPC call.
+    All other spaces protocol data — blocks, transactions, spaces, rollouts,
+    root anchors with empty pointers root — indexes normally. A future
+    rebuild of `horologger/spaces` with a newer spaced will let us drop these
+    patches.

 ## What Is Unchanged from Upstream

@@ -142,11 +246,11 @@ the same idempotent init logic runs and reuses the existing credentials in

 After install:

-1. Run the **Show Web UI Credentials** action (the install task surfaces it).
-2. Open the Web UI from the StartOS dashboard.
+1. Run the **Show Space-CLI Web UI Credentials** action (the install task surfaces it).
+2. Open the Space-CLI Web UI from the StartOS dashboard.
 3. Log in with `admin` and the displayed password.
 4. Use `spaces <subcommand>` -- it expands to
-   `space-cli --chain mainnet --rpc-cookie /data/mainnet/.cookie <subcommand>`.
+   `space-cli --chain mainnet --rpc-user "$SPACED_RPC_USER" --rpc-password "$SPACED_RPC_PASSWORD" <subcommand>`.

 Examples:

@@ -161,32 +265,86 @@ spaces walletbalance default
 ```yaml
 package_id: spaces
 upstream_version: subspacesplus
-image: docker.io/horologger/spaces:v0.0.9s
+images:
+  spaces: docker.io/horologger/spaces:v0.0.9s
+  postgres: docker.io/postgres:16.3
+  indexer-go: docker.io/golang:1.23-alpine
+  explorer-ui: docker.io/node:20-alpine
 architectures: [x86_64, aarch64]
 volumes:
  main: /data
 ports:
  ui: 8080
-  spaced_rpc: 7225  # loopback only
+  explorer_ui: 3000     # external, only useful while explorer enabled
+  spaced_rpc: 7225      # loopback only
+  postgres: 5432        # loopback only
 dependencies:
  - bitcoind
-startos_managed_env_vars:
+spaced_env_vars:
  - SPACED_CHAIN
  - SPACED_DATA_DIR
  - SPACED_RPC_BIND
  - SPACED_RPC_PORT
  - SPACED_RPC_URL
+  - SPACED_RPC_USER
+  - SPACED_RPC_PASSWORD
  - SPACED_BLOCK_INDEX
  - SPACED_BITCOIN_RPC_URL
  - SPACED_BITCOIN_RPC_USER
  - SPACED_BITCOIN_RPC_PASSWORD
-  - SPACED_RPC_COOKIE
  - BTC_RPC_HOST
  - BTC_RPC_PORT
  - BTC_RPC_USER
  - BTC_RPC_PASSWORD
  - APP_USER
  - APP_PASSWORD
+  - DB_URL
+postgres_env_vars:
+  - POSTGRES_USER
+  - POSTGRES_PASSWORD
+  - POSTGRES_DB
+  - PGDATA
+explorer:
+  default_enabled: false   # opt-in via Enable Embedded Explorer action
+  store_field: enableExplorer
+  bundled_when_disabled: true   # images stay in .s9pk; data on /data preserved
+indexer:
+  source_repo: spacesprotocol/explorer-indexer
+  pinned_commit: 00ae1e548734d93f1a8bb9f48d2290f459e12b35
+  install_dir: /data/explorer-indexer
+  bin_dir: /data/explorer-indexer/bin
+  schema_dir: /data/explorer-indexer/sql/schema
+  language: Go
+  build_image: golang:1.23-alpine
+  migrations_tool: goose
+  goose_version: v3.24.3
+explorer_ui:
+  source_repo: randomlogin/explorer
+  pinned_commit: c827da1754c3cba5c5507d2c29f21b8fa231344d
+  install_dir: /data/explorer-ui
+  build_dir: /data/explorer-ui/build
+  language: TypeScript (SvelteKit)
+  build_image: node:20-alpine
+  build_command: PUBLIC_BTC_NETWORK=mainnet npm run build
+  daemon_command: node /data/explorer-ui/build
+  port: 3000
+  managed_env_vars:
+    - POSTGRES_URI
+    - BITCOIN_NODE_URI
+    - BITCOIN_NODE_USER
+    - BITCOIN_NODE_PASSWORD
+    - SPACES_NODE_URI
+    - RPC_USER
+    - RPC_PASSWORD
+    - ACTIVATION_BLOCK_HEIGHT
+    - FAST_SYNC_BLOCK_HEIGHT
+    - UPDATE_DB_INTERVAL
+    - MEMPOOL_CHUNK_SIZE
+  defaults:
+    ACTIVATION_BLOCK_HEIGHT: '871222'  # spaces mainnet activation
+    FAST_SYNC_BLOCK_HEIGHT: '864000'
+    UPDATE_DB_INTERVAL: '5'
+    MEMPOOL_CHUNK_SIZE: '200'
 actions:
  - reset-password
  - show-credentials
@@ -196,4 +354,10 @@ actions:
  - reset-spaced-state
  - export-wallet
  - import-wallet
+  - enable-explorer
+  - disable-explorer
+  - show-db-credentials
+  - reset-db-state
+  - reset-indexer-state
+  - reset-explorer-state
 ```