From e41ff312212990fc193495a09d5187e3028b4b5c Mon Sep 17 00:00:00 2001 From: spacesops Date: Wed, 20 May 2026 15:43:38 -0400 Subject: [PATCH] Added optional Explorer/Indexer --- README.md | 220 +++++++-- startos/actions/disableExplorer.ts | 37 ++ startos/actions/enableExplorer.ts | 37 ++ startos/actions/exportWallet.ts | 15 +- startos/actions/importWallet.ts | 20 +- startos/actions/index.ts | 12 + startos/actions/resetDbState.ts | 58 +++ startos/actions/resetExplorerState.ts | 58 +++ startos/actions/resetIndexerState.ts | 58 +++ startos/actions/resetPassword.ts | 2 +- startos/actions/showCredentials.ts | 4 +- startos/actions/showDbCredentials.ts | 81 ++++ startos/actions/showPassword.ts | 4 +- startos/actions/syncStatus.ts | 17 +- startos/fileModels/storeJson.ts | 16 + startos/i18n/dictionaries/default.ts | 64 ++- startos/init/index.ts | 6 + startos/init/taskSeedDb.ts | 20 + startos/init/taskSeedEnableExplorer.ts | 13 + startos/init/taskSeedSpacedAuth.ts | 19 + startos/interfaces.ts | 26 +- startos/main.ts | 625 ++++++++++++++++++++++++- startos/manifest/index.ts | 12 + startos/utils.ts | 36 ++ startos/versions/index.ts | 4 +- startos/versions/v0.0.9.0.a1.ts | 14 - startos/versions/v0.0.9.1.ts | 23 + 27 files changed, 1431 insertions(+), 70 deletions(-) create mode 100644 startos/actions/disableExplorer.ts create mode 100644 startos/actions/enableExplorer.ts create mode 100644 startos/actions/resetDbState.ts create mode 100644 startos/actions/resetExplorerState.ts create mode 100644 startos/actions/resetIndexerState.ts create mode 100644 startos/actions/showDbCredentials.ts create mode 100644 startos/init/taskSeedDb.ts create mode 100644 startos/init/taskSeedEnableExplorer.ts create mode 100644 startos/init/taskSeedSpacedAuth.ts delete mode 100644 startos/versions/v0.0.9.0.a1.ts create mode 100644 startos/versions/v0.0.9.1.ts diff --git a/README.md b/README.md index ceb8a46..e39dbec 100644 --- a/README.md +++ b/README.md @@ -37,13 +37,21 @@ through a browser-based terminal. | Field | Value | | --- | --- | -| Image | `docker.io/horologger/spaces` | +| Spaces image | `docker.io/horologger/spaces` | +| PostgreSQL image | `docker.io/postgres:16.3` | +| Indexer-Go image | `docker.io/golang:1.23-alpine` | +| Explorer-UI image | `docker.io/node:20-alpine` | | Architectures | `linux/amd64`, `linux/arm64` | -| Entrypoint | StartOS-managed (image entrypoint is **not** used) | +| Entrypoint | StartOS-managed (image entrypoints are **not** used directly) | -The image bundles `spaced`, `space-cli`, `bitcoin-cli`, `gotty`, `node`, `npm`, -`screen`, and a small shell environment. StartOS ignores the image's -`docker_entrypoint.sh`; daemons are defined in `startos/main.ts`. +The Spaces image bundles `spaced`, `space-cli`, `bitcoin-cli`, `gotty`, `node`, +`npm`, `screen`, and a small shell environment. StartOS ignores the Spaces +image's `docker_entrypoint.sh`; daemons are defined in `startos/main.ts`. The +PostgreSQL image is the upstream `postgres:16.3`, launched via its official +`docker-entrypoint.sh`. The Indexer-Go image is the upstream +`golang:1.23-alpine`, used both to compile the indexer's `sync` binary + +`goose` migrator (one-time, at install) and to run the compiled binaries on +each start. ## Volume and Data Layout @@ -51,53 +59,115 @@ The image bundles `spaced`, `space-cli`, `bitcoin-cli`, `gotty`, `node`, `npm`, | --- | --- | --- | | `/data` | `main` | Spaces data directory (`SPACED_DATA_DIR`), wallets, indexes, and `store.json` | | `/data/mainnet/.cookie` | `main` | Spaced RPC cookie (auto-generated by `spaced` at startup) | -| `/data/store.json` | `main` | StartOS-managed credentials (web-UI password + bitcoind RPC user/password) | +| `/data/store.json` | `main` | StartOS-managed credentials (web-UI password, bitcoind RPC user/password, PostgreSQL user/password/database) | +| `/data/postgres` | `main` | PostgreSQL data directory (`PGDATA`). Owned by uid 999 (`postgres`). The whole `main` volume is backed up, so the database is included. | +| `/data/explorer-indexer` | `main` | Spaces-protocol explorer-indexer source (`spacesprotocol/explorer-indexer`), fetched from GitHub on first start. Contains `cmd/sync/`, `pkg/`, `sql/schema/`, `go.mod`, etc. | +| `/data/explorer-indexer/bin/sync` | `main` | Compiled indexer sync binary (`go build ./cmd/sync`). | +| `/data/explorer-indexer/bin/goose` | `main` | Compiled goose migrator (`go install github.com/pressly/goose/v3/cmd/goose`). Used by `indexer-migrate` to apply `sql/schema/*.sql`. | +| `/data/explorer-indexer/.installed-sha` | `main` | Marker file recording the pinned commit + build-ID currently extracted. If the build-ID in the package code differs from the marker, the next start wipes and re-fetches. | +| `/data/explorer-ui` | `main` | SvelteKit explorer source (`randomlogin/explorer`), fetched on first explorer-enable. Contains `src/`, `static/`, `server.js`, `package.json`, etc. | +| `/data/explorer-ui/build` | `main` | SvelteKit node-adapter build output. `node build` (i.e. `node /data/explorer-ui/build`) runs the production server. | +| `/data/explorer-ui/node_modules` | `main` | npm dependencies (~300-500 MB). Re-fetched on `Reset Explorer UI State`. | +| `/data/explorer-ui/.installed-sha` | `main` | Marker file for the pinned explorer commit + build-ID. | ## Installation and First-Run Flow On the first install, StartOS: 1. Seeds `store.json.password` with a random 32-char alphanumeric admin password - and creates a critical task that points the user at **Show Web UI Credentials** + and creates a critical task that points the user at **Show Space-CLI Web UI Credentials** so the password can be copied before login. 2. Seeds `store.json.btcAuth` with a random `spaces:` RPC credential pair and creates a critical cross-service task on **Bitcoin** that runs bitcoind's `generate-rpc-dependent` action to register the credentials in `bitcoin.conf`. -3. Launches `spaced` as a managed daemon (no `screen`, no shell auto-start). -4. Launches the `gotty` web terminal once `spaced` is up. +3. Seeds `store.json.dbAuth` with a random PostgreSQL password (username + `postgres`, database `spacesprotocol_explorer`). No task is created — the + credentials are only used internally by the embedded daemon. +4. Runs the `postgres-chown` oneshot to create `/data/postgres` with the right + ownership, then launches the PostgreSQL daemon on loopback 5432. +5. Seeds `store.json.spacedAuth` with a random `spaces:` credential + pair. Spaced is configured to require these via `SPACED_RPC_USER` / + `SPACED_RPC_PASSWORD` (cookie auth is **not** used in this package), so the + gotty terminal's `spaces` alias and the indexer share a single auth path. +6. Seeds `store.json.enableExplorer = false`. The embedded explorer + (PostgreSQL + Go indexer) is **opt-in**. +7. Launches `spaced` as a managed daemon (no `screen`, no shell auto-start) + and the `gotty` web terminal once the bashrc oneshot completes. -The user is **not** prompted to choose a chain or RPC mode -- this package is +The user is **not** prompted to choose a chain or RPC mode — this package is mainnet-only. +### Enabling the embedded explorer + +Run the **Enable Embedded Explorer** action. That flips +`store.json.enableExplorer = true` and the service auto-restarts with the full +daemon graph: + +8. Postgres-chown oneshot creates `/data/postgres` with correct ownership. +9. PostgreSQL daemon starts on loopback `127.0.0.1:5432`. +10. `indexer-fetch` downloads the pinned `spacesprotocol/explorer-indexer` + tarball into `/data/explorer-indexer` and applies two compatibility + patches for our older spaced binary (see Limitations). +11. `indexer-build` runs `CGO_ENABLED=0 go build -o /data/explorer-indexer/bin/sync ./cmd/sync` + and `GOBIN=/data/explorer-indexer/bin go install github.com/pressly/goose/v3/cmd/goose@v3.24.3`. +12. `indexer-cleanup-legacy` checks for and removes leftover TS-indexer schema + if present (from earlier package versions). +13. `indexer-migrate` runs `goose ... up` to apply `sql/schema/*.sql`. +14. `indexer` daemon (`/data/explorer-indexer/bin/sync`) starts polling + bitcoind + spaced and writes blocks, transactions, spaces, rollouts, etc. + to PostgreSQL. +15. `explorer-fetch` downloads the pinned `randomlogin/explorer` tarball into + `/data/explorer-ui` (skipped if marker already matches). +16. `explorer-install` runs `npm install` + `PUBLIC_BTC_NETWORK=mainnet npm run build` + to produce `/data/explorer-ui/build/`. Skipped if `build/index.js` already + exists. First-time install can take 1-3 minutes for the npm download. +17. `explorer-ui` daemon (`node /data/explorer-ui/build`) starts on port 3000. + The SvelteKit app reads exclusively from PostgreSQL — it does not talk to + spaced or bitcoind directly. The UI link will appear in the StartOS + dashboard alongside the gotty terminal. + +Run **Disable Embedded Explorer** to turn it back off; on-disk data at +`/data/postgres`, `/data/explorer-indexer`, and `/data/explorer-ui` is +preserved. + ## Configuration Management | StartOS-Managed | Upstream-Managed | | --- | --- | | Web-UI username / password (`admin` + generated password) | `spaced` runtime tuning via `SPACED_*` env vars in the image | | Bitcoin RPC username / password (registered on bitcoind) | Wallet creation, bidding, and registration -- all driven via `space-cli` inside the terminal | -| Chain selection (locked to `mainnet`) | `space-cli` flags and subcommands | +| PostgreSQL username / password / database (loopback only) | `space-cli` flags and subcommands | +| Chain selection (locked to `mainnet`) | | | Spaced data directory and RPC bind | | ## Network Access and Interfaces | Interface | Port | Protocol | Exposure | Notes | | --- | --- | --- | --- | --- | -| Web UI (gotty terminal) | 8080 | HTTP | LAN / `.local` / Tor / clearnet (via StartOS) | Basic auth: `admin:` | -| Spaced RPC | 7225 | HTTP JSON-RPC | **Loopback only** | Internal use, cookie-authenticated | +| Space-CLI Web UI (gotty terminal) | 8080 | HTTP | LAN / `.local` / Tor / clearnet (via StartOS) | Basic auth: `admin:` | +| Explorer Web UI | 3000 | HTTP | LAN / `.local` / Tor / clearnet (via StartOS) | SvelteKit explorer. No built-in auth — relies on StartOS interface exposure controls. Only useful while the embedded explorer is enabled (otherwise nothing is listening). | +| Spaced RPC | 7225 | HTTP JSON-RPC | **Loopback only** | Internal use, static-cred-authenticated via `SPACED_RPC_USER` / `SPACED_RPC_PASSWORD` from `store.json.spacedAuth` | +| PostgreSQL | 5432 | Postgres wire protocol | **Loopback only** (`listen_addresses=127.0.0.1`) | Username/password from `store.json.dbAuth`; full `DB_URL` exported into the web terminal | ## Actions | ID | Name | Visibility | Availability | Purpose | | --- | --- | --- | --- | --- | -| `reset-password` | Reset Web UI Password | Enabled | Any | Regenerates the web-UI password and restarts the terminal daemon | -| `show-credentials` | Show Web UI Credentials | Hidden | Any | Surfaces the current `admin` username + masked password (launched by the first-install task) | -| `show-password` | Show Web UI Password | Enabled | Any | Same as `show-credentials` but visible in the actions list, for routine re-display of the current admin credentials | +| `reset-password` | Reset Space-CLI Web UI Password | Enabled | Any | Regenerates the Space-CLI Web UI password and restarts the terminal daemon | +| `show-credentials` | Show Space-CLI Web UI Credentials | Hidden | Any | Surfaces the current `admin` username + masked password (launched by the first-install task) | +| `show-password` | Show Space-CLI Web UI Password | Enabled | Any | Same as `show-credentials` but visible in the actions list, for routine re-display of the current admin credentials | | `set-bitcoin-rpc` | Set up Bitcoin RPC | Enabled | Any | Re-invokes bitcoind's `generate-rpc-dependent` with the stored credentials. Safe to call repeatedly. | | `sync-status` | Sync Status | Enabled | Only running | Runs `space-cli getserverinfo` inside the daemon container and returns the JSON output | | `reset-spaced-state` | Reset Spaced State | Enabled | Any | Deletes `/data/mainnet/` so spaced resyncs its index from spaces' anchor. Preserves `store.json` (passwords + RPC credentials). Use when spaced crash-loops on a stale or corrupt index. | | `export-wallet` | Export Wallet | Enabled | Only running | Runs `space-cli exportwallet /data/mainnet/wallets_backup/default.json` and surfaces the resulting JSON as a masked/copyable result. The file is also persisted inside the volume at that path. | | `import-wallet` | Import Wallet | Enabled | Only running | Accepts a pasted JSON payload (textarea), writes it to `/data/mainnet/wallets_backup/default.json` (rotating the existing file to `.bakNNN`), rotates `/data/mainnet/wallets/default` to `.bakNNN`, then runs `space-cli importwallet` + `loadwallet`. | +| `enable-explorer` | Enable Embedded Explorer | Enabled (hidden when already on) | Any | Sets `store.enableExplorer = true` and triggers a service restart so the full daemon graph (postgres + indexer + indexer-sync HC) takes effect. | +| `disable-explorer` | Disable Embedded Explorer | Enabled (hidden when already off) | Any | Sets `store.enableExplorer = false` and triggers a service restart so the indexer and postgres daemons stop. On-disk data at `/data/postgres` and `/data/explorer-indexer` is preserved. | +| `show-db-credentials` | Show Database Credentials | Enabled | Any | Surfaces the PostgreSQL username, password, database, and full `DB_URL` (all copyable; secrets masked). Credentials exist regardless of whether the explorer is currently enabled. | +| `reset-db-state` | Reset Database State | Enabled | Any | Deletes `/data/postgres` so the next start re-initializes an empty database. Warning-gated. `store.json` is preserved. | +| `reset-indexer-state` | Reset Indexer State | Enabled | Any | Deletes `/data/explorer-indexer` so the next start re-fetches the indexer source and rebuilds the sync + goose binaries. Useful when bumping the pinned commit. PostgreSQL data is preserved. Warning-gated. | +| `reset-explorer-state` | Reset Explorer UI State | Enabled | Any | Deletes `/data/explorer-ui` so the next start re-fetches the SvelteKit source and rebuilds the bundle. PostgreSQL data is preserved. Warning-gated. | ## Backups and Restore @@ -110,23 +180,57 @@ the same idempotent init logic runs and reuses the existing credentials in | ID | Display | Grace period | Behaviour | | --- | --- | --- | --- | +| `postgres` (daemon `ready`) | Database | 60 s | TCP listen on `127.0.0.1:5432` | | `spaced` (daemon `ready`) | Spaced RPC | 120 s | TCP listen on `127.0.0.1:7225` | +| `indexer` (daemon `ready`) | Indexer Process | 60 s | Daemon liveness only — health is reported by `indexer-sync` below | +| `explorer-ui` (daemon `ready`) | Explorer Web UI | 60 s | TCP listen on `0.0.0.0:3000`. Only present when embedded explorer is enabled. | | `web-terminal` (daemon `ready`) | Web Interface | default | TCP listen on `0.0.0.0:8080` | -| `sync` (standalone) | Spaced Sync | 300 s | Exec `space-cli getserverinfo`; reports `success` when `ready=true`, otherwise `loading` with progress percentage | +| `sync` (standalone) | Spaced Sync | 30 s | Exec `space-cli --output-format json getserverinfo`; reports `success` when `ready=true && progress=100%`, otherwise `loading` with progress percentage | +| `indexer-sync` (standalone) | Indexer Sync | 60 s | `psql` the `blocks` table for `MAX(height) WHERE orphan = FALSE AND height >= 0`; cross-check against spaced's current tip via `space-cli getserverinfo`. `success` when within 5 blocks of the tip; otherwise `loading` with explicit lag. | ## Limitations and Differences 1. **Mainnet only.** Testnet, testnet4, signet, and regtest are not exposed. -2. **The image's `docker_entrypoint.sh` is not used.** Its auto-start of +2. **The Spaces image's `docker_entrypoint.sh` is not used.** Its auto-start of `spaced` inside `screen` would conflict with the StartOS-managed daemon. -3. **Web UI is a terminal, not a graphical app.** All Spaces operations happen +3. **Space-CLI Web UI is a terminal, not a graphical app.** All Spaces operations happen via `space-cli` (aliased as `spaces` inside the shell). -4. **Only port 8080 (gotty) is exposed.** Spaced RPC, and any other ports - present in the image (e.g. 22253, 3000, 5173, 8081) are not bound. +4. **Only port 8080 (gotty) is exposed externally.** Spaced RPC (7225) and + PostgreSQL (5432) are loopback-only, and any other ports present in the + image (e.g. 22253, 3000, 5173, 8081) are not bound. 5. **Bitcoin Core 31.x is the only supported dependency.** Earlier majors are not allowed by the manifest version range. 6. **The web terminal is independent of spaced.** Gotty stays reachable even when `spaced` is crash-looping, so you can always shell in to diagnose. +7. **The embedded explorer is opt-in.** Fresh installs run in spaces-only + mode (spaced + gotty). Run **Enable Embedded Explorer** to start + PostgreSQL + the Go indexer; **Disable Embedded Explorer** stops them. The + `postgres` and `indexer-go` images are bundled in the .s9pk regardless so + toggling is instant, no re-download. Indexed data is preserved across + toggles. +8. **PostgreSQL is embedded, not a separate service.** There is no + cross-service dependency on a postgres .s9pk; the database is local to this + package, lives at `/data/postgres`, and is intended for the embedded + indexer (and any future explorer UI). Connect to it from inside the gotty + terminal with `psql $DB_URL` (only meaningful while the explorer is + enabled). +9. **Enabling the explorer requires internet on its first start.** The pinned + commit of `spacesprotocol/explorer-indexer` is downloaded from GitHub into + `/data/explorer-indexer`, and `go build` + `go install github.com/pressly/goose/v3/cmd/goose` + fetch Go modules. After that the indexer runs offline; the only network it + needs is bitcoind (already a StartOS dependency) and the local spaced RPC. +10. **Explorer first-enable latency is ~2-5 minutes.** The Go indexer needs to + download modules and compile from scratch the first time you turn it on. + Subsequent restarts (and toggle off→on) skip `indexer-fetch` + + `indexer-build` because the binaries already exist on the volume. Use the + **Reset Indexer State** action to force a rebuild. +11. **Subspaces pointers data is not indexed.** Our spaced binary predates the + pointers feature. The indexer source is patched during `indexer-fetch` to + treat `ptrs_root` as optional and to skip the `getptrblockmeta` RPC call. + All other spaces protocol data — blocks, transactions, spaces, rollouts, + root anchors with empty pointers root — indexes normally. A future + rebuild of `horologger/spaces` with a newer spaced will let us drop these + patches. ## What Is Unchanged from Upstream @@ -142,11 +246,11 @@ the same idempotent init logic runs and reuses the existing credentials in After install: -1. Run the **Show Web UI Credentials** action (the install task surfaces it). -2. Open the Web UI from the StartOS dashboard. +1. Run the **Show Space-CLI Web UI Credentials** action (the install task surfaces it). +2. Open the Space-CLI Web UI from the StartOS dashboard. 3. Log in with `admin` and the displayed password. 4. Use `spaces ` -- it expands to - `space-cli --chain mainnet --rpc-cookie /data/mainnet/.cookie `. + `space-cli --chain mainnet --rpc-user "$SPACED_RPC_USER" --rpc-password "$SPACED_RPC_PASSWORD" `. Examples: @@ -161,32 +265,86 @@ spaces walletbalance default ```yaml package_id: spaces upstream_version: subspacesplus -image: docker.io/horologger/spaces:v0.0.9s +images: + spaces: docker.io/horologger/spaces:v0.0.9s + postgres: docker.io/postgres:16.3 + indexer-go: docker.io/golang:1.23-alpine + explorer-ui: docker.io/node:20-alpine architectures: [x86_64, aarch64] volumes: main: /data ports: ui: 8080 - spaced_rpc: 7225 # loopback only + explorer_ui: 3000 # external, only useful while explorer enabled + spaced_rpc: 7225 # loopback only + postgres: 5432 # loopback only dependencies: - bitcoind -startos_managed_env_vars: +spaced_env_vars: - SPACED_CHAIN - SPACED_DATA_DIR - SPACED_RPC_BIND - SPACED_RPC_PORT - SPACED_RPC_URL + - SPACED_RPC_USER + - SPACED_RPC_PASSWORD - SPACED_BLOCK_INDEX - SPACED_BITCOIN_RPC_URL - SPACED_BITCOIN_RPC_USER - SPACED_BITCOIN_RPC_PASSWORD - - SPACED_RPC_COOKIE - BTC_RPC_HOST - BTC_RPC_PORT - BTC_RPC_USER - BTC_RPC_PASSWORD - APP_USER - APP_PASSWORD + - DB_URL +postgres_env_vars: + - POSTGRES_USER + - POSTGRES_PASSWORD + - POSTGRES_DB + - PGDATA +explorer: + default_enabled: false # opt-in via Enable Embedded Explorer action + store_field: enableExplorer + bundled_when_disabled: true # images stay in .s9pk; data on /data preserved +indexer: + source_repo: spacesprotocol/explorer-indexer + pinned_commit: 00ae1e548734d93f1a8bb9f48d2290f459e12b35 + install_dir: /data/explorer-indexer + bin_dir: /data/explorer-indexer/bin + schema_dir: /data/explorer-indexer/sql/schema + language: Go + build_image: golang:1.23-alpine + migrations_tool: goose + goose_version: v3.24.3 +explorer_ui: + source_repo: randomlogin/explorer + pinned_commit: c827da1754c3cba5c5507d2c29f21b8fa231344d + install_dir: /data/explorer-ui + build_dir: /data/explorer-ui/build + language: TypeScript (SvelteKit) + build_image: node:20-alpine + build_command: PUBLIC_BTC_NETWORK=mainnet npm run build + daemon_command: node /data/explorer-ui/build + port: 3000 + managed_env_vars: + - POSTGRES_URI + - BITCOIN_NODE_URI + - BITCOIN_NODE_USER + - BITCOIN_NODE_PASSWORD + - SPACES_NODE_URI + - RPC_USER + - RPC_PASSWORD + - ACTIVATION_BLOCK_HEIGHT + - FAST_SYNC_BLOCK_HEIGHT + - UPDATE_DB_INTERVAL + - MEMPOOL_CHUNK_SIZE + defaults: + ACTIVATION_BLOCK_HEIGHT: '871222' # spaces mainnet activation + FAST_SYNC_BLOCK_HEIGHT: '864000' + UPDATE_DB_INTERVAL: '5' + MEMPOOL_CHUNK_SIZE: '200' actions: - reset-password - show-credentials @@ -196,4 +354,10 @@ actions: - reset-spaced-state - export-wallet - import-wallet + - enable-explorer + - disable-explorer + - show-db-credentials + - reset-db-state + - reset-indexer-state + - reset-explorer-state ``` diff --git a/startos/actions/disableExplorer.ts b/startos/actions/disableExplorer.ts new file mode 100644 index 0000000..46da282 --- /dev/null +++ b/startos/actions/disableExplorer.ts @@ -0,0 +1,37 @@ +import { storeJson } from '../fileModels/storeJson' +import { i18n } from '../i18n' +import { sdk } from '../sdk' + +export const disableExplorer = sdk.Action.withoutInput( + // id + 'disable-explorer', + + // metadata + async ({ effects }) => { + const current = await storeJson.read((s) => s.enableExplorer).once() + return { + name: i18n('Disable Embedded Explorer'), + description: i18n( + 'Stop the embedded PostgreSQL + Go indexer. Indexed data on disk (/data/postgres, /data/explorer-indexer) is preserved and will be reused if the explorer is re-enabled later. Service restarts automatically.', + ), + warning: null, + allowedStatuses: 'any', + group: null, + visibility: current === true ? 'enabled' : 'hidden', + } + }, + + // run + async ({ effects }) => { + await storeJson.merge(effects, { enableExplorer: false }) + + return { + version: '1', + title: i18n('Success'), + message: i18n( + 'Embedded explorer disabled. The service is restarting in spaces-only mode (spaced + web terminal). Run "Enable Embedded Explorer" to turn it back on.', + ), + result: null, + } + }, +) diff --git a/startos/actions/enableExplorer.ts b/startos/actions/enableExplorer.ts new file mode 100644 index 0000000..f6764dc --- /dev/null +++ b/startos/actions/enableExplorer.ts @@ -0,0 +1,37 @@ +import { storeJson } from '../fileModels/storeJson' +import { i18n } from '../i18n' +import { sdk } from '../sdk' + +export const enableExplorer = sdk.Action.withoutInput( + // id + 'enable-explorer', + + // metadata + async ({ effects }) => { + const current = await storeJson.read((s) => s.enableExplorer).once() + return { + name: i18n('Enable Embedded Explorer'), + description: i18n( + 'Turn on the embedded PostgreSQL + Go indexer. Spaces protocol data (blocks, transactions, spaces, rollouts) will be indexed locally for use by the future explorer web UI. Service restarts automatically.', + ), + warning: null, + allowedStatuses: 'any', + group: null, + visibility: current === true ? 'hidden' : 'enabled', + } + }, + + // run + async ({ effects }) => { + await storeJson.merge(effects, { enableExplorer: true }) + + return { + version: '1', + title: i18n('Success'), + message: i18n( + 'Embedded explorer enabled. The service is restarting; the indexer will begin fetching blocks shortly. First-time start can take several minutes for the Go build.', + ), + result: null, + } + }, +) diff --git a/startos/actions/exportWallet.ts b/startos/actions/exportWallet.ts index ffee668..a236cc3 100644 --- a/startos/actions/exportWallet.ts +++ b/startos/actions/exportWallet.ts @@ -1,3 +1,4 @@ +import { storeJson } from '../fileModels/storeJson' import { i18n } from '../i18n' import { sdk } from '../sdk' import { dataDir, SPACED_CHAIN } from '../utils' @@ -23,6 +24,18 @@ export const exportWallet = sdk.Action.withoutInput( // run async ({ effects }) => { + const spacedAuth = await storeJson.read((s) => s.spacedAuth).once() + if (!spacedAuth) { + return { + version: '1', + title: i18n('Failure'), + message: i18n('Could not export wallet: ${error}', { + error: 'spacedAuth missing from store.json', + }), + result: null, + } + } + const mounts = sdk.Mounts.of().mountVolume({ volumeId: 'main', subpath: null, @@ -32,7 +45,7 @@ export const exportWallet = sdk.Action.withoutInput( const cmd = [ `mkdir -p ${BACKUP_DIR}`, - `/root/.cargo/bin/space-cli --chain ${SPACED_CHAIN} --rpc-cookie ${dataDir}/${SPACED_CHAIN}/.cookie exportwallet ${BACKUP_PATH}`, + `/root/.cargo/bin/space-cli --chain ${SPACED_CHAIN} --rpc-user ${spacedAuth.username} --rpc-password ${spacedAuth.password} exportwallet ${BACKUP_PATH}`, ].join(' && ') const result = await sdk.SubContainer.withTemp( diff --git a/startos/actions/importWallet.ts b/startos/actions/importWallet.ts index f07e938..7d624b6 100644 --- a/startos/actions/importWallet.ts +++ b/startos/actions/importWallet.ts @@ -1,3 +1,4 @@ +import { storeJson } from '../fileModels/storeJson' import { i18n } from '../i18n' import { sdk } from '../sdk' import { dataDir, SPACED_CHAIN } from '../utils' @@ -7,7 +8,6 @@ const { InputSpec, Value } = sdk const BACKUP_DIR = `${dataDir}/${SPACED_CHAIN}/wallets_backup` const BACKUP_PATH = `${BACKUP_DIR}/default.json` const WALLET_DIR = `${dataDir}/${SPACED_CHAIN}/wallets/default` -const COOKIE_PATH = `${dataDir}/${SPACED_CHAIN}/.cookie` const inputSpec = InputSpec.of({ walletJson: Value.textarea({ @@ -63,9 +63,23 @@ export const importWallet = sdk.Action.withInput( } } + const spacedAuth = await storeJson.read((s) => s.spacedAuth).once() + if (!spacedAuth) { + return { + version: '1', + title: i18n('Failure'), + message: i18n('Could not import wallet: ${error}', { + error: 'spacedAuth missing from store.json', + }), + result: null, + } + } + // base64-encode so shell quoting can never break on user content. const b64 = Buffer.from(input.walletJson, 'utf8').toString('base64') + const cliAuth = `--rpc-user '${spacedAuth.username}' --rpc-password '${spacedAuth.password}'` + const script = `set -e mkdir -p '${BACKUP_DIR}' @@ -93,8 +107,8 @@ if [ -d "$WDIR" ]; then fi # Import + load via spaced RPC. -/root/.cargo/bin/space-cli --chain ${SPACED_CHAIN} --rpc-cookie '${COOKIE_PATH}' importwallet "$BACKUP" -/root/.cargo/bin/space-cli --chain ${SPACED_CHAIN} --rpc-cookie '${COOKIE_PATH}' loadwallet +/root/.cargo/bin/space-cli --chain ${SPACED_CHAIN} ${cliAuth} importwallet "$BACKUP" +/root/.cargo/bin/space-cli --chain ${SPACED_CHAIN} ${cliAuth} loadwallet ` const res = await sdk.SubContainer.withTemp( diff --git a/startos/actions/index.ts b/startos/actions/index.ts index 3e31ce6..3a8d6f4 100644 --- a/startos/actions/index.ts +++ b/startos/actions/index.ts @@ -1,10 +1,16 @@ import { sdk } from '../sdk' +import { disableExplorer } from './disableExplorer' +import { enableExplorer } from './enableExplorer' import { exportWallet } from './exportWallet' import { importWallet } from './importWallet' +import { resetDbState } from './resetDbState' +import { resetExplorerState } from './resetExplorerState' +import { resetIndexerState } from './resetIndexerState' import { resetPassword } from './resetPassword' import { resetSpacedState } from './resetSpacedState' import { setBitcoinRpc } from './setBitcoinRpc' import { showCredentials } from './showCredentials' +import { showDbCredentials } from './showDbCredentials' import { showPassword } from './showPassword' import { syncStatus } from './syncStatus' @@ -17,3 +23,9 @@ export const actions = sdk.Actions.of() .addAction(resetSpacedState) .addAction(exportWallet) .addAction(importWallet) + .addAction(enableExplorer) + .addAction(disableExplorer) + .addAction(showDbCredentials) + .addAction(resetDbState) + .addAction(resetIndexerState) + .addAction(resetExplorerState) diff --git a/startos/actions/resetDbState.ts b/startos/actions/resetDbState.ts new file mode 100644 index 0000000..9f3dea3 --- /dev/null +++ b/startos/actions/resetDbState.ts @@ -0,0 +1,58 @@ +import { i18n } from '../i18n' +import { sdk } from '../sdk' +import { dataDir, pgDataDir } from '../utils' + +export const resetDbState = sdk.Action.withoutInput( + // id + 'reset-db-state', + + // metadata + async ({ effects }) => ({ + name: i18n('Reset Database State'), + description: i18n( + 'Wipe /data/postgres so PostgreSQL re-initializes from scratch.', + ), + warning: i18n( + 'This deletes all PostgreSQL data on disk. The next start will re-create an empty database. store.json (passwords + RPC credentials) is preserved.', + ), + allowedStatuses: 'any', + group: null, + visibility: 'enabled', + }), + + // run + async ({ effects }) => { + const res = await sdk.SubContainer.withTemp( + effects, + { imageId: 'spaces' }, + sdk.Mounts.of().mountVolume({ + volumeId: 'main', + subpath: null, + mountpoint: dataDir, + readonly: false, + }), + 'spaces-reset-db', + (subc) => subc.exec(['rm', '-rf', pgDataDir], { user: 'root' }), + ) + + if (res.exitCode !== 0) { + return { + version: '1', + title: i18n('Failure'), + message: i18n('Could not wipe PostgreSQL state: ${error}', { + error: (res.stderr ?? '').toString() || `exit ${res.exitCode}`, + }), + result: null, + } + } + + return { + version: '1', + title: i18n('Success'), + message: i18n( + 'PostgreSQL data has been wiped. Start (or restart) the service to re-initialize the database.', + ), + result: null, + } + }, +) diff --git a/startos/actions/resetExplorerState.ts b/startos/actions/resetExplorerState.ts new file mode 100644 index 0000000..fd1feda --- /dev/null +++ b/startos/actions/resetExplorerState.ts @@ -0,0 +1,58 @@ +import { i18n } from '../i18n' +import { sdk } from '../sdk' +import { dataDir, EXPLORER_DIR } from '../utils' + +export const resetExplorerState = sdk.Action.withoutInput( + // id + 'reset-explorer-state', + + // metadata + async ({ effects }) => ({ + name: i18n('Reset Explorer UI State'), + description: i18n( + 'Wipe /data/explorer-ui so the next start re-fetches the explorer source and rebuilds it from scratch.', + ), + warning: i18n( + 'This deletes the cached explorer source and the built SvelteKit bundle. The next start will need internet access to re-download from GitHub and to fetch npm dependencies. PostgreSQL data is preserved.', + ), + allowedStatuses: 'any', + group: null, + visibility: 'enabled', + }), + + // run + async ({ effects }) => { + const res = await sdk.SubContainer.withTemp( + effects, + { imageId: 'explorer-ui' }, + sdk.Mounts.of().mountVolume({ + volumeId: 'main', + subpath: null, + mountpoint: dataDir, + readonly: false, + }), + 'spaces-reset-explorer', + (subc) => subc.exec(['rm', '-rf', EXPLORER_DIR], { user: 'root' }), + ) + + if (res.exitCode !== 0) { + return { + version: '1', + title: i18n('Failure'), + message: i18n('Could not wipe explorer state: ${error}', { + error: (res.stderr ?? '').toString() || `exit ${res.exitCode}`, + }), + result: null, + } + } + + return { + version: '1', + title: i18n('Success'), + message: i18n( + 'Explorer cache has been wiped. Start (or restart) the service to re-fetch and rebuild from GitHub.', + ), + result: null, + } + }, +) diff --git a/startos/actions/resetIndexerState.ts b/startos/actions/resetIndexerState.ts new file mode 100644 index 0000000..de65ae0 --- /dev/null +++ b/startos/actions/resetIndexerState.ts @@ -0,0 +1,58 @@ +import { i18n } from '../i18n' +import { sdk } from '../sdk' +import { dataDir, INDEXER_DIR } from '../utils' + +export const resetIndexerState = sdk.Action.withoutInput( + // id + 'reset-indexer-state', + + // metadata + async ({ effects }) => ({ + name: i18n('Reset Indexer State'), + description: i18n( + 'Wipe /data/explorer-indexer so the next start re-fetches the indexer source and rebuilds the sync + goose binaries.', + ), + warning: i18n( + 'This deletes the cached indexer source and compiled binaries. The next start will need internet access to re-download from GitHub and to fetch Go modules. PostgreSQL data (the indexed blocks themselves) is preserved — use Reset Database State if you also want to clear that.', + ), + allowedStatuses: 'any', + group: null, + visibility: 'enabled', + }), + + // run + async ({ effects }) => { + const res = await sdk.SubContainer.withTemp( + effects, + { imageId: 'spaces' }, + sdk.Mounts.of().mountVolume({ + volumeId: 'main', + subpath: null, + mountpoint: dataDir, + readonly: false, + }), + 'spaces-reset-indexer', + (subc) => subc.exec(['rm', '-rf', INDEXER_DIR], { user: 'root' }), + ) + + if (res.exitCode !== 0) { + return { + version: '1', + title: i18n('Failure'), + message: i18n('Could not wipe indexer state: ${error}', { + error: (res.stderr ?? '').toString() || `exit ${res.exitCode}`, + }), + result: null, + } + } + + return { + version: '1', + title: i18n('Success'), + message: i18n( + 'Indexer cache has been wiped. Start (or restart) the service to re-fetch and rebuild from GitHub.', + ), + result: null, + } + }, +) diff --git a/startos/actions/resetPassword.ts b/startos/actions/resetPassword.ts index 1d4169a..18c2ff0 100644 --- a/startos/actions/resetPassword.ts +++ b/startos/actions/resetPassword.ts @@ -9,7 +9,7 @@ export const resetPassword = sdk.Action.withoutInput( // metadata async ({ effects }) => ({ - name: i18n('Reset Web UI Password'), + name: i18n('Reset Space-CLI Web UI Password'), description: i18n( 'Generate a new admin password for the Spaces web terminal', ), diff --git a/startos/actions/showCredentials.ts b/startos/actions/showCredentials.ts index 4cb9d80..579575d 100644 --- a/startos/actions/showCredentials.ts +++ b/startos/actions/showCredentials.ts @@ -9,7 +9,7 @@ export const showCredentials = sdk.Action.withoutInput( // metadata async ({ effects }) => ({ - name: i18n('Show Web UI Credentials'), + name: i18n('Show Space-CLI Web UI Credentials'), description: i18n( 'Display the username and password for the Spaces web terminal', ), @@ -25,7 +25,7 @@ export const showCredentials = sdk.Action.withoutInput( return { version: '1', - title: i18n('Show Web UI Credentials'), + title: i18n('Show Space-CLI Web UI Credentials'), message: i18n( 'Use these credentials to log in to the Spaces web terminal.', ), diff --git a/startos/actions/showDbCredentials.ts b/startos/actions/showDbCredentials.ts new file mode 100644 index 0000000..198e8c1 --- /dev/null +++ b/startos/actions/showDbCredentials.ts @@ -0,0 +1,81 @@ +import { storeJson } from '../fileModels/storeJson' +import { i18n } from '../i18n' +import { sdk } from '../sdk' +import { POSTGRES_PORT } from '../utils' + +export const showDbCredentials = sdk.Action.withoutInput( + // id + 'show-db-credentials', + + // metadata + async ({ effects }) => ({ + name: i18n('Show Database Credentials'), + description: i18n( + 'Display the PostgreSQL username, password, database, and connection URL.', + ), + warning: null, + allowedStatuses: 'any', + group: null, + visibility: 'enabled', + }), + + // run + async ({ effects }) => { + const dbAuth = await storeJson.read((s) => s.dbAuth).once() + const username = dbAuth?.username ?? '' + const password = dbAuth?.password ?? '' + const database = dbAuth?.database ?? '' + const url = dbAuth + ? `postgres://${dbAuth.username}:${dbAuth.password}@127.0.0.1:${POSTGRES_PORT}/${dbAuth.database}` + : '' + + return { + version: '1', + title: i18n('Show Database Credentials'), + message: i18n( + 'Use these credentials to connect to the Spaces PostgreSQL database (loopback only inside the container).', + ), + result: { + type: 'group', + value: [ + { + type: 'single', + name: i18n('Username'), + description: null, + value: username, + masked: false, + copyable: true, + qr: false, + }, + { + type: 'single', + name: i18n('Password'), + description: null, + value: password, + masked: true, + copyable: true, + qr: false, + }, + { + type: 'single', + name: i18n('Database'), + description: null, + value: database, + masked: false, + copyable: true, + qr: false, + }, + { + type: 'single', + name: i18n('Connection URL'), + description: null, + value: url, + masked: true, + copyable: true, + qr: false, + }, + ], + }, + } + }, +) diff --git a/startos/actions/showPassword.ts b/startos/actions/showPassword.ts index d09cc2a..69d00a8 100644 --- a/startos/actions/showPassword.ts +++ b/startos/actions/showPassword.ts @@ -9,7 +9,7 @@ export const showPassword = sdk.Action.withoutInput( // metadata async ({ effects }) => ({ - name: i18n('Show Web UI Password'), + name: i18n('Show Space-CLI Web UI Password'), description: i18n( 'Display the existing username and password for the Spaces web terminal.', ), @@ -25,7 +25,7 @@ export const showPassword = sdk.Action.withoutInput( return { version: '1', - title: i18n('Show Web UI Password'), + title: i18n('Show Space-CLI Web UI Password'), message: i18n( 'Use these credentials to log in to the Spaces web terminal.', ), diff --git a/startos/actions/syncStatus.ts b/startos/actions/syncStatus.ts index 566e895..d0f464a 100644 --- a/startos/actions/syncStatus.ts +++ b/startos/actions/syncStatus.ts @@ -1,3 +1,4 @@ +import { storeJson } from '../fileModels/storeJson' import { i18n } from '../i18n' import { sdk } from '../sdk' import { dataDir, SPACED_CHAIN } from '../utils' @@ -20,6 +21,16 @@ export const syncStatus = sdk.Action.withoutInput( // run async ({ effects }) => { + const spacedAuth = await storeJson.read((s) => s.spacedAuth).once() + if (!spacedAuth) { + return { + version: '1', + title: i18n('Failure'), + message: i18n('Could not query spaced. Is the service running?'), + result: null, + } + } + const res = await sdk.SubContainer.withTemp( effects, { imageId: 'spaces' }, @@ -35,8 +46,10 @@ export const syncStatus = sdk.Action.withoutInput( '/root/.cargo/bin/space-cli', '--chain', SPACED_CHAIN, - '--rpc-cookie', - `${dataDir}/${SPACED_CHAIN}/.cookie`, + '--rpc-user', + spacedAuth.username, + '--rpc-password', + spacedAuth.password, 'getserverinfo', ]), ) diff --git a/startos/fileModels/storeJson.ts b/startos/fileModels/storeJson.ts index b5e3ba0..d61d904 100644 --- a/startos/fileModels/storeJson.ts +++ b/startos/fileModels/storeJson.ts @@ -10,6 +10,22 @@ const shape = z.object({ }) .nullable() .catch(null), + dbAuth: z + .object({ + username: z.string(), + password: z.string(), + database: z.string(), + }) + .nullable() + .catch(null), + spacedAuth: z + .object({ + username: z.string(), + password: z.string(), + }) + .nullable() + .catch(null), + enableExplorer: z.boolean().nullable().catch(null), }) export const storeJson = FileHelper.json( diff --git a/startos/i18n/dictionaries/default.ts b/startos/i18n/dictionaries/default.ts index b7bfe88..966aeb4 100644 --- a/startos/i18n/dictionaries/default.ts +++ b/startos/i18n/dictionaries/default.ts @@ -17,12 +17,68 @@ const dict = { 'getserverinfo non-JSON. stdout=${stdout} stderr=${stderr}': 11, 'Spaced Sync health check crashed: ${error}': 53, 'space-cli exited ${code}: ${error}': 54, - 'Show Web UI Password': 55, + 'Show Space-CLI Web UI Password': 55, 'Display the existing username and password for the Spaces web terminal.': 56, + Database: 57, + 'postgres is ready': 58, + 'postgres is not ready': 59, + 'Show Database Credentials': 60, + 'Display the PostgreSQL username, password, database, and connection URL.': + 61, + 'Reset Database State': 62, + 'Wipe /data/postgres so PostgreSQL re-initializes from scratch.': 63, + 'This deletes all PostgreSQL data on disk. The next start will re-create an empty database. store.json (passwords + RPC credentials) is preserved.': + 64, + 'PostgreSQL data has been wiped. Start (or restart) the service to re-initialize the database.': + 65, + 'Could not wipe PostgreSQL state: ${error}': 66, + 'Use these credentials to connect to the Spaces PostgreSQL database (loopback only inside the container).': + 67, + 'Connection URL': 68, + 'Indexer Process': 69, + 'indexer process is running': 70, + 'Indexer Sync': 71, + 'indexer psql exited ${code}: ${error}': 72, + 'indexer has not run a sync cycle yet.': 73, + 'indexer has not yet committed any blocks (spaced tip ${tip}).': 74, + 'indexer caught up at block ${end} (spaced tip ${tip}).': 76, + 'indexer at block ${end}, ${lag} behind spaced tip ${tip}.': 77, + 'Indexer Sync health check crashed: ${error}': 78, + 'Reset Indexer State': 79, + 'Wipe /data/explorer-indexer so the next start re-fetches the indexer source and rebuilds the sync + goose binaries.': + 80, + 'This deletes the cached indexer source and compiled binaries. The next start will need internet access to re-download from GitHub and to fetch Go modules. PostgreSQL data (the indexed blocks themselves) is preserved — use Reset Database State if you also want to clear that.': + 81, + 'Indexer cache has been wiped. Start (or restart) the service to re-fetch and rebuild from GitHub.': + 82, + 'Could not wipe indexer state: ${error}': 83, + 'Enable Embedded Explorer': 84, + 'Turn on the embedded PostgreSQL + Go indexer. Spaces protocol data (blocks, transactions, spaces, rollouts) will be indexed locally for use by the future explorer web UI. Service restarts automatically.': + 85, + 'Embedded explorer enabled. The service is restarting; the indexer will begin fetching blocks shortly. First-time start can take several minutes for the Go build.': + 86, + 'Disable Embedded Explorer': 87, + 'Stop the embedded PostgreSQL + Go indexer. Indexed data on disk (/data/postgres, /data/explorer-indexer) is preserved and will be reused if the explorer is re-enabled later. Service restarts automatically.': + 88, + 'Embedded explorer disabled. The service is restarting in spaces-only mode (spaced + web terminal). Run "Enable Embedded Explorer" to turn it back on.': + 89, + 'Explorer Web UI': 90, + 'explorer UI is ready': 91, + 'explorer UI is not ready': 92, + 'SvelteKit explorer for the Spaces protocol. Reads from the embedded PostgreSQL populated by the indexer. Only useful while the embedded explorer is enabled.': + 93, + 'Reset Explorer UI State': 94, + 'Wipe /data/explorer-ui so the next start re-fetches the explorer source and rebuilds it from scratch.': + 95, + 'This deletes the cached explorer source and the built SvelteKit bundle. The next start will need internet access to re-download from GitHub and to fetch npm dependencies. PostgreSQL data is preserved.': + 96, + 'Explorer cache has been wiped. Start (or restart) the service to re-fetch and rebuild from GitHub.': + 97, + 'Could not wipe explorer state: ${error}': 98, // interfaces.ts - 'Web UI': 12, + 'Space-CLI Web UI': 12, 'Browser terminal that exposes space-cli inside the Spaces container.': 13, // dependencies.ts / tasks @@ -30,9 +86,9 @@ const dict = { 'Spaces needs an admin password for the web terminal': 15, // actions - 'Reset Web UI Password': 16, + 'Reset Space-CLI Web UI Password': 16, 'Generate a new admin password for the Spaces web terminal': 17, - 'Show Web UI Credentials': 18, + 'Show Space-CLI Web UI Credentials': 18, 'Display the username and password for the Spaces web terminal': 19, 'Set up Bitcoin RPC': 20, 'Re-run the bitcoind RPC credential setup for Spaces': 21, diff --git a/startos/init/index.ts b/startos/init/index.ts index 3dd81a2..f614cb6 100644 --- a/startos/init/index.ts +++ b/startos/init/index.ts @@ -5,6 +5,9 @@ import { setInterfaces } from '../interfaces' import { sdk } from '../sdk' import { versionGraph } from '../versions' import { taskBtcAuth } from './taskBtcAuth' +import { taskSeedDb } from './taskSeedDb' +import { taskSeedEnableExplorer } from './taskSeedEnableExplorer' +import { taskSeedSpacedAuth } from './taskSeedSpacedAuth' import { taskSetPassword } from './taskSetPassword' export const init = sdk.setupInit( @@ -14,6 +17,9 @@ export const init = sdk.setupInit( setDependencies, actions, taskBtcAuth, + taskSeedDb, + taskSeedSpacedAuth, + taskSeedEnableExplorer, taskSetPassword, ) diff --git a/startos/init/taskSeedDb.ts b/startos/init/taskSeedDb.ts new file mode 100644 index 0000000..b9b62d4 --- /dev/null +++ b/startos/init/taskSeedDb.ts @@ -0,0 +1,20 @@ +import { storeJson } from '../fileModels/storeJson' +import { sdk } from '../sdk' +import { POSTGRES_DB, POSTGRES_USER, randomPassword } from '../utils' + +export const taskSeedDb = sdk.setupOnInit(async (effects) => { + const existing = await storeJson.read((s) => s.dbAuth).once() + if (existing) return + + await storeJson.merge( + effects, + { + dbAuth: { + username: POSTGRES_USER, + password: randomPassword(), + database: POSTGRES_DB, + }, + }, + { allowWriteAfterConst: true }, + ) +}) diff --git a/startos/init/taskSeedEnableExplorer.ts b/startos/init/taskSeedEnableExplorer.ts new file mode 100644 index 0000000..a935858 --- /dev/null +++ b/startos/init/taskSeedEnableExplorer.ts @@ -0,0 +1,13 @@ +import { storeJson } from '../fileModels/storeJson' +import { sdk } from '../sdk' + +export const taskSeedEnableExplorer = sdk.setupOnInit(async (effects) => { + const existing = await storeJson.read((s) => s.enableExplorer).once() + if (existing !== null && existing !== undefined) return + + await storeJson.merge( + effects, + { enableExplorer: false }, + { allowWriteAfterConst: true }, + ) +}) diff --git a/startos/init/taskSeedSpacedAuth.ts b/startos/init/taskSeedSpacedAuth.ts new file mode 100644 index 0000000..e7e7b18 --- /dev/null +++ b/startos/init/taskSeedSpacedAuth.ts @@ -0,0 +1,19 @@ +import { storeJson } from '../fileModels/storeJson' +import { sdk } from '../sdk' +import { randomPassword } from '../utils' + +export const taskSeedSpacedAuth = sdk.setupOnInit(async (effects) => { + const existing = await storeJson.read((s) => s.spacedAuth).once() + if (existing) return + + await storeJson.merge( + effects, + { + spacedAuth: { + username: 'spaces', + password: randomPassword(), + }, + }, + { allowWriteAfterConst: true }, + ) +}) diff --git a/startos/interfaces.ts b/startos/interfaces.ts index 79a9719..1fcc642 100644 --- a/startos/interfaces.ts +++ b/startos/interfaces.ts @@ -1,6 +1,6 @@ import { i18n } from './i18n' import { sdk } from './sdk' -import { uiPort } from './utils' +import { EXPLORER_PORT, uiPort } from './utils' export const setInterfaces = sdk.setupInterfaces(async ({ effects }) => { const uiMulti = sdk.MultiHost.of(effects, 'ui-multi') @@ -8,7 +8,7 @@ export const setInterfaces = sdk.setupInterfaces(async ({ effects }) => { protocol: 'http', }) const ui = sdk.createInterface(effects, { - name: i18n('Web UI'), + name: i18n('Space-CLI Web UI'), id: 'ui', description: i18n( 'Browser terminal that exposes space-cli inside the Spaces container.', @@ -23,5 +23,25 @@ export const setInterfaces = sdk.setupInterfaces(async ({ effects }) => { const uiReceipt = await uiMultiOrigin.export([ui]) - return [uiReceipt] + const explorerMulti = sdk.MultiHost.of(effects, 'explorer-multi') + const explorerMultiOrigin = await explorerMulti.bindPort(EXPLORER_PORT, { + protocol: 'http', + }) + const explorer = sdk.createInterface(effects, { + name: i18n('Explorer Web UI'), + id: 'explorer', + description: i18n( + 'SvelteKit explorer for the Spaces protocol. Reads from the embedded PostgreSQL populated by the indexer. Only useful while the embedded explorer is enabled.', + ), + type: 'ui', + masked: false, + schemeOverride: null, + username: null, + path: '', + query: {}, + }) + + const explorerReceipt = await explorerMultiOrigin.export([explorer]) + + return [uiReceipt, explorerReceipt] }) diff --git a/startos/main.ts b/startos/main.ts index ea58102..b7ee76b 100644 --- a/startos/main.ts +++ b/startos/main.ts @@ -6,6 +6,29 @@ import { BITCOIND_RPC_HOSTNAME, BITCOIND_RPC_PORT, dataDir, + EXPLORER_BUILD_DIR, + EXPLORER_BUILD_ID, + EXPLORER_DIR, + EXPLORER_MARKER, + EXPLORER_NETWORK, + EXPLORER_PORT, + EXPLORER_TARBALL_URL, + INDEXER_ACTIVATION_HEIGHT, + INDEXER_BIN_DIR, + INDEXER_BUILD_ID, + INDEXER_DIR, + INDEXER_FAST_SYNC_HEIGHT, + INDEXER_GOOSE_BIN, + INDEXER_MARKER, + INDEXER_MEMPOOL_CHUNK_SIZE, + INDEXER_SCHEMA_DIR, + INDEXER_SYNC_BIN, + INDEXER_TARBALL_URL, + INDEXER_UPDATE_INTERVAL, + pgDataDir, + POSTGRES_DB, + POSTGRES_PORT, + POSTGRES_USER, SPACED_CHAIN, spacedRpcPort, uiPort, @@ -15,13 +38,20 @@ export const main = sdk.setupMain(async ({ effects }) => { console.info(i18n('Starting Spaces!')) const store = await storeJson.read().const(effects) - if (!store?.password || !store?.btcAuth) { - // taskSetPassword + taskBtcAuth both seed these in init; if they aren't - // populated yet, init hasn't finished — let StartOS restart us. + if ( + !store?.password || + !store?.btcAuth || + !store?.dbAuth || + !store?.spacedAuth + ) { + // taskSetPassword + taskBtcAuth + taskSeedDb + taskSeedSpacedAuth all seed + // these in init; if they aren't populated yet, init hasn't finished — let + // StartOS restart us. throw new Error('Spaces store.json is not yet populated.') } - const { password: APP_PASSWORD, btcAuth } = store + const { password: APP_PASSWORD, btcAuth, dbAuth, spacedAuth } = store + const enableExplorer = store.enableExplorer === true const spacedEnv = { SPACED_CHAIN, @@ -33,7 +63,8 @@ export const main = sdk.setupMain(async ({ effects }) => { SPACED_BITCOIN_RPC_URL: `http://${BITCOIND_RPC_HOSTNAME}:${BITCOIND_RPC_PORT}`, SPACED_BITCOIN_RPC_USER: btcAuth.username, SPACED_BITCOIN_RPC_PASSWORD: btcAuth.password, - SPACED_RPC_COOKIE: `/data/mainnet/.cookie`, + SPACED_RPC_USER: spacedAuth.username, + SPACED_RPC_PASSWORD: spacedAuth.password, // legacy aliases for `bitcoin-cli` / shell helpers that read these names BTC_RPC_HOST: BITCOIND_RPC_HOSTNAME, BTC_RPC_PORT: String(BITCOIND_RPC_PORT), @@ -41,6 +72,7 @@ export const main = sdk.setupMain(async ({ effects }) => { BTC_RPC_PASSWORD: btcAuth.password, APP_USER, APP_PASSWORD, + DB_URL: `postgres://${dbAuth.username}:${dbAuth.password}@127.0.0.1:${POSTGRES_PORT}/${dbAuth.database}`, } const mounts = sdk.Mounts.of().mountVolume({ @@ -64,10 +96,65 @@ export const main = sdk.setupMain(async ({ effects }) => { 'terminal-sub', ) + const postgresSub = await sdk.SubContainer.of( + effects, + { imageId: 'postgres' }, + mounts, + 'postgres-sub', + ) + + const postgresEnv = { + POSTGRES_USER: dbAuth.username, + POSTGRES_PASSWORD: dbAuth.password, + POSTGRES_DB: dbAuth.database, + PGDATA: pgDataDir, + } + + const indexerSub = await sdk.SubContainer.of( + effects, + { imageId: 'indexer-go' }, + mounts, + 'indexer-sub', + ) + + const explorerSub = await sdk.SubContainer.of( + effects, + { imageId: 'explorer-ui' }, + mounts, + 'explorer-sub', + ) + + const explorerEnv = { + DB_URL: `postgres://${dbAuth.username}:${dbAuth.password}@127.0.0.1:${POSTGRES_PORT}/${dbAuth.database}?sslmode=disable`, + PUBLIC_BTC_NETWORK: EXPLORER_NETWORK, + PORT: String(EXPLORER_PORT), + HOME: '/root', + } + + const postgresUri = `postgres://${dbAuth.username}:${dbAuth.password}@127.0.0.1:${POSTGRES_PORT}/${dbAuth.database}?sslmode=disable` + + const indexerEnv = { + POSTGRES_URI: postgresUri, + BITCOIN_NODE_URI: `http://${BITCOIND_RPC_HOSTNAME}:${BITCOIND_RPC_PORT}`, + BITCOIN_NODE_USER: btcAuth.username, + BITCOIN_NODE_PASSWORD: btcAuth.password, + SPACES_NODE_URI: `http://127.0.0.1:${spacedRpcPort}`, + RPC_USER: spacedAuth.username, + RPC_PASSWORD: spacedAuth.password, + ACTIVATION_BLOCK_HEIGHT: INDEXER_ACTIVATION_HEIGHT, + FAST_SYNC_BLOCK_HEIGHT: INDEXER_FAST_SYNC_HEIGHT, + UPDATE_DB_INTERVAL: INDEXER_UPDATE_INTERVAL, + MEMPOOL_CHUNK_SIZE: INDEXER_MEMPOOL_CHUNK_SIZE, + PATH: `${INDEXER_BIN_DIR}:/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin`, + HOME: '/root', + GOPATH: '/root/go', + GOBIN: INDEXER_BIN_DIR, + } + const bashrc = [ 'export PATH=/root/.cargo/bin:/data/bin:/usr/local/bin:/usr/bin:/bin', "export PS1='spaces:\\w$ '", - `alias spaces='space-cli --chain ${SPACED_CHAIN} --rpc-cookie ${dataDir}/${SPACED_CHAIN}/.cookie '`, + `alias spaces='space-cli --chain ${SPACED_CHAIN} --rpc-user "$SPACED_RPC_USER" --rpc-password "$SPACED_RPC_PASSWORD" '`, 'cat < { 'EOF', ].join('\n') + if (!enableExplorer) { + // Spaces-only mode: spaced + gotty terminal. No PostgreSQL, no indexer. + // User can flip the toggle via the Enable Embedded Explorer action; + // store.json.enableExplorer is read via .const() so the merge triggers + // an automatic service restart and the full chain takes over. + return sdk.Daemons.of(effects) + .addOneshot('bashrc', { + subcontainer: termSub, + exec: { + command: ['bash', '-c', `cat > /root/.bashrc <<'SPACES_BASHRC_EOF' +${bashrc} +SPACES_BASHRC_EOF`], + user: 'root', + }, + requires: [], + }) + .addDaemon('spaced', { + subcontainer: spacedSub, + exec: { + command: ['/root/.cargo/bin/spaced'], + env: spacedEnv, + }, + ready: { + display: i18n('Spaced RPC'), + fn: () => + sdk.healthCheck.checkPortListening(effects, spacedRpcPort, { + successMessage: i18n('spaced RPC is ready'), + errorMessage: i18n('spaced RPC is not ready'), + }), + gracePeriod: 120_000, + }, + requires: [], + }) + .addDaemon('web-terminal', { + subcontainer: termSub, + exec: { + command: [ + 'gotty', + '--port', + String(uiPort), + '-c', + `${APP_USER}:${APP_PASSWORD}`, + '--permit-write', + '--reconnect', + '/bin/bash', + ], + env: spacedEnv, + }, + ready: { + display: i18n('Web Interface'), + fn: () => + sdk.healthCheck.checkPortListening(effects, uiPort, { + successMessage: i18n('The web terminal is ready'), + errorMessage: i18n('The web terminal is not ready'), + }), + }, + requires: ['bashrc'], + }) + .addHealthCheck('sync', { + ready: { + display: i18n('Spaced Sync'), + fn: async () => { + try { + const probe = await spacedSub.exec( + [ + '/root/.cargo/bin/space-cli', + '--chain', + SPACED_CHAIN, + '--rpc-user', + spacedAuth.username, + '--rpc-password', + spacedAuth.password, + '--output-format', + 'json', + 'getserverinfo', + ], + {}, + ) + const stderr = (probe.stderr ?? '').toString().trim() + const stdoutText = (probe.stdout ?? '').toString() + const stdoutTrimmed = stdoutText.trim() + if (probe.exitCode !== 0) { + return { + result: 'failure', + message: i18n('space-cli exited ${code}: ${error}', { + code: String(probe.exitCode), + error: (stderr || stdoutTrimmed || '').slice(0, 200), + }), + } + } + let parsed: { + ready?: boolean + progress?: number + chain?: { blocks?: number; headers?: number } + } + try { + parsed = JSON.parse(stdoutText) + } catch { + return { + result: 'failure', + message: i18n( + 'getserverinfo non-JSON. stdout=${stdout} stderr=${stderr}', + { + stdout: (stdoutTrimmed || '').slice(0, 160), + stderr: (stderr || '').slice(0, 160), + }, + ), + } + } + const progress = Math.min( + 100, + Math.max(0, Math.round((parsed.progress ?? 0) * 100)), + ) + const blocks = parsed.chain?.blocks ?? 0 + const headers = parsed.chain?.headers ?? 0 + if (parsed.ready === true && progress >= 100) { + return { + result: 'success', + message: i18n( + 'spaced is fully synced (blocks ${blocks} / headers ${headers}).', + { blocks: String(blocks), headers: String(headers) }, + ), + } + } + return { + result: 'loading', + message: i18n( + 'spaced is indexing: ${pct}% (blocks ${blocks} / headers ${headers}).', + { + pct: String(progress), + blocks: String(blocks), + headers: String(headers), + }, + ), + } + } catch (e) { + return { + result: 'failure', + message: i18n('Spaced Sync health check crashed: ${error}', { + error: (e as Error)?.message ?? String(e), + }), + } + } + }, + gracePeriod: 30_000, + trigger: sdk.trigger.cooldownTrigger(30_000), + }, + requires: ['spaced'], + }) + } + + // enableExplorer = true: full chain — spaced + terminal + embedded + // PostgreSQL + Go indexer + indexer-sync health check. return sdk.Daemons.of(effects) .addOneshot('bashrc', { subcontainer: termSub, @@ -91,6 +331,40 @@ SPACES_BASHRC_EOF`], }, requires: [], }) + .addOneshot('postgres-chown', { + subcontainer: postgresSub, + exec: { + command: [ + 'bash', + '-c', + `mkdir -p ${pgDataDir} && chown -R postgres:postgres ${pgDataDir} && chmod 700 ${pgDataDir}`, + ], + user: 'root', + }, + requires: [], + }) + .addDaemon('postgres', { + subcontainer: postgresSub, + exec: { + command: [ + 'docker-entrypoint.sh', + 'postgres', + '-c', + 'listen_addresses=127.0.0.1', + ], + env: postgresEnv, + }, + ready: { + display: i18n('Database'), + fn: () => + sdk.healthCheck.checkPortListening(effects, POSTGRES_PORT, { + successMessage: i18n('postgres is ready'), + errorMessage: i18n('postgres is not ready'), + }), + gracePeriod: 60_000, + }, + requires: ['postgres-chown'], + }) .addDaemon('spaced', { subcontainer: spacedSub, exec: { @@ -108,6 +382,215 @@ SPACES_BASHRC_EOF`], }, requires: [], }) + .addOneshot('indexer-fetch', { + subcontainer: indexerSub, + exec: { + command: [ + 'sh', + '-c', + `set -eu; \ + echo "indexer-fetch: image diagnostics..."; \ + cat /etc/os-release 2>/dev/null | head -3 || echo "no /etc/os-release"; \ + echo " go: $(go version 2>/dev/null || echo MISSING)"; \ + echo " git: $(git --version 2>/dev/null || echo MISSING)"; \ + echo " curl: $(curl --version 2>/dev/null | head -1 || echo MISSING)"; \ + echo " wget: $(wget --version 2>/dev/null | head -1 || echo MISSING)"; \ + echo " tar: $(tar --version 2>/dev/null | head -1 || echo MISSING)"; \ + mkdir -p ${INDEXER_DIR} ${INDEXER_BIN_DIR}; \ + if [ -f ${INDEXER_MARKER} ] && [ "$(cat ${INDEXER_MARKER})" = "${INDEXER_BUILD_ID}" ] && [ -x ${INDEXER_SYNC_BIN} ] && [ -x ${INDEXER_GOOSE_BIN} ]; then \ + echo "indexer-fetch: ${INDEXER_BUILD_ID} already present, skipping."; \ + exit 0; \ + fi; \ + echo "indexer-fetch: downloading ${INDEXER_BUILD_ID}..."; \ + rm -rf ${INDEXER_DIR}; \ + mkdir -p ${INDEXER_DIR} ${INDEXER_BIN_DIR}; \ + if command -v curl >/dev/null 2>&1; then \ + curl -fsSL '${INDEXER_TARBALL_URL}' -o /tmp/explorer-indexer.tar.gz; \ + elif command -v wget >/dev/null 2>&1; then \ + wget -q -O /tmp/explorer-indexer.tar.gz '${INDEXER_TARBALL_URL}'; \ + else \ + echo "indexer-fetch: ERROR no curl or wget available"; exit 1; \ + fi; \ + tar -xzf /tmp/explorer-indexer.tar.gz --strip-components=1 -C ${INDEXER_DIR}; \ + rm -f /tmp/explorer-indexer.tar.gz; \ + echo "indexer-fetch: patching types.go to make ptrs_root optional (older spaced compatibility)..."; \ + awk 'BEGIN{patched=0} /if aux.PointersRoot == nil \\{/ {skip=2; patched++; next} skip>0 {skip--; next} /ra\\.PointersRoot = \\*aux\\.PointersRoot/ {print "\\tif aux.PointersRoot != nil { ra.PointersRoot = *aux.PointersRoot }"; next} {print} END{if(patched<2){print "PATCH-FAIL: expected 2 ptrs_root blocks, got " patched > "/dev/stderr"; exit 1}}' ${INDEXER_DIR}/pkg/node/types.go > ${INDEXER_DIR}/pkg/node/types.go.new; \ + mv ${INDEXER_DIR}/pkg/node/types.go.new ${INDEXER_DIR}/pkg/node/types.go; \ + if grep -q 'missing required field: ptrs_root' ${INDEXER_DIR}/pkg/node/types.go; then \ + echo "indexer-fetch: ERROR ptrs_root check still present after patch"; exit 1; \ + fi; \ + echo "indexer-fetch: patching store.go to skip getptrblockmeta (older spaced has no pointer-block RPC)..."; \ + awk 'BEGIN{patched=0} /spacesPtrBlock, err := sc.GetPtrBlockMeta/ {skip=8; patched++; next} skip>0 {skip--; next} {print} END{if(patched<1){print "PATCH-FAIL: expected GetPtrBlockMeta block, got " patched > "/dev/stderr"; exit 1}}' ${INDEXER_DIR}/pkg/store/store.go > ${INDEXER_DIR}/pkg/store/store.go.new; \ + mv ${INDEXER_DIR}/pkg/store/store.go.new ${INDEXER_DIR}/pkg/store/store.go; \ + if grep -q 'GetPtrBlockMeta' ${INDEXER_DIR}/pkg/store/store.go; then \ + echo "indexer-fetch: ERROR GetPtrBlockMeta call still present after patch"; exit 1; \ + fi; \ + echo "${INDEXER_BUILD_ID}" > ${INDEXER_MARKER}; \ + echo "indexer-fetch: done."`, + ], + user: 'root', + }, + requires: [], + }) + .addOneshot('indexer-build', { + subcontainer: indexerSub, + exec: { + command: [ + 'sh', + '-c', + `set -eu; \ + if [ -x ${INDEXER_SYNC_BIN} ] && [ -x ${INDEXER_GOOSE_BIN} ]; then \ + echo "indexer-build: sync + goose already built, skipping."; \ + exit 0; \ + fi; \ + cd ${INDEXER_DIR}; \ + mkdir -p ${INDEXER_BIN_DIR}; \ + echo "indexer-build: building sync binary..."; \ + CGO_ENABLED=0 go build -o ${INDEXER_SYNC_BIN} ./cmd/sync; \ + echo "indexer-build: installing goose..."; \ + CGO_ENABLED=0 GOBIN=${INDEXER_BIN_DIR} go install github.com/pressly/goose/v3/cmd/goose@v3.24.3; \ + ls -l ${INDEXER_BIN_DIR}; \ + echo "indexer-build: done."`, + ], + env: indexerEnv, + user: 'root', + }, + requires: ['indexer-fetch'], + }) + .addOneshot('indexer-cleanup-legacy', { + subcontainer: postgresSub, + exec: { + command: [ + 'sh', + '-c', + `set -eu; \ + PSQL="psql -U $POSTGRES_USER -h 127.0.0.1 -p 5432 -d $POSTGRES_DB -tA"; \ + HAS_LEGACY=$($PSQL -c "SELECT (to_regclass('public.block_stats') IS NOT NULL) OR (to_regclass('public.spaces_history') IS NOT NULL)" 2>/dev/null || echo f); \ + HAS_GOOSE=$($PSQL -c "SELECT to_regclass('public.goose_db_version') IS NOT NULL" 2>/dev/null || echo f); \ + if [ "$HAS_GOOSE" != "t" ] && [ "$HAS_LEGACY" = "t" ]; then \ + echo "indexer-cleanup-legacy: dropping leftover TS indexer schema..."; \ + $PSQL -c "DROP TABLE IF EXISTS block_stats, syncs, spaces_history, spaces, transactions, blocks CASCADE; DROP EXTENSION IF EXISTS pg_trgm;"; \ + echo "indexer-cleanup-legacy: done."; \ + else \ + echo "indexer-cleanup-legacy: nothing to clean (HAS_GOOSE=$HAS_GOOSE HAS_LEGACY=$HAS_LEGACY)."; \ + fi`, + ], + env: { + ...postgresEnv, + PGPASSWORD: dbAuth.password, + }, + user: 'postgres', + }, + requires: ['postgres'], + }) + .addOneshot('indexer-migrate', { + subcontainer: indexerSub, + exec: { + command: [ + 'sh', + '-c', + `set -eu; \ + echo "indexer-migrate: applying goose migrations..."; \ + ${INDEXER_GOOSE_BIN} -dir ${INDEXER_SCHEMA_DIR} postgres "$POSTGRES_URI" up; \ + echo "indexer-migrate: done."`, + ], + env: indexerEnv, + user: 'root', + }, + requires: ['indexer-build', 'postgres', 'indexer-cleanup-legacy'], + }) + .addDaemon('indexer', { + subcontainer: indexerSub, + exec: { + command: [INDEXER_SYNC_BIN], + env: indexerEnv, + }, + ready: { + display: i18n('Indexer Process'), + fn: () => ({ + result: 'success', + message: i18n('indexer process is running'), + }), + gracePeriod: 60_000, + }, + requires: ['indexer-migrate', 'postgres', 'spaced'], + }) + .addOneshot('explorer-fetch', { + subcontainer: explorerSub, + exec: { + command: [ + 'sh', + '-c', + `set -eu; \ + mkdir -p ${EXPLORER_DIR}; \ + if [ -f ${EXPLORER_MARKER} ] && [ "$(cat ${EXPLORER_MARKER})" = "${EXPLORER_BUILD_ID}" ] && [ -d ${EXPLORER_BUILD_DIR} ]; then \ + echo "explorer-fetch: ${EXPLORER_BUILD_ID} already present, skipping."; \ + exit 0; \ + fi; \ + echo "explorer-fetch: downloading ${EXPLORER_BUILD_ID}..."; \ + rm -rf ${EXPLORER_DIR}; \ + mkdir -p ${EXPLORER_DIR}; \ + if command -v curl >/dev/null 2>&1; then \ + curl -fsSL '${EXPLORER_TARBALL_URL}' -o /tmp/explorer-ui.tar.gz; \ + elif command -v wget >/dev/null 2>&1; then \ + wget -q -O /tmp/explorer-ui.tar.gz '${EXPLORER_TARBALL_URL}'; \ + else \ + echo "explorer-fetch: ERROR no curl or wget"; exit 1; \ + fi; \ + tar -xzf /tmp/explorer-ui.tar.gz --strip-components=1 -C ${EXPLORER_DIR}; \ + rm -f /tmp/explorer-ui.tar.gz; \ + echo "${EXPLORER_BUILD_ID}" > ${EXPLORER_MARKER}; \ + echo "explorer-fetch: done."`, + ], + user: 'root', + }, + requires: [], + }) + .addOneshot('explorer-install', { + subcontainer: explorerSub, + exec: { + command: [ + 'sh', + '-c', + `set -eu; \ + cd ${EXPLORER_DIR}; \ + if [ -d node_modules ] && [ -d build ] && [ -f build/index.js ]; then \ + echo "explorer-install: already built, skipping."; \ + exit 0; \ + fi; \ + echo "explorer-install: npm install (this may take a few minutes)..."; \ + npm install --no-audit --no-fund; \ + echo "explorer-install: building (PUBLIC_BTC_NETWORK=${EXPLORER_NETWORK})..."; \ + PUBLIC_BTC_NETWORK=${EXPLORER_NETWORK} npm run build; \ + if [ ! -f ${EXPLORER_BUILD_DIR}/index.js ]; then \ + echo "explorer-install: ERROR build/index.js not produced"; \ + ls -la ${EXPLORER_BUILD_DIR} || true; \ + exit 1; \ + fi; \ + echo "explorer-install: done."`, + ], + env: explorerEnv, + user: 'root', + }, + requires: ['explorer-fetch'], + }) + .addDaemon('explorer-ui', { + subcontainer: explorerSub, + exec: { + command: ['node', EXPLORER_BUILD_DIR], + env: explorerEnv, + }, + ready: { + display: i18n('Explorer Web UI'), + fn: () => + sdk.healthCheck.checkPortListening(effects, EXPLORER_PORT, { + successMessage: i18n('explorer UI is ready'), + errorMessage: i18n('explorer UI is not ready'), + }), + gracePeriod: 60_000, + }, + requires: ['explorer-install', 'postgres'], + }) .addDaemon('web-terminal', { subcontainer: termSub, exec: { @@ -143,8 +626,10 @@ SPACES_BASHRC_EOF`], '/root/.cargo/bin/space-cli', '--chain', SPACED_CHAIN, - '--rpc-cookie', - `${dataDir}/${SPACED_CHAIN}/.cookie`, + '--rpc-user', + spacedAuth.username, + '--rpc-password', + spacedAuth.password, '--output-format', 'json', 'getserverinfo', @@ -229,4 +714,128 @@ SPACES_BASHRC_EOF`], }, requires: ['spaced'], }) + .addHealthCheck('indexer-sync', { + ready: { + display: i18n('Indexer Sync'), + fn: async () => { + try { + // Go indexer has no syncs table; track progress by max(blocks.height) + // of non-orphan rows. + const psql = await postgresSub.exec( + [ + 'psql', + '-U', + POSTGRES_USER, + '-d', + POSTGRES_DB, + '-h', + '127.0.0.1', + '-p', + String(POSTGRES_PORT), + '-t', + '-A', + '-F', + '|', + '-c', + "SELECT COALESCE(MAX(height), 0) FROM blocks WHERE orphan = FALSE AND height >= 0;", + ], + { env: { PGPASSWORD: dbAuth.password } as Record }, + ) + + if (psql.exitCode !== 0) { + return { + result: 'failure', + message: i18n('indexer psql exited ${code}: ${error}', { + code: String(psql.exitCode), + error: ((psql.stderr ?? '') as string).toString().slice(0, 200) || + '', + }), + } + } + + const out = (psql.stdout ?? '').toString().trim() + if (!out) { + return { + result: 'loading', + message: i18n('indexer has not run a sync cycle yet.'), + } + } + const parts = out.split('|') + const endBlockHeight = parseInt(parts[0] ?? '0', 10) || 0 + const ageSec = -1 // Go indexer has no per-sync timestamp surface + + const probe = await spacedSub.exec( + [ + '/root/.cargo/bin/space-cli', + '--chain', + SPACED_CHAIN, + '--rpc-user', + spacedAuth.username, + '--rpc-password', + spacedAuth.password, + '--output-format', + 'json', + 'getserverinfo', + ], + {}, + ) + let spacedBlocks = 0 + if (probe.exitCode === 0) { + try { + const parsed = JSON.parse((probe.stdout ?? '').toString()) + spacedBlocks = parsed?.chain?.blocks ?? 0 + } catch { + /* ignore */ + } + } + + if (endBlockHeight === 0) { + return { + result: 'loading', + message: i18n( + 'indexer has not yet committed any blocks (spaced tip ${tip}).', + { tip: String(spacedBlocks) }, + ), + } + } + + const lag = Math.max(0, spacedBlocks - endBlockHeight) + if (spacedBlocks > 0 && lag <= 5) { + return { + result: 'success', + message: i18n( + 'indexer caught up at block ${end} (spaced tip ${tip}).', + { + end: String(endBlockHeight), + tip: String(spacedBlocks), + }, + ), + } + } + + return { + result: 'loading', + message: i18n( + 'indexer at block ${end}, ${lag} behind spaced tip ${tip}.', + { + end: String(endBlockHeight), + lag: String(lag), + tip: String(spacedBlocks), + }, + ), + } + } catch (e) { + return { + result: 'failure', + message: i18n('Indexer Sync health check crashed: ${error}', { + error: (e as Error)?.message ?? String(e), + }), + } + } + }, + gracePeriod: 60_000, + trigger: sdk.trigger.cooldownTrigger(30_000), + }, + requires: ['indexer', 'postgres'], + }) }) diff --git a/startos/manifest/index.ts b/startos/manifest/index.ts index cd73a3e..033b7ed 100644 --- a/startos/manifest/index.ts +++ b/startos/manifest/index.ts @@ -17,6 +17,18 @@ export const manifest = setupManifest({ source: { dockerTag: 'horologger/spaces:v0.0.9s' }, arch: ['x86_64', 'aarch64'], }, + postgres: { + source: { dockerTag: 'postgres:16.3' }, + arch: ['x86_64', 'aarch64'], + }, + 'indexer-go': { + source: { dockerTag: 'golang:1.23-alpine' }, + arch: ['x86_64', 'aarch64'], + }, + 'explorer-ui': { + source: { dockerTag: 'node:20-alpine' }, + arch: ['x86_64', 'aarch64'], + }, }, alerts: { install: null, diff --git a/startos/utils.ts b/startos/utils.ts index 8466b84..29d6a46 100644 --- a/startos/utils.ts +++ b/startos/utils.ts @@ -16,6 +16,42 @@ export const BITCOIND_RPC_PORT = 8332 export const SPACED_CHAIN = 'mainnet' +export const POSTGRES_PORT = 5432 +export const POSTGRES_USER = 'postgres' +export const POSTGRES_DB = 'spacesprotocol_explorer' +export const pgDataDir = '/data/postgres' + +export const INDEXER_REPO = 'spacesprotocol/explorer-indexer' +export const INDEXER_GIT_SHA = '00ae1e548734d93f1a8bb9f48d2290f459e12b35' +// Bump the suffix to force a full re-fetch + rebuild of /data/explorer-indexer. +export const INDEXER_BUILD_ID = `${INDEXER_GIT_SHA}-g3` +export const INDEXER_TARBALL_URL = `https://github.com/${INDEXER_REPO}/archive/${INDEXER_GIT_SHA}.tar.gz` +export const INDEXER_DIR = '/data/explorer-indexer' +export const INDEXER_BIN_DIR = '/data/explorer-indexer/bin' +export const INDEXER_SYNC_BIN = '/data/explorer-indexer/bin/sync' +export const INDEXER_GOOSE_BIN = '/data/explorer-indexer/bin/goose' +export const INDEXER_SCHEMA_DIR = '/data/explorer-indexer/sql/schema' +export const INDEXER_MARKER = '/data/explorer-indexer/.installed-sha' + +// Spaces protocol mainnet activation block (per spacesprotocol/explorer-indexer +// env.example). Indexer skips fast-sync below FAST_SYNC and starts indexing +// spaces data at ACTIVATION. +export const INDEXER_ACTIVATION_HEIGHT = '871222' +export const INDEXER_FAST_SYNC_HEIGHT = '864000' +export const INDEXER_UPDATE_INTERVAL = '5' +export const INDEXER_MEMPOOL_CHUNK_SIZE = '200' + +export const EXPLORER_REPO = 'randomlogin/explorer' +export const EXPLORER_GIT_SHA = 'c827da1754c3cba5c5507d2c29f21b8fa231344d' +// Bump suffix to force re-fetch + rebuild of /data/explorer-ui. +export const EXPLORER_BUILD_ID = `${EXPLORER_GIT_SHA}-e1` +export const EXPLORER_TARBALL_URL = `https://github.com/${EXPLORER_REPO}/archive/${EXPLORER_GIT_SHA}.tar.gz` +export const EXPLORER_DIR = '/data/explorer-ui' +export const EXPLORER_BUILD_DIR = '/data/explorer-ui/build' +export const EXPLORER_MARKER = '/data/explorer-ui/.installed-sha' +export const EXPLORER_PORT = 3000 +export const EXPLORER_NETWORK = 'mainnet' + export function randomPassword() { // bitcoind's generate-rpc-dependent action validates the password against // /^[A-Za-z0-9_-]+$/, so the charset must stay in that set. diff --git a/startos/versions/index.ts b/startos/versions/index.ts index 7893a04..d6d9e8b 100644 --- a/startos/versions/index.ts +++ b/startos/versions/index.ts @@ -1,7 +1,7 @@ import { VersionGraph } from '@start9labs/start-sdk' -import { v_0_0_9_0_a1 } from './v0.0.9.0.a1' +import { v_0_0_9_1 } from './v0.0.9.1' export const versionGraph = VersionGraph.of({ - current: v_0_0_9_0_a1, + current: v_0_0_9_1, other: [], }) diff --git a/startos/versions/v0.0.9.0.a1.ts b/startos/versions/v0.0.9.0.a1.ts deleted file mode 100644 index eb301e0..0000000 --- a/startos/versions/v0.0.9.0.a1.ts +++ /dev/null @@ -1,14 +0,0 @@ -import { IMPOSSIBLE, VersionInfo } from '@start9labs/start-sdk' - -export const v_0_0_9_0_a1 = VersionInfo.of({ - version: '0.0.9:0-alpha.1', - releaseNotes: { - en_US: `- Add "Reset Spaced State" action so a corrupt /data/mainnet/ index can be wiped from the UI. -- Web terminal no longer requires spaced to be healthy, so gotty stays reachable when spaced crash-loops. -- Initial build bundles spaced + space-cli from horologger/spaces:v0.0.9s, managed spaced daemon (mainnet only), gotty browser terminal with admin basic auth, and the Bitcoin Core 31.x dependency.`, - }, - migrations: { - up: async ({ effects }) => {}, - down: IMPOSSIBLE, - }, -}) diff --git a/startos/versions/v0.0.9.1.ts b/startos/versions/v0.0.9.1.ts new file mode 100644 index 0000000..f39aac3 --- /dev/null +++ b/startos/versions/v0.0.9.1.ts @@ -0,0 +1,23 @@ +import { IMPOSSIBLE, VersionInfo } from '@start9labs/start-sdk' + +export const v_0_0_9_1 = VersionInfo.of({ + version: '0.0.9:1', + releaseNotes: { + en_US: `- Embedded explorer (PostgreSQL + Go indexer + SvelteKit web UI) is now **opt-in**. Fresh installs run in spaces-only mode (spaced + gotty terminal). New "Enable Embedded Explorer" action turns the whole bundle on; "Disable Embedded Explorer" turns it back off. Both auto-restart the service so the new daemon graph takes effect. Indexed data on disk is preserved across toggles. +- Embedded SvelteKit explorer web UI (randomlogin/explorer @ c827da175). Fetched into /data/explorer-ui on first enable, \`npm install\` + \`npm run build\` produce a SvelteKit node-adapter bundle, and \`node build\` runs as a managed daemon on port 3000. Exposed as a StartOS interface so it appears alongside the gotty terminal in the dashboard. Reads exclusively from the embedded PostgreSQL; does not talk to spaced or bitcoind directly. New "Reset Explorer UI State" action wipes the cache for a clean rebuild. Adds a fourth manifest image \`node:20-alpine\`. +- Embedded spaces-protocol explorer indexer (Go, spacesprotocol/explorer-indexer @ 00ae1e548). When enabled, on first start its source tarball is fetched into /data/explorer-indexer, the \`sync\` binary and \`goose\` migrator are built into /data/explorer-indexer/bin, the goose-managed SQL schema is applied to the embedded PostgreSQL, and the sync binary runs as a managed daemon that polls bitcoind + spaced and writes to the database. +- Two patches applied to the indexer source during \`indexer-fetch\` to handle our older spaced binary: \`pkg/node/types.go\` makes \`ptrs_root\` optional in RootAnchor, and \`pkg/store/store.go\` skips the \`getptrblockmeta\` RPC call. Subspaces pointer data won't be indexed; everything else (blocks, transactions, spaces, rollouts, root anchors) does. +- Spaced is now configured with static RPC credentials (SPACED_RPC_USER / SPACED_RPC_PASSWORD seeded into store.json.spacedAuth) instead of cookie auth, so the indexer and the gotty terminal share a single auth path. space-cli inside the terminal aliases to \`--rpc-user "$SPACED_RPC_USER" --rpc-password "$SPACED_RPC_PASSWORD"\`. +- New "Indexer Sync" standalone health check queries the blocks table for the highest non-orphan block and cross-references against spaced's tip; success when within 5 blocks, otherwise loading with explicit lag. +- New "Reset Indexer State" action wipes /data/explorer-indexer so the next start re-fetches the pinned commit and rebuilds the binaries. +- Embedded PostgreSQL 16.3 daemon on loopback 127.0.0.1:5432 (data at /data/postgres). "Show Database Credentials" + "Reset Database State" actions. POSTGRES_URI exported into the gotty terminal. +- Indexer container is a third manifest image (\`golang:1.23-alpine\`) used both to build the binaries and to run \`sync\`. +- "Reset Spaced State" action so a corrupt /data/mainnet/ index can be wiped from the UI. +- Web terminal no longer requires spaced to be healthy, so gotty stays reachable when spaced crash-loops. +- Initial build bundles spaced + space-cli from horologger/spaces:v0.0.9s, managed spaced daemon (mainnet only), gotty browser terminal with admin basic auth, and the Bitcoin Core 31.x dependency.`, + }, + migrations: { + up: async ({ effects }) => {}, + down: IMPOSSIBLE, + }, +})