---
title: MoonStack Update & Changelog
slug: moonstack-update
status: active
owner: hazem
updated: 2026-06-18   # docroot-ready packaging SHIPPED + PROVEN by 2.2.16; composer-from-web fixed (CLI-php + --no-scripts)
refs:
  - moon-erp-be/app/MoonStack/
  - moon-erp-be/app/MoonStack/Packaging/PackageBuilder.php
  - moon-erp-be/public/moonstack-setup.php
  - moon-erp-be/public/moonstack-update.php
  - moon-erp-be/config/moonstack.php
  - moon-erp-be/docs/moonstack/CHANGELOG.md
  - moon-erp-be/docs/moonstack/conventions.md
plans:
  - https://moonui.elbaset.com/knowledge-base/plans/moonstack-docroot-ready-plan.html        # docroot-ready package fix (planned, owner OK pending)
  - https://moonui.elbaset.com/knowledge-base/plans/moonstack-plan.html
  - https://moonui.elbaset.com/knowledge-base/plans/moonstack-update-seeder-fix-plan.html   # seeder-gap fix (done)
  - https://moonui.elbaset.com/knowledge-base/plans/moonstack-release-update-flow.html      # how build/release/update works + stability proposals
related:
  - middleware
---

## Context
MoonStack = the isolated self-host install + auto-update system (`App\MoonStack\*`, `config/moonstack.php`, `public/moonstack-{setup,update}.php`, `moonstack:*` commands). Clients (elmadina, moontest, …) run the same app and update via signed delta/full packages + a `versions.json` manifest. Goal: updates must be **fast + crash-safe** (on failure → immediate rollback to last working version).

## How it works
- **Install** (`Installer`): migrate → seed `config('moonstack.installer.seeders')` → first company/admin.
- **Update** (`public/moonstack-update.php`, orchestrates with the new code): backup → maintenance down → apply files (delta = changed files only; full = swap code, preserve DATA) → composer (only if `composer.lock` changed) → **migrate** → caches → up → finalize version. Any failure → `BackupManager.restore()` (code + DB) → up.
- **Delta updates** copy only changed files (no 15k-vendor churn). Frontend = delete-then-copy (no stale chunks). `config.json`/storage never overwritten.

## The what's-new / changelog process (قاعدة: أي تعديل يتسجّل)
- After EVERY user-facing change: add a one-line bullet to `moon-erp-be/docs/moonstack/CHANGELOG.md` under `## [Unreleased]`. Log EVERYTHING (UI tweaks, fixes, removals) — don't pre-judge.
- `moonstack:ship` attaches `[Unreleased]` to the release, promotes it under a dated version heading; the release page shows/edits it in a textarea; the in-app Updates screen shows it aggregated across pending versions.
- **Middleware changes are separate** → log them in `lis-middleware/CHANGELOG.md`, NOT here (middleware isn't distributed by MoonStack).

## Decisions & why
- **Crash-safe rollback** is non-negotiable (the updater is "اساس التواصل" with the fleet — a bug here bricks clients).
- **Delta-first** because full updates crawled/hung on shared hosting (vendor copy).
- **Reference seeders now run on update too** (was install-only — that was the seeder-gap bug, now fixed; see above).

## Release pipeline (build side) — direction (2026-06-17)
- **Today:** `moonstack:build` packages from `base_path()` on moonui (a snapshot of the working copy) + `MOONSTACK_FRONTEND_DIST` (the built Angular) + vendor → signed zip → `versions.json`. GitHub is NOT in the update path.
- **VERIFIED FACT:** GitHub `main` (both repos) holds **SOURCE ONLY** — no `vendor`, no built `public/app`. The built `/app` lives in `public_html` which has **no git remote** (not pushed). So a release needs a build step regardless.
- **Direction (multi-dev):** build releases from `main` via a runner/CI — checkout BE+FE at a tag → `composer install --no-dev` + `ng build` → `moonstack:build --source=<be> --frontend=<fe-dist> --sign`. Reproducible, no manual assembly on moonui. **One code change needed:** add `--source` to `moonstack:build` (PackageBuilder already accepts `opts['source']`). Full write-up: the `moonstack-release-update-flow.html` plan.
- **Plan:** merge `lis/lab-updates` → `main` once MoonStack work is done; main becomes the release source.
- ✅ **Release web page `public/.../moonstack-release.php` (on moonui, served at `moonui.elbaset.com/moonstack-release.php`) now does it all (2026-06-17, BE `87877ff2e`):** it runs `moonstack:ship` and the page adds **Channel (stable/beta canary)** + **Build source (current server = fast / from git = reproducible, with a git-ref)**. It does NOT git-pull or ng-build on the "current" path (packages the live state + prebuilt `/app`); the "git" path runs `scripts/moonstack-release-from-git.sh` (clone+composer+ng build→ship --source). `moonstack:ship` gained `--source=`. The page self-resolves repo URLs from the live repos' remotes (no tokens in the page). **CLI still works for Claude:** `moonstack:ship <v> --channel=beta [--source=…]` or the script directly.
- ⚠️ The page edits make it root-owned (suexec → 500); **chown moonui:moonui moonstack-release.php** after editing.

## ✅ Stability system — DONE (2026-06-17, BE commit `9805bb0ee`, pushed `lis/lab-updates`) — pending release
Plan doc: `knowledge-base/plans/moonstack-release-update-flow.html`. Built in parallel (4 agents, isolated files), each reviewed by me + verified:
1. ✅ **Build-from-git** — `moonstack:build --source=` (PackageBuilder already accepted `opts['source']`) + `scripts/moonstack-release-from-git.sh` (clone BE+FE at a ref → `composer install --no-dev` → `ng build` → `moonstack:build --source --frontend --sign` → `moonstack:release`; env URLs, no secrets, trap cleanup). Verified: `--source` registered. **Use:** `MOON_BE_REPO=… MOON_FE_REPO=… scripts/moonstack-release-from-git.sh <ver> [ref]`.
2. ✅ **Post-update health-check + auto-rollback** — `moonstack-update.php` block 8c after `up`: probes `config('app.url')+/up` per `config('moonstack.update.health_check')` ({enabled,path:/up,attempts:3,timeout:10}). Rolls back to the safe point **ONLY** when every attempt returns HTTP ≥500; any 2xx/3xx/4xx or connection failure = inconclusive (no rollback); whole block try/caught (can't crash the updater); sets `$finished` before exit (no double-rollback via the shutdown net).
3. ✅ **Backup retention** — `BackupManager::prune($base,$current)` keeps newest `retention_count` (default 3), hardened: realpath direct-children only, no symlink follow, never deletes the just-created backup, non-fatal.
4. ✅ **Idempotency test** — `tests/Feature/MoonStack/UpdaterSeedersIdempotencyTest` (sqlite) — each `updater.seeders` class re-runnable, no dup. **Passes (8 assertions via `vendor/bin/pest`).** NOTE: `php artisan test` returns 255/no-output as `runuser` here — use `vendor/bin/pest <file>` to run tests.
- ✅ **Canary/channel rollout — ALREADY BUILT** (discovered 2026-06-17, not new code): `moonstack:ship/release --channel=…` → `Publisher` writes each release with its channel into `versions.json` and computes `latest` PER channel; `UpdateClient::check()` filters by `config('moonstack.update.channel')` (env `MOONSTACK_CHANNEL`, default `stable`). **To canary:** `moonstack:ship <ver> --channel=beta` → set a test client's `MOONSTACK_CHANNEL=beta` → it updates; stable clients don't see it. Promote = ship the same version to `stable`. Signing key (`MOONSTACK_PRIVATE_KEY`) is SET on moonui; `MOONSTACK_FRONTEND_DIST=/home/moonui/public_html/app`. Published latest (stable) = 2.2.10.
- Config keys: `update.health_check` added; `backup.retention_count` + `update.channel` pre-existed.

## Canary test setup + ⚠️ the shared-manifest gotcha (2026-06-17)
- ✅ **moontest = the canary**: set `MOONSTACK_CHANNEL=beta` in `/home/moontest/public_html/moonerp/.env` (live app; version 2.2.6; config not cached). Verified `channel=beta`.
- ✅ **Superset client shipped** (`49b67a2fd`): a beta client sees beta+stable (highest wins); stable sees stable only. UpdateClient now computes the highest VISIBLE release directly from `releases[]` (ignores `manifest['latest']`).
- ⚠️ **GOTCHA — old clients + single `manifest['latest']`:** the Publisher writes ONE `manifest['latest']` = the LAST publish's channel/version. Clients on OLD code (pre-`49b67a2fd`: moontest 2.2.6, elmadina, …) key off `manifest['latest']` + strict channel match. So **publishing a beta to the SHARED `moonstack/versions.json` sets latest=beta-version → stable old clients then find no stable release matching latest → they see "no update"** until the version is promoted to stable. Benign (manual updates, no wrong install, recovers on promote) but real.
- **Safe canary = SEPARATE beta manifest (recommended):** Publisher accepts a `dir`/`base_url` override (or env `MOONSTACK_RELEASE_DIR`/`MOONSTACK_RELEASE_URL`). Publish beta to `…/moonstack-beta/versions.json`, point moontest's `MOONSTACK_UPDATE_URL` there. Stable `moonstack/versions.json` is NEVER touched → ZERO risk to elmadina/stable. Then promote = ship the version to the stable manifest normally.
- After the fleet is on `49b67a2fd`+ the gotcha is moot (clients compute direct), but the separate-manifest canary is the clean approach regardless.

## ✅ DOCROOT-READY PACKAGING — FIXED + PROVEN END-TO-END (2026-06-18; owner chose the visual-clean WRAPPED layout)
**🟢 PROVEN by a full from-git release of 2.2.16** (clean `main` checkout → composer → ng build → ship → wrap → publish): both `moon-erp-v2.2.16.zip` (updater) **and `moon-erp-v2.2.16-setup.zip` (docroot-ready installer)** published, `versions.json latest=2.2.16` carries `setup_url`/`setup_sha256`/`setup_size`, and the setup-zip's verified top-level = **only `.htaccess` + `core/`** (with `core/public/moonstack-setup.php`, `core/vendor/autoload.php`, `core/artisan`, `core/public/app/`). To get here, the from-git `composer install` had to be fixed for the web context — **TWO root causes (package:discover phone-home + composer under a NON-CLI SAPI php); fix = `${PHP} ${COMPOSER_BIN} … --no-scripts`** — full trace in [[dev-workflow]] (composer-from-web bullet). **Lesson: any composer step run from FPM/web MUST invoke `<cli-php> <composer.phar>`, never the bare phar (its `#!/usr/bin/env php` shebang picks the CGI/FPM SAPI → "cannot run on non-CLI SAPIs").**
**Owner decision:** wanted **visual cleanliness + correct paths** (not just a shim), and **two build modes must both emit it**. **Implemented = a SECOND artifact `moon-erp-v<ver>-setup.zip`** (the docroot-ready installer for humans) built by `App\MoonStack\Packaging\InstallerWrapper` after the normal publish.

**🔼 UPDATE (2026-06-18, commit `9642319c0` on `main`+`hazemdev`): the MAIN `moon-erp-v<ver>.zip` is now ALSO docroot-ready** — `PackageBuilder::build()` injects a root `.htaccess` (→ `public/`, via `PackageBuilder::rootHtaccess()`) into every FULL build. **Why:** the owner kept extracting the natural download `moon-erp-vX.zip` (not the `-setup.zip`) → raw Laravel at the webroot → `…/moonstack-setup.php` 404 and, after setup, the SPA 404'd its assets (`/app/chunk-*.js` requested at the root because the FE base-href is `/app/`, but the files live at `/public/app/…`). Owner requirement, verbatim: **"install → works with ZERO manual edits, and the URL must be `https://<site>/app/`."** The injected shim makes `/app/*` rewrite to `public/app/*` (URL stays `/app/`), everything else → `public/index.php`, secrets denied. **Fleet-safe:** deltas DON'T include it (existing installs already have it); a full-package update lands it at `base_path()/.htaccess` — correct for root-docroot installs, harmless (above docroot) for installs already pointing at `public/`. **So the `-setup.zip` (clean wrapped `core/`) and the main `…zip` (raw + root `.htaccess`) BOTH now work extracted into `public_html`.** Verified: a built package contains a root `.htaccess` at top level alongside `artisan`. ⚠️ The owner builds the release themselves — this only ships once they cut a new version (> 2.2.17).
- **Setup-zip layout (verified):** root `.htaccess` (rewrite all → `core/public/` + deny `.env`/secrets) + the WHOLE Laravel app under `core/`. Extract into `public_html` → webroot has **only `.htaccess` + `core/`** (clean); `<domain>/moonstack-setup.php` works; core unreachable; the INSTALLED tree is plain standard Laravel under `core/` so the **installer + updater run unchanged** (`base_path()`=core, `public_path()`=core/public, entry `$basePath=dirname(__DIR__)`=core resolves standard).
- **InstallerWrapper** (`app/MoonStack/Packaging/InstallerWrapper.php`): memory-safe — extracts the full zip to a temp dir beside the output, re-zips every entry under `core/` via `ZipArchive::addFile` (deferred reads), adds the root `.htaccess` (`rootHtaccess()`). Isolated re-stream → never touches PackageBuilder's delta/index logic. `ShipCommand::handle()` wires it (after the updater-bootstrap publish) → `Publisher::publishSetup()` adds `setup_url`+`setup_sha256`+`setup_size` to the release entry in `versions.json`.
- **Installer correctness fixes shipped:** `Installer::linkStorage()` (storage:link, best-effort) + `finalize()` writes BOTH markers (`storage/moonstack/installed` AND legacy `storage/installed` that `index.php:19` reads); `RequirementsChecker::check()` now fails loudly if `vendor/autoload.php` is missing (catches truncated extracts); `PackageBuilder::build()` guards `with_vendor && !is_dir(source/vendor)`.
- **New install runbook:** download `…-setup.zip` → extract into `public_html` → open `/moonstack-setup.php`. No docroot change, no `/public/` digging. (Shared `public/index.php` NOT edited — Ahmed-isolation.)
- After editing build-side classes: `composer dump-autoload` is `--classmap-authoritative` (run `bash local-deploy.sh`) or the new class is "not found". Ships via next release (current-server packages the live tree; from-git needs commit to hazemdev).

### (original diagnosis kept for reference)
**Full plan:** [`plans/moonstack-docroot-ready-plan.html`](../plans/moonstack-docroot-ready-plan.html).
**Symptom (owner, on a NEW client `prod.elbaset.com`, cPanel user `prod`, docroot `/home/prod/public_html`):** downloaded `moon-erp-v2.2.14/15.zip`, extracted into `public_html` → got a **raw Laravel app at the webroot** (`app/ config/ public/ vendor/ artisan .env.example …` all top-level). So `prod.elbaset.com/moonstack-setup.php` → **404** (the file is really `/public/moonstack-setup.php` → 200), and `.env`/`config`/`storage` are **web-exposed** (no root `.htaccess`).
**Root cause (verified):** `config/moonstack.php → package.include` (L149-156) enumerates Laravel's root dirs verbatim; `PackageBuilder::addDir()` (L208) zips each at its **bare relative path** with NO prefix → **zip root === Laravel root, `public/` is a subfolder**, FE at `public/app`. There is **no relayout step**. The package was authored for a **"docroot = `…/public`"** layout (the pilot install required a **manual cPanel docroot repoint to `…/public`** — `self-hosted-distribution-plan`), which the owner didn't/can't do. (Confirmed from the live `moon-erp-v2.2.15.zip`: 15,978 files, raw-root top-level, `vendor/` IS inside = 11,206 entries → the prod "missing vendor" was an **incomplete extraction** of the ~164MB zip, not a build defect.)
**Both build modes converge on `PackageBuilder::build()`** (`current-server` packages `base_path()`; `from-git` packages a clean checkout via `--source`) → **one fix covers both**; the shell script + release page never touch layout.
**Updater is the constraint (the fleet lifeline):** it applies the payload at `base_path($rel)` + writes FE to `public_path('app')` = `<root>/public/app`, with **zip paths relative to the Laravel root** → any fix MUST keep payload paths un-wrapped or it breaks deltas + the updater. (It's mostly layout-agnostic: locates `$basePath` via `dirname(__DIR__)` then a one-level sibling `glob`; `HOME=dirname($basePath)`.)
**CHOSEN design = root `.htaccess` docroot-shim** (safe, one file, zero updater risk): keep the package standard-Laravel and **inject a `.htaccess` at the zip ROOT** that rewrites all → `public/` (`RewriteCond %{REQUEST_URI} !^/public/` + `RewriteRule ^(.*)$ public/$1 [L]`, plus deny `.env`/dotfiles). Extract into `public_html` → works at the root + core protected (every path forced under `public/` → core files 404), **no manual docroot change**. Harmless for old `docroot=public/` installs (it sits above their docroot). **Rejected:** manual docroot repoint (needs root/WHM), wrap-core-in-subfolder / core-above-webroot (both break `public_path()` + delta path-mapping → fleet risk). Visual declutter (core not at webroot) deferred as optional (separate installer-zip vs update-zip).
**Fix hooks (one-shot):** (A) `PackageBuilder::build()` ~L130-138 `addFromString('.htaccess', <shim>)` on full builds; (B) ~L74 guard: fail build if `withVendor && !is_dir("$source/vendor")`; (C) `Installer::install()/finalize()` — add `storage:link` + write **both** markers (`storage/moonstack/installed` AND `storage/installed`); (D) `RequirementsChecker::check()` — add `vendor/autoload.php` + layout check (it currently passes "green" then dies at boot). **Do NOT edit shared `public/index.php`** (Ahmed-isolation). New install runbook = extract full zip into `public_html` → open `/moonstack-setup.php` → done.
**Secondary bugs found (fold into the same batch):** marker mismatch — installer writes `storage/moonstack/installed` but `index.php:19` reads old `storage/installed` (masked only because the `.env`-exists clause also gates); **no `storage:link`** in the MoonStack installer (old `install.php` had it); updater seeders are a strict subset of installer seeders + non-fatal (see seeder gap below).

## 🗺️ Architecture map (file:line) — edit fast, don't re-audit
*(built from a 5-agent code audit 2026-06-18; keep current as you touch these.)*

**BUILD / PACKAGE** — one funnel: everything goes through `App\MoonStack\Packaging\PackageBuilder::build()`.
- `addDir()` L193-216 walks an include dir → `$rel = substr($abs,strlen($source)+1)` L208 → `addFileEntry()` L181-191 (`addFile` at L188, **memory-safe**; records sha1 in `$index`, skips unchanged vs `$baseline` for deltas). Single-file includes added at L81.
- `addExternalDir()` L218-242 ships the FE at `frontend_target` (=`public/app`, `config moonstack.php:178` / builder L35); `index.html`+`.htaccess` force-shipped L237.
- `addStorageSkeleton()` L280-293 (full builds only, L130-133): boot-ready `storage/*`+`bootstrap/cache` `.gitignore`s.
- Delta build (`--delta-from`): only changed files; `delta.json` = `deleted` (L103 `array_diff` of baseline−index) + `composer_changed` (L122/245-262). `*.files.json` baseline written L149. **All zip paths are Laravel-root-relative (un-prefixed) — the updater depends on this.**
- `config moonstack.php`: `package.include` L149-156, `package.exclude` L159-167, `package.frontend.target` L178. Vendor guard L~41 (`with_vendor && !is_dir source/vendor`).
- **Docroot-ready installer** = `InstallerWrapper::wrap()` (separate, post-build): full zip → `…-setup.zip` (`core/`-prefixed + root `.htaccess` from `rootHtaccess()`).

**PUBLISH** — `App\MoonStack\Packaging\Publisher`: `publish()` (copies zip+sha+sig, upserts release entry, `latest`=highest semver in channel via `highestVersion()`), `publishDelta()` (attaches to `release.deltas[from]`), `publishUpdater()` (the signed self-heal `moonstack-update.php` bootstrap), `publishSetup()` (adds `setup_url`/`setup_sha256`/`setup_size`). Orchestrated by `App\MoonStack\Console\ShipCommand::handle()`: stamp FE version → changelog notes → `builder->build` (full, signed) → `publisher->publish` → publish updater bootstrap L88-97 → **publish setup zip** L99+ → deltas from `recentDeltaSources()` (last 6).

**INSTALL (fresh)** — entry `public/moonstack-setup.php` (path-detect L11-19: `$basePath=dirname(__DIR__)`, else sibling-`glob` for artisan+composer.json; AJAX `?action=requirements|test_db|install`). Drives `App\MoonStack\Installer\Installer::install()` L54: `migrate` → `createCompanyAndAdmin` (id=1) → `seed` (`config installer.seeders`) → `ensureOwnerRole` → `writeFrontendConfig` (writes `public/app/assets/config.json` apiUrl=`app.url`+`/api`) → `linkStorage` (new) → `finalize` (`VersionTracker::sync` writes `storage/moonstack/version.json`; markers `storage/moonstack/installed` + legacy `storage/installed`). Reqs in `RequirementsChecker::check()` (PHP≥8.2, 12 exts, vendor/autoload, writable paths). Shared `public/index.php` install-gate L18-21 (`!storage/installed && !.env`)||`!vendor/autoload` → 302 `install.php` (Ahmed's — **use `/moonstack-setup.php` instead**). **Don't edit `public/index.php` (Ahmed-isolation).**

**UPDATE** — entry `public/moonstack-update.php` (same path-detect; install-gate L22-25 `!storage/moonstack/installed`→`/moonstack-setup.php`; `msEnv()` L43-48 `HOME=dirname($basePath)`; `msSelfRefresh` L134-187; `msRecoverStaleLock` L101-124). `case 'apply'` (L243+): check+`Updater::chooseSource` (delta if `deltas[current]` else full) → preflight → download+verify sha+**sig** → backup (`BackupManager`) → lock → `artisan down` → apply (`Updater::applyDelta`/`applyFiles`, paths via `base_path($rel)`/`public_path()`) → composer dump-autoload **iff composer_changed** → `migrate --force` (fatal) → **`moonstack:sync-reference --force`** (definition seeders, non-fatal, L383) → `optimize:clear`+`up` → health-check (`/up`, auto-rollback only if ALL 5xx) → `Updater::finalize` (`VersionTracker::sync`, rewrite FE config, rewrite marker, clear lock). **Version source of truth = `storage/moonstack/version.json:app_version`** (`VersionTracker::appVersion` L36-39). "Update available" = `UpdateClient` highest release in `visibleChannels()` (pre-release ring sees [ring,stable]), NOT `manifest.latest`.

**CONFIG `config/moonstack.php`:** `version` (env `MOONSTACK_VERSION`); `paths.{home,version_file,installed_marker}` L~30-36; `manifest.{protected_paths,frontend_paths,public_skip}` (CODE/DATA boundary read by `DistributionManifest`); `package.{include,exclude,frontend,output,name}`; `release.{dir,base_url,changelog}`; `installer.seeders` (incl `RolePermissionSeeder`, `*SettingDefinitionSeeder`); `updater.seeders` (**definitions ONLY**, no RolePermission); `update.{channel,manifest url,public_key}`; health-check keys.

**ISOLATION (absolute):** Ahmed = `public/{install,update,deploy}.php`, `App\Services\*`, `config/{license,update}.php`, `moon:*`/`license:*`, phone-home in `routes/console.php`. MoonStack = `App\MoonStack\*`, `config/moonstack.php`, `public/moonstack-{setup,update}.php`, `moonstack:*`, `storage/moonstack/*`. `public/index.php` is SHARED — never edit it for MoonStack.

**TWO PACKAGE FLAVORS (key mental model):** updater payload `moon-erp-vX.zip` = standard-Laravel, paths relative to the installed `base_path()` (applied in place). Human installer `moon-erp-vX-setup.zip` = same content under `core/` + root `.htaccess` (extract-and-go). One PackageBuilder + one InstallerWrapper, both build modes.

## Gotchas
- **The setup-zip is the human installer** (`…-setup.zip`, docroot-ready); the plain `…-zip` is the updater payload (raw Laravel, applied in place — do NOT hand it to a human to extract into public_html).
- **`exec()` ignores PHP `timeout`** → wrap long steps as `timeout 300 bash -c '<cmd>'`.
- **alt-php82 vs ea-php82 drift** (mbstring missing on alt → `mb_split` 500s) → pin `ea-php82` for the vhost.
- **CHANGELOG must be `moonui:moonui`-owned** or `ship` (runs as moonui) silently fails to promote `[Unreleased]` → notes pile up across versions.
- **elmadina has TWO databases**: LIVE app = `elmadina_db` (`public_html/.env`); stale/abandoned = `elmadina_app` (`moon_erp/.env`, old admin `admin@elmadina.com`). Always use `elmadina_db` / the API. elmadina API = `https://elmadina.elbaset.com/api`; admin login = `admin@gt4.com / 123456789`. Run artisan as `runuser -u elmadina`.

## ✅ Update self-test recipe — prove an UPDATE carried new CODE + DB + UI (built-in)
There is a tiny self-test harness baked in to verify a deployed update actually applied on a client:
- **Endpoint:** `GET /api/moonstack-selftest` (public, no auth) → `{php, db, app_version}`. `php` = a hard-coded marker string in `routes/api.php` (proves the new BE **code** is live); `db` = the latest `marker` row from table `moonstack_update_tests` (proves the **migration** ran); `app_version` = `config('moonstack.version')`.
- **To run a fresh round:** (1) bump the `php` marker in `routes/api.php` (e.g. `php-v2.2.19-ok`); (2) add a NEW migration that inserts a NEW `db` marker into `moonstack_update_tests` (e.g. `db-v2.2.19-ok`) — a new migration file = a new *pending* migration the updater's `migrate` will run; (3) for a visible **UI** proof, add a marker to a no-auth screen (login page `src/app/features/auth/login/login.component.html`). Pattern done 2026-06-18: php→`php-v2.2.19-ok`, migration `2026_06_18_160000_moonstack_update_test_marker_v2_2_19`, login badge `🔄 UPDATE-TEST ✓`.
- **Verify after updating a client:** `curl https://<client>/api/moonstack-selftest` shows the new php+db markers, and the login screen shows the UI badge. (Markers are throwaway — drop the table/route/badge when done.)

### Build-mode propagation (current-server vs from-git) — how a change reaches the package
- **current-server** ship packages the **live working tree** (`base_path()`) + the **prebuilt** frontend at **`MOONSTACK_FRONTEND_DIST=/home/moonui/public_html/app`**. ⚠️ **So a FE/UI change is NOT in a current-server build until you `npx ng build --base-href /app/` and redeploy the dist into `/app`** (BE changes need no rebuild — they're read straight from the tree). Uncommitted local edits ARE included (handy for a quick test before pushing).
- **from-git** ship CLONES the ref (`main`) and **rebuilds the FE itself** (`ng build`) — so a FE change only needs to be **committed + pushed** (no manual `/app` rebuild); BE changes likewise must be pushed.
- Net: test loop = edit → (FE: rebuild `/app`) → build **current-server** → verify → `git push` hazemdev → build **from-git** → verify.

## ✨ Setup page loading feedback (2026-06-18) — `public/moonstack-setup.php`
The install step (`?action=install` = migrate + **seed**, the slow part) used to show only a static "Installing…" button → users couldn't tell if it hung. Added a CSS **spinner + live elapsed-seconds counter + rotating status hints** (`startInstalling()`/`stopInstalling()` around the `api('install')` call). It's client-side only (the install is one synchronous AJAX call; no per-row server progress without splitting the action) — the **live `Ns elapsed` counter is the real "not hung" signal**. Ships in the package like any `public/` file.

## ⚡ Update speed — full vs delta + two trims (2026-06-18)
- A client within the **last 6 published versions** gets a **delta** (only changed files) → fast; its backup is `backupLight` (already skips vendor). A client further behind falls back to the **FULL** package → slow: it tars (backup) AND `cp -a` (apply) ALL ~16k files incl `vendor` (108MB). So the FIRST update of a long-stale client is a one-time slow full; every later update is a fast delta. **Proven on moontest:** first jump = full + ~3min vendor tar, then `2.2.19→2.2.20` = **delta + fast**. (Delta window = `recentDeltaSources($version, 6)` in `ShipCommand`.)
- **Trim 1 — exclude `public/documentation`** (config `moonstack.package.exclude`): ~35MB / 561 dev API-doc files (scribe/openapi/postman) clients don't need at runtime → package **48.6MB→43.6MB, 15979→15419 files** → faster backup+apply for fulls, smaller deltas if docs change. Frontend (`public/app`) + root `.htaccess` still shipped (verified).
- **Trim 2 — skip `vendor` in the FULL backup when `composer.lock` is unchanged**: `BackupManager::backup($v, $skipVendor)` + `tarCode($dir, $skipVendor)` filter out `vendor`; `moonstack-update.php` computes `$skipVendor = msComposerUnchanged($zip, $basePath)` (sha256 of the package's `composer.lock` == the installed one). When deps don't change, vendor stays valid for a rollback too → the slow ~108MB tar is skipped (3min→seconds). Defaults to **false** (back vendor up) whenever uncertain. Both are BE edits in the live tree → ship in the next build; the updater trims take effect on the update AFTER a client receives the new `moonstack-update.php`/`BackupManager`.

## ✅ THE SEEDER GAP — FIXED (2026-06-17, commit `5b6eda047`, pushed `lis/lab-updates`)
Implemented exactly the plan below: `installer.seeders` += `LabSettingDefinitionSeeder`; new `config('moonstack.updater.seeders')` (definition seeders only); `moonstack:sync-reference` command (idempotent, per-seeder non-fatal); `moonstack-update.php` runs it after migrate **best-effort/non-fatal** (timeout-wrapped — can't roll back a good update); **bridge migration** `2026_06_17_220000_moonstack_sync_reference_definitions` runs the same seeders so OLD-updater clients catch up on next update. Verified crash-safe: bridge ran clean on moonui, command idempotent (defs stayed 58, no dup). **Pending: release** (clients get it on next update; the bridge catches them up immediately). Crash-safety reviewed against the full updater flow (self-heal, stale-lock recovery, shutdown-net rollback, backup) — the seed step uses the same safe `exec + timeout 300` pattern as migrate and never throws.

## Background — the bug that was fixed
**Updates ran `migrate` but NEVER ran seeders.** Setting DEFINITIONS are seeded via `SettingDefinitionSeeder` / `LabSettingDefinitionSeeder` (idempotent `updateOrCreate`), NOT migrations → **new definitions never reach updated clients**. Even fresh installs missed LIS defs (`installer.seeders` lacked `LabSettingDefinitionSeeder`). Proof: elmadina on **2.2.8 (latest)** had only **14/53** lis definitions → most lab settings invisible/inert (`SettingsService::get()` returns null without a definition).

**Fix plan (NOT yet implemented — sensitive, touches the fleet updater):**
1. Add `LabSettingDefinitionSeeder` to `installer.seeders`.
2. New `config('moonstack.updater.seeders')` = idempotent **definition seeders ONLY** (`SettingDefinitionSeeder`, `LabSettingDefinitionSeeder`). NOT `RolePermissionSeeder` — it re-assigns user roles, unsafe to re-run.
3. Run them after `migrate` in `moonstack-update.php`, **best-effort/non-fatal** (a seed failure must NOT trigger rollback of an otherwise-good update).
4. **Bridge migration** that runs the definition seeders → existing fleet (on the OLD updater) catches up on next update, since `migrate` always runs.

(elmadina was fixed manually 2026-06-17: ran `LabSettingDefinitionSeeder` → 58 settings; transferred 46 non-ID `lis.*` settings from moonui company 4, excluding `*_account_id`/`*_cost_center_id`/`*_price_list_id` to protect its accounting; `allow_self_verification=true`.)

## Links
- **Docroot-ready package plan (2026-06-18): https://moonui.elbaset.com/knowledge-base/plans/moonstack-docroot-ready-plan.html**
- Plan: https://moonui.elbaset.com/knowledge-base/plans/moonstack-plan.html
- Seeder-fix plan: https://moonui.elbaset.com/knowledge-base/plans/moonstack-update-seeder-fix-plan.html
- Changelog source: `moon-erp-be/docs/moonstack/CHANGELOG.md`
