chat_grid/plans/item-architecture-refactor-plan.md

# Server-First Item Architecture Refactor Plan

## Goal
Make the **server the only source of truth** for item definitions, schema, defaults, options, validation rules, and editable behavior. The client should consume server definitions and provide UX/rendering/audio only.

This plan removes client fallback definitions and introduces a repeatable, consistent item authoring structure so adding new item types is low-risk and uniform.

## Target End State

### 1) Source of truth
- Server owns, for each item type:
  - Type id, label, tooltip
  - Full property schema (value type, required/optional, min/max/step, maxLength, enum options)
  - Defaults (global + per-item initial params)
  - Editability and read-only behavior
  - Validation, normalization, migration policy (if any)
  - Runtime actions (`use`, optional custom actions)
  - Capability list
- Server sends this as canonical `uiDefinitions` + schema metadata on `welcome` (or equivalent bootstrap).

### 2) Client model
- Client has no static fallback item definitions.
- If schema payload is missing/invalid, item features are unavailable (explicit error/status), not silently guessed.
- Client property editor and item menus are metadata-driven.
- Client runtime behavior modules remain for UX/audio only (e.g., piano local mode), keyed by server type ids.

### 3) Repeatable item authoring
Adding an item type uses one standard server folder/template and a short checklist, with **auto-discovery** on server boot (no manual registry edits).

---

## Proposed Architecture

## A) Server: Item Type Package Contract (Auto-Discovered)
Create/standardize per-item server packages under something like:
- `server/app/items/types/<item_type>/`

Each item type package exports the same contract:
- `definition.py`
  - `type_id`, `label`, `tooltip`
  - `schema` (properties + metadata)
  - `defaults`
  - `editable_properties`
  - `capabilities`
- `validator.py`
  - `validate_create(params) -> normalized_params`
  - `validate_update(existing, patch) -> normalized_params`
  - Must drop unknown keys by default.
- `actions.py`
  - `use(context, item, client, payload?) -> result`
  - optional `custom_actions` handlers
- `ui.py` (optional if definition is enough)
  - transforms schema -> `uiDefinitions` payload fragments

A central loader in server scans `server/app/items/types/*` at boot and imports one plugin entrypoint per folder (for example `plugin.py` with `ITEM_TYPE_PLUGIN` export).

The discovered plugins are then assembled into an in-memory registry object exposing:
- validation hooks
- defaults
- ui definitions
- capabilities
- action dispatch

This means:
- adding a new item folder + plugin file is sufficient for server registration
- no hand-edited master list is required

## B) Server: Strict Params Hygiene
In update flow:
- Build next params by applying patch into current params.
- Run through type validator that:
  - strips unknown keys
  - normalizes known keys
  - enforces types/ranges/options
- Persist only validated output.

No raw client params should persist.

## C) Server: Save Strategy
Replace synchronous `save_state()` every mutation with coalesced writes:
- mark dirty on mutation
- debounce write (e.g., 100-300ms)
- cap max delay (e.g., 1-2s)
- flush on shutdown/signal

This preserves durability while reducing event-loop blocking.

## D) Client: Schema-Driven UI Runtime
Refactor client item registry/editor to consume server schema only.

Client keeps:
- Presentation helpers
- Generic item behavior path driven only by schema/metadata
- Optional per-item UX runtime modules only where needed (example: piano key mode)

Client removes:
- static defaults/options/editability lists as authority
- fallback-driven assumptions
- requirement for per-item client modules when behavior is generic

Property editor logic becomes generic:
- `valueType: boolean` -> toggle
- `valueType: list` + `options` -> list select
- `valueType: number` + `range` -> numeric editor/stepper
- `valueType: text/sound` + `maxLength` -> text editor
- `readonly` -> blocked edit with status

Special-case handlers only for UX extras (e.g., live preview for certain fields).

### D.1) Dependent Property Rules
Add dependency metadata to server schema so client can hide dependent fields generically.

Recommended metadata fields per property:
- `visibleWhen`: simple predicate (for example `{ directional: true }`)

Example:
- `facing` has `visibleWhen: { directional: true }`
- when `directional` is `false`, `facing` is hidden
- when a controlling property changes, the property menu is recomputed immediately so visibility updates live

## E) Protocol Shape (Recommended)
Ensure `welcome.uiDefinitions` includes enough to be complete:
- `itemTypes[]`
  - `type`, `label`, `tooltip`
  - `editableProperties[]`
  - `propertyMetadata{ key -> { valueType, tooltip, range, maxLength, options?, readonly?, visibleWhen? } }`
  - `globalProperties`
  - `capabilities`
- `itemTypeOrder[]`

Optional future:
- `schemaVersion` for compatibility checks.

---

## Phased Implementation Plan

## Phase 0: Preconditions and guardrails
1. Document canonical schema contract in `docs/item-schema.md`.
2. Add tests asserting unknown keys are rejected/stripped per type.
3. Add tests asserting `uiDefinitions` completeness for all registered types.

Deliverable:
- Locked schema contract and tests before heavy refactor.

## Phase 1: Server type package standardization + auto-discovery
1. Standardize all existing item types to same package contract.
2. Move any remaining type-specific logic out of generic server paths into per-type packages.
3. Add auto-discovery loader APIs:
- `get_type_definition(type_id)`
- `validate_update(type_id, existing, patch)`
- `build_ui_definitions()`
4. Loader scans item folders at startup and registers plugins automatically.

Deliverable:
- Uniform server-side item modules for all current item types.

## Phase 2: Strict validation and unknown-key stripping
1. Enforce strict allowed-key filtering in per-type validators.
2. Fail/strip behavior decision:
- recommended: strip unknown keys on load/update, optionally log at debug level.
3. Backfill tests for each type.

Deliverable:
- No unsupported params can persist.

## Phase 3: Client removes authority/fallback definitions
1. Remove client hardcoded item defaults/options as authoritative data.
2. Keep only bootstrap guards:
- if schema missing/invalid, fail item UX with explicit status (no fallback behavior).
3. Refactor `itemRegistry` to be a runtime cache of server definitions.

Deliverable:
- Client item UI driven entirely by server payload.

## Phase 4: Metadata-driven property editor + visibility dependencies
1. Replace key-specific submit/toggle/list branches with generic metadata-based handlers.
2. Keep a small optional hook map:
- `onPropertyPreviewChange(type,key,value)` for UX preview.
3. Implement `visibleWhen` semantics with live menu recompute when controlling values change.
4. Verify all current item properties work with generic editor.

Deliverable:
- Adding a new field usually requires server changes only.

## Phase 5: Behavior registry completion (optional modules)
1. Keep one generic behavior path that works for items with no special runtime.
2. Keep per-item behavior modules only when UX/audio runtime is truly custom.
3. Ensure all runtime hooks are accessed via registry interfaces.
4. Remove any remaining type checks in `main.ts` and shared handlers.

Deliverable:
- `main.ts` stays orchestration-only.

## Phase 6: Coalesced persistence
1. Implement debounced save queue in server item service.
2. Add durability tests (flush on shutdown).
3. Add config knobs (debounce ms, max delay).

Deliverable:
- lower save overhead under bursty updates.

---

## Repeatable New Item Template

When adding a new item type:

1. Server
- Create `server/app/items/types/<new_type>/`
- Implement:
  - `definition.py`
  - `validator.py`
  - `actions.py`
  - `plugin.py` (entrypoint export for auto-discovery)
- Add tests:
  - create defaults
  - update validation (valid + invalid + unknown keys)
  - `use` behavior
  - `uiDefinitions` fields present

2. Client
- Add `client/src/items/types/<newType>/behavior.ts` only if custom UX runtime exists.
- Prefer zero client type-specific code for generic items.
- If behavior module is needed, register via behavior loader pattern.
- No hardcoded property logic in editor.

3. Docs
- Update `docs/item-types.md`
- Update `docs/item-schema.md`
- Update controls docs if keybindings changed.

---

## Risks and Mitigations

1. Risk: temporary client breakage when fallback removed.
- Mitigation: explicit schema-required startup check and clear status error.

2. Risk: inconsistent schema during deploy rollover.
- Mitigation: include `schemaVersion` and reject incompatible client/server combinations with clear reconnect message.

3. Risk: over-generalized editor misses edge-case UX.
- Mitigation: keep small per-item preview hooks while generic editor handles core commit logic.

4. Risk: debounced persistence data loss on crash.
- Mitigation: short debounce + max-delay + flush on shutdown.

---

## Suggested Execution Order for Your Repo (Practical)
1. Implement strict unknown-key stripping on server (highest impact, lowest UX risk).
2. Implement server plugin auto-discovery for item type folders.
3. Convert client item registry to require server schema payload (remove fallback authority).
4. Make item property editor fully metadata-driven with dependency rules.
5. Finalize optional client behavior modules (only for custom UX items like piano).
6. Add coalesced persistence.

---

## Definition of Done
- Server item validators fully define accepted params and drop unknowns.
- Server item types are boot-loaded from folder plugins (no manual master registry edits).
- `uiDefinitions` is complete and authoritative for all item UI config.
- Client contains no authoritative item defaults/options/editability outside server payload.
- Client has no fallback schema path.
- New item addition follows one template with predictable files/tests.
- `main.ts` has no item-type-specific runtime branches.

---

## Implementation Update (2026-02-24)

### Completed
- Phase 0:
  - Added server-side contract coverage for `uiDefinitions` completeness.
  - Added/kept tests for unknown-key stripping and validation behavior.
- Phase 1:
  - Server item plugins are auto-discovered from `server/app/items/types/*/plugin.py`.
  - Registry now builds type order/modules from discovered plugins.
- Phase 2:
  - Unknown params are stripped by validators and use-path updates are revalidated before persist.
- Phase 3:
  - Client item registry now requires server `uiDefinitions`; no fallback item-definition authority.
  - Missing/invalid schema now disables item menus with explicit status.
- Phase 4:
  - Property editor behavior is metadata-driven by `valueType/range/options/maxLength`.
  - `visibleWhen` now survives protocol parsing and item property rows recompute live after updates.
  - List options are now carried in `propertyMetadata[key].options` (no separate `propertyOptions` map).
- Phase 5:
  - Client runtime behavior remains modular per item via behavior registry; `main.ts` orchestration no longer carries item-specific business branches.
- Phase 6:
  - Coalesced/debounced state saving implemented.
  - Flush-on-shutdown implemented.
  - Save timing now configurable via:
    - `storage.state_save_debounce_ms`
    - `storage.state_save_max_delay_ms`
 - Additional:
   - Added plugin contract tests to validate required item package files/exports.
   - Added unknown-item-type rejection for `item_add`.
   - Added `capabilities` in `welcome.uiDefinitions.itemTypes`.
   - Removed hardcoded client/server item-type enum restrictions (string-based ids + runtime known-type checks).

### Notes
- Client item-specific runtime is now reduced to only `piano`; simple items (`dice`, `wheel`, `clock`, `radio_station`, `widget`) run through generic client flows with no custom behavior module.
- Server item implementations now live inside per-type folders (`server/app/items/types/*/definition.py`, `validator.py`, `actions.py`).
- Plugins now compose module surfaces via a shared helper (`server/app/items/types/plugin_helpers.py`), so per-type `module.py` is no longer required.
- Added docs sample folder at `docs/examples/item-type-sample/` and updated template docs to reflect the package layout.
## Completion Review (2026-02-24)

### Status against phases
- **Phase 0 (docs + guardrails):** ✅ Complete. `docs/item-schema.md` exists and server tests cover UI schema completeness + unknown-key stripping behavior.
- **Phase 1 (server package standardization + auto-discovery):** ✅ Complete. Item types are standardized under `server/app/items/types/*` with `plugin.py` entrypoints and discovered by `server/app/items/registry.py`.
- **Phase 2 (strict validation + unknown-key stripping):** ✅ Complete. Type validators enforce allowed keys and normalize values; tests verify unknown params are stripped.
- **Phase 3 (client removes fallback authority):** ✅ Complete. Client consumes `welcome.uiDefinitions` and disables item menus when schema is missing.
- **Phase 4 (metadata-driven editor + visibility dependencies):** ✅ Complete. Generic metadata-driven editing is in place and `visibleWhen` survives client protocol parsing.
- **Phase 5 (behavior registry completion):** ✅ Complete. Runtime behavior is registry-driven with generic path + optional piano module.
- **Phase 6 (coalesced persistence):** ✅ Complete. Debounced/max-delay state save and flush-on-shutdown are implemented in server lifecycle.

### Recommendations / Cleanup (remaining)
1. **Eliminate manual property label mapping in client (medium):** either include labels for all properties in server metadata or auto-humanize keys so new properties are readable without code changes.
2. **Strengthen new-item completeness checks (low):** extend the plugin contract tests to optionally assert docs coverage and richer protocol examples per discovered type.

### Item 1-6 Status (explicit)
1. **visibleWhen end-to-end:** ✅ Done.
2. **Remove hardcoded item-type literals/enums:** ✅ Done (string-based ids + runtime known-type checks).
3. **Include capabilities in `uiDefinitions.itemTypes`:** ✅ Done.
4. **Move list options into property metadata:** ✅ Done (`propertyMetadata[key].options` only).
5. **Remove manual client property-label mapping:** ❌ Not fully done yet. Metadata labels are supported, but legacy hardcoded label map still exists as fallback.
6. **New-item completeness validation check:** ✅ Baseline done (plugin contract tests added for required files/exports).  
Potential enhancement: add docs/protocol example assertions for each discovered type.