16. Dev/Staging Subscription Remediation & Entitlement Verification

16.1 Purpose and scope

Following Chapter 15's deployment-readiness pass, the dev tenant catalog was reported to contain stale TenantSubscription references blocking true Finance-tier live testing. This chapter documents: the corrected root-cause investigation, the new root-admin-only Preview/Apply remediation tool built to fix this class of problem safely in any environment, and a full live re-verification of all four packages end-to-end (registration → activation → entitlement → onboarding → Starter Kit → Cash/Bank GL linking) on four fresh, safe test tenants.

This is a data-integrity and tooling phase — no package definition, entitlement rule, or limit was changed.

16.2 Root cause, corrected

The prior phase's "16 stale tenants" finding was itself based on an incorrect diagnostic query that joined TenantSubscription.PackageId directly to Saas.SubscriptionPackages.Id. That join is wrong — Chapter 3 §3.3 already documents the real chain correctly:

TenantSubscription.PackageId → SubscriptionPlanPackage.Id
  → SubscriptionPlanPackage.PackageId → SubscriptionPackage.Id

Re-running the diagnosis with the correct join chain, this task independently re-confirmed:

• There is no live evidence of the previously-reported "16 stale tenants" — the dev tenant catalog had, in the meantime, been intentionally reduced by the user to just the root tenant (000111); the other tenant databases were deliberately left orphaned/untouched and are explicitly out of scope for this phase (not restored, not reconnected, not inferred from).
• A genuinely dangling TenantSubscription.PackageId (pointing at a SubscriptionPlanPackage row that no longer exists) could not be manufactured or observed live without corrupting real data, so this specific failure mode is verified by unit test only (see §16.4).
• The root tenant (000111) itself currently has no subscription in Status = Active. This is a pre-existing, low-risk condition (the root tenant is never gated by package entitlement — see Chapter 6) but the new remediation tool correctly flags it rather than silently guessing a target package for it.

Lesson carried forward: always re-derive a diagnostic query from the entity's real [ForeignKey] attribute (or this chapter) before trusting a prior phase's raw SQL — a plausible-looking join on a misleadingly-named column (PackageId on TenantSubscription does not point at SubscriptionPackage) is exactly the kind of mistake that produces a confident but false finding.

16.3 Why FName is the stable package identity

SubscriptionPackage has no Code/Key column (Chapter 3 §3.1). Numeric Ids and SubscriptionPlanPackage.Ids are stable within one environment but are not safe to hardcode across environments or to assume unchanged after a data cleanup — exactly the kind of drift that triggered this phase. FName (the English package name — e.g. "Shumoul Starter") is the only human-stable, cross-environment identity for a package, and is therefore the required target key for any remediation mapping (see §16.4). A numeric TargetPlanPackageId may still be supplied explicitly when an operator has a specific plan-package row (e.g. Annual vs Monthly) in mind, but TargetPackageFName is the default, safer path.

16.4 SubscriptionRemediation — Preview/Apply tool

A new root-admin-only surface, Shumoul.Framework.MultiTenancy.Api.SubscriptionRemediation (Shumoul.Framework.MultiTenancy.Api 1.0.124), for diagnosing and — only when explicitly enabled and explicitly mapped — repairing a tenant's subscription-package reference.

POST api/Saas/SubscriptionRemediation/Preview   — MustHavePermission(Permissions.SubscriptionRemediation.View)
POST api/Saas/SubscriptionRemediation/Apply      — MustHavePermission(Permissions.SubscriptionRemediation.Apply)

Request body (both actions): TenantIds (empty = scan every tenant), Mappings (TenantId → TargetPackageFName or explicit TargetPlanPackageId, plus a free-text Reason), CorrelationId.

16.4.1 Safety gates

• Preview never writes anything, regardless of environment or flag state — it is pure diagnosis.
• Apply is disabled by default everywhere, gated by a host-level (not per-tenant) config flag: SubscriptionRemediationSettings:AllowApply (default false). This is a plain IOptions<T> binding in Startup.cs, the same pattern as the sibling apply-bridge settings — there is no HttpClient/bridge involved because this feature never leaves the central Saas database.
• Apply never guesses a target package from a tenant's name, activity, or any heuristic — an ambiguous or unmapped tenant is always reported as SkippedRequiresExplicitMapping, never silently remediated to a default.
• Remediation reuses the existing, already-audited ActivateLicenceAsync convention exactly (Tenant Entitlement Provisioning Phase 2 family of work): the old subscription row is superseded (Status = Renewed, IsActive = false), a new row is created (Status = Active, IsActive = true, linked via FromSubscriptionId), and a TenantSubscriptionLog entry is written — all inside one transaction. No existing subscription row is ever deleted or mutated in place.
• Apply is fully idempotent: re-running it against an already-remediated tenant reports ValidNoChange, never a duplicate remediation.

16.4.2 Result status vocabulary

Status	Meaning
ValidNoChange	Tenant's active subscription already resolves correctly — nothing to do
Remediated	Existing subscription superseded, new Active row created pointing at the resolved plan-package
Created	Tenant had no subscription at all; a new Active row was created
SkippedRequiresExplicitMapping	Tenant's target package is ambiguous and no mapping was supplied — never guessed
SkippedNotAllowedInEnvironment	Apply attempted while AllowApply = false
SkippedInvalidTargetPackage	Supplied TargetPackageFName/TargetPlanPackageId does not resolve to a real active plan-package row
SkippedConflictMultipleActiveSubscriptions	More than one Status = Active subscription found for the tenant — a data anomaly that must be resolved manually, never auto-picked
SkippedUnknownState	Diagnosis could not classify the tenant's state confidently
Failed	An error occurred during remediation; the transaction was rolled back

16.4.3 Live Preview result, this task (current tenant set)

Running Preview with no TenantIds filter (scan all) against the current dev tenant set — the root tenant plus the four newly-registered test tenants (§16.6) — returned:

• 4 × ValidNoChange — the four freshly-registered/activated test tenants, each correctly resolving to its intended package with no remediation needed.
• 1 × SkippedRequiresExplicitMapping — the root tenant (000111), correctly flagged rather than guessed, per §16.2.
• 0 stale/dangling references were found live in this environment — consistent with §16.2's corrected root-cause finding.

16.4.4 Unit test coverage

10 tests in TenantSubscriptionRemediationServiceShould cover both Preview and Apply, including a manufactured dangling-PackageId scenario (a real SubscriptionPlanPackage row is seeded then deleted via a raw-SQL statement, since SQLite's own FK cascade would otherwise remove the dependent subscription row too — production does not enforce/cascade this the same way): stale-reference detection, no-active detection, multiple-active-subscriptions conflict detection, the AllowApply gate, explicit-mapping requirement, successful remediation by FName, idempotency on a second run, no-op on an already-valid subscription, Preview never writing under any input, and invalid-target-package reporting. All 10 pass.

16.5 Safe test-contact policy (binding for all future dev/staging verification)

A previous live verification pass used a guessed test phone number that triggered a real WhatsApp notification to a real person. Going forward, for any phase that registers a tenant, verifies OTP, or otherwise exercises a notification-sending flow in dev/staging:

• Use only test phone numbers and emails that have been explicitly confirmed safe in a prior session or by the user in the current one. Never invent, guess, or increment a phone number or email address — see [Feedback: Test Phone Number Safety] (internal memory note) for the incident this policy traces back to.
• If no safe contact is available for a needed scenario, stop and ask — do not substitute a plausible-looking one.
• Prefer reusing an existing safe dev tenant over registering a new one; only register a new tenant when the existing catalog cannot cover the scenario (as was the case this phase, once the prior dev tenant catalog had been intentionally reduced to just the root tenant).

16.6 Live re-verification — four fresh test tenants, one per package

Per explicit instruction, four new tenants were registered — one per current package — using only pre-approved safe contacts, through the real, unmodified registration → OTP-verification → auto-activation pipeline (POST api/Tenant/Registration/Register → POST api/Tenants/Verification → DemoAutoActivation, gated by SubscriptionSettings.Enable_Auto_Activate_When_Verified). No shortcut bypassed real business logic; the OTP value itself was read from the tenant's own VerificationToken column (a reasonable, narrow substitution for lacking inbox access to a real device, not a bypass of the verification endpoint or its validation).

Package	Tenant Id	Result
Shumoul Starter	183581	Registered, verified, auto-activated
Shumoul POS	510611	Registered, verified, auto-activated
Shumoul Finance Starter	171994	Registered, verified, auto-activated
Shumoul Finance Advanced	793084	Registered, verified, auto-activated (second attempt — first tenant name collided with an existing name and was rejected by the existing uniqueness validator, not a bug)

16.6.1 Entitlement matrix — confirmed live, zero cross-tier leakage

Package	Entitled feature keys (live-confirmed)
Shumoul Starter	PointofSalesApp, ProductsManagement, SalesManagement
Shumoul POS	InventoryManagement, PointofSalesApp, ProductsManagement, SalesManagement
Shumoul Finance Starter	AccountManagement, PayablesManagement, ReceivablesManagement
Shumoul Finance Advanced	AccountManagement, PayablesManagement, ProjectManagement, ReceivablesManagement

16.6.2 Operational defaults / Starter Kit / Cash-Bank linking — confirmed live

Read-only verification against the shared tenant database, across all four tenants, after running onboarding and/or TenantOperationalDefaults apply for each:

Tenant	Package	Warehouse	CashAccount	BankAccount	CashAccountCurrency	BankAccountCurrency
183581	Starter	1	1	0	0	0
510611	POS	1	1	0	0	0
171994	Finance Starter	1	1	1	1	1
793084	Finance Advanced	1	1	1	1	1

Exactly matches expectation: only the two AccountManagement-entitled (Finance-tier) tenants receive a BankAccount and the Cash/Bank GL-currency links (Cash/Bank GL-Currency Linking phase); Starter/POS correctly receive Warehouse + CashAccount only, with zero finance-account leakage.

16.6.3 Onboarding entitlement filtering — confirmed live (Starter tenant)

For tenant 183581 (Starter), a full onboarding pass (status → survey → answers → recommendation → apply → status) was run through the real authenticated host:

• recommendation returned a settingsPatch limited to CashierAppsSettings, ProductSettings, DiscountSettings, TaxSettings — with explicit warnings confirming CostCenterSettings and FinancialSettings were stripped specifically because they are not included in the tenant's active subscription, exactly the entitlement-filtering behavior documented in Chapter 9.
• apply succeeded ("Onboarding settings applied."); final status returned "status":"Applied", "appliedProfileCode":"RestaurantCafe".
• This satisfies the "at least one Starter/POS tenant" onboarding-verification requirement for this phase; the POS tenant's onboarding flow was not separately exercised this pass.

16.6.4 No side effects — confirmed via read-only SQL

Across all four test tenants, read-only counts confirm zero financial or ERP-side records were created as a byproduct of this verification pass:

Check	Result (all 4 tenants)
GeneralLedgerJournalEntries	0
PaymentVouchers / ReceiptVouchers	0
StockItemTransactions	0
AccountOpeningBalances	0
AppNotifications (ERP-side)	0

No notification was sent beyond the already-authorized registration/OTP messages to the four pre-approved safe contacts (§16.5).

16.7 Known issue — shared-DB cross-tenant settings cache (FIXED — see Chapter 17)

While verifying tenant 793084 (Finance Advanced), CashBankAccountingLink initially reported MissingDependency immediately after TenantOperationalDefaults had just created the required CashAccount/BankAccount rows in the same request — a symptom, not a real missing dependency.

Root cause, confirmed by direct DB inspection plus a controlled experiment: all four test tenants share one physical database (UsingSharedDb = 1) in this dev environment. An IAppSettingService/ CashierAppsSettings-shaped in-process cache keyed insufficiently for shared-DB tenants processed sequentially in the same host process can serve a stale/incorrect value to the second tenant processed. Killing and restarting the host process, then processing the previously-failing tenant first (alone) in the fresh process, succeeded correctly — proving this is a same-process, cross-tenant cache artifact, not a defect in the Cash/Bank GL-Currency Linking logic itself or in any prior-phase code.

Status: fixed in the very next phase. Every AppSettings-related cache key across AppSettingService, TenantSettingService, CashierUserSettingService, DeviceSettingService, and PrinterSettingService now embeds the resolved tenant identity via a new centralized IAppSettingCacheKeyBuilder, so two tenants sharing one host process (isolated-DB or shared-DB) can no longer read or invalidate each other's cached settings. Full root cause, design, tests, and live re-verification (using this exact tenant pair) are in Chapter 17 — Tenant-Isolated AppSettings Caching.

16.8 Build and test verification (this pass)

Repo	Command	Result
MultiTenancyApi	dotnet build --no-restore	0 errors
MultiTenancyApi	dotnet test --no-build	baseline 1165 total / 6 failed, plus 10 new remediation tests — all 10 new tests pass, zero new failures
BackEnd	dotnet restore	Succeeded
BackEnd	dotnet build --no-restore -p:TreatWarningsAsErrors=false	0 errors
BackEnd	dotnet test --no-build	baseline 815 total / 35 failed — unchanged, zero new failures

16.9 Outcome

• The previously-reported "16 stale tenants" finding does not hold up under a corrected diagnostic query and is superseded by this chapter.
• A safe, disabled-by-default, idempotent, root-admin-only Preview/Apply remediation tool now exists for this entire class of problem in any future environment (Shumoul.Framework.MultiTenancy.Api 1.0.124).
• All four current packages are live-reconfirmed end-to-end on fresh, safe test tenants with zero cross-tier entitlement leakage and zero unintended financial/ERP side effects.
• One real, previously-unknown bug was discovered and documented (§16.7); it was intentionally left unfixed in this phase (outside its data-integrity/verification scope) and closed in the very next phase — see Chapter 17.