Checklist: Progressive SOA Decomposition (TDD)

This checklist provides a step-by-step, deep-dive process for decomposing Framework M Business modules (like WMS, People, and Finance) into Macroservices (SOA) while maintaining a standard monolith mode for “Indie” users.

Showcase Scope: This document covers the SOA decomposition of domain modules. Extremely high-throughput specialized microservices (e.g., Book-Keeper, Stock-Keeper built on Litestar/TigerBeetle) are considered External Adapters to this ecosystem.

---

Part 1: The Contract (Shared Protocols)

Goal: Define WHAT a module does, not HOW it does it. This avoids direct library imports.

• 1.1 Identify the Cross-Module Call
  • Example: WMS calls Finance for ledger syncing (CustomerService).
  • Example: Finance calls Core for party details.
• 1.2 Create a Shared Protocol
  • Place in libs/<provider>/src/<provider>/interfaces/.
  • Define a Python Protocol (interface).
  • Use only standard Python types or shared DTOs (no DocType imports if possible).
• 1.3 Zero-Import Check
  • Ensure the consumer (WMS) does NOT import from the provider’s services or doctypes directly. It should only import the Protocol.

---

Part 2: The “Indie” Adapter (Local Monolith)

Goal: Logic for when everything runs in a single process.

• 2.1 Red Test: Local Adapter
  • Create libs/<provider>/tests/test_local_adapter.py.
  • Assert the adapter fulfills the protocol.
• 2.2 Green Code: Local Implementation
  • Create libs/<provider>/src/<provider>/adapters/local_<name>.py.
  • This implementation calls the local module’s services/controllers directly.
• 2.3 Refactor
  • High coverage for the local path.

---

Part 3: The “Enterprise” Adapter (NATS/Remote)

Goal: Logic for when modules are separate processes.

• 3.1 Red Test: NATS Adapter
  • Create libs/<provider>/tests/test_nats_adapter.py.
  • Mock the NATS event bus.
  • Assert that calling the protocol publishes a message to the correct subject.
• 3.2 Green Code: NATS Implementation
  • Create libs/<provider>/src/<provider>/adapters/nats_<name>.py.
  • Use framework-m-standard’s NATS bridge.
  • Pattern A (Consistency): Use bus.request for real-time lookups (e.g. Finance calling Core for Supplier terms).
  • Pattern B (CQRS): Use bus.subscribe for local projections (e.g. WMS building local Item read-models).
  • Scaling: Background subscribers MUST specify a queue group name (e.g., queue='wms-service') to enable NATS load-balancing across replicated pods.
• 3.3 Refactor
  • Ensure timeouts and error handling for network calls are in place.

---

Part 4: Dependency Injection (Constructor Injection)

Goal: Remove the “hidden” dependencies (singletons/direct imports).

• 4.1 Red Test: Service Isolation
  • Run the consumer service (MemberService) in a test that wipes the provider’s lib from sys.modules.
  • It MUST fail with ImportError if it still has a direct import.
• 4.2 Green Code: Refactor to DI
  • Add the Protocol to the service’s __init__.
  • Replace direct calls (e.g., bookkeeper_client.sync_account) with self.ledger.sync_account.
• 4.3 Container Wiring
  • Register the service in the Container using container.register(...).
• 4.4 Container Wiring
  • Register the service in the Container using container.register(...).
  • Ensure any environment-specific overrides (Indie vs Enterprise) are applied here.

---

Part 11: Identified Service Protocols (The Spoonfed List)

Goal: Explicit list of domain services to be abstracted into protocols for SOA decomposition.

11.1 Finance Domain Boundaries

• ExternalLedgerProtocol: Primary interface for the external BookKeeperClient (TigerBeetle).
• BudgetProtocol:
  • Service: finance.services.budget_utilization
  • Methods: update_budget_utilization(company, account, cost_center, fiscal_year, amount, reverse=False)
• FiscalProtocol:
  • Service: finance.services.fiscal_year
  • Methods: get_fiscal_year(date, company), is_period_closed(date, company)

11.2 WMS Domain Boundaries

• ExternalStockProtocol: Primary interface for the external StockKeeperClient (Litestar/KurrentDB).
• InventoryProtocol:
  • Service: wms.services.inventory_service
  • Methods: get_stock_balance(item, warehouse), create_sle(...), get_bin(item, warehouse)
• OnboardingProtocol:
  • Service: wms.services.product_onboarding_service
  • Methods: submit_application(app), evaluate_pre_screening(...), record_screening(...), management_decide(...)
• LifecycleProtocol:
  • Service: wms.services.outlet_lifecycle_service
  • Methods: submit_registration(reg), cs_approve_registration(reg), agm_approve_registration(...), disburse_loan(loan, reference)
• DemandProtocol:
  • Service: wms.services.demand_service
  • Methods: calculate_predicted_demand(...), generate_distribution_center_demand(dc, from_date, to_date)

---

Part 5: The “Mode” Toggle (The Switch)

Goal: Toggle between Indie and Enterprise via configuration.

• 5.1 Red Test: Mode Switching
  • Create apps/business-m/tests/test_mode_toggle.py.
  • Set os.environ["M_MODE"] = "enterprise".
  • Resolve the protocol from the container -> Assert it is the NatsAdapter.
  • Set os.environ["M_MODE"] = "indie".
  • Resolve -> Assert it is the LocalAdapter.
• 5.2 Green Code: Container Logic
  • Update apps/business-m/src/business_m/container.py (or main init).
  • Add conditional bindings based on environment variables.

---

Part 6: The “Unified” Startup (Indie & Replicated Mode)

Goal: Highly efficient “Golang-like” single-process startup that scales horizontally via replication.

• 6.1 Bootstrap Script (Unified Runner)
  • Create libs/<provider>/src/<provider>/main.py.
  • Use the create_app factory from Framework M.
  • Implement the Unified Async Loop to run API and Background Workers in the same process:

    async def main():
        # 1. Start Background Listeners (Task)
        broker = get_broker()
        worker_task = asyncio.create_task(_run_worker_async(broker))
        # 2. Start Web Server (Blocking)
        app = create_app(serve_static=False)
        config = uvicorn.Config(app)
        server = uvicorn.Server(config)
        await server.serve()

• 6.2 Replication & Scaling Test
  • Run 3 instances of the unified process.
  • Send 3 NATS messages and 3 API requests.
  • Success Criteria: API requests are distributed by the Ingress; NATS messages are distributed by the Queue Group.
• 6.3 Standalone Test
  • Send a NATS message to an isolated provider -> Verify it executes.

---

Part 7: Final Verification

• 7.1 Indie Mode Smoke Test: Monolith works, ACID transactions hold.
• 7.2 Enterprise Mode Smoke Test: Services talk over NATS, logs show RPC calls.

---

Part 8: Protocol Spoonfeeding (The Blueprints)

Goal: Concrete interfaces for WMS and Finance to be implemented as standalone services.

8.1 LedgerProtocol (Finance)

Path: libs/finance/src/finance/interfaces/ledger.py

from typing import Protocol, Any
from datetime import date
from decimal import Decimal

class LedgerProtocol(Protocol):
    async def sync_account(self, name: str, account_type: str, company: str, is_group: bool, parent_account: str | None) -> str: ...
    async def post_gl_entry(self, company: str, posting_date: date, debit_account: str, credit_account: str, amount: Decimal, narration: str, reference: str | None = None, cost_center: str | None = None, party_type: str | None = None, party: str | None = None) -> str: ...
    async def cancel_gl_entry(self, gl_entry_id: str) -> None: ...
    async def post_journal_entry(self, company: str, posting_date: date, entry_type: str, narration: str, legs: list[dict[str, Any]]) -> str: ...
    async def cancel_journal_entry(self, gl_batch_id: str) -> None: ...
    async def get_trial_balance(self, company: str, fiscal_year: str, as_of_date: date) -> list[Any]: ...
    async def post_period_closing(self, company: str, fiscal_year: str, closing_account: str, net_pl_amount: Decimal, posting_date: date) -> str: ...
    async def get_batch_gl_entries(self, gl_batch_id: str) -> list[Any]: ...

8.2 StockProtocol (WMS)

Path: libs/wms/src/wms/interfaces/stock.py

from typing import Protocol, Optional
from decimal import Decimal

class StockProtocol(Protocol):
    async def transfer_stock(self, tenant_id: str, item: str, from_zone: Optional[str], to_zone: Optional[str], qty: Decimal, rate: Decimal, reference: str) -> str: ...
    async def get_balance(self, tenant_id: str, item: str, zone: str) -> Decimal: ...
    async def get_ledger(self, tenant_id: str, item: str, zone: str, limit: int = 100) -> list[dict]: ...

---

Part 9: Resilience (The Retry-Cooldown Loop)

Goal: Ensure network-bound adapters don’t fail on transient glitches.

• 9.1 Define Retry Logic
  • Use exponential backoff (e.g., 1s, 2s, 4s).
  • Target: asyncio.TimeoutError, OSError, ConnectionError (transient NATS/network failures).
• 9.2 TDD: The “Green” Path with Failure
  • Red Test: Mock a TimeoutError for the first 2 calls, then success.
  • Success Criteria: Service retries twice automatically, waits the cooldown, and eventually returns the success result.
• 9.3 TDD: The “Hard” Failure
  • Red Test: Mock TimeoutError consistently for all calls.
  • Success Criteria: Service exhausts retries and raises a clear ResilienceError.
• 9.4 Implementation Strategy
  • Implemented @retry_with_backoff in libs/finance/src/finance/utils/resilience.py.
  • Parallel copies in libs/wms/src/wms/utils/resilience.py and libs/m_adapters/src/m_adapters/resilience.py.
  • Applied to all methods of all NATS adapters in finance, wms, and m_adapters.

---

Part 10: Downstream Failure Handling

Goal: Decide what happens when the remote service is DEAD, not just transiently slow.

• 10.1 Classify the Operation
  • Critical (Synchronous): Payment Voucher, Stock Transfer. If Ledger/Stock-Keeper is down, the operation MUST fail.
  • Non-Critical (Eventually Consistent): Account creation, Item onboarding. The record can be saved locally, but marked as “Pending Sync”.
• 10.2 Implement “Hard Fail” for Criticals
  • PaymentVoucher.on_submit and on_cancel: ResilienceError → ValidationError (rolls back DB transaction).
  • StockEntry.on_submit and on_cancel: ResilienceError → ValidationError (rolls back DB transaction).
• 10.3 Implement “Soft Fail” for Non-Criticals
  • Added sync_status field (Synced, Pending, Error) to Account and Outlet DocTypes.
  • Account.after_save: catches ResilienceError and BookKeeperSyncError, sets sync_status="Pending", surfaces warning.
  • Outlet.after_save: soft-fail stub with sync_status="Pending" pattern (NATS call wired in Part 12).
• 10.4 The Outbox Pattern (Enterprise Grade)
  • Instead of calling the NATS adapter directly in the request cycle, save the intent to an Outbox table.
  • A background “Relay” service reads the outbox and retries the NATS call until success.

---

Part 16: External Microservice Integration

Goal: Connect Framework M Macroservices to Specialized, high-performance external engines.

• 16.1 Define the External Protocol
  • e.g., BookKeeperClient for TigerBeetle.
  • These protocols are fundamentally different from internal SOA. They map to the external engine’s API, not a internal DocType.
• 16.2 Implement Async Drivers
  • Use high-performance Python drivers (e.g., msgspec for serialization, uvloop if needed).
• 16.3 Isolate the Interaction
  • Interactions with these microservices should ideally be handled asynchronously (via Outbox) to prevent the framework’s synchronous handlers from blocking.

---

Part 12: The Enterprise Runner (Uvicorn Standalone)

Goal: Run a library as a self-contained microservice.

• 12.1 The App Wrapper
  • Libraries (libs/wms) are code, but apps/<service> are the executors.
  • They must fulfill the WebProtocol.
• 12.2 Standalone Initialization
  • In app.py, initialize the framework: app = framework_m.init_app("<service_name>").
  • The WebAdapter (located at framework_m_standard/adapters/web/app.py) is used to implement the WebProtocol.
  • Bind Enterprise Adapters (NATS/GRPC) to the domain Protocols.
• 12.3 Running via Uvicorn
  • Command: uvicorn <package>.app:app --host 0.0.0.0 --port <PORT>
• 12.4 Comparison: Indie (Unified) vs Enterprise (Split)
  • Indie (Standard Replicated): python main.py -> Unified process (API + Worker). Scale by replicating the Pod.
  • Enterprise (Isolated Scaling): Split into --mode=api and --mode=worker targets.
    • Why?: Resource Partitioning (e.g., Workers need 4GB RAM for reports, APIs need 512MB for high-concurrency).
    • NATS Synergy: Both modes use the same Queue Group name, ensuring messages are load-balanced correctly regardless of pod topology.

---

Part 13: UI Decomposition (The Remote MFE)

Goal: Serve and register the Micro-Frontend (MFE) from the standalone service.

• 13.1 Build the MFE Remote
  • Run the MFE build for the module: cd libs/wms/frontend && pnpm build:mfe.
  • Verify remoteEntry.js is generated in src/wms/static/mfe/.
• 13.2 Host the MFE Assets
  • Ensure the standalone service (apps/wms-service) is configured to serve static files from the library’s static/ directory.
  • Or upload to a CDN/S3 bucket for “Level 5” deployments.
• 13.3 Register with the Shell Host
  • Access the main Shell Host’s configuration (e.g., K8s ConfigMap or .env).
  • Add the service to FRAMEWORK_M_MFE_REMOTES:

    { "wms": "http://wms-service:8000" }

• 13.4 Enable Proxy Fallback (Optional)
  • If the browser cannot reach wms-service:8000 directly, enable the Shell Proxy on the main host: FRAMEWORK_M_MFE_PROXY_ENABLED=true
• 13.5 Verify UI Discovery
  • Open http://shell-host:8000/api/v1/frontend/remotes.
  • Assert it contains: "wms": "http://wms-service:8000/mfe/wms/1.2.3/remoteEntry.js".
• 13.6 Smoke Test the Desk
  • Log in to the Desk UI.
  • Navigate to a WMS-specific route (e.g., /wms/inventory).
  • Verify the remote MFE loads and communicates with the decomposed backend over NATS.

---

Part 14: Data Sovereignty & Distributed Reads

Goal: Transition from SQL Foreign Keys (Monolith) to distributed logical references.

14.1 Logical References

Instead of hardcoding database-level Foreign Keys, services use Plain String IDs to reference data across domains.

libs/wms/src/wms/doctypes/stock_entry.py

class StockEntry(BaseDocType):
    supplier_id: str  # Stored as plain UUID/string

14.2 Resolution Patterns

The Application Adapter (see Part 3) is responsible for resolving these IDs based on the deployment mode:

• Pattern A: Request/Response (RPC):

  • Usecase: When we need the absolute latest, consistent state from the source.
  • Implementation: The adapter dispatches a NATS bus.request("rpc.Supplier.get", {"id": ...}).
  • Showcase: Finance resolving Supplier terms for a payment.

• Pattern B: CQRS Projections (Local Read-Models):

  • Usecase: When we need high-throughput autonomy and survival.
  • Implementation: The service maintains a local projection_suppliers table, updated by listening to Supplier.updated events.
  • Showcase: WMS resolving Item names during high-speed stock intake.

---

Part 15: Resolved Case Study - StockEntry -> Supplier

15.1 The Application Showcase Path

• Field Definition: StockEntry.supplier_id is defined as a plain string.
• Pattern A (RPC) Verification:
  • Verified the NatsSupplierAdapter correctly resolves the full partner object via NATS RPC.
• Pattern B (CQRS) Verification:
  • Implemented a background NATS subscriber (subscribe_supplier_events) in wms.services.projection_worker that listens to doc.create, doc.update, and doc.delete Supplier events and enqueues update_projection_supplier jobs.
  • Enriched ProjectionSupplier read-model with supplier_name, is_active, and last_synced_at fields for full offline autonomy.
  • StockEntryController.before_save now validates both existence and active status against the local projection.
  • Verified (tests/test_part15_cqrs_verification.py): GRN entry completes using local projection even if Business M is offline; 7 TDD scenarios cover enriched upsert, idempotency, offline GRN success, unknown-supplier rejection, inactive-supplier guard, delete-event handling, and timestamp freshness.

15.2 The Pattern Selection Matrix

Scenario	Recommended Pattern	Reason
Domain Masters (Supplier)	NATS RPC	Consistency over autonomy.
Transactional Masters (Item)	CQRS Projection	High throughput and local survival.
External Engines (Ledger)	Specialized Protocol	External microservices are independent out-of-band handlers.

---

Part 16: External Microservice Integration

Goal: Bridge Framework M Macroservices to high-performance engines.

• 16.1 Book-Keeper Integration (TigerBeetle)
  • Use the ExternalLedgerProtocol to ship GL batches to the Litestar service.
• 16.2 Outbox Strategy
  • All external calls must be wrapped in an Outbox/Relay pattern to ensure transactional integrity in the Macroservice.