Multi-Agent�Reinforcement Learning
Reading the inventory environment code through a MARL lens
COMPACT CODE WALKTHROUGH
What it does
Defines one SKU's state, one order action, one reward and a transition.
Why it matters
It is a reusable foundation for adding interacting supply-chain agents.
Tutorial focus: distinguish the MARL-ready wrapper from the multi-agent coordination that still needs to be implemented.
1
1. Where the multi-agent aspect is—and isn't
Read the class name in context: the implemented interface is still single-agent.
2
Present in the code
• One SKU series is validated�• One action enters `step(action)`�• One state vector of 11 features�• One scalar reward is returned
Not yet present
• No collection of agents�• No per-agent observations/actions�• No joint action or agent turn-taking�• No agent-specific/shared reward design
So: `MARLEnvironment` is the wrapper's name, but the shown implementation does not yet establish MARL.
This is not a flaw if the purpose is to build a baseline first. It becomes a MARL environment when multiple decision-makers interact through shared supply-chain dynamics.
2. What one “agent” sees and controls
The current observation/action/reward loop
3
Observation: 11 features
• Inventory on hand�• Pipeline quantity�• Inventory position�• Adaptive reorder point�• Forecast lead-time demand�• Uncertainty (`sigma`)�• Beta / smoothed beta / regime�• Demand lags: 1 and 7
Action: 26 choices
`ACTIONS = 0, 2, 4, …, 50`��The policy selects an action index; `resolve_action()` maps it to an order quantity.
Reward: one scalar
Negative penalties for:�• Ending inventory�• Backlog�• Placing an order�• Service-target violation��Optionally add normalized economic cost via `alpha`.
This is a standard single-agent RL contract: observation → action → transition → reward + next observation.
3. What `step()` does
The wrapper delegates physical inventory dynamics to Stage 9.1.
4
1
Resolve action
Index → order quantity
→
2
Call Stage 9.1
`base_env.step(order_quantity=…)`
→
3
Read transition
Received units, demand, served demand, inventory, backlog
→
4
Compute reward
Service violation + base penalty + optional cost penalty
→
5
Return result
`next_state, reward, done, info`
For MARL, this transition would need to account for all agents' actions together—not just one `order_quantity`.
4. Mapping the wrapper to a four-echelon MARL system
A proposed extension—not behavior already implemented in the supplied code.
5
Retailer
Customer demand�Retail stock / backlog
↔
Wholesaler
Retailer orders�Wholesale stock / pipeline
↔
Distributor
Wholesale orders�Distribution inventory
↔
Manufacturer
Production / capacity�Finished goods
Each agent needs its own decision interface:
Local observation
What this echelon can know
Local action
Order / produce / allocate
Reward
Local cost or shared system cost
5. Minimum code changes to make it genuinely MARL
Preserve Stage 9.1 transition mechanics where possible; extend the interface.
6
A. Joint action
Replace a single `action` with a mapping such as `{retailer: aR, wholesaler: aW, ...}`.
B. Per-agent observations
Return an observation dictionary keyed by agent; choose local or shared information deliberately.
C. Reward design
Return per-agent rewards or one shared reward; state the cooperation/competition objective.
D. Training loop
Train multiple policies (e.g., independent DQN) or use CTDE for centralized training and decentralized execution.
Also define agent lifecycle/termination, pipeline and order-flow rules between echelons, and metrics such as total cost, service level and bullwhip effect.
Takeaway
Use this code to teach the boundary between an RL environment and a MARL environment.
7
The supplied code defines a strong starting interface for inventory-control RL.
Already implemented
One controller · 11-feature state · discrete orders · reward · transition logging
Still needed for MARL
Multiple decision-makers · coupled actions · inter-agent flows · reward/training design
Suggested teaching sequence: explain the current wrapper → add a second agent → implement joint transitions → compare independent DQN with CTDE.