Research Challenges in Reinforcement Learning for Autonomous Network Operations�NTT’s Research and Development Activities
Yoichi Matsuo, NTT Network Service Systems Lab., NTT inc.
27th, October 2025
© NTT, Inc. 2025
Outline
© NTT, Inc. 2025
1
NTT
NTT
NTT DOCOMO
NTT EAST
NTT WEST
A holding company including R&D department
R&D
A mobile carriers
on cellular services
Regional carriers responsible for local fixed-line and fiber-optic services
…
© NTT, Inc. 2025
2
R&D activities for network management
Control operations
Database
External
Information
API
API
API
API
Generative-AI
API
Analysis・Decision
Failure handling
Control network
Network
Data collection・
Visualization
Zero-touch operations
Network-AI Infrastructure
Network�Technology
Generative-AI
×
Network-AI�Technology
=
Chaos engineering
automation technology
Anomaly detection,
fault device identification
Network equipment
parameter optimization
Recommendation technology for
failure handling procedures
Resource management
Use-case using RL
© NTT, Inc. 2025
3
Objective and challenges of resource management
© NTT, Inc. 2025
4
Objective and challenges of resource management
© NTT, Inc. 2025
5
Use-case1�Safe multi-agent deep reinforcement learning for dynamic virtual network allocation
A. Suzuki and S. Harada, “Safe multi-agent deep reinforcement learning for dynamic virtual network allocation,” in Proc. IEEE GLOBECOM, 2020, pp. 1–7.
© NTT, Inc. 2025
Network resource management in NFV
User B
User B
VNF
VNF
VM
User A
VNF
VNF
Internet
Physical network
Virtual networks
User A
Internet
①Server resource�allocation
②Network resource�allocation
Traffic: 100Mbps
Traffic: 50Mbps
VM: Virtual Machine
VNF: Virtual Network Functions
© NTT, Inc. 2025
7
Virtual network allocation problem
Residual: 5 10 15
Server resource allocation
VNF
VNF
3 CPU
Network resource allocation
Traffic:� 100 Mbps
Internet
Goal
Start
Internet
VNF
VNF: Virtual Network Functions
© NTT, Inc. 2025
8
Motivation | Dynamic virtual network allocation
Congestion
Demand
Time
Static�Allocation
Actual Demands
Inefficiency
Demand
Time
Dynamic�Allocation
Actual Demands
Re-allocation
every t
Δt
Static allocation
Dynamic allocation
© NTT, Inc. 2025
9
Difficulty | Trade-off relationship
Demand
Time
Allocated�resource
Actual�demands
(2) Inefficiency� by low-immediacy
for high-efficiency
© NTT, Inc. 2025
10
Key Idea | Reinforcement Learning (RL)
Demand
Time
Allocated�resource
Actual traffic �demands
Training
Immediate decision
based on pre-training
© NTT, Inc. 2025
11
Challenge Safety
⇒ We proposes “safe” multi-agent deep reinforcement learning (Safe-MADRL)
[1] R. Mijumbi et al., “Design and evaluation of algorithms for mapping and scheduling of virtual network functions,” in Proc. IEEE NetSoft, 2015, pp. 1–9.
[2] R. Mijumbi et al., “A neuro-fuzzy approach to self-management of virtual network resources,” Expert systems with applications, vol. 42, no. 3, pp. 1376–1390, 2015.
[3] P. T. A. Quang et al., “A deep reinforcement learning approach for VNF forwarding graph embedding,” IEEE Trans. Netw. Service Manag., vol. 16, no. 4, pp. 1318–1331, 2019.
© NTT, Inc. 2025
12
Method Safe Multi-agent Deep RL
© NTT, Inc. 2025
13
Challenge Number of actions
[1] M. Dolati et al., “DeepViNE:Virtual network embedding with deep reinforcement learning,” in Proc. IEEE INFOCOM Workshops, 2019, pp. 879–885.
[2] Z. Yan et al., “Automatic virtual network embedding: A deep reinforcement learning approach with graph convolutional networks,” IEEE J. Sel. Areas Commun., 2020.
[3] A. Suzuki et al., “Safe multi-agent deep reinforcement learning for dynamic virtual network allocation,” in Proc. IEEE GLOBECOM, 2020, pp. 1-7.
© NTT, Inc. 2025
14
Challenge | Number of candidate actions
t=1
t=2
t=3
VN3
VN4
VN1
VN2
B
B
A
A
B
A
A
B
A
B
A
B
Server�A
Server�B
VN
User
Node
VM
t=1
t=2
t=3
VN10
VN1
VN2
B
A
A
A
A
B
B
A
B
…
Simple Network Example
(#Server=2)
…
2^4 actions
2^10 actions
When #VN=4
When #VN=10
VM: Virtual Machine
© NTT, Inc. 2025
15
Method | Safe Multi-agent Deep RL
t=1
t=2
t=3
VN3
VN4
VN1
VN2
B
B
A
A
B
A
A
B
A
B
A
B
2^4 actions
t=1
t=2
t=3
B
B
A
A
B
A
A
B
A
B
A
B
VN3
VN4
VN1
VN2
2^2 actions
2^2 actions
User
Server�A
Server�B
x4
User Node
Single-Agent
Multi-Agent
VM
VM: Virtual Machine
© NTT, Inc. 2025
16
Method | Modeling
© NTT, Inc. 2025
17
Method | Procedure
© NTT, Inc. 2025
18
Evaluation | Evaluation conditions
20
20
20
20
20
20
20
20
20
0.4G
0.4G
0.4G
0.4G
0.2G
0.8G
0.4G
0.4G
0.2G
0.8G
0.4G
0.8G
User
VM
Traffic:� 0-200 Mbps
VM size:� 3-5
VM: Virtual Machine
Steps
Traffic Demand [Mbps]
Steps
Steps
© NTT, Inc. 2025
19
Evaluation | Comparison methods
© NTT, Inc. 2025
20
Evaluation | Performance
© NTT, Inc. 2025
21
© NTT, Inc. 2025
22
Objective and challenges of resource management
© NTT, Inc. 2025
23
Use-case 2�Integrating state prediction into the Deep Reinforcement Learning for the Autoscaling of Core Network Functions
Y. Matsuo, J. Singh, S. Verma and G. Fraysse, "Integrating state prediction into the Deep Reinforcement Learning for the Autoscaling of Core Network Functions," NOMS, 2023, pp. 1-5.
© NTT, Inc. 2025
Motivation
P-GW
S-GW
MME
UE
eNodeB
MME
MME
S-GW
S-GW
P-GW
P-GW
UE
eNodeB
MME
S-GW
S-GW
P-GW
P-GW
MME
Increase the traffic load
UE makes traffic load
Mobility Management Entity (MME)
Serving Gateway (SGW)
Packet data network Gateway (PGW)
© NTT, Inc. 2025
25
Reinforcement Learning (RL)-based scaling
environment
Policy
P-GW
S-GW
MME
UE
eNodeB
MME
MME
S-GW
S-GW
P-GW
P-GW
UE
eNodeB
MME
S-GW
S-GW
P-GW
P-GW
MME
© NTT, Inc. 2025
26
Scaling process delay
P-GW
S-GW
MME
UE
eNodeB
MME
S-GW
S-GW
P-GW
P-GW
MME
Decision to create a new FN instance
Initializing process
takes time
P-GW
S-GW
MME
UE
eNodeB
MME
S-GW
S-GW
P-GW
P-GW
MME
Initialization process is finished
© NTT, Inc. 2025
27
Problem of classical RL setting
# of UEs connected to NFs
(2) NFs crash due to heavy load
(1) Action
Increase the NFs
© NTT, Inc. 2025
28
Proposed method
© NTT, Inc. 2025
29
D3QN-based scaling method
© NTT, Inc. 2025
30
Experimental setting
NFV platform
Traffic load
Comparing method
© NTT, Inc. 2025
31
Experimental results
© NTT, Inc. 2025
32
Conclusion
© NTT, Inc. 2025
33