1 of 34

Research Challenges in Reinforcement Learning for Autonomous Network Operations�NTT’s Research and Development Activities

Yoichi Matsuo, NTT Network Service Systems Lab., NTT inc.

27th, October 2025

© NTT, Inc. 2025

2 of 34

Outline

  • Introduction
  • Challenges of applying RL for real network operations
  • Use-case 1. Safe multi-agent deep reinforcement learning for dynamic virtual network allocation
  • Use-case 2. Integrating state prediction into the Deep Reinforcement Learning for the Autoscaling of Core Network Functions
  • Conclusion

© NTT, Inc. 2025

1

3 of 34

NTT

  • NTT group is one of the largest telecommunications networks and ICT solutions companies in Japan.

​

​

NTT

NTT DOCOMO

NTT EAST

NTT WEST

A holding company including R&D department

R&D

A mobile carriers

on cellular services

Regional carriers responsible for local fixed-line and fiber-optic services

…

© NTT, Inc. 2025

2

4 of 34

R&D activities for network management

  • NTT R&D is aiming for zero-touch operations using many types of AI-based methods.
  • Deep reinforcement learning is one of promising techniques.

Control operations

Database

External

Information

API

API

API

API

Generative-AI

API

Analysis・Decision

Failure handling

Control network

Network

Data collection・

Visualization

Zero-touch operations

Network-AI Infrastructure

Network�Technology

Generative-AI

×

Network-AI�Technology

=

Chaos engineering

automation technology

Anomaly detection,

fault device identification

Network equipment

parameter optimization

Recommendation technology for

failure handling procedures

Resource management

Use-case using RL

© NTT, Inc. 2025

3

5 of 34

Objective and challenges of resource management

  • Objective
      • Our research aims to optimize resource management of future networks
      • The optimal management refers to the allocation of virtual networks and tasks that maximizes the objective function (e.g., resource utilization) while satisfying constraints (e.g., resource capacity)
  • Challenges
    • Practical constraints when applying resource management using Deep reinforcement learning (DRL) to real network operations.
    • Safety and combinatorial explosion of the action space (Use-case 1)
    • Control and actuation latency from decision to application in real networks (Use-case 2)

​

© NTT, Inc. 2025

4

6 of 34

Objective and challenges of resource management

  • This presentation introduces two specific use-cases about network resource management using DRL
      • Use-case 1. Safety control in dynamic virtual network allocation
      • Use-case 2. Control considering prediction in autoscaling of core network functions

© NTT, Inc. 2025

5

7 of 34

Use-case1�Safe multi-agent deep reinforcement learning for dynamic virtual network allocation

A. Suzuki and S. Harada, “Safe multi-agent deep reinforcement learning for dynamic virtual network allocation,” in Proc. IEEE GLOBECOM, 2020, pp. 1–7.

© NTT, Inc. 2025

8 of 34

Network resource management in NFV

  • Network functions virtualization (NFV) provides various network services by flexibly combining multiple virtual network functions (VNFs).
  • To provide carrier-grade services, network carrier must optimally embed virtual networks (VNs) into underlying physical servers and networks. -> Virtual network allocation problem

User B

User B

VNF

VNF

VM

User A

VNF

VNF

Internet

Physical network

Virtual networks

User A

Internet

①Server resource�allocation

②Network resource�allocation

Traffic: 100Mbps

Traffic: 50Mbps

VM: Virtual Machine

VNF: Virtual Network Functions

© NTT, Inc. 2025

7

9 of 34

Virtual network allocation problem

  • Virtual network (VN) = Virtual nodes + Virtual links
  • VN allocation = Server resource allocation + Network resource allocation
    • Objective: Minimize: Max. server utilization + Max. link utilization
    • Constraints: Server capacity, Link capacity

Residual: 5 10 15

Server resource allocation

VNF

VNF

3 CPU

Network resource allocation

Traffic:� 100 Mbps

Internet

Goal

Start

Internet

VNF

VNF: Virtual Network Functions

© NTT, Inc. 2025

8

10 of 34

Motivation | Dynamic virtual network allocation

  • Network traffic and computing demand dramatically have been changing.
  • Static virtual network allocation leads to inefficient resource utilization and congestion
  • Dynamic virtual network allocation for time-varying demand becomes important

Congestion

Demand

Time

Static�Allocation

Actual Demands

Inefficiency

Demand

Time

Dynamic�Allocation

Actual Demands

Re-allocation

every t

Δt

Static allocation

Dynamic allocation

© NTT, Inc. 2025

9

11 of 34

Difficulty | Trade-off relationship

  • Finding optimal virtual network allocation problem is NP-Hard
  • Dynamic virtual network allocation simultaneously requires efficiency and immediacy of virtual network allocation, even though both are in a trade-off relationship

Demand

Time

Allocated�resource

Actual�demands

(2) Inefficiency� by low-immediacy

  1. Increase computational time

for high-efficiency

© NTT, Inc. 2025

10

12 of 34

Key Idea | Reinforcement Learning (RL)

  • RL can quickly optimize the network resources by learning the relationship between demand patterns and optimal allocation in advance

Demand

Time

Allocated�resource

Actual traffic �demands

Training

Immediate decision

based on pre-training

© NTT, Inc. 2025

11

13 of 34

Challenge Safety

  • Method: Deep RL-based dynamic virtual network allocation method
  • However, there are two challenges in applying RL to dynamic virtual network allocation
  • Challenge 1: Safety
    • Violations of constraints in the carrier network can lead to large-scale congestion
    • RL agent is no guarantee to satisfy constraints
  • Previous studies
    • There have been several studies on dynamic virtual network allocation using RL (e.g., [1]-[3])
    • However, none of these methods have addressed safety considerations.

⇒ We proposes “safe” multi-agent deep reinforcement learning (Safe-MADRL)

[1] R. Mijumbi et al., “Design and evaluation of algorithms for mapping and scheduling of virtual network functions,” in Proc. IEEE NetSoft, 2015, pp. 1–9.

[2] R. Mijumbi et al., “A neuro-fuzzy approach to self-management of virtual network resources,” Expert systems with applications, vol. 42, no. 3, pp. 1376–1390, 2015.

[3] P. T. A. Quang et al., “A deep reinforcement learning approach for VNF forwarding graph embedding,” IEEE Trans. Netw. Service Manag., vol. 16, no. 4, pp. 1318–1331, 2019.

© NTT, Inc. 2025

12

14 of 34

Method Safe Multi-agent Deep RL

  •  

© NTT, Inc. 2025

13

15 of 34

Challenge Number of actions

  • Method: Deep RL-based dynamic virtual network allocation method
  • However, there are two challenges in applying RL to dynamic virtual network allocation
  • Challenge 2: Number of candidate actions
    • The numbers of VNs, VMs, and possible routes in the carrier network are enormous.
    • The performance of RL drastically decreases as the number of candidate actions increases
  • Previous studies
    • Several studies [1]-[3] have discussed and addressed this problem of number of actions
    • However, previous methods compress the action space using handcrafted features, which degrades the performance of virtual network allocation

[1] M. Dolati et al., “DeepViNE:Virtual network embedding with deep reinforcement learning,” in Proc. IEEE INFOCOM Workshops, 2019, pp. 879–885.

[2] Z. Yan et al., “Automatic virtual network embedding: A deep reinforcement learning approach with graph convolutional networks,” IEEE J. Sel. Areas Commun., 2020.

[3] A. Suzuki et al., “Safe multi-agent deep reinforcement learning for dynamic virtual network allocation,” in Proc. IEEE GLOBECOM, 2020, pp. 1-7.

© NTT, Inc. 2025

14

16 of 34

Challenge | Number of candidate actions

  • As the number of virtual networks (VNs) increases, the candidate action exponentially increases, and the performance of RL drastically decreases.

t=1

t=2

t=3

VN3

VN4

VN1

VN2

B

B

A

A

B

A

A

B

A

B

A

B

Server�A

Server�B

VN

User

Node

VM

t=1

t=2

t=3

VN10

VN1

VN2

B

A

A

A

A

B

B

A

B

…

Simple Network Example

(#Server=2)

…

2^4 actions

2^10 actions

When #VN=4

When #VN=10

VM: Virtual Machine

© NTT, Inc. 2025

15

17 of 34

Method | Safe Multi-agent Deep RL

  •  

t=1

t=2

t=3

VN3

VN4

VN1

VN2

B

B

A

A

B

A

A

B

A

B

A

B

2^4 actions

t=1

t=2

t=3

B

B

A

A

B

A

A

B

A

B

A

B

VN3

VN4

VN1

VN2

2^2 actions

2^2 actions

User

Server�A

Server�B

x4

User Node

Single-Agent

Multi-Agent

VM

VM: Virtual Machine

© NTT, Inc. 2025

16

18 of 34

Method | Modeling

  •  

© NTT, Inc. 2025

17

19 of 34

Method | Procedure

  •  

© NTT, Inc. 2025

18

20 of 34

Evaluation | Evaluation conditions

  •  

20

20

20

20

20

20

20

20

20

0.4G

0.4G

0.4G

0.4G

0.2G

0.8G

0.4G

0.4G

0.2G

0.8G

0.4G

0.8G

User

VM

Traffic:� 0-200 Mbps

VM size:� 3-5

VM: Virtual Machine

Steps

Traffic Demand [Mbps]

Steps

Steps

© NTT, Inc. 2025

19

21 of 34

Evaluation | Comparison methods

  •  

© NTT, Inc. 2025

20

22 of 34

Evaluation | Performance

  • Average performance: safe MADRL > MADRL >> Static
    • A series of actions (called an episode E) is repeatedly executed until learning is complete
    • At E=5000, constraint violation of safe MADRL already decreased to 1/100 that of Static

​

© NTT, Inc. 2025

21

23 of 34

 

  • Computation time: Static = safe MADRL = MADRL >> Exhaustive search
    • It includes the time to decide action, update the next state, and evaluate performance
    • Update time to the next state is dominant in three methods

​

​

​

​

​

​

© NTT, Inc. 2025

22

24 of 34

Objective and challenges of resource management

  • This presentation introduces two specific use-case about network resource management using deep reinforcement learning (DRL)
      • Use-case 1. Safety control in dynamic virtual network allocation
      • Use-case 2. Control considering prediction in autoscaling of core network functions

© NTT, Inc. 2025

23

25 of 34

Use-case 2�Integrating state prediction into the Deep Reinforcement Learning for the Autoscaling of Core Network Functions

Y. Matsuo, J. Singh, S. Verma and G. Fraysse, "Integrating state prediction into the Deep Reinforcement Learning for the Autoscaling of Core Network Functions," NOMS, 2023, pp. 1-5.

© NTT, Inc. 2025

26 of 34

Motivation

  • Increasing or decreasing capacity of core Network Functions (NFs) is important to adapt the NFs to the incoming traffic load by scaling in or out the NFs.
  • Excessive capacity wastes resources while limited capacity leads to degradation in Quality of Service (QoS).

P-GW

S-GW

MME

UE

eNodeB

MME

MME

S-GW

S-GW

P-GW

P-GW

UE

eNodeB

MME

S-GW

S-GW

P-GW

P-GW

MME

Increase the traffic load

UE makes traffic load

Mobility Management Entity (MME)

Serving Gateway (SGW)

Packet data network Gateway (PGW)

© NTT, Inc. 2025

25

27 of 34

Reinforcement Learning (RL)-based scaling

  • Many RL-based scaling methods have developed.

​

environment

 

Policy

 

P-GW

S-GW

MME

UE

eNodeB

MME

MME

S-GW

S-GW

P-GW

P-GW

UE

eNodeB

MME

S-GW

S-GW

P-GW

P-GW

MME

© NTT, Inc. 2025

26

28 of 34

Scaling process delay

  • Scaling process delay between the decision to create a new NF instance and the moment this new NF instance can handle the traffic.
    • Launching the NF instance and initializing process such as loading the necessary configurations and network registrations.

P-GW

S-GW

MME

UE

eNodeB

MME

S-GW

S-GW

P-GW

P-GW

MME

Decision to create a new FN instance

Initializing process

takes time

P-GW

S-GW

MME

UE

eNodeB

MME

S-GW

S-GW

P-GW

P-GW

MME

Initialization process is finished

© NTT, Inc. 2025

27

29 of 34

Problem of classical RL setting

  • The selected action by RL is less useful in the real system due to this delay.
    • In our platform, we need a little more than 120 seconds for scaling in or out NFs.
    • Although the RL decides to increase the instance, NFs crashed before the increased instance was Initialized.
  • Before the effect of selected action comes into play, the state for which the action was taken has changed already.

# of UEs connected to NFs

(2) NFs crash due to heavy load

(1) Action

Increase the NFs

© NTT, Inc. 2025

28

30 of 34

Proposed method

  •  

© NTT, Inc. 2025

29

31 of 34

D3QN-based scaling method

  •  

© NTT, Inc. 2025

30

32 of 34

Experimental setting

NFV platform

  • We prepared MME, SGW, and PGW as NFs using the Magma, Open Source software on Flexible Engine, Cloud Infrastructure as a Service

Traffic load

  • We generated the traffic load based on the sine wave, which represents the timestamps of user device attach requests to MME.
    • A crash might occur on the MME if an appropriate number of instances aren’t running

Comparing method

  • We prepared Basic D3QN which does not use the predicted next state.

© NTT, Inc. 2025

31

33 of 34

Experimental results

  • State-predicted D3QN method improves the average reward by approximately 30%.
    • The state-predicted D3QN method handled more UEs than basic method

© NTT, Inc. 2025

32

34 of 34

Conclusion

  • NTT R&D is aiming for Zero-touch operation using AI methodologies.
  • DRL is one of promising techniques for our goal.
  • Safety, action space, and delay are challenging for applying DRL to real network operations.
  • Our team is developing several methods to address these issues.

​

​

© NTT, Inc. 2025

33