1 of 20

​

​

​

1

2 of 20

Karkala Shashank Hegde

�

2

PhD Student at Robotic Embedded Systems Lab

Annenberg Fellow

USC Viterbi

�

3 of 20

About me

Education:

      • Bachelor of Technology - National Institute of Technology Karnataka, 2017
      • Master of Science - University of Southern California, 2021
      • (Current) PhD - University of Southern California

​

Experience:

      • Fidelity Investments: Software Engineer, 2017-2019
      • Stealth Startup: Data Scientist, 2021

3

4 of 20

Why AI?

​

“One day, computers will do everything smarter than us, like playing chess or driving a car. They will even be able to write books and music and create art“ - OpenAI’s GPT3

​

​

4

5 of 20

How do humans learn to do tasks?

�

5

�PRACTICE!

6 of 20

What is (Deep) Reinforcement Learning?

�

  • AI agent gets to interact with its surroundings.
  • The agent’s action gets a feedback
  • The agent’s actions are optimized for better feedback

​

6

7 of 20

What tasks can AI solve with DRL

​

  1. Potentially any Decision-making process
  2. What do we need? Either:
    • Abundant data of task being completed

Or

    • Access to a hyper realistic simulation (or reality itself) of the task

​

7

8 of 20

Overview of Use cases

​

  1. Self-Driving Vehicles
  2. Multi Agent Systems
  3. Robotic Learning
  4. Video Games
  5. Portfolio Management

8

9 of 20

Use Case 1: Self driving vehicles, 2020�

  1. We created an AI that learnt to drive in simulation (TORCS) on its own!
  2. The AI observed images as input and predicted actions as steering angles

​

9

10 of 20

Use Case 2: Multi Agent systems, 2020�

  1. AI agents need to collaborate and work together to solve a common goal
  2. Predator prey – The red dots need to learnt  to work together to capture the green dot
  3. Report Link: Competetive and Co-operative Multi Agent Reinforcement Learning

10

11 of 20

Use Case 3: Robotic Learning, 2022�

  1. The Quadruped learns to hurdle over the obstacle
  2. The actions here are joint angles

​

*current work

11

12 of 20

Use Case 4: Video Games, 2021�

  1. Use Voice commands to control an AI game bot
  2. The AI bot could also remember AI commands over long periods of time
  3. Project Link: Agents that listen

12

13 of 20

Use Case 5: Finance, Portfolio Management�

  1. Risk aware portfolio construction using deep deterministic policy, 2017

​

  • Use AI to optimize a portfolio over time

​

​

13

14 of 20

  • Risk aware portfolio construction using deep deterministic policy, 2017

​

  • Challenges:
    1. RL learns from failure; we cannot let it loose on the real world
    2. We need tons of data to model the environment
    3. We need a simulation that mimics the real world

​

​

14

15 of 20

  • Risk aware portfolio construction using deep deterministic policy, 2017

​

  • Environment Modelling:
  • We picked 20 securities from 31 Dec 2001 to 31 Dec 2016.
  • Goal: create an optimal portfolio balancing strategy
  • Created a simulation environment, to analyze portfolios
  • The AI is given an option every week to rebalance its portfolio
  • The AI tries to optimize the Sortino ratio for the past week

​

​

​

15

16 of 20

  • Risk aware portfolio construction using deep deterministic policy, 2017

​

  • AI modeling:
    • The agent is a Deep Neural network that analyses the market (20 stocks) for the last 14 days
    • The action is a vector of length 20, each entry in the vector represents the amount each stock's weight in the portfolio
    • To account for longer movements in data (longer than 14 days), we use Long Short Term Memory (LSTM) inside the DNN

​

​

16

17 of 20

  • Risk aware portfolio construction using deep deterministic policy, 2017

​

  • Algorithm:
    1. We use Deep Deterministic Policy Gradients, since state-actions are continuous
    2. We have 2 DNN:
      1. Actor: Takes as input the market's current state and gives the agent's action (portfolio strategy)
      2. Critic: Takes as input the market's current state, and the agent's portfolio strategy, and gives a "goodness" score
    3. In regular DL, we have a loss function which we try to reduce
    4. Here, The Actor DNN tries to increase the predicted "goodness" score from the critic

​

​

​

17

18 of 20

  • Risk aware portfolio construction using deep deterministic policy, 2017

​

Results:

  1. With this set up, we back tested on simulated unseen data.
  2. Our AI got about 260% ROI in simulation

​

​

18

19 of 20

  • Risk aware portfolio construction using deep deterministic policy, 2017

​

Limitations and scope for improvement:

  1. This was done ignoring slippage
  2. These were market orders
  3. Does not account for black swan events
  4. Does not account for unseen stocks

POC for future work:

  1. This method should scale up with more data
  2. Modern algorithms should give better performance w.r.t generalization,  data efficiency and speed

​

19

20 of 20

​

​

Questions?

20