Publication date

Training compute (FLOPs)

Method 1 Training cost (2020 USD)

Method 2 Training cost (2020 USD)

Organization(s)

Organization Categorization

Training core-hours

Inclusion criteria

Training dataset

Training dataset size (datapoints)

Inference compute (FLOPs)

Training time (hours)

Equivalent training time (hours)

Inference time (ms)

Training dataset size (GB)

Dense or sparse model

Training objective

Compute Sponsor Categorization

R Raina, A Madhavan, AY Ng

Large-scale Deep Unsupervised Learning using Graphics Processors

Character recognition

IDSIA ; University of Lugano & SUPSI

Dan Claudiu Ciresan, Ueli Meier, Luca Maria Gambardella, Juergen Schmidhuber

Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition

University of Montreal

X Glorot, Y Bengio

Understanding the difficulty of training deep feedforward neural networks

RNN 500/10 + RT09 LM (NIST RT05)

Brno University of Technology, Johns Hopkins University

T. Mikolov, M. Karafiat, L. Burget, J. Cernock ´ y, and S. Khudanpur

Recurrent neural network based language model.

KN5 LM + RNN 400/10 (WSJ)

Brno University of Technology, Johns Hopkins University

T. Mikolov, M. Karafiat, L. Burget, J. Cernock ´ y, and S. Khudanpur

Recurrent neural network based language model.

Character recognition

D Ciregan, U Meier, J Schmidhuber

Multi-column Deep Neural Networks for Image Classification

Character recognition

University of Toronto

GE Hinton, N Srivastava, A Krizhevsky

Improving neural networks by preventing co-adaptation of feature detectors

Image classification

University of Toronto

Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton

ImageNet Classification with Deep Convolutional Neural Networks

V Mnih, K Kavukcuoglu, D Silver, A Graves

Playing Atari with Deep Reinforcement Learning

Dan C. Cireşan, Alessandro Giusti, Luca M. Gambardella, Jürgen Schmidhuber

Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks

ICPR 2012 mitosis detection competition winner

Semantic embedding

T Mikolov, I Sutskever, K Chen, GS Corrado

Distributed Representations of Words and Phrases and their Compositionality

Predict nearby words

Recurrent Neural Network

MD Zeiler, R Fergus

Visualizing and Understanding Convolutional Networks

Predict nearby words

Entity embedding

Industry - Academia Collaboration

Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko

Translating Embeddings for Modeling Multi- relational Data

Univeristy of Amsterdam

DP Kingma, M Welling

Auto-Encoding Variational Bayes

Image Classification with the Fisher Vector: Theory and Practice

Image Classifcation

Universidad Nacional de Cordoba, Xerox Research Centre Europe, Inteligent Systems Lab Amsterdam, University of Amsterdam, LEAR Team, INRIA Grenoble

Industry - Academia Collaboration

orge Sanchez, Florent Perronnin, Thomas Mensink, Jakob Verbeek

Image Classification with the Fisher Vector: Theory and Practice

Image generation

Universite de Montréal

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio

Generative Adversarial Networks

Image classification

Microsoft, Xi’an Jiaotong University, University of Science and Technology of China

Industry - Academia Collaboration

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

Universite de Montréal, Jacobs University Bremen

D Bahdanau, K Cho, Y Bengio

Neural Machine Translation by Jointly Learning to Align and Translate

WMT'14 + selection

University of Oxford

Karen Simonyan; Andrew Zisserman

Very Deep Convolutional Networks for Large-Scale Image Recognition

I Sutskever, O Vinyals, QV Le

Sequence to Sequence Learning with Neural Networks

Image classification

University of Amsterdam, OpenAI, University of Toronto

Industry - Academia Collaboration

Adam: A Method for Stochastic Optimization

Image classification

Microsoft research

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

GoogLeNet / InceptionV1

Image classification

Google, University of Michigan, University of North Carolina

Industry - Academia Collaboration (Industry leaning)

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich

Going deeper with convolutions

Google DeepMind

D Silver, A Huang, CJ Maddison, A Guez, L Sifre

Mastering the game of Go with deep neural networks and tree search

SOTA improvement

Speech recognition

Baidu Research- Silicon Valley AI Lab

D Amodei, S Ananthanarayanan

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Image classification

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Deep Residual Learning for Image Recognition

D Silver, A Huang, CJ Maddison, A Guez, L Sifre

Mastering the game of Go with deep neural networks and tree search

Microsoft research, Tsinghua university

Industry - Academia Collaboration (Industry leaning)

Jifeng Dai, Y. Li, Kaiming He, and Jian Sun

R-fcn: Object detection via region-based fully convolutional networks.

PASCAL VOC (2007 and 2012 vesrions) + MS COCO

Part-of-sentence tagging model

University of Toronto

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton

Layer Normalization.

Named Entity Recognition model

Named Entity Recognition model

University of Toronto

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton

Layer Normalization.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, Jeffrey Dean

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Image classification

Xception: Deep Learning with Depthwise Separable Convolutions

Barret Zoph, Quoc V. Le

Neural Architecture Search with Reinforcement Learning

Carnegie Mellon University

N Brown, T Sandholm, S Machine

Libratus: The Superhuman AI for No-Limit Poker

SOTA improvement

D Silver, J Schrittwieser, K Simonyan, I Antonoglou

Mastering the game of Go without human knowledge

University of Alberta, Charles University, Czech Technical University

Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, Michael Bowling

DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker

Language modelling / Machine translation

Google Brain, Jagiellonian University, Cracow

Industry - Academia Collaboration (Industry leaning)

N Shazeer, A Mirhoseini, K Maziarz, A Davis

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Long Short-Term Memory Mixture-Of-Experts

Google Brain ; Google Research

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

Attention Is All You Need

Google Research, CMU

Industry - Academia Collaboration

ChenSun,AbhinavShrivastava,SaurabhSingh,andAbhinavGupta

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era.

A Radford, K Narasimhan, T Salimans, I Sutskever

D Silver, J Schrittwieser, K Simonyan, I Antonoglou

Mastering the game of Go without human knowledge

Johns Hopkins University, Stanford, Google AI

Industry - Academia Collaboration (Industry leaning)

C Liu, B Zoph, M Neumann, J Shlens

Progressive Neural Architecture Search

D Silver, T Hubert, J Schrittwieser, I Antonoglou

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, Koray Kavukcuoglu

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Image classification

Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le

Regularized Evolution for Image Classifier Architecture Search

University of Washington

Joseph Redmon, Ali Farhadi

YOLOv3: An Incremental Improvement

A Radford, K Narasimhan, T Salimans, I Sutskever

Improving Language Understanding by Generative Pre-Training

Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, Thore Graepel

Human-level performance in first-person multiplayer games with population-based deep reinforcement learning

Image generation

Heriot-Watt University, DeepMind

Industry - Academia Collaboration

A Brock, J Donahue, K Simonyan

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Next sentence prediction

J Devlin, MW Chang, K Lee, K Toutanova

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Decoupled weight decay regularization

Image classification

University of Freiburg

Ilya Loshchilov and Frank Hutter

Decoupled weight decay regularization.

DeepMind, University of Oxford, Google Brain, Carnegie Mellon University,

Industry - Academia Collaboration (Industry leaning)

The Hanabi Challenge: A New Frontier for AI Research

A Radford, J Wu, R Child, D Luan, D Amodei

Language Models are Unsupervised Multitask Learners

Han Cai, Ligeng Zhu, and Song Han

ProxylessNAS: Direct neural architecture search on target task and hardware

Tel Aviv University, MIT

Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson.

Cross-lingual alignment of contextual word embeddings, with applications to zero- shot dependency parsing.

Performing image classification and object detection on mobile devices

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, Quoc V. Le

MnasNet: Platform-Aware Neural Architecture Search for Mobile

Performing image classification and object detection on mobile devices

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, Quoc V. Le

MnasNet: Platform-Aware Neural Architecture Search for Mobile

Recommendation

M Naumov, D Mudigere, HJM Shi, J Huang

Deep Learning Recommendation Model for Personalization and Recommendation Systems

M Jaderberg, WM Czarnecki, I Dunning, L Marris

Human-level performance in 3D multiplayer games with population-based reinforcement learning

Object recognition

Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfre- und, Josh Tenenbaum, and Boris Katz

Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

B Baker, I Kanitscheider, T Markov, Y Wu

Emergent Tool Use From Multi-Agent Autocurricula

Megatron-LM (Original, 8.3B)

M Shoeybi, M Patwary, R Puri, P LeGresley

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

M Shoeybi, M Patwary, R Puri, P LeGresley

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Neural architecture search for computer vision

Brown and Facebook AI Research

Industry - Academia Collaboration (Academia leaning)

Linnan Wang, Yiyang Zhao, Yuu Jinnai, Yuandong Tian, Rodrigo Fonseca1

AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search

Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, Lei Zhang

Solving Rubik’s Cube with a Robot Hand

Text autocompletion

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colossal Clean Crawled Corpus (C4)

Transformer (encoder-decoder performed best)

Text autocompletion

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colossal Clean Crawled Corpus (C4)

Transformer (encoder-decoder performed best)

Oriol Vinyals,Igor Babuschkin,Wojciech M. Czarnecki,Michaël Mathieu,Andrew Dudzik,Junyoung Chung,David H. Choi,Richard Powell,Timo Ewalds,Petko Georgiev,Junhyuk Oh,Dan Horgan,Manuel Kroiss,Ivo Danihelka,Aja Huang,Laurent Sifre,Trevor Cai,John P. Agapiou,Max Jaderberg,Alexander S. Vezhnevets,Rémi Leblond,Tobias Pohlen,Valentin Dalibard,David Budden,Yury Sulsky,James Molloy,Tom L. Paine,Caglar Gulcehre,Ziyu Wang,Tobias Pfaff,Yuhuai Wu,Roman Ring,Dani Yogatama,Dario Wünsch,Katrina McKinney,Oliver Smith,Tom Schaul,Timothy Lillicrap,Koray Kavukcuoglu,Demis Hassabis,Chris Apps,David Silver

Grandmaster level in StarCraft II using multi-agent reinforcement learning

J Schrittwieser, I Antonoglou, T Hubert, K Simonyan

Mastering Atari Go Chess and Shogi by Planning with a Learned Model

SOTA improvement

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung,Przemysław “Psyho" Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, Susan Zhang

Dota 2 with Large Scale Deep Reinforcement Learning

SOTA improvement

J Raiman, S Zhang, F Wolski

Dota 2 with Large Scale Deep Reinforcement Learning

SOTA improvement

Recommendation

D Mudigere, Y Hao, J Huang, A Tulloch

High- performance, Distributed Training of Large scale Deep Learning Recommendation Models

Protein folding prediction

Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander W. R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Koray Kavukcuoglu & Demis Hassabis

Improved protein structure prediction using potentials from deep learning

Text autocompletion

Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

Towards a Human-like Open-Domain Chatbot

Evolved Transformer seq2seq model

Google research, Toyota Technological Institute at Chicago

Industry - Academia Collaboration

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.

Text autocompletion

Turing-NLG: A 17-billion-parameter language model by Microsoft

Next token prediction

Protein generation

Salesforce research, Stanford

Industry - Academia Collaboration

A Madani, B McCann, N Naik, NS Keskar

ProGen: Language Modeling for Protein Generation

Text autocompletion

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei

Language models are Few- Shot Learners

CommonCrawl; WebText2; Books1; Books2; Wikipedia

MIT-IBM Watson AI Lab

Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han

Once for all: Train one network and specialize it for efficient deployment.

Image completion

Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, Ilya Sutskever

Generative Pretraining from Pixels

Image completion

Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, Ilya Sutskever

Generative Pretraining from Pixels

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Image representation

Google Research, Brain Team

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Speech completion

Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

SOTA Improvement

Relation Extraction

Tsinghua University, Princeton, Mila- Quebec AI, University de Montreal, HEC, CIFAR

Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang.

KEPLER: A Unified Model for Knowledge Embedding and Pre- trained Language Representation.

Wikipedia+BookCorpus

Tsinghua University, BAAI

Industry - Academia Collaboration

Z Zhang, X Han, H Zhou, P Ke, Y Gu, D Ye, Y Qin, Y Su

CPM: A Large-scale Generative Chinese Pre-trained Language Model

Left-To-Right Transformer Decoder

American University of Beirut

W Antoun, F Baly, H Hajj

AraGPT2: Pre-Trained Transformer for Arabic Language Generation

Recommendation

D Mudigere, Y Hao, J Huang, A Tulloch

Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models

Text autocompletion

William Fedus, Barret Zoph, Noam Shazeer

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Switch Transformer

Zero-shot image classification

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

Learning Transferable Visual Models From Natural Language Supervision

Custom image-text pairs from the internet

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever

Zero-Shot Text-to-Image Generation

Image Classification

Google AI, Brain team

Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, and Quoc V. Le

Meta pseudo labels

SOTA Improvement

Research collective

Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi LiaoZhiwei WangXin JiangZhenzhang YangKaisheng WangXiaoda ZhangChen LiZiyan GongYifan YaoXinjing HuangJun WangJianfeng YuQi GuoYue YuYan ZhangJin WangHengtao TaoDasen YanZexuan YiFang PengFangqing JiangHan ZhangLingfeng DengYehong ZhangZhe LinChao ZhangShaojie ZhangMingyue GuoShanzhi GuGaojun FanYaowei WangXuefeng JinQun LiuYonghong Tian

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

unidirectional transformer decoder

Research collective

Aran Komatsuzaki

Technical University of Munich, Med AI Technology, Google AI, NVIDIA, Oak Ridge National Laboratory

Industry - Academia Collaboration

A Elnaggar, M Heinzinger, C Dallago, G Rihawi

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Y Sun, S Wang, S Feng, S Ding, C Pang

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Transformer-XL: Transformer with auxilary recurrence memory module