1 of 11

Using LLMs to build your own Copilots

Vinayak Hegde

CTO-in-Residence

Microsoft for Startups

2 of 11

Agenda

How LLMs work (short intro)
The challenges of working with LLMs (Hallucinations)
How to reduce Hallucination using Retrieval Augmented Generation (RAG)
How can you can finetune LLMs to give give more coherent answers
What is the Copilot pattern for building creative/enterprise apps?
How does the Copilot pattern work from a UX point of view to augment human in the loop.
What are the different elements of the Copilot stack and how do they interplay with each other?
Some learnings from building copilots so far? (eg related to Responsible AI)

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

3 of 11

Artificial Intelligence

Machine Learning

Deep Learning

1956

Artificial Intelligence

the field of computer science that seeks to create intelligent machines that can replicate or exceed human intelligence

1997

Machine Learning

subset of AI that enables machines to learn from existing data and improve upon that data to make decisions or predictions

2017

Deep Learning

a machine learning technique in which layers of neural networks are used to process data and make decisions

2021

Generative AI

Create new written, visual, and auditory content given prompts or existing data.

Generative AI

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

Narrative/talk track:

What is a brief history of artificial intelligence?

Artificial intelligence (AI) has a long and storied history that spans many decades. Some of the earliest work in AI was done in the 1950s and 1960s, when researchers first began to explore the possibility of creating machines that could "think" like humans.

During the 1950s and 1960s, AI research focused primarily on "symbolic" AI, which involved creating symbolic representations of knowledge and using them to solve problems. Researchers in this field developed early versions of what are now known as expert systems, which are computer programs that mimic the decision-making abilities of human experts in a specific domain.

In the 1970s and 1980s, AI research began to shift away from symbolic AI and towards "sub-symbolic" AI, which involves using statistical methods and mathematical models to solve problems. This shift was driven in part by the realization that symbolic AI was too complex and difficult to scale, and in part by the development of new computer hardware that made it possible to process large amounts of data quickly.

In the 1990s and 2000s, AI research continued to evolve, and new subfields began to emerge. One of these was "machine learning," which focuses on creating algorithms that allow computers to learn from data without being explicitly programmed. Another was "neural networks," which are mathematical models inspired by the structure and function of the human brain.

Recently, Generative Models like GPT-3 and DALL-E, Variational Autoencoders, Generative Adversarial Networks and many more have risen and continue to advance the field of AI. These models are able to generate new examples that are similar to examples from a given dataset. It is being used in diverse fields such as natural language processing, computer vision, speech recognition and many more.

Overall, AI has come a long way since its inception in the 1950s, and it continues to evolve and improve at a rapid pace today.

What is Generative AI and where does it fit into the scope of AI?

Generative AI is a subfield of Artificial Intelligence that focuses on creating systems that can generate new examples, such as images, text, or speech, that are similar to examples from a given dataset. The goal of generative AI is to develop algorithms that can learn the underlying probability distribution of a given dataset and use this knowledge to generate new examples that are similar to the examples in the dataset.

Generative AI models can be classified into two broad categories: generative models and discriminative models. Generative models such as GPT-3, DALL-E, Variational Autoencoders etc learns the underlying probability distribution of the dataset and can generate new examples from the learned distribution. Discriminative models, such as deep neural networks, learn to differentiate between different classes or categories, and are generally used for tasks such as image or speech recognition.

Generative AI sits on top of other areas of AI like unsupervised learning, supervised learning and reinforcement learning. It relies on the capability of those models to extract useful features from the data and use that for learning the underlying probability distribution of the data which is used for generthe potential to be used in a wide range of applications, such as computer vision, natural language processing, speech recognition, and many more.

ating new examples.

Overall, generative AI is a rapidly growing field within AI, and it has

4 of 11

Traditional model development�High cost and slow deployment—each service is trained disjointly

FLOWER

PLAYING SOCCER

EAGLE

DEPLOYMENTS

Tagging �Services

Spatial Analysis �Services

Accessibility �Services

Spatial �Presenter

Azure Search, �Video Indexer

TASKS

Classification

Object Detection

Object Tracking

Action Recognition

Entities

Topics

Sentiments

INDIVIDUAL MODEL �(DISJOINTLY)

Classification Model

Detection �Model

Tracking �Model

Action �Model

Entity Recognition

Topic Classification

Sentiment Analysis

TRAINING DATA

(w/ ANNOTATION)

Tagging data

Detection data

Tracking data

Action �data

Entity �data

Topic �data

Sentiment data

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

5 of 11

Foundation Models

Data

Text

Images

Speech

Structured Data

3d Signals

Tasks

Question and Answering

Sentiment Analysis

Information Extraction

Image Captioning

Object Recognition

Training

Adaptation

Foundation Model

Instruction Follow

Transformer Model

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

The future of AI is models that are flexible, reusable and that can be applied to just about any domain or industry task.

Foundation models are at the heart of many of the latest breakthroughs. These models have been influential in the development of artificial intelligence systems, and have played a major role in changing how AI systems are constructed since their introduction in 2018.

A foundation model is a machine learning model that is trained on a large amount of unlabeled data, allowing it to be adapted to a wide variety of tasks. These models capture general patterns and structure of data, and aim to replace task-specific models by unifying different tasks and modalities into one, eliminating the need to train individual models and integrate several models together. An example is a large language foundation model that can perform various NLP tasks and can be used out-of-the-box due to its extensive training.

Foundation models use self-supervised learning and transfer learning to apply information learned in one situation to another situation or task.

Training these models can be complex and resource-intensive as they are typically large and require a significant amount of data. The scale of these models also plays a big role in the training process. So lets talk about that.

6 of 11

Copilot

An application using modern AI �to assist you with complex cognitive tasks 

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

7 of 11

Augment users
Make people more efficient
Automate some mundane tasks
Makes it easy to find signal inside the noise
Use human language to query

Copilot

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

8 of 11

Copilot accelerated experiences

Creative assistant

Content generation

Provides recommendations, �with human in the loop

Uses natural language

Across multiple sources

From

To

Tools

Product specific

Data input

Following commands

Uses specialized languages

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

Copilot transforms how we work. With copilot accelerated experiences we can move away from product specific tools that require niche expertise to focusing on turning our ideas into reality.

In today’s data and analytics tools people can barely scratch the surface of what’s possible – because just knowing how to best use a tool is a full-time job.

But, with Copilot, you can turn your words into the most powerful productivity tool on the planet.

This let’s us move from needing to master complex tools, to working alongside a creative assistant. From being siloed in a specific product to working with a cross-cutting experience. From being focused on manually building artifacts to refining automatically generated content.

And most importantly, with Copilot you’re always in control. You decide what to keep, modify, or discard.

9 of 11

Copilot in Power BI

Create beautiful and insightful reports just by chatting with Copilot

Define metrics and calculations for your data model just by describing them in natural language

Use Copilot to find �and summarize insights�in your data

Stay focused on your business outcomes and unlock insights in your data with Copilot

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

10 of 11

Copilot stack

(detail)

AI infrastructure

Foundation models

AI �safety

BYO models

Hosted fine-tuned foundation models

Hosted foundation models

Plugin extensibility

Plugin execution

Grounding

Metaprompt

Prompt & response filtering

Orchestration

Copilot frontend

UX

MICROSOFT CONFIDENTIAL – NON-DISCLOSURE AGREEMENT REQUIRED

11 of 11

Azure OpenAI Docs - https://aka.ms/oai/docs
Azure OpenAI Pricing – https://aka.ms/oai/pricing
GPT-4 Report - https://cdn.openai.com/papers/gpt-4.pdf
Sparks of AGI - Early experiments with GPT-4 - https://arxiv.org/pdf/2303.12712.pdf
Building NLP with OpenAI - https://www.youtube.com/watch?v=_KGuW7y3I6s
The Copilot system - https://www.youtube.com/watch?v=E5g20qmeKpg

MS Build 2023 Videos

Kevin Scott keynote - https://www.youtube.com/watch?v=FyY0fEO5jVY
Scott Guthrie Keynote - https://www.youtube.com/watch?v=KMOV1Zy8YeM
State of GPT by Anrej Karpathy - https://www.youtube.com/watch?v=bZQun8Y4L2A

Further Information and resources