Build with AI

Build agent with Gemini/Gemma and AutoGen

Jimmy Liao

Generative AI Evangelist

Taoyuan

Deck Link

April 13, 2024

Discord for GDG Taiwan

• ​

https://discord.com/invite/google-dev-community

https://discord.gg/6xGhqZf6J8

In the upcoming series of session/workshops

• 使用 Gemini Pro Vision API 打造出國旅遊的小幫手
• Gemini多模態RAG應用/ Gemini Code Assist 應用介紹
• AI Your Summer

Before start - check your GCP access

https://github.com/jimmyliao/genai-gdg

已經有 Google Cloud billing account ，而且可以打開取得產生 API Key 的權限了嗎？

​

https://aistudio.google.com/

Temp access for this workshop

為了 Workshop方便，會提供講師的 Google Cloud Project，加入學員的 email。預計課程後一個禮拜會關掉，之後請改用自己的帳號與 Project。

請用 QRCode 填入你的 Email

​

Agenda

1. Intro to the Gemini Pro API
2. Hands-on: AI Studio with Gemini Pro, Vision, Prompt Library
3. Hands-on: Simple Chat with Gemini Pro
4. Hands-on: Image Recognition with Gemini Pro Vision API
5. Hands-on: RAG Concept with Gemini Pro API
6. Hands-on: Multi-Agent, Gemma, Ollama, and AutoGen

​

https://github.com/jimmyliao/genai-gdg

Gemini Models

and APIs

Taoyuan

AI Studio

https://aistudio.google.com/

​

• Generate API Keys
• Create, test, save prompts
• Customize models in minutes
• Generate starter code

From Prompt Library

Prompting

​

Chained prompts

• make a plan, then execute it

Context

• Few shot prompts

Generation parameters

• Temperature, Safety, Top-P, Top-K

Search and Information Synthesis

​

BYO Data

• Models have knowledge cut-offs
• LLMs are not fact engines
• No exposure to private data

Use the prompt’s context window

• Instructions + Context + Question all in the prompt
• Easy to implement
• No extra code, just ask

Gemini 1.5 Pro!

Use the prompt’s content window

• gemini-1.5-pro: 1M tokens.
• Remember: Everything is tokens
• Using context may be more flexible than Retrieval Augmented Generation (RAG)
• More input context means higher latency

​

​

AQA Attributed Question Answering

with inline chunks

• Instruction + Chunks + Question in one request

​

• AQA model is specifically tuned for RAG

​

• AQA is a new feature

​

Use AQA with inline chunks

• Simple to use

​

• Example: handle chunks returned by a search tool

​

• Limited by the request size (Max 4MB)
• Inefficient when asking about the same data repeatedly

Use AQA with the retriever service

• Corpus < Document < Chunk hieararchy

​

• Pass a reference to a document or corpus to the generate_answer function

Function Calling

• Describe external functions to the model.
• The model may ask you to call the function to help it respond to your queries.
• Endless possibilities for integrating external tools.

Function Calling - Basic

• How?
• The google.generativeai SDK will inspect the function’s type hints to determine the schema.
• Allowed types are limited:

AllowedTypes = (

int | float | str | bool | list | dict )

Function Calling - Basic

• What happened?
• Use the chat history to find out.
• The chat history collects all the function calls and responses that tool place.

Function Calling - Schema

• JSON object

Function Calling - Structured data

• Structured data extraction.
• Ask the model to do it and return JSON.

Before function calling

Function Calling - Structured data

• Asking for JSON often works.
• Function calling lets you strictly describe the schema.
• With a strict description, can strictly enforce that what gets returned.

with function calling declaration

Function Calling - Structured data

• Why?
• Function calling doesn’t return a string.
• return data-structures: You don’t parse text.

Here how function calling with SDK…

Image understanding

Gemini 1.5 Pro

Gemini 1.5 Pro

Image understanding

• Images are just tokens in the input

​

• Can be used for instructions, context or query subject

Time to

Get Hands Dirty

Kaohsiung

Task 1: AI Studio and Prompting

Generate API Key

Gemini Pro

• Playground
• Generate Code
• 1.0, 1.5

Gemini Vision

• Image

Prompt Library

Task 2: Python SDK part 1 - Chat

Notebook => intro_gemini_chat.ipynb

Model parameters

Stream

Chat History

Use with LangChain

​

Estimated: 10 mins

Task 3: Python SDK part 2 - Vision

Notebook =>

https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_pro_vision_python.ipynb

Image (Local) -> Text

Image + Text -> Text

Video -> Text

​

Estimated: 20 mins

Task 4: cURL to know all Gemini Pro APIs

Notebook => intro_gemini_curl.ipynb

1. text -> text
2. image -> text
3. chat (multi-turn, role)
4. function calling
5. vision
1. image -> text
2. video -> text

Estimated: 15 mins

Task 5: Build RAG application (LMM)

• Purpose link
• Notebook: Gemini-lmm.ipynb
• Remember to use TPU runtime
• Estimated: 20 mins

​

(https://github.com/jimmyliao/genai-gdg/blob/main/gemini/rag-intro/gemini-lmm.ipynb)

Gemma

Taoyuan

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

https://www.youtube.com/watch?v=60V70JqGkuU

Setup Tokenizer

Create Pipeline

Text Generation

Create Message and Prompt

Ollama

Taoyuan

Run Your Local LLM!

AI Agent

Taoyuan

AI Agent

• ​

Difference

​

​

​

​

​

​

​

​

Source: Youtube video of Andrew Ng at Sequoia Capital (Link in References)

AI Agent Example

• ​

AutoGPT

​

​

Langroid

​

​

AutoGen

​

​

Agentic Workflow (example)

https://www.linkedin.com/pulse/introducing-ai-agents-agentic-workflows-future-ken-rheingans-1rwce/

​

Related publication

• Agent / Multi-Agent Application 淺談系列 - 1 (https://jimmyliaonet.substack.com/p/agent-multi-agent-application-1 )
• ​

Task 6: Multi-Agent - Gemma, Ollama, and AutoGen

• Notebook
• Remember to use TPU runtime
• Estimated: 30~60 mins

(https://colab.research.google.com/github/yeyu2/Youtube_demos/blob/main/autogen-ollama-gemma.ipynb )

References

Official

• Gemini Official Cloud Training doc
• Google Developers Group training deck (remade)
• Multi-Agent demo https://www.youtube.com/watch?v=RC6OFzyHYpY

​

​

DM for detail

We are working on * initiatives around GenAI