1 of 22

Large Language Models Can Self-Synthesize Training Data for Low-Resource Languages

Mathias Stenlund

University of Iceland

1/8

2 of 22

Default LLM training paradigm

Massive collections of text: books, articles, websites etc.

Data:

Pretraining

Pretrained LLM

Action:

2/8

3 of 22

Default LLM training paradigm

Massive collections of text: books, articles, websites etc.

Data:

Pretraining

Pretrained LLM

Instruction-response pairs

Data:

Instruction-tuning

Instruction-tuned LLM

Action:

2/8

4 of 22

Default LLM training paradigm

Instruction-response pairs

Data:

Instruction-tuning

Instruction-tuned LLM

Instruction-tuning LLMs

Requires scarce instruction-response datasets
Collecting human-authored data is expensive and time-consuming
Could synthetic data be an option even for smaller languages?

Action:

2/8

5 of 22

Self-synthesised Magpie approach

(Xu et al., 2024)

3/8

6 of 22

Self-synthesised Magpie approach

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

(Xu et al., 2024)

= User template

3/8

7 of 22

Self-synthesised Magpie approach

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

What is the current population of Hella?

(Xu et al., 2024)

= User template

3/8

8 of 22

Self-synthesised Magpie approach

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

What is the current population of Hella?

<|eot_id|>

(Xu et al., 2024)

= User template

3/8

9 of 22

Self-synthesised Magpie approach

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

What is the current population of Hella?

<|eot_id|>

(Xu et al., 2024)

= User template

Llama Output

As of 2019, the population of Hella is 865.

3/8

10 of 22

Self-synthesised Magpie approach

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

(Xu et al., 2024)

= User template

3/8

11 of 22

Self-synthesised Magpie approach

Llama Output

(Xu et al., 2024)

Write me a poem about a black and white bird where every word starts with the letter C.

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

= User template

3/8

12 of 22

Generating instructions in target languages

(Xu et al., 2024)

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

= System template

= User template

4/8

13 of 22

Generating instructions in target languages

(Xu et al., 2024)

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Ett samtal mellan en nyfiken användare och en hjälpsam AI-assistent.

= System template

= User template

English: A conversation between a curious user and a helpful AI assistant

4/8

14 of 22

Generating instructions in target languages

(Xu et al., 2024)

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Ett samtal mellan en nyfiken användare och en hjälpsam AI-assistent.

<|eot_id|>

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

= System template

= User template

English: A conversation between a curious user and a helpful AI assistant

4/8

15 of 22

Generating instructions in target languages

(Xu et al., 2024)

Llama-3.3-70B-Instruct input:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Ett samtal mellan en nyfiken användare och en hjälpsam AI-assistent.

<|eot_id|>

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

Llama outputs instruction in Swedish

Vad var järnridån för något?

= System template

= User template

English: A conversation between a curious user and a helpful AI assistant

4/8

16 of 22

Task type distribution

Original implementation

5/8

17 of 22

Targeted system prompt pool

Default system prompt

A conversation between a curious user and an AI assistant.

6/8

- What is the talled structure on Earth?

18 of 22

Targeted system prompt pool

Default system prompt

A conversation between a curious user and an AI assistant.

Math system prompt

A conversation between a math student and an AI tutor.

6/8

- What is the talled structure on Earth?

- How do I define a prime number?

19 of 22

Targeted system prompt pool

Default system prompt

A conversation between a curious user and an AI assistant.

Math system prompt

A conversation between a math student and an AI tutor.

Creative writing prompt

A conversation between a professional writer brainstorming ideas and an AI-assistant.

6/8

- What is the talled structure on Earth?

- How do I define a prime number?

- Write a short story about a squirrel.

20 of 22

Task type distribution

Original implementation

7/8

21 of 22

Task type distribution

Original implementation

W. Targeted system prompts

7/8

22 of 22

Reference page

Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y. and Lin, B.Y., 2024. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464.

8/8