Introduction to Scripture Forge Drafting

Jeff Webster (Seed Company)�Adapted from:�Michael Martin & David Baines (SIL)

Outline

• Outcomes of workshop
• Orientation to AI drafting and Scripture Forge
• How does Scripture Forge drafting work?
• What can I do with Scripture Forge drafts?
• Feedback from Editing AI drafts
• Results from Using AI drafts
• What are drafts like?
• How can I get better drafts?
• Q&A
• More information

Workshop Outcomes

• Teams will have a basic understanding of how AI drafts are created and how they relate to their existing translation.
• Teams will be comfortable using SF to generate AI drafts, applying to their projects, and syncing to Paratext
• Teams will have experience editing the AI drafts into quality first drafts, and understand how AI drafts fit into their translation workflow.
• Teams will objectively evaluate their AI drafts and reflect on their quality and how helpful they might be.
• Teams will have a rough comparison of both speed and quality differences/similarities between manual drafting and using AI drafts. (probably can’t achieve this one very well)
• Teams will know how useful AI drafts were for them, and where usefulness was seen, be ready to immediately apply them to their workflow.

Orientation

Scripture Forge in Bible Translation

Scripture Forge

Scripture Forge for Drafting and Editing

Scripture Forge and the Adoption Curve

​

Who else is using Scripture Forge drafting?

How does Scripture Forge drafting work?

You are the craftsman, AI is the tool in your hand.

As the craftsman…

​

• You decide whether using the AI tool is a helpful for your work. �
• The AI tool will never replace your knowledge and creativity as the workman, it needs your mind and hand to guide it.

​

​

​

What is Scripture AI Forge Drafting?

What it is not –

• Translation based on linguistic knowledge

​

• A “generative” model like ChatGPT

​

• A replacement for translators

​

What it is –

• Deep Learning: mathematical analysis of the relationship between two texts
• An “encoder-decoder” translation model: output depends on input
• Studies a pattern → Predicts an output

​

Deep Learning: The Basic Idea

“Cat”

“Dog”

“Cat”

“Dog”

Deep Learning in Translation

Cat

Dog

Good morning

I love you

Chat

Chien

Bonjour

Je t'aime

How are you today?

Comment allez-vous�aujourd'hui?

Deep Learning in Scripture Forge

Our Father in heaven ...

Your kingdom come ...

Your will be done ...

Give us today our ...

Notre père qui êtes aux cieux ...

Ton royaume vient ...

Votre volonté sera faite ...

Donnez-nous aujourd'hui notre ...

Forgive us our debts

Pardonne-nous nos dettes ...

Scripture Forge drafting �Deep learning with 8000 New Testament verses

How does Scripture Forge Drafting work?

THE BASE MODEL:

NLLB, No Language Left Behind (Meta)

Data from ~ 200 world languages

​

FINE-TUNING:

Parallel Bible texts�(Model text + your project)

​

INPUT

• Model text: a source version with trusted exegesis and style
• Completed books from your project

​

OUTPUT

• Rough draft of next Bible book, translated from the model text

​

Rough drafts contain errors and require editing by skilled translators

​

STEP 1: Fine-tune Meta’s NLLB model using your Bible translation

STEP 2: Draft a book using the fine-tuned model

How does Scripture Forge Drafting work?

NT (Target Language)

NT (Model Text)

GEN (Model Text)

Step 1. Fine-Tune

Step 2. Create Draft

GEN Draft�(Target Language)

What can I do with Scripture Forge drafts?

What can I do with Scripture Forge Drafts?

• Direct editing

Edit a draft to begin translating your next book in Scripture Forge or Paratext.

• Drafts as resources

Create one or more drafts to consult for ideas as you translate.

• Drafts for checking

Compare a completed book with a draft during the checking process to improve quality and consistency.

• Back translation drafting

Create starter drafts of back translation books that maintain the style and notation choices of your existing back translation.

1. Review and edit in Scripture Forge

2. Download

3. Add to

project

Four options for newly-created drafts

4. Add to

AI drafts

project

Keeping in Sync

• Paratext users keep their projects up-to-date using Send/Receive
• Paratext servers store the master copy of the project
• Scripture Forge is another Paratext ‘user’
• Scripture Forge’s Sync with Paratext is like a Paratext Send/Receive
• Use Scripture Forge Sync with Paratext:
• Before creating a draft
• After adding a draft to your project

Paratext

ABC

ABC

Scripture Forge

ABC

Feedback from Editing AI Drafts?

Scripture Forge Drafts–Translator Feedback

Translator Feedback

• “It sound[s] like our language.”�
• “Good range of words provided by the AI makes the translation richer.” �
• “I can't tell you how exciting this is. The draft from the [source] was excellent. I'm really impressed!”�
• “When drafting manually, I could only complete at most 20 verses a day, but with AI-generated drafts, I’ve been able to work through 30 to 40 verses. I’ve felt that working with AI has made the task much easier and more accessible…. I’ve found AI to be highly effective, and I’ve realized that I can accomplish much more than I expected with its help.

​

Scripture Forge Drafts–Consultant Feedback

Consultant Feedback

• "When I first looked at the AI draft of Jonah I started laughing to myself because of its quality. I am very excited about the possibilities ahead, both for the Xerente people and for hundreds of other peoples around the world!”--Werner�
• "The AI-generated draft saves the team approximately 85% of their time while maintaining a natural flow and consistency with key terms. From a consultant’s perspective, AI-generated drafts are excellent when supported by sufficient base text. The better the quality of the available data, the better the results.”--Jamuna�
• "I have observed notable differences in the quality of their AI-generated drafts. My work with the Rose and Thamne teams has been particularly impressive; their drafts were of very high quality–more accurate [than pre-AI], pretty clear and sounded very natural.”--Kunsang

​

Results from Using AI Drafts

5 Languages in South Asia

Prescience Labs Study: 11 Languages, 4 Countries

Acceleration

​

Quality

​

What are drafts like?

What are Scripture Forge Drafts like?

Common error types

• Names
• numbers
• Kinship terms
• Incorrect word choice
• Added or missing words
• Repeated words
• Grammar formation
• Syntax/word order

Common strengths

• Naturalness
• Consistency of style, word choice, key terms.
• Consistency with previous books.
• Formatting, punctuation

Example: Proper names

Unfamiliar names are replaced with different names, while familiar names are translated correctly.

Example: Numbers

​

​

Number changed (numerals replaced with words).

Other number transfers correctly..

Example: Punctuation, sentence structure

Partial sentence becomes 2 complete sentences.

Example: Kinship terms

Mother-in-law is correct in verse 14.

​

Sister-in-law becomes Mother-in-law in verse 15.

​

Known limitations of Scripture Forge drafts

• Footnotes are not transferred to the draft.
• Key terms are not used for fine tuning for most teams.
• Time: Creating a draft takes about 2.5 hours (longer if the server is busy).

Managing Drafts for Resources / Checking

How does Back Translation Drafting work?

GEN (draft)�Scripture Forge

Back Translation Language

NT

Back Translation Project

Back Translation Language

NT

Your Project

Your Language

GEN

Your Source

Your Language

Step 1. Fine-Tune

Step 2. Create Draft

How will our process need to change?

• Only the METHOD of making the FIRST DRAFT is different. All other steps stay the same.
• The order of first draft steps may change, for example:

Example steps when drafting by hand:

• Read the primary source version.
• Read other versions to understand text.
• Consult exegetical resources.
• Draft a verse that sounds natural.
• Check and edit the AI draft for accuracy, clarity, naturalness, grammar and spelling.

​

Example steps when editing a AI draft:

• Read the primary source version.
• Read other versions to understand text.
• Edit the AI draft for accuracy, clarity, naturalness, grammar and spelling.
• Verify exegesis by consulting exegetical resources.

​

How can I get better drafts?

Improving your draft

• Reference project(s) &�Drafting source
• Training books
• Paratext key terms
• Name translation pairs
• Other language data
• Paratext checks

​

​

Choosing the Reference Project(s) and Drafting Source

Single reference options

• Model text (i.e., primary source)
• Other text, Same language as model text
• English translation
• Closely related language translation

Multiple reference options

• Model text + back translation

Drafting Source options

• Model text
• Same language as the model text

Training Books

• Which books should be used for fine-tuning?
• Complete - yes!
• What about in-progress books?
• More data / not final
• Partially complete books?
• Genesis passages
• Portions of Psalms
• Out of date back translation books?

Can Paratext Key Terms Improve My Draft?

• Drafting may use project key term renderings for fine-tuning
• Automatic, but it depends on …
• Key terms list type
• Reference project language
• Only proper name key terms
• Renderings must prepared for the book being drafted
• Mostly impacts new proper names in the book being drafted

Name Translation Pairs Can Improve Quality

• Add name translations for the fine-tuning step (by admin)
• New names for next book
• Loaded during draft configuration
• Two columns file format
• Source language name
• Target language name
• CSV, TSV, xls/xlsx format
• Used on each draft request (until changed)

Name Translation Pairs

• Data must be parallel in the source and target language.

For fine-tuning the English-to-Hindi model to improve the draft of Esther:

Name Translation Pairs

Other language data

• Add sentence pairs for the fine-tuning step (by admin)
• E.g., poetry samples
• Loaded during draft configuration
• Two columns file format
• Source language sentence
• Target language sentence
• CSV, TSV, xls/xlsx format
• Used on each draft request (until changed)

Questions?

More information

Scripture Forge Information and Help

• Click the “?” for help:
• Help and learning: https://help.scriptureforge.org
• Report a problem:

Email help@scriptureforge.org

• Scripture Forge Questions?

Email: support_team@scriptureforge.org

• Scripture Forge Updates: https://software.sil.org/ScriptureForge

​