1 of 65

SIL Converters

There’s more than meets the eye!

1

2 of 65

What will be covered

What is it, and where did it come from? Brief history
Why is it even useful? “What can it do for me?”
Overview of the Converters (Transduction Engines)

Setting up a Convertor

Demo of SIL Converters used in various tools

Installation tips

What else is included with SIL Converters?
Finding help,and getting support

2

3 of 65

3

4 of 65

Brief History

4

5 of 65

Where it all began

Humble beginnings �back in May 2002�

“What we need is �Consistent Changes” �within Microsoft Word…

Done!

5

6 of 65

The end of the road for legacy encodings

Driven by a sense of�urgency as legacy�non-standard �encoding issues �started to bite us…

“..stops working with XP,�so we need Unicode-�based solutions NOW!”

6

7 of 65

How it grew over the years

CC table (Consistent Changes)
TECkit maps (more powerful than CC and even bidirectional)
Data Conversion (Word) Macro - very popular as it helped convert complex documents into Unicode without losing the formatting.
Bulk Word Document Converter - great for churning through lots of docs
Bulk SFM Converter - great for multiple scripture files (in Paratext)
ICU Converters and Transliterators
Daisy-Chained converter - for multiple sub-steps in the conversion process
Perl ● Python ● Regular Expressions

7

Later someone told us about TECkit maps, so Bob added that.

Then we needed something anyone can use, so the Word macro "Data Conversion Macro" was born; and was very popular when people were doing their Legacy to Unicode conversions about 20 years ago.

That was great if you only had a few documents to do, but Bob developed a BULK Conversion tool that would automatically convert as many documents as you needed (one after the other). He even had a tool to go looking for Legacy fonts in *any* documents on your computer in order to convert them to Unicode.

Soon the ICU converters and transliterators were plugged in too. This gave us access to a whole host of existing tools that others had developed for switching between scripts, and encodings. But they weren't always exactly what we wanted; they needed to be tweaked.

So along came the Daisy-Chained converter; allowing you to use a standard ICU converter to do the bulk of the hard work, and then a simple tidy up step afterwards to handle the specific tweaks you needed to make. (show an example on the slide)

Why not call a Perl expression, or a Python script, or better still, a Regular Expression. Gradually all these have been added in.

8 of 65

Conceptual Framework

A very simple concept.

Programs send data to any�one of the converters, which �process it and send it back.

8

Data Conversion Macro
Bulk Word Doc Converter
Bulk SFM Converter
etc.

CC (Consistent Changes)
TECkit
ICU Converters
ICU Transliterators
Perl
Python
Regular Expressions

9 of 65

Conceptual Framework

9

Data Conversion Macro
Bulk Word Doc Converter
Bulk SFM Converter
etc.

CC (Consistent Changes)
TECkit
ICU Converters
ICU Transliterators
Perl
Python
Regular Expressions

I suppose this is a little bit like the customers who only want to deal with ONE interface (let’s call that Amazon!) but need to be able to shop from a host of different suppliers.

With Amazon, you use their familiar interface to send them money, they do all the hard work of talking to the supplier behind the scenes for you, and bingo - you get back whatever you asked for.

With SIL converters, you send your data to EncConverters using a nice familiar interface, and it works behind the scenes to improve/change/fix your data and sends it right back to you.

(click) there’s actually a lot more “clients” in blue, and a lot more “suppliers” (or transduction engines) in red -- and this diagram is woefully out of date, but gives you an idea.

I hope the overview helps as we move forward.

10 of 65

What can it do for me?

10

11 of 65

What can it actually do?

Integrated in other software:

Paratext
FLEx
AdaptIt
OneStory Editor
MS Office (Word, Excel, Publisher, Access)
Open/Libre Office (Writer, Calc)

Bulk Word Doc Converter
Bulk SFM Converter
XML Document Converter
Clipboard Converter
Data Conversion Macro
+ some other tools

CC Table
TECkit map
AdaptIt Knowledge Base
ICU Transliterator
Perl Expression
Python Script
RegEx (find & replace)
Online converters

Bing Translator
DeepL Translator
Google Translate

…and a few more

11

So, yes, it is a simple concept; but in reality there are many more places where you will encounter it. And once you figure out how significant it is that all these things on the left (click) can talk to all these “transduction engines” (or converters) on the right, then the world becomes an interesting place.

For those of you who know the concept of LEGO, anything can be built by connecting the right things to each other. Your data is no longer constrained to where you initially typed it in. It can be sucked out, transformed and brough back in with some “value add”. That’s the piece that you need to understand.

Notice that some of SIL’s core software for lexicography and translation have already incorporated the EncConverters (which is the core) within them. And there are actually a whole bunch of different kinds of converters. We’ll come to those later.

Note that structured data files can also be transformed (data which isn’t even associated with specific software)

SFM files, XML data files, .doc / .docx (Word) documents (without even running Word).

In fact, any text in the clipboard (copy-convert-paste) using the Clipboard converter

Before we get stuck into technical details, I’d like to show you some examples of what I’ve used it for.

12 of 65

Example #1 - rather complex, but good fun!

12

My personal experience of using the tool: Adilabad Gondi Phrasebook. Our language learning phrases just kept on growing. English, and Gondi, then Hindi and Telugu, then another language called Kolami. Before we knew it, we had 5 languages going, in 3 different scripts.

If you do the math, you'll see that's 15 things to keep in parallel! All too hard, especially while you’re still editing.

“But what if we only needed to keep the 5 core languages in sync, and could GENERATE the other 10?”

We set up an Excel spreadsheet with the 5 languages side by side, and the other 10 in columns off to the right. But these other 10 columns of phrases were only TRANSLITERATIONS of the 'original 5 languages' generated automatically using Converters from the 5.

It was then easy to focus on getting the 5 core columns right, and eventually we generated several different publications via mail-merge in Word (and an App) from the spreadsheet. The published phrasebook was an instant hit that sold out very quickly, but continues to be very popular today as an App.

13 of 65

Example #2 - spellings

“I need a tool that switches between British English spelling conventions and American English spellings.”

AND��“A single tool that works in multiple places with minimum effort.”

13

differences-between-british-and-american-english

Paratext, FLEx, Word, Excel, e-mails.

Imagine that you want an easy-to-use tool that switches between British English spelling conventions and American English spellings. Is that easier to imagine, and closer to home? The problem is that you need to use this tool in a bunch of different places. You don’t want to have to invent a different solution in each program you use. Sometimes you’re in FLEx (and you need to tidy up your colleague’s [bad?] spellings in the dictionary), sometimes in Word documents, in e-mails, or in the Paratext back translation, and so on. You need a single tool that works in multiple places with minimum effort. Here’s my starting point. Note that because this is a bi-directional TECkit mapping we can go BOTH ways across the atlantic! But beware of words like kerb/curb and check/cheque. It is easy to get tripped up!

14 of 65

Example #2 - spellings (now available in Paratext)

One of the Project Types in Paratext is:��Transliteration (using Encoding Converter)

It takes data from one project, converts it, and places the converted data in another project!

GNTUK → British2American → GNTUS

There’s a lot more to it than that, but this is �just to illustrate HOW converters get used in Paratext.

14

15 of 65

15

16 of 65

Or… Tidy up a Back Translation

16

17 of 65

17

Or use a Converter to Transliterate scripts that you otherwise would not be able to read.��(It is a common need in South Asia to publish the same scripture text in more than one script.)

18 of 65

18

19 of 65

Setting up a Converter

19

20 of 65

Adding a Converter to the Repository

SIL Converters has some standard converters, but allows you to add custom converters to its repository.

When you click on� Add Converter�it needs to know what kind of converter (Transduction Engine) you’re going to add.

20

21 of 65

Adding a Converter to the Repository

21

22 of 65

Adding a Converter to the Repository

22

?

23 of 65

Overview of the Converters

23

the 16�different�types of

24 of 65

SIL-specific Converters

CC Table

bulk find-&-replace tool (‘a’ > ‘b’)

TECkit map

for more complex rule-based transformations, �and TECkit maps are often also reversible

AdaptIt Knowledge Base Converter

leverage work that was done in AdaptIt for adapting whole words or phrases between two languages or two scripts in an AdaptIt project

24

CC: There's a UI around CC that is in the latest version. It is called "SpellFixer" from the Clipboard EncConverter to "Add Substitution" or "Find/Replace" in the Ptx BackTranslation Helper.

You create a "Find What" string and a "Replace With", which can either be as is or with word boundaries and it can be added as a subsequent step of a compound converter to do fixups after some other conversion (as you mentioned above, but the point here is there's a UI way to deal with the CC table).

TECkit Map: Worth noting that it can handle either a .map file or a compiled .tec file. It is also smart enough to know if the .map file has changed and automatically compiles the .tec file for you behind the scenes. [So I tend to always call a .map file unless I don’t actually have access to the source .map and only have someone else’s .tec file.]��AdaptIt: The primary purpose of this transducer is to leverage work that was done in AdaptIt for adapting whole words or phrases between two languages or two scripts in an AdaptIt project. �There are two main uses of this: � 1. translation (which normally involves a one-to-many mapping of words between languages), and � 2. transliteration (which normally involves a one-to-one mapping).

25 of 65

AdaptIt Knowledge Base Converter with OSE

OneStory Editor’s glossing tool being used to disambiguate where a one-to-many relationship needs human interaction

25

Here’s an example of the AdaptIt Knowledge Base Converter being called from another program called OneStory Editor, so that the glosses can be updated.

The output of the Convert call is a perhaps less than useful string in the one-to-many scenario that mimics the old CarlaStudio Ample ambiguity output: %<count>%<translation1>%<translation2>...% (e.g. %4%from%with%by%DAT%). This kind of output is primarily useful in applications, and the only one using it currently is the OneStory Editor’s glossing tool (shown here):

Notice the 4th word on the top line has 5 distinct translations for the Hindi word /पर/, that are shown in the box below separated by the vertical bar character. If you click in that box, you would get a drop down menu like the one shown above giving you an easy way to choose which one fits the current line of the story.

If the AdaptIt project is a transliteration project, where there’s a one-to-one mapping between source and target language, then it could be used to transliterate from one language to another, e.g. in Paratext. But of course, that would only work if every word that you want to transliterate is already in the AI knowledgebase.

The other main use for the AdaptIt transducer will be discussed below when talking about the primary-fallback converter.

26 of 65

Programming Languages

Perl Expression

one or more expressions, usually substitutions

Python Script

with this converter, you can make SIL Converters do almost anything you want (using common libs)

Regular Expression Find and Replace (ICU)

extremely powerful, & quick to configure if you already have a good handle on RegEx (patterns & behavior are based on Perl’s regular expressions)

26

27 of 65

Perl Expression to strip Paratext markup

27

$strInOut =~ s/^\\c \d+//g;

$strInOut =~ s/^\\(pc|pi1|pi) ?//g;

$strInOut =~ s/^\\[psm] ?//g;

$strInOut =~ s/\\v \d+-?\d{0,3} //g;

$strInOut =~ s/\\[pm] //g;

$strInOut =~ s/\\q\d //g;

$strInOut =~ s/ *\\rq .*?\\rq//g;

$strInOut =~ s/\\qt (.*?)\\qt*/$1/g;

$strInOut =~ s/\\fig .*?\\fig//g;

$strInOut =~ s/\\[fx] .*?\\[fx]\*//g;

$strInOut =~ s/\\v[ap] .*?\\v[ap]\* *//g;

$strInOut =~ s/[\x{2020}\x{2021}\x{200c}\x{200d}\x{230a}\x{230b}]//g;

$strInOut =~ s/\*//g;

28 of 65

ICU-based Converters

ICU Transliterator

great for doing off-the-shelf transliteration with �a very impressive list of mostly complex scripts

ICU Converter

not much use any more (c.f. Code Page Converter)

ICU Boundary Analysis/Break Iterator

for languages (like Thai and Chinese) that don't otherwise use spaces between words; can be used to split them based on ICU dictionaries

28

IBM's library of I nternational C omponents for U nicode

The ICU-based converters give us the ability to do all kinds of amazing transformations. ICU is now a well-known standard library, produced by IBM. It stands for: International Components for Unicode, and there’s a link there if you want to explore the power of these tools from ICU, all of which we can now access from within our own software. This is mindblowing when you consider what is possible.

ICU Transliterator - by far the most useful converter in my context. “Any-Latin” is one of the most useful converters, but there are loads of others too. Great for doing off-the-shelf transliteration, and checkout the impressive list of scripts.�

ICU Converter - not really much use (cf. code page converter) - don’t bother unless you’re a developer needing to convert, say from UTF32 to UTF8 or vice versa; but you’ve got other tools to do that anyway.

ICU Boundary Analysis/Break Iterator

This is for languages like Thai and Chinese that don't otherwise use spaces between words. They can be used to split them based on ICU dictionaries. This is used by the OneStory Editor in projects that use the Thai script and Chinese also.

29 of 65

29

IBM's library of I nternational C omponents for U nicode

30 of 65

Combining Converters

Primary-Fallback Converter

try this (primary) converter first, but if nothing changes, then use this other (fallback) converter

Compound (daisy-chained) Converter

combines 2 or more existing converters to make a new converter which does all the steps at once
extremely useful, and worth learning about

Code Page Converter

not much use any more

30

Compound Converter Demo (by Den, 4m22s)

Primary-Fallback Converter - try this, and if nothing changes, then use this other converter.

(Add a slide to say when this might get used.) - be ready for a question, but don’t make a big deal of it.

When one particular converter doesn’t quite do what you want it to, and you need to do more than one step. Link/group multiple converters together so that it (eventually) happens in one step.

Compound (daisy-chained) Converter - do this, then that, and finally something else (2 or more steps)

Maybe use Den's 5-min presentation on Daisy-Chaining converters: Compound Converter Demo (4 mins)

System converters

Code Page Converters - “I can't think of any reason you'd ever use them... A developer might need to do code page conversions, but he has more direct ways of doing it than using EncConverter…”

31 of 65

Primary-Fallback Converter calling an

31

AdaptIt Knowledge Base Converter

Bob: I have a good use case for this converter type is for when the transliteration between two scripts is not perfectly algorithmic (e.g. Devanagari and Arabic). That is, lexical substitutions are needed for those words for which the correspondence can’t be predicted by a TECkit map (or other) algorithm.

In the Devanagari to Arabic case, there are cases where 4-5 different characters or diacritics in Devanagari merge to a single character in the ABS (e.g. nun for nasalization). And there are other cases, where a single /s/ symbol in Devanagari may map to one of several different characters in ABS depending on where the word originally came from.

The easiest way to create a transliteration in this case would be to use a tool like AdaptIt. You load the text into an AdaptIt documents, set up the algorithmic converter (e.g. the teckit map, UDev2UAbs.tec) to preprocess the source word in the target box. If the algorithmic approach works, you just press Enter or Tab and move on to the next word. If it doesn’t work, you type in the correct correspondence, and then hit Enter to capture the couplet in the AI knowledge base.

This may seem much more labor intensive than other options for creating a transliteration, but sometimes it’s necessary, and with a tool like AdaptIt, at least, you only have to do each word form once. If it comes up again, AdaptIt will skip right past it (as long as there are not multiple target words for the same source word, which there shouldn’t be in a strict transliteration project).

Once you have the entire corpus finished in this way, you can add the AdaptIt knowledge base and the TECkit map as two separate EncConverters, and then add them to a primary-fallback converter: the AI converter as the primary (for lexical substitutions), and the TECkit map as the fallback (for the algorithmic conversion. When converting a word, if there is a lexical exception in the AI knowledge base converter, it processes the word. If not, it falls back to the algorithm in the TECkit map. Of course, if both fail on some new word, then you have to add the new combination to the map in some way.

There is a way in the Setup tab of the AdaptIt Knowledge Base Converter to edit the Knowledge Base and add correspondences:

32 of 65

Web-based Converters Translators

Bing Translator

multilingual translations with broad language support and continuous improvements (Microsoft)

DeepL Translator

provides highly accurate and natural-sounding translations using advanced neural networks

Google Translate

automated translation for numerous languages with varying degrees of accuracy and naturalness

32

Now we get to the latest generation of converters which obviously, only work if you’re online!

Bing Translator, DeepL Translator, Google Translate

Note that there are limits on how much data we can translate (each month) for free. If you need this feature a lot, then go and get your own paid-for key. (Show to how to do this…)

And perhaps one day in the not too distant future: SIL’s AI service (called Serval) might be a possibility too.

OPTIONAL: [Also note that there’s the old Technical Hindi (Google group) Html Converter which is basically only for Indic legacy fonts to Unicode.

(This is not actually an online converter.) The java script code to do the transducing is inside the html webpage, but the webpage is only hosted on a google file site as a document rather than a website. So you download the webpage that has the encoding change/font you want, create a converter referring to it, specify the functions to the forward and sometimes the reverse conversion, and then you can use it in the various EncConverter client applications.]

33 of 65

Paratext Plugin: Back Translation Helper

33

Bing

DeepL

Paratext

Source BT

Play video

34 of 65

Subtle differences of meaning!

34

Play video

35 of 65

The principle of 2 or 3 witnesses…

35

Play video

36 of 65

Balancing accuracy with naturalness

36

Play video

37 of 65

Demo of various tools

37

38 of 65

Live demonstration

Installation Tips (enabling the right features)
Using Translation Helper dialog in Word
Doing RegEx Find and Replace operations in Word
Using SIL Converters in Excel (to obfuscate email addresses) with:

selected cells
ConvertString function

XML converter calling AI translators to localize Transcelerator �questions into an LWC other than English (major regional languages)

38

39 of 65

39

40 of 65

Using a converter with OneStory Editor

40

41 of 65

Using a converter with OneStory Editor

41

42 of 65

Using a converter with OneStory Editor

42

43 of 65

Installation Tips�Enabling the right features

43

44 of 65

Overview of some other tools

Clipboard EncConverter
FLEx
SFM
AdaptIt
Bulk Word Document Converter
Technical Hindi Google Group converter

44

45 of 65

Clipboard EncConverter

Copy source text → Right-click on to Convert � → Select the appropriate Convertor → Paste converted text

45

Clipboard converter - is one of my favorites for one-off tasks (and relatively small amounts of data).

It sits quietly in the system tray area waiting to be called on.

A great way for testing a new converter too - try it on a subset of your data

Also a quick way to add (or install) new converters

It also has hooks into the 'SpellFixer' tool (the UI on a CC table) where you can copy the FindWhat text to the clipboard, and left-click on the ClipboardEncConverter, and it'll bring up the UI dialog an initialize the FindWhat box w/ the clipboard data and give you a place to put the replacement... to make it easy to add new find/replace couplets to your SpellFixer project. This is esp. useful for Indic lgs for which there are so many distinct and yet valid ways of spelling things to map from forms you want to avoid to the 'received' form...

46 of 65

Install (or Uninstall) Converters as Needed

46

47 of 65

Accessing SIL Converters in FLEx

When in Bulk Edit mode, �you can an call �a “Process” (i.e. an SIL Converter)

47

48 of 65

Accessing SIL Converters in FLEx

48

49 of 65

Accessing SIL Converters in FLEx

49

50 of 65

Converting SFM databases

50

51 of 65

Converting SFM databases

51

52 of 65

XML Document Converter

Easy to use with click and select options.

But also supports XPath Expressions for advanced users.

52

53 of 65

XML Document Converter (sample data)

53

54 of 65

Bulk Word Document Converter

Select several Docs

(let it scan them…)

Select the Converter(s) for each font/row

Optionally set the �new Font to apply

Click on the Save icon

54

55 of 65

Linguistic Tools in LibreOffice: software.sil.org/oolt/

55

56 of 65

SIL Converters in Word: Multiple tools

56

57 of 65

SIL Converters in Word: RegEx Find/Replace

57

58 of 65

Using a converter with AdaptIt

58

59 of 65

What else is included?

TECkit map editor (helps you build and test your mapping)
Discourse Chart Helper (another very useful utility for analysis)
Consistent Spelling Checker (now redundant as Paratext does a better job)

59

60 of 65

TECkit map editor

60

61 of 65

Discourse Chart Helper

Helps prepare discourse analysis charts to study discourse features of a vernacular language text

61

62 of 65

62

63 of 65

Finding Help and Getting Support

The built-in documentation is very comprehensive �(please refer to it before asking for support)

Also check out the FAQs:�software.sil.org/silconverters/silconverters-faq/

If all else fails, e-mail: silconverters_support@sil.org �for limited support.

63

software.sil.org/silconverters

64 of 65

64

Hunger is the best cook

65 of 65

SIL Converters

There’s always more …

65

Thanks for attending!

tiny.cc/silconv