Building Native Extensions
This Could Take A While ...
Mike Dalessio
@flavorjones
RubyConf 2021
Mike Dalessio
@flavorjones
I work at Shopify
I maintain Nokogiri
Preface
bit.ly/rcee-gems
flavorjones/ruby-c-extensions-explained
Only a Few Basic Patterns
“Isolated”
“System”
“Packaged Source”
“Packaged Tarball”
“Precompiled”
Ruby Kaigi talk
Was a
HOW-TO
Today we’ll also cover the
“WHY?”
Things I will talk about today
What is a
C Extension?
Show me a
C extension!
Let’s look at the “Isolated”
gem
I need more details, plz!
What can go wrong
when installing C extensions?
1. Compiler Toolchain Not Installed
What problems can a C Extension solve?
Talk to�Third-party
Libraries
Examples of gems using third-party libraries
nokogiri
psych
sqlite3
rmagick
grpc
libxml2, libxslt, libgumbo
libyaml
libsqlite3
libMagick
libgrpc
Are there other ways to call third-party libraries?
Show me
how to use an external library!
bit.ly/rcee-gems
flavorjones/ruby-c-extensions-explained
Strategy #1: “system” libraries
gem install rcee_system
What tends to break when using the “System” strategy?
Strategy #2: “Packaged” libraries
gem install rcee_packaged_tarball
Good and Bad about the “Packaged Tarball” Strategy
Good
Bad
User Satisfaction
Strategy #3: “Precompiled” libraries
gem install rcee_precompiled
“Precompiled” strategy risks
Ruby version 🗹
Platform 🗹
System libraries (gnu vs musl) 😜
Mitigation:
Test All The F’n TimE
Unfortunately, this doesn’t currently test musl/alpine
Why Most Gems Shouldn’t Be Precompiled
“Goodbye Fat gem” raises some good points
“Goodbye Fat gem” raises some good points
“Goodbye Fat gem” raises some good points
“Goodbye Fat gem” raises some good points
“Goodbye Fat gem” raises some good points
“Goodbye Fat gem” raises some good points
Why Precompiled Nokogiri is a Good Thing™
Why Precompiled Nokogiri is a Good Thing™
(For me)
Fewer Support Issues
In 2020, just over 1 per week.
In 2021, just over 1 per month.
Why Precompiled Nokogiri is a Good Thing™
(For you)
Fewer Struggling Users
YoY, page hits on installation docs are down ~30%
(Page hits on nokogiri.org are up ~20% in the same period)
Fewer Angry Complaints
Nearly zero four-letter-words uttered on Twitter
(except for folks on older versions! 🤪)
Why Precompiled Nokogiri is a Good Thing™
(For the universe)
Nokogiri is Webscale™, I guess
Nokogiri precompiled downloads since v1.11.0 (2021-01-03):
Total: 60,743,405
x86_64-linux: 56,055,797 (88.38%)�x86_64-darwin: 3,124,231 (4.93%)�java: 1,065,024 (1.68%)�x64-mingw32: 326,319 (0.51%)�arm64-darwin: 137,295 (0.22%)�x86-mingw32: 18,721 (0.03%)�x86-linux: 16,018 (0.03%)
Compare to vanilla ruby: 2,685,027 (4.23%)
Back-of-the-envelope Power Calculations
Intel(R) Core(TM) i7-1065G7 CPU�rated at 15W TDP
60_000_000 * 15W * 11s
= 2.75 MWh over 10 months
Back-of-the-envelope Power Calculations
2.75 MWh over 10 months is …�
a refrigerator running for the same amount of time.
Back-of-the-envelope Power Calculations
2.75 MWh over 10 months is …�
560 pounds of carbon emissions
(about half of just my one-way flight from Newark to Denver)
Back-of-the-envelope Power Calculations
2.75 MWh over 10 months is …�
Not much, I guess? I thought it would be more.
Back-of-the-envelope Time Calculations
Default dev configuration on a modern laptop taks 82s to install Nokogiri (not precompiled)
With 8 cores, it’s 48s
Back-of-the-envelope Time Calculations
If even 10% of installations are humans (and not automated CI/CD),�that’s ~$13M USD in saved human labor.
(But even CI/CD has humans waiting for it sometimes, right?)
Precompiled Nokogiri is a Good Thing™
A word about Trust
What’s in the box?
Why should you trust what’s in the box?
Do you trust the source
(the author)?
Do you trust the chain of custody�(the delivery system)?
Why do you trust this guy?�
Does he follow basic security hygiene?
Is he liable to be blackmailed?
Does He have illegal Gambling Debts?
You can trust this guy! ☑�
I’ve been around for a while
I have a history of shipping Nokogiri
You have met me in person, probably
Don’t worry, IMMAGOODGUY!
Why do you trust the supply chain?�
Have the maintainers all enabled MFA?
Was the gem signed?�Did you check the signature?�OR: did you verify checksums?
Probably, trust the supply chain? 😎�
Nokogiri maintainers use MFA!
Opted into new “MFA Required” feature!
Checksums always provided!
I think you are all too trusting,
And I just want you to be a little more paranoid
Multi-Factor Authentication
MFA makes it hard to impersonate you
Gem Signing
Gem signing and verification is wonky and very few people do it.
Find me and I will talk to you about this!
Coming Soon!
rake-compiler-dock
Support for ARM64 Linux!
rake-compiler-dock
Easier use in CI/CD pipelines!
rake-compiler-dock
Ship 3.1 support early this year? Maybe?�(Hi, Lars!)
Musl vs glibc
This is a pretty big problem.
Thank you!�
Q&A