Fetching Data from APIs using Mojolicious
Boyd Duffee, LPW 2025
Getting Started
tl;dr - just go out and buy Mojolicious Web Clients by brian d foy and follow your nose.
… the second half talks about testing your client!
Today I’m talking about REST, not SOAP. That’s a story for another time.
Live Now on CPAN
Astro::ADS
WebService::OurWorldInData
Thank you!
CPAN Testing Service for finding all my install fails.
No Dev key found …
Dev key in file not the environment …
Etc, etc…
Write a Base Class to hold the attributes every http request uses:
1a. Scafolding
Write wrapper functions around your REST calls so you Don’t Repeat Yourself. This gives consistency to your interface.
1b. Scafolding
my $tx = $self->ua->build_tx( GET => $url );
$tx->req->headers->authorization( 'Bearer ' . $self->token );
warn $tx->req->to_string if $DEBUG; #optional
Tip
To add headers, you need to build the transaction object $tx yourself.
Using built-in get() or post() sends the request immediately.
Authorization
my $tx = $self->ua->build_tx( GET => $url );
$tx->req->headers->authorization( 'Bearer ' . $self->token );
try { $tx = $self->ua->start($tx) }
catch ($error) {
carp "Got this error: ", $error;
}
# suppress warning for native perl 5.36 try/catch
no if $] >= 5.018, 'warnings', 'experimental';
Tip
$self->ua->start($tx) tells the UserAgent to make the request.
Feature::Compat::Try; gives you the new try/catch syntax on Perls older than v5.36
Error handling
The response to your API call is a result, but not always provided in the desired format.
2. Results
package MyClient::Role::ResultMapper;
use Moo::Role;
use strictures 2;
use MyClient;
sub parse_response {
my ($self, $json) = @_;
return unless exists $json->{responseHeader};
@{$result_params}{ qw<q rows> } =
@{$json->{response}{params}}{ qw<q rows> };
Tip
strictures 2 turns on strict and makes most warnings fatal.
Oooh, here’s a lovely hash slice mapping values 2 levels up from the json response
MyClient::Role::ResultMapper
The internet is not always with us, so use a module that listens to the HTTP traffic, record the conversation and play back the responses instead of constantly hammering the target website:
3. Testing
package Test::MyClient;
# set env variable for playback only
$ENV{DEV_KEY} = 'A_very_long_string'
unless $ENV{LWP_UA_MOCK} eq 'record'
|| $ENV{LWP_UA_MOCK} eq 'passthrough';
1;
Tip
Use the real DEV_KEY for talking to the live service.
Also a good place for start-up and tear-down methods to make your tests consistent.
t/lib/Test/MyClient.pm
use lib 't/lib';
use Test::MyClient;
BEGIN {
$ENV{ LWP_UA_MOCK } ||= 'playback';
$ENV{ LWP_UA_MOCK_FILE } ||= __FILE__.'-lwp-mock.out';
}
...
Tip
This snippet is from the docs on LWP::UserAgent::Mockable which controls behaviour with environment variables instead of code.
t/endpoint.t
...
skip_all('No API key found in test suite')
unless $ENV{DEV_KEY};
my $ua = Mojo::UserAgent::Mockable->new(
mode => 'lwp-ua-mockable',
ignore_headers => 'all'
);
my $client = MyClient->new( q => 'search_term',
ua => $ua
);
Tip
The skip_all turns a CPANTS fail report into an unknown report
LWP mode checks environment variables
Ignore the headers
Use mocked UserAgent in tests!
t/endpoint.t
Protect your users from the network, but not from themselves. Pass the API errors through for sensible debugging.
4. Error Testing
use Test2::Tools::Exception;
subtest 'Bad Key - Authorisation failure' => sub {
local $ENV{SECRET_DEV_KEY} = 'BAD';
like(
warning { $client->method() },
qr/HTTP Error: $expected_error/,
'Caught dev token error'
);
Tip
Make a single test file for each hard-to-reproduce network error.
Record when you can, playback at leisure.
t/useragent.t
Scrub your DEV_KEY with a
Git pre-commit hook
before committing it to the repository
Security Tip
Do Not commit your dev key to a public repository.
Recording your mocks saved the http headers in the mock file – which included the Authorization.
Ooops!
s/Bearer \w{10,}/
Bearer TOKEN_REMOVED/g
Tip
My auth header starts with Bearer and my 40 character dev_key.
Image: https://xkcd.com/208
use v5.20;
use FindBin qw($Bin);
use File::Find;
use Mojo::File;
# for non-destructive testing
my $sub = sub {
my $name = $File::Find::name;
say $name if $name =~ /out$/;
};
...
Tip
Find all recorded mocks.
Save this script in
.git/hooks/pre-commit
scrub_mock_headers.pl
...
my $scrub = sub {
next unless $File::Find::name =~ /mock\.out$/;
my $path = Mojo::File->new( $File::Find::name );
my $mock = $path->slurp;
say 'Scrubbing ', $path->basename
if $mock =~ s/Bearer \w{10,}/Bearer TOKEN_REMOVED/g;
$path->spew( $mock )
or warn "Error writing to $path: $!\n";
};
find $scrub, "$Bin/../t";
And saved as .git/hooks/pre-commit
Software engineers,
ask me
How long does it take
To write an endpoint?
Thank You
You have been enthralled and entertained by
Boyd Duffee
I am DUFFEE on CPAN, duffee on github, gitlab and dev.to
I have started to write sciency things at
https://perldatascience.wordpress.com/
And yes, I am currently looking for work ;)
boyd.duffee@gmail.com