A practical and effortless
way to interact with AWS S3 in PHP
PUG ROMA, 25th June 2019
Who Am I?
Mauro Cassani
Back-end dev
@Translated
What is AWS S3?
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics.
Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9's) of durability, and stores data for millions of applications for companies all around the world.
How AWS S3 does work?
Every interaction with AWS S3 is done by a http call against �a RESTful API:
S3
API
Client
PUT /ObjectName HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
...
How files in AWS S3 are organized?
bucket-1
folder/to/file.txt
BUCKET
OBJECT(S)
folder/to/file2.txt
folder/file.txt
bucket-2
folder/to/file.txt
folder/to/file2.txt
folder/file.txt
How files in AWS S3 are organized?
folder/to/path/file.txt
folder/to/path/another-file.ext
folder/to/path/third-file.ext
folder/to/file.txt
folder/to/another-file.ext
folder/to/third-file.ext
folder/file.txt
folder/another-file.ext
folder/third-file.ext
file.txt
another-file.ext
third-file.ext
PREFIX
OBJECT
folder/to/path/
folder/to/path/
folder/to/path/
folder/to/
folder/to/
folder/to/
folder/
folder/
folder/
/
/
/
The official repository of the �AWS SDK for PHP
OFFICIAL PAGE
https://aws.amazon.com/sdk-for-php/
GITHUB
https://github.com/aws/aws-sdk-php
DEVELOPER GUIDE https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/welcome.html
The official repository of the �AWS SDK for PHP: how it works?
Take a look at this sample code:
<?php�// Require the Composer autoloader.�require 'vendor/autoload.php';��use Aws\S3\S3Client;��$credentials = new Aws\Credentials\Credentials('key', 'secret');
// Instantiate an Amazon S3 client.�$s3 = new S3Client([� 'credentials' => $credentials� 'version' => 'latest',� 'region' => 'us-west-2'�]);
The official repository of the �AWS SDK for PHP: how it works?
For example try to upload a file:
<?php�// Upload a publicly accessible file. The file size and type are determined by the SDK.
try {� $s3->putObject([� 'Bucket' => 'my-bucket',� 'Key' => 'my-object',� 'Body' => fopen('/path/to/file', 'r'),� 'ACL' => 'public-read',� ]);�} catch (Aws\S3\Exception\S3Exception $e) {� echo "There was an error uploading the file.\n";�}
The official repository of the �AWS SDK for PHP: how it works?
The AWS Client uses the command pattern:
<?php�// This expression is equivalent to the previous one�$s3->getCommand('PutObject', [� 'Bucket' => 'my-bucket',� 'Key' => 'my-object',� 'Body' => fopen('/path/to/file', 'r'),� 'ACL' => 'public-read',�]);
// and so on…�$s3->getCommand('UploadPart', [� ...�]);�
The official repository of the �AWS SDK for PHP: how it works?
Thanks to __call() method invoked in Aws\AwsClientTrait:
<?php�// ...�public function __call($name, array $args)�{� if (substr($name, -5) === 'Async') {� $name = substr($name, 0, -5);� $isAsync = true;� }� if (!empty($this->aliases[ucfirst($name)])) {� $name = $this->aliases[ucfirst($name)];� }� $params = isset($args[0]) ? $args[0] : [];� if (!empty($isAsync)) {� return $this->executeAsync(� $this->getCommand($name, $params)� );� }
return $this->execute($this->getCommand($name, $params));
}
�
Simple S3: a PHP client built on top of official AWS SDK
To instantiate Simple3\Client do the following:
<?php�use SimpleS3\Client;
// To instantiate the Client do the following:�$s3Client = new Client(� 'YOUR_ACCESS_ID',� 'YOUR SECRET_ID',� [� 'version' => 'latest',� 'region' => 'us-west-2',� ]�);
�
composer require mauretto78/simple-s3
Simple S3: how it works?
The magic method __call() in SimpleS3\Client �instantiate a SimpleS3\Command\CommandHandler:
<?php�// ...�public function __call($name, $args)� {� $params = isset($args[0]) ? $args[0] : [];� $commandHandler = 'SimpleS3\\Commands\\Handlers\\'.ucfirst($name);� if (false === class_exists($commandHandler)) {� throw new \InvalidArgumentException($commandHandler . ' is not a valid command name. Please refer to README to get the complete command list.');� }� /** @var CommandHandler $commandHandler */� $commandHandler = new $commandHandler($this);� if ($commandHandler->validateParams($params)) {� return $commandHandler->handle($params);� }� }
�
Simple S3: the public methods list (1)
clearBucket �copyItem�copyInBatch�createBucketIfItDoesNotExist�createFolder�deleteBucket�deleteFolder�deleteItem�downloadItem�enableAcceleration�getBucketLifeCycle�getBucketSize�getCurrentItemVersion�getItem�getItemsInABucket�getPublicItemLink�hasBucket
METHOD
DESCRIPTION
clear a bucket from all files�copy an item from a bucket to another one�copy in batch items from a bucket to another one�create a bucket if it does not exists�create an empty folder in a bucket if it does not exists�delete a bucket�delete a folder�delete an item�download an item�enable the acceleration mode for a bucket�get the bucket lifecycle configuration�get the size (in Bytes) of files in a bucket�get the latest version of an item�get all informations for an item�get an array of items in a bucket�get the public link to download the item�check if a bucket exists
Simple S3: the public methods list (2)
hasFolder�hasItem�isBucketVersioned�openItem�restoreItem�setBucketLifecycleConfiguration �setBucketVersioning�transfer�uploadItem�uploadItemFromBody
METHOD
DESCRIPTION
check if a folder exists�check if an item exists�check if bucket has versioned enabled�get the content of an item�try to restore an item from archive�set bucket lifecycle configuration�set the bucket versioning�transfer content from/to buckets�upload an item to a bucket from a file�upload an item to a bucket from the body content
AWS S3: buckets safe �naming rules
Amazon S3 defines a bucket name as a series of one or more labels, separated by periods, that adhere to the following rules:
Warning
Because S3 allows your bucket to be used as a URL that can be accessed publicly, the bucket name that you choose must be globally unique
Simple S3: create buckets
PUT / HTTP/1.1
Host: BucketName.s3.amazonaws.com
Content-Length: length
Date: date
Authorization: authorization string
$s3Client->createBucketIfItDoesNotExist([� 'bucket' => 'bucket-name', // [REQUIRED]�]);
�
To create a bucket:
You use the createBucketIfItDoesNotExist method, which creates the bucket only if it does not exists:
AWS S3: bucket lifecycle setup
PUT /?lifecycle HTTP/1.1
Host: bucketname.s3.amazonaws.com
Content-Length: length
Date: date
Authorization: authorization string
Content-MD5: MD5
<LifecycleConfiguration>
<Rule>
...
</Rule>
<Rule>
...
</Rule>
...
</LifecycleConfiguration>
To setup properly a bucket lifecycle:
Simple S3: a basic bucket�lifecycle setup
Here is a very simple configuration:
$s3Client->setBucketLifeCycleConfiguration([� 'bucket' => 'bucket', // [REQUIRED]� 'rules' => [ // [OPTIONAL] � [� "ID" => "Move rotated logs to Glacier",� "Prefix" => "rotated/",� "Status" => "Enabled",� "Transitions"=> [� [� "Date" => "2015-11-10T00:00:00.000Z",� "StorageClass" => "GLACIER"� ]� ]� ],� [� "ID" => "Move old versions to Glacier",� "Status" => "Enabled",� "Prefix" => "",� "NoncurrentVersionTransitions" => [� [� "NoncurrentDays" => 2,� "StorageClass" => "GLACIER"� ]� ], � ]� ]�]);
Simple S3: check if a bucket exists
...�// Check if an object exists in a bucket�$s3Client->hasBucket([� 'bucket' => 'your-bucket', // [REQUIRED] �]);
To check if a bucket exists you perform an HEAD request:
HEAD / HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Authorization: authorization string
And you use hasBucket method:
AWS S3: Object Key Naming �Guidelines
You can use any UTF-8 character in an object key name. However, using certain characters in key names may cause problems with some applications and protocols. The following guidelines help you maximize compliance with DNS, web-safe characters, XML parsers, and other APIs.
Alphanumeric characters |
| Special �characters |
|
Safe Characters
The following character sets are generally safe for use in key names:
AWS S3: Object Key Naming �Guidelines
The following are examples of valid object key names:
And these are not valid object key names:
Simple S3: escaping object�names
Escaping object names is entirely up to you.
You can use the provided SimpleS3\Components\Encoders\UrlEncoder class, or inject in Client your own encoder if you prefer, but please note that it MUST implement SimpleS3\Components\Encoders\SafeNameEncoderInterface:
...
use SimpleS3\Components\Encoders\UrlEncoder;
$encoder = new UrlEncoder();
$s3Client->addEncoder($encoder);
AWS S3: upload objects
PUT /my-image.jpg HTTP/1.1
Host: myBucket.s3.amazonaws.com
Date: Wed, 12 Oct 2009 17:50:00 GMT
Authorization: authorization string
Content-Type: text/plain
Content-Length: 11434
x-amz-meta-author: Janet
Expect: 100-continue
[11434 bytes of object data]
This is an example of a complete http request to upload a file to s3:
AWS S3: Multipart �object upload
POST /ObjectName?uploadId=UploadId HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: Date
Content-Length: Size
Authorization: authorization string
<CompleteMultipartUpload>
<Part>
<PartNumber>PartNumber</PartNumber>
<ETag>ETag</ETag>
</Part>
...
</CompleteMultipartUpload>
1) start a multipart upload; 2) upload single parts; 3) finalize the upload
PUT /my-movie.m2ts?partNumber=1&uploadId=VCVsb2FkIElEIGZvciBlbZZpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZR HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: Mon, 1 Nov 2010 20:34:56 GMT
Content-Length: 10485760
Content-MD5: pUNXr/BjKK5G2UKvaRRrOA==
Authorization: authorization string
***part data omitted***
POST /ObjectName?uploads HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date�Authorization: authorization string
Simple S3: upload objects
You can use uploadItem or uploadItemFromBody methods. �For files larger than 6Mb a multipart upload will be done automatically:
$source = __DIR__ .'/support/files/txt/test.txt';
// upload an object from a file
$s3Client->uploadItem([
'bucket' => 'bucket-name', // [REQUIRED]
'key' => 'key.txt', // [REQUIRED]
'source' => $source // [REQUIRED]
]);
// upload an object from a body
$s3Client->uploadItemFromBody([
'bucket' => 'bucket-name', // [REQUIRED]
'key' => 'key.txt', // [REQUIRED]
'body' => 'This is a simple text' // [REQUIRED]
]);
Simple S3: object metadata
You can setup your custom metadata. They MUST start by x-amz-meta-:
PUT /my-image.jpg HTTP/1.1
Host: myBucket.s3.amazonaws.com
Date: Wed, 12 Oct 2009 17:50:00 GMT
Authorization: authorization string
x-amz-meta-your-meta-1: your meta data
x-amz-meta-your-meta-2: your meta data
...�Body omitted
$s3Client->uploadItem([
'bucket' => 'bucket-name', // [REQUIRED]
'key' => 'key.txt', // [REQUIRED]
'source' => $source, // [REQUIRED]
'meta' => [ // [OPTIONAL]
'key' => 'value'
]
]);
You pass the meta array in uploadItem method:
Simple S3: copying objects
To copy an object from a bucket to another one:
PUT /destinationObject HTTP/1.1
Host: destinationBucket.s3.amazonaws.com
x-amz-copy-source: /source_bucket/sourceObject
x-amz-metadata-directive: metadata_directive
x-amz-copy-source-if-match: etag
x-amz-copy-source-if-none-match: etag
x-amz-copy-source-if-unmodified-since: time_stamp
x-amz-copy-source-if-modified-since: time_stamp
<request metadata>
Authorization: authorization string �Date: date
$s3Client->copyItem([� 'source_bucket' => 'source-bucket', // [REQUIRED]� 'source' => 'source-key', // [REQUIRED]� 'target_bucket' => 'target-bucket', // [REQUIRED]� 'target' => 'target-key' // [REQUIRED]�]);
You use copyItem method:
Simple S3: transfer objects
You can use transfer method to transfer file from/to buckets. It’s NOT PERMITTED file transfer between buckets.
This method uses internally \Aws\S3\Transfer to manage the file transfer:
// In this example files were transfered from a bucket to local folder
$source = 's3://your-bucket';
$dest = __DIR__ . '/support/files/transfer';
$s3Client->transfer([
'dest' => $dest, // [REQUIRED]
'source' => $source // [REQUIRED]
]);
Simple S3: restore objects
To restore an object from a storage class:
POST /ObjectName?restore&versionId=VersionID HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Authorization: authorization string
Content-MD5: MD5
<RestoreRequest>
<Days>2</Days>
<GlacierJobParameters>
<Tier>Bulk</Tier>
</GlacierJobParameters>
</RestoreRequest>
$s3Client->restoreItem([� 'bucket' => 'your-bucket', // [REQUIRED] � 'key' => 'to-restore', // [REQUIRED] � 'tier' => 'Expedited', // [OPTIONAL,Expedited|Standard|Bulk] � 'days' => 5, // [OPTIONAL, default is 2] �]);
You use restoreItem method:
Simple S3: list objects in a bucket
To get the list of objects in a bucket:
GET /?list-type=2&delimiter=/ HTTP/1.1
Host: bucket-name.s3.amazonaws.com
x-amz-date: 20160430T235931Z
Authorization: authorization string
...
$s3Client->getItemsInABucket([� 'bucket' => 'your-bucket', // [REQUIRED] � 'prefix' => 'prefix/', // [OPTIONAL]� 'delimiter' => '/', // [OPTIONAL]�]);
You use the getItemsInABucket method. You can specify a prefix:
Simple S3: list objects in a bucket(2)
...
$s3Client->getItemsInABucket([� 'bucket' => 'your-bucket', // [REQUIRED] � 'prefix' => 'prefix/', // [OPTIONAL] � 'hydrate' => false // [OPTIONAL] �]);
// this will return an indexed array of object keys with that prefix:
[� 'prefix/key-1',� 'prefix/key-2',� 'prefix/key-3',� ...�]
With hydrate mode set to false the Client return an indexed array of keys:
Simple S3: list objects in a bucket(3)
...
$s3Client->getItemsInABucket([� 'bucket' => 'your-bucket', // [REQUIRED] � 'prefix' => 'prefix/', // [OPTIONAL] � 'hydrate' => true // [OPTIONAL] �]);
// this will return an associative array of objects with that prefix:
[� 'prefix/key-1' => '{Aws\ResultInterface}',� 'prefix/key-2' => '{Aws\ResultInterface}',� 'prefix/key-3' => '{Aws\ResultInterface}',� ...�]
With hydrate mode set to true the Client return an associative array:
Simple S3: check if an object exists
...�// Check if an object exists in a bucket�$s3Client->hasItem([� 'bucket' => 'your-bucket', // [REQUIRED] � 'key' => 'prefix/to/file.ext', // [REQUIRED]�]);
To check if an object exists you perform an HEAD request:
HEAD /ObjectName HTTP/1.1
Host: BucketName.s3.amazonaws.com
Authorization: authorization string �Date: date
And you use hasItem method:
Simple S3: open files
...�// Open a file�$s3Client->openItem([� 'bucket' => 'your-bucket', // [REQUIRED] � 'key' => 'prefix/to/file.ext', // [REQUIRED]�]);
To get the file full content:
GET /ObjectName HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Authorization: authorization string�Range:bytes=byte_range
You use openItem method:
Simple S3: download files
// Download a file�$s3Client->downloadItem([� 'bucket' => 'your-bucket', // [REQUIRED] � 'key' => 'prefix/to/file.ext', // [REQUIRED]� 'save_as' => 'file-name-txt' // [OPTIONAL]�]);
It’s very easy to download a file.
GET /ObjectName HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Authorization: authorization string�Range:bytes=byte_range
Content-Disposition: attachment; filename="filename.jpg"
You simply have to pass bucket and keyname to downloadItem method:
Simple S3: deleting objects
// Delete an object in a bucket
$s3Client->deleteItem([
'bucket' => 'bucket-name', // [REQUIRED]
'key' => 'keyname', // [REQUIRED]
]);
To delete an object in bucket:
DELETE /ObjectName HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Content-Length: length
Authorization: authorization string
You simply have to pass bucket and keyname to deleteItem method.
Simple S3: deleting and clearing�buckets
$s3Client->deleteBucket([
'bucket' => 'bucket-name' // [REQUIRED]
]);
To delete a bucket:
DELETE / HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Authorization: authorization string
$s3Client->clearBucket([
'bucket' => 'bucket-name' // [REQUIRED]
]);
To clear a bucket from file use clearBucket method:
AWS S3: bucket versioning
bucket
DEFAULT BEHAVIOUR
(1)
(2)
rose.txt
rose.txt
rose.txt
bucket
WITH VERSIONING
(1)
(2)
rose.txt
rose.txt
rose.txt
versionId: 123465
rose.txt
versionId: 1654321
Simple S3: bucket versioning
PUT /?versioning HTTP/1.1
Host: bucket.s3.amazonaws.com
Date: Wed, 01 Mar 2006 12:00:00 GMT
Authorization: authorization string
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>Enabled</Status>
</VersioningConfiguration>
$s3Client->setBucketVersioning([� 'bucket' => 'bucket-name', // [REQUIRED] � 'md5' => 'string' // [OPTIONAL] � 'mfa' => 'auth string' // [OPTIONAL]� 'mfa_delete' => 'Disabled' // [OPTIONAL] Enabled|Disabled� 'status' => 'Enabled' // [OPTIONAL] Enabled|Disabled �]);
�
To setup bucket versioning:
You use setBucketVersioning method:
Simple S3: bucket versioning (2)
Now, when you use getItemsInABucket method, a <VERSION_ID> tag will be added to keys:
...
// getItemsInABucket() will return something like this:
$notHydrated = [
'key<VERSION_ID=123456789>',
'key<VERSION_ID=234567890>',
'key<VERSION_ID=345678901>',
];
$hydrated = [
'key<VERSION_ID=123456789>' => 'content',
'key<VERSION_ID=234567890>' => 'content',
'key<VERSION_ID=345678901>' => 'content',
];
Simple S3: bucket versioning (3)
And now you can use getCurrentItemVersion method to get the latest version of an object:
$s3Client->getCurrentItemVersion([� 'bucket' => 'bucket-name', // [REQUIRED]� 'key' => 'keyname', // [REQUIRED]�]);
�
Simple S3: logging
You can inject your logger to log every time a command is executed. If you enable the logger all the Exceptions are catched and logged.
Please note that your logger MUST be PSR-3 compliant:
use Monolog\Handler\StreamHandler;
use Monolog\Logger;
use SimpleS3\Client;
...
// $logger MUST implement Psr\Log\LoggerInterface
$logger = new Logger('channel-test');
$logger->pushHandler(new StreamHandler(__DIR__.'/../log/test.log', Logger::DEBUG));
$s3Client->addLogger($logger);
Simple S3: caching
In order speed up data retrieval, you can inject a cache handler. �Please note that the cache MUST implement SimpleS3\Components\Cache\CacheInterface. �
use SimpleS3\Components\Cache\RedisCache;
$redis = new Predis\Client();
$cacheAdapter = new RedisCache($redis);
$s3Client->addCache($cacheAdapter);
// this will get keys from cache
$s3Client->getItemsInABucket([
'bucket' => 'your-bucket',
'prefix' => 'prefix/',
'hydrate' => true
]);
// this will EVER get keys from S3
$s3Client->getItemsInABucket(['bucket' => 'your-bucket']);
Simple S3: caching, how it works
HASH
SET OF VALUES
Bucket-name::prefix {"prefix/to/file.ext": ""} � {"prefix/to/file2.ext": ""} � {"prefix/to/file3.ext": ""} � {"prefix/to/file4.ext" ""}
Bucket-name::prefix2 {"prefix2/to/file.ext": ""} � {"prefix2/to/file2.ext" ""} � {"prefix2/to/file3.ext": ""} � {"prefix2/to/file4.ext": ""}
{
md5
Cache system uses Redis hashed lists:
Simple S3: caching, benchmark
Comparison of getItemsInABucket response time:
getItemsInABucket() response time
Simple S3: commands
If you have an application which uses Symfony Console, you have some commands available:
Thank you!
Any questions?
I am here for this :)