Hacking at 0300

Using S3 files in PHP

Danny — Tue, 03 Apr 2012 19:37:49 +0000

As I wrote, I'm using Amazon S3 to store files that are too expensive to keep on my web server, with the plan of having frequently-updated files on the server and relatively constant stuff on S3. The address for my S3 server is bililite.s3.amazonaws.com, which is stored in the global variable $_SERVER['CDN'].

So to include a file, I would do:

$filename = '/toinclude.php';
if (file_exists($_SERVER['DOCUMENT_ROOT'].$filename)){
  $filename = $_SERVER['DOCUMENT_ROOT'].$filename;
}else{
  $filename = $_SERVER['CDN'].$filename;
}
include ($filename);

Which I use often enough to want to generalize it into a class.

The other thing that would be useful is a directory listing, which is harder than it sounds for S3 since it has no directory structure; it's just a flat database of keys (the equivalent of filenames) and values (the files themselves). Thus bililite.s3.amazonaws.com/images/silk/add.png has S3 return the file labelled /images/silk/add.png; it has no inherent relationship to /images/silk/delete.png or /images/silk/.

The key is that just retrieving the server URL returns an XML listing of all the files, and there is an API to limit the files returned. bililite.s3.amazonaws.com returns all the files (up to a numerical limit; see below). bililite.s3.amazonaws.com?prefix=images/silk/ returns all the filenames that start with images/silk/ (note no leading slash). That's not quite enough, since it gives us sub-folders as well, but the delimiter parameter tells S3 to group all files that contain the delimiter after the prefix into one entry in the XML list. That's the equivalent of a subfolder. So bililite.s3.amazonaws.com?prefix=images/silk/&delimiter=/ gives us the list we want.

One more subtlety: S3 returns a maximum of 1000 names, then sets a flag in the XML to say the list was truncated. You can then ask for the next 1000 by naming the last returned file.

The documentation is pretty opaque, but it's all in there.

I put it all together into an abstract class that just handles the file part (assuming that any CDN would work the same way, just appending the host name to the file name) and a concrete class that handles the S3-specific directory-simulating parts. See the source code. The method names are meant to parallel the built-in PHP functions.

constructor

$s3 = new S3('http://bililite.s3.amazonaws.com');

realpath

$path = $s3->realpath('/toinclude.php');
include($path);
// or
$content = file_get_contents($path);

returns the real path for the file, either from $_SERVER['DIRECTORY_ROOT'] or the S3 root passed in with the constructor. In other words, if the file exists on the web server, realpath returns something like '/public/www/toinclude.php' and if it does note, returns something like 'http://bililite.s3.amazonaws.com/public/www/toinclude.php'. Note that if the file does not exist on the web server, this will return the path on the S3 root without checking if the file actually exists; use file_exists for that.

isCDN

$flag = $s3->isCDN($s3->realpath('/toinclude.php')); returns FALSE if the path represents a file on the web server (i.e. from $_SERVER['DIRECTORY_ROOT']), TRUE otherwise (note that it does not check if the file actually exists on the S3 server). Note also that this requires the path returned by realpath.

file_exists

$flag= $s3->file_exists('/toinclude.php'); returns TRUE if the file exists on the web server or the S3 server (this does check the S3 server).

filemtime

$timestamp = $s3->filemtime('/toinclude.php'); returns the time the file was last modified.

scandir

$files = $s3->scandir('/images/'); returns an array of names of files that exist either on the web server or the S3 server (it's the union of the directory contents).

This assumes that the ACL (access control list) for the files is set to allow anonymous reading; if not, use Donovan Schönknecht's excellent S3 class. Of course, you'd have to rename one of these classes to avoid the conflict (or use namespaces).

Hope this helps someone.

Using NFS.net with Amazon S3

Danny — Mon, 02 Apr 2012 21:28:22 +0000

Since I started using nearlyfreespeech.net 3 months ago, I've been very pleased. Getting them to set up access with a private key was straightforward and the email support person was prompt, helpful and friendly. The only downsides are the safe mode restrictions, which I have been easily able to work around, and the expensive storage ($1/MB/month), which would add up quickly with all the icons and fonts I'm serving with the webservices.

So I put them onto Amazon's S3 at bililite.s3.amazonaws.com with the intent of using that like a Content Delivery Network (though it isn't really unless I pay for CloudFront as well)—static, large files should come tranparently from S3 while the dynamic site runs on NFS.net.

I do this with a bit of .htaccess hackery. It's harder to create or modify files on S3, so files that are in active developement are on the webserver. I want to serve those files if they exist. Only if the desired files do not exist do I want to get them from S3. Unfortunately, NFS.net does not support mod_proxy, so the redirecting is not transparent (and we can't do things that require same-origin security). But for images and the like, this works:

SetEnvIf Request_URI . CDN=http://bililite.s3.amazonaws.com
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} ^/(images|inc|fonts)/
RewriteRule . %{ENV:CDN}%{REQUEST_URI}

Line 1 creates a variable named CDN. The directive SetEnv would be more natural to use as SetEnv CDN http://bililite.s3.amazonaws.com, but URL rewriting is done before SetEnv runs. The newer SetEnvIf runs early enough for the variable to be used for rewriting, but it's conditional, so we use a dummy condition: REQUEST_URI ., which means "If the requested URI matches any character"

Line 4 tests whether the requested file exists on the server. Only if it does not exist (!-f) is the next line tested, which is whether the file is in any of the CDN directories.

If it is to be redirected, create the new URL by concatenating the CDN variable with the requested URI, which does not contain the protocol or hostname. Thus http://bililite.com/images/silk/add.png has a REQUEST_URI of /images/silk/add.png and the rewritten URL is http://bililite.s3.amazonaws.com/images/silk/add.png.

The advantage of setting a variable in the .htaccess (aside from having the "magic constant" at the top of the file") is that this is passed to the PHP code as imagecreatefrompng($_SERVER['CDN'].'/images/silk/add.png'). So the name of the S3 server is written in just one place, with no need to change multiple files if it changes.

Inline Audio Player

Danny — Mon, 02 Apr 2012 16:08:52 +0000

Download the code.

Demo.

Download the WP Audio Player Standalone.

So the rabbi asked me to add the ability to play the audio on the taamim page (basically, a long list of short MP3's) directly on the page, rather than click the link to open a new page. No problem, right? We're living in an HTML5 world, so I should be able to do:

$('a[href$=mp3]').each(function(){
  $('').attr({src: this.href, controls: 'controls'}).insertBefore(this);
});

And everything ought to work: browsers that can't handle elements get nothing, modern browsers get a modern audio player. Nice progressive enhancement.

But of course, it's not that easy. Webkit (Chrome, Safari) supports MP3 playing, but Firefox does not (and won't), and Internet Explorer only does for IE9 and up, and I have to support Windows XP and IE8 (source; consistent with my experimentation). I don't like the ed players, so I'll go with Flash. I like the player that Language Log uses, and viewing the source tells me that's WPAudioplayer, which has a standalone version that requires just two files, the 11-kb swf file and a 12-kb javascript file.

To use it, include the javascript with a