Nginx image processing server with OpenResty and Lua

Today I’ll be showing you how to create a fast on the fly image processing server. The whole system can be created in less than 100 lines of code.

We’ll be using
OpenResty
, an enhanced distribution of Nginx. We’ll also need to write a little bit of
Lua
to get all the functionality we want. Lastly, we’ll be using this
Lua ImageMagick binding
. If you’re not familiar with any of these that’s OK, I’ll be showing you how to get everything running from scratch.

Because the entire system isn’t that many lines of code I’ll show everything first and you can read on if you want to learn more about how it works.

You can also find the code in this
Git repository
.

# These three directives should be tweaked for production
error_log stderr notice;
daemon off;
events { }

http {
  include /usr/local/openresty/nginx/conf/mime.types;

  server {
    listen 80;

    location @image_server {
      content_by_lua_file "serve_image.lua";
    }

    location ~ ^/images/(?[^/]+)/(?[^/]+)/(?.*.(?[a-z_]*))$ {
      root cache;
      set_md5 $digest "$size/$path";
      try_files /$digest.$ext @image_server;
    }
  }

}
local sig, size, path, ext =
  ngx.var.sig, ngx.var.size, ngx.var.path, ngx.var.ext

local secret = "hello_world" -- signature secret key
local images_dir = "images/" -- where images come from
local cache_dir = "cache/" -- where images are cached

local function return_not_found(msg)
  ngx.status = ngx.HTTP_NOT_FOUND
  ngx.header["Content-type"] = "text/html"
  ngx.say(msg or "not found")
  ngx.exit(0)
end

local function calculate_signature(str)
  return ngx.encode_base64(ngx.hmac_sha1(secret, str))
    :gsub("[+/=]", {["+"] = "-", ["/"] = "_", ["="] = ","})
    :sub(1,12)
end

if calculate_signature(size .. "/" .. path) ~= sig then
  return_not_found("invalid signature")
end

local source_fname = images_dir .. path

-- make sure the file exists
local file = io.open(source_fname)

if not file then
  return_not_found()
end

file:close()

local dest_fname = cache_dir .. ngx.md5(size .. "/" .. path) .. "." .. ext

-- resize the image
local magick = require("magick")
magick.thumb(source_fname, size, dest_fname)

ngx.exec(ngx.var.request_uri)

What’s an image processing server?

An image processing server is a web application that is concerned with taking an image path along with a set of manipulation instructions and returning the manipulated image.

A good example is user avatar images. If you let your users upload their own images then the images probably come in a handful of different sizes and formats. When displaying the image you might have it on many different pages and require many different sizes. In order to avoid resizing up front you can use an image processing server to get the image sizes you want on demand just by requesting a special URL.

Additionally, if the URL is requested multiple times the resized image should be cached so it can be returned to the user instantly.

The first step to this project is to design a URL structure. In this tutorial I’ll use the following format:

/images/SIGNATURE/SIZE/PATH

Given an image,
leafo.jpg
, and a desired size,
100x100
, we might request the URL:

/images/abcd123/100x100/leafo.png

You’ll notice I’ve included a section for a signature in the URL. We’ll be using some basic cryptography to ensure a stranger can’t request images of any size. This is an important thing to consider as image processing can take a lot of CPU power. If someone were to write a malicious script that iterates over a large quantity of image sizes they could max out your CPU in an attempt to perform a
Denial-of-service attack
.

The signature is the result of a cryptographic function run on a portion of the URL (the size and path) and a secret key. To verify that the URL of the resized image is valid you need to perform a simple assertion:

assert(calculate_signature("100x100/leafo.png") == "abcd123")

Lastly, it’s worth mentioning image sources and caching. For simplicity the images will be loaded directly from local disk. Although it’s perfectly possible to load them from external places, like S3 or other URLs, it wont be covered in this tutorial.

Modified images will be cached to disk, and cache expiration wont be covered. This is perfectly fine for most cases.

Installation requirements

You can download the latest version of OpenResty from here:
http://openresty.org/#Download

Installation is simple, after extracting the archive just run

$ ./configure --with-luajit
$ make
$ make install

You can find more detailed installation instructions
on the official site
.

On my system this places OpenResty at
/usr/local/openresty/
, so when it comes time to run Nginx we’ll be running:
/usr/local/openresty/nginx/sbin/nginx

OpenResty comes with Lua, so the last component is the
ImageMagick binding
.

If you’re familiar with Lua already, you can use LuaRocks to do the install:

luarocks install --server=http://rocks.moonscript.org magick

If not, you can just copy
this file
into the directory along with your project, renaming it to
magick.lua
.

Nginx configuration

Our Nginx configuration is concerned with serving cached images or executing our Lua script for un-cached images. It’s listed in full at the top of the post, but here I’ll step all the pieces explain their roles.

error_log stderr notice;
daemon off;
events { }

These are basic settings that I use for doing Nginx configuration development. I leave most settings default but I disable the daemon and make sure information is printed to standard out. When deploying your Nginx application you’ll want to spend some time adding some additional directives to make sure it can run as fast as possible.

http {
  include /usr/local/openresty/nginx/conf/mime.types;

The
http
block defines our HTTP settings, the only thing to be done here is include the
mime.types
file. This is a file that comes with Nginx. When Nginx serves a file from disk is uses the mime types configuration file to correctly set the
Content-type
header.

server {
  listen 80;

The
server
block configures our server. For illustrative purposes I’ve bound to port 80 (even though port 80 is the default). In the server block we declare

location

blocks, which are destinations for requests.

location @image_server {
  content_by_lua_file "serve_image.lua";
}

The first location defined is called
@image_server
. The
@
signifies a named location. A named location can not be externally accessed by any URL, it can only be called upon by other locations. We’ll execute this location when the file we want doesn’t exist in the cache.

location ~ ^/images/(?[^/]+)/(?[^/]+)/(?.*.(?[a-z_]*))$ {

This location matches our image URLs. The regular expression here is quite big so let’s step through it.

Following the word
location
is
~
, this instructs Nginx to perform case sensitive regular expression matching against the incoming request path.

As I mentioned before our URL structure is:

/images/SIGNATURE/SIZE/PATH

And the regular expression is:

^/images/(?[^/]+)/(?[^/]+)/(?.*.(?[a-z_]*))$

^/images/
will match the left hand side of the path. Following that is a series of named capture groups that match each part of the request path separated by
/
. Although it would be possible to shorten the regular expression by avoiding the named captures, I decided to use them because they’ll help with the readability of our script.

$(?[^/]+)
is a named capture group. It captures as many characters as it can that aren’t
/
and assigns it to the name
sig
. The
size
named capture works the same.

(?.*.(?[a-z_]*))$
captures the path of our image, which is everything else in the URL. It contains an inner named capture group to extract the extension of the image.

Named capture groups are interesting in Nginx becuase their results can be used directly as variables. For example, if we were using the
echo
module, we could just dump the extension of the request using
echo $ext;
in the Nginx configuration.

Now the body of the location:

root cache;
set_md5 $digest "$size/$path";
try_files /$digest.$ext @image_server;

This will only execute if the regular expression from above matches.

root cache;

This sets the directory of the cache where files will be searched using
try_files
.

set_md5 $digest "$size/$path";

This calculates a hash of the image path and size. The MD5 digest is used as the name of the file in the cache.

try_files /$digest.$ext @image_server;

Finally we use

try_files

to either load the existing image from the cache or pass the request off to the
@image_server
location we defined earlier.

Our Lua script runs when the
@image_server
location is executed. Its job is to verify the signature, ensure the image exists, then resize and serve the image.

The script should be saved in the current directory of Nginx, normally next to the
nginx.conf
. I’ve also called the script
serve_image.lua
. It must match what is referenced in the configuration.

local sig, size, path, ext =
  ngx.var.sig, ngx.var.size, ngx.var.path, ngx.var.ext

local secret = "hello_world" -- signature secret key
local images_dir = "images/" -- where images come from
local cache_dir = "cache/" -- where images are cached

The first step is to set some variables that will be used. The named capture group variables are pulled in as local variables to make their access more convenient.

local function return_not_found(msg)
  ngx.status = ngx.HTTP_NOT_FOUND
  ngx.header["Content-type"] = "text/html"
  ngx.say(msg or "not found")
  ngx.exit(0)
end

If an invalid URL is accessed the server should gracefully show a 404 message. The function
return_not_found
sets the correct status code, prints a message and exits.

local function calculate_signature(str)
  return ngx.encode_base64(ngx.hmac_sha1(secret, str))
    :gsub("[+/=]", {["+"] = "-", ["/"] = "_", ["="] = ","})
    :sub(1,12)
end

This is the function that signs our URL using the secret key. I’ve opted to take the first 12 characters of the base64 encoded result of the HMAC-SHA1.

Additionally I use
gsub
to translate characters that have special meanings in URLs to avoid any potential URL encoding issues.

Now that everything has been declared we can continue on with the logic of the file.

if calculate_signature(size .. "/" .. path) ~= sig then
  return_not_found("invalid signature")
end

Here we verify that the signature is correct. If it’s not what we expect based on the rest of the URL then a 404 is returned.

local source_fname = images_dir .. path

-- make sure the file exists
local file = io.open(source_fname)

if not file then
  return_not_found()
end

file:close()

These lines of check for the existence the file. In Lua we can check if a file is readable by trying to open it. If the file can’t be opened we abort. We don’t need to read the file here so we close.

local dest_fname = cache_dir .. ngx.md5(size .. "/" .. path) .. "." .. ext

-- resize the image
local magick = require("magick")
magick.thumb(source_fname, size, dest_fname)

dest_fname
is set to the same hashed name we searched for in our Nginx configuration. The file can be picked up automatically by Nginx
try_files
on any subsequent requests.

Now that the request has been verified it’s time to do the resize. We pass the size string directly into Magick’s

thumb
function

. This gives us nice syntax for various types of resizes and crops, like
100x100
for a resize, or
10x10+5+5
for a crop.

ngx.exec(ngx.var.request_uri)

Now that the image is written we are ready to display it to the browser. Here I’ve trigger a request to the current location,
request_uri
. Normally this would trigger a loop error but, because we’ve written the cached file,
try_files
will return the file and skip the Lua script.

Running the server

Now were ready to try it out. We’ll run Nginx isolated in its own directory. This directory should start out with
nginx.conf
,
serve_image.lua
, and an
images
directory.

Before starting the server you should place some images in the
images
directory.

You should be inside of the directory where we want the server to run before running the following commands.

Create the cache directory:

$ mkdir cache

Initialize some files our configuration requires for starting:

$ mkdir logs
$ touch logs/error.log

Now start the server:

$ /usr/local/openresty/nginx/sbin/nginx -p "$(pwd)" -c "nginx.conf"

Assuming the server has started we can now access the server. For example, if you have an image
leafo.jpg
you might resize it by going to the following URL:
http://localhost/images/LMzEhc_nPYwX/80×80/leafo.jpg
.

That’s all there is to it. With some minor tweaks to the initialization in
nginx.conf
your server is ready to go live.

There are a couple additional things you could also do:

If you already have an Nginx installation you could integrate this code into it so you don’t have to run separate Nginx processes.

If you are using the image server with another web application you’ll need to write the
calculate_signature
function inside of your application so you can generate valid URLs.

If you’re concerned about the cache taking up too much space with unused image sizes you could look into creating a system that deletes unused cached entries.

Thanks for reading, leave a comment if you any suggestions or are confused about anything.

稿源:leafo.net (源链) | 关于 | 阅读提示

本站遵循[CC BY-NC-SA 4.0]。如您有版权、意见投诉等问题,请通过eMail联系我们处理。
酷辣虫 » 综合编程 » Nginx image processing server with OpenResty and Lua

喜欢 (0)or分享给?

专业 x 专注 x 聚合 x 分享 CC BY-NC-SA 4.0

使用声明 | 英豪名录

登录

忘记密码 ?

切换登录

注册