TruerWords Logo

Search TruerWords

Sign Up  Log On

“How to Serve Compressed Data with mod_gzip and Apache 1.3 on Mac OS X”

From: Seth Dillingham In Response To: Top of Thread.  
Date Posted: Thursday, November 16, 2006 10:04:07 PM Replies: 0
Enclosures: None.

Some of us have recently been discussing the size of the Prototype library (my preferred JavaScript library for DHTM/AJAX). Proponents of some of the other libraries play up their smaller file sizes, and this is a real issue for some people.

Not that I think it's a serious issue. I don't. It's mostly one of marketing and spin. One way the other libraries keep their sizes down is with the various code-squeezing and obfuscation tools. Effective, but it makes a nightmare of debugging… as if debugging web apps wasn't already hard enough!

Sam is firmly against modifying prototype.js to take advantage of the obfuscators' compression-like features. I'm not sure I agree with his reasoning, but he's certainly right about one thing: the spec on which all this stuff is built (HTTP) has already solved the problem for us via content negotiation. All modern browsers and most servers are already designed to take advantage of it.

Content Negotiation

The general feature which makes it possible is called "content negotiation." Briefly, when the browser requests a URL, it also specifies what flavors of content it's willing to accept, via various headers included with the request. I'm not going to spelunk into the depths of content negotiation here, but suffice to say that the browsers can specify their preferred language (like English or French), MIME types they know how to read (like text/html), and encoding techniques they know how to decode. Here's an example from Firefox:

Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*

See the "Accept-Encoding" line? It says it'll take gzip and deflate (which are essentially the same thing so we're just going to take about gzip).

The browser has told the server that it will accept files which have been "encoded" with gzip. What it calls "encoded," we call "compressed." Prototype.js, which is normally about 52 kb, compresses down to about 12 kb with gzip. Very nice.

mod_gzip is pretty clever, too. If the URL that was requested points to a "static file" (that is, a file on disk rather than a page generated dynamically), then it will look for and serve a file with the same name but with '.gz' tacked onto the end: prototype.js.gz. The browser receives the 12 kb version of prototype.js over the network, and then decompresses it back into the 52 kb version. The result is that the browser gets the same data in a lot less time, and less bandwidth has been used.

Unfortunately, most servers are not configured to take advantage of this. Apache 2 comes with mod_deflate, but it's off by default. The Mac includes Apache 1.3, but that doesn't come with mod_gzip or mod_deflate (mod_deflate is just mod_gzip, updated for Apache 2).

Compile and Install

To install mod_gzip for Apache 1.3 on your Mac, follow these steps (note that I assume you have installed the developer tools on your mac):

  1. Download the source for mod_gzip from sourceforge.

  2. Expand the archive, and move into that directory

  3. Edit the Makefile. Two lines need to be changed:

    • the location of APXS is wrong for most macs

    • the command which runs APXS to build the module is also wrong

  4. run "make" (or you could just run the command we edited in the Makefile, as it's just one line)

  5. run "sudo make install" to install at /usr/libexec/httpd/ and update your httpd.conf (it just adds some comment lines, we'll edit the file ourselves, next)

If you're ready to do it yourself, you can follow along with me here:

Last login: Thu Nov 16 18:05:33 on ttyp1
Welcome to Darwin!

[seth@Outer-Ring ~] (1) $ cd /usr/local/src
[seth@Outer-Ring /usr/local/src] (2) $ curl -O % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 134k 100 134k 0 0 247k 0 --:--:-- --:--:-- --:--:-- 612k
[seth@Outer-Ring /usr/local/src] (3) $ tar -xzf mod_gzip-
[seth@Outer-Ring /usr/local/src] (4) $ cd mod_gzip-
[seth@Outer-Ring /usr/local/src/mod_gzip-] (5) $ vim Makefile    (see below)
[seth@Outer-Ring /usr/local/src/mod_gzip-] (6) $ make /usr/sbin/apxs -c mod_gzip.c mod_gzip_debug.c mod_gzip_compress.c gcc -DDARWIN -DUSE_HSREGEX -DUSE_EXPAT -I../lib/expat-lite -g -Os -pipe -DHARD_SERVER_LIMIT=2048 -DEAPI -DSHARED_MODULE -I/usr/include/httpd -c mod_gzip.c gcc -DDARWIN -DUSE_HSREGEX -DUSE_EXPAT -I../lib/expat-lite -g -Os -pipe -DHARD_SERVER_LIMIT=2048 -DEAPI -DSHARED_MODULE -I/usr/include/httpd -c mod_gzip_debug.c gcc -DDARWIN -DUSE_HSREGEX -DUSE_EXPAT -I../lib/expat-lite -g -Os -pipe -DHARD_SERVER_LIMIT=2048 -DEAPI -DSHARED_MODULE -I/usr/include/httpd -c mod_gzip_compress.c cc -bundle -undefined suppress -flat_namespace -Wl,-bind_at_load -o mod_gzip_compress.o mod_gzip_debug.o mod_gzip.o
[seth@Outer-Ring /usr/local/src/mod_gzip-] (7) $ ls ChangeLog Makefile~ mod_gzip.o mod_gzip_debug.c Makefile docs mod_gzip_debug.h Makefile.libdir mod_gzip.c mod_gzip_compress.c mod_gzip_debug.o Makefile.tmpl mod_gzip.h mod_gzip_compress.o
[seth@Outer-Ring /usr/local/src/mod_gzip-] (8) $ sudo make install Password: **** /usr/sbin/apxs -A -i [preparing module `gzip' in /private/etc/httpd/httpd.conf] cp /usr/libexec/httpd/ chmod 755 /usr/libexec/httpd/ cp /private/etc/httpd/httpd.conf /private/etc/httpd/httpd.conf.bak cp /private/etc/httpd/ /private/etc/httpd/httpd.conf rm /private/etc/httpd/

The Updated Makefile

# this is the original line
#	$(APXS) -Wc,-Wall,-O3,-fomit-frame-pointer,-pipe -c mod_gzip.c mod_gzip_debug.c mod_gzip_compress.c -o
# this line works well for both the PPC and the Intel macs
	$(APXS) -c mod_gzip.c mod_gzip_debug.c mod_gzip_compress.c
# want a Universal binary? use this line instead of the above
#	$(APXS) -Wl,-arch -Wl,i386 -Wl,-arch -Wl,ppc -Wc,-arch -Wc,ppc -Wc,-arch -Wc,i386 -c mod_gzip.c mod_gzip_debug.c mod_gzip_compress.c -o
	$(APXS) -A -i
	$(RM) -rf *.o
distclean: clean
	$(RM) -f

Credit to Marc Liyanage for the command for making a Universal Binary... but be warned that it won't work on pre-Tiger systems (anything before 10.4).


Depending on your architecture (PPC or Intel) and OS (I tested 10.3 and 10.4), you may see various warnings when you run 'make'. Most of them were on Tiger (OS X 10.4), and were related to the mod_gzip headers providing terms which are also defined in headers that mod_gzip #includes in its source. You can safely ignore these warnings.

That's the first half: the module is installed and ready for action. Now we need to update httpd.conf to activate the module, and we'll add some configuration to tell the module what should and should not be compressed.

Configure in httpd.conf

httpd.conf is the Apache's config file. (If you didn't already know that, you're probably in the wrong place...)

  1. Open httpd.conf in your editor of choice (I used BBEdit locally, and vim on my server)

  2. Search for "gzip_module". The first instance is on a line that reads, "#LoadModule gzip_module libexec/httpd/"

    Note: It's essential that mod_gzip be the last module loaded/added. Apache hands control off to the modules in the order they're listed, and data compression must be the very last step.

  3. Uncomment that line.
    (That line was added when we ran 'sudo make install', but it's commented out. Just remove the '#'.)

  4. Find the next line with 'gzip_module'. This one reads, "#AddModule mod_gzip.c"

  5. Uncomment that line, too (same reason as last time)

  6. Add the configuration section for mod_gzip, as shown below, somewhere in httpd.conf. It follows all the standard rules of apache configuration. I put it right after the end of the browser configuration directives (search for a line that says, "# End of browser customization directives")

  7. Save the changes

  8. Restart Apache (either stop/start Personal Web Sharing in the system preferences, or run sudo apachectl graceful

  9. Test it! Any file types that you've said can be handled by mod_gzip should be compressed on the fly (or, if there's a .gz version of the file already available, it will serve that instead). In Firefox, load up any page and type command-i (or control-i on Windows) to see the Page Info window. The file size is shown, and it should be significantly less than the size of the file on disk.

Configuration for mod_gzip

Here's a sample configuration section for the mod_gzip module. None of this is official, and you're unlikely to break anything if you configure your own with different options.

<IfModule mod_gzip.c>
    mod_gzip_on                                     Yes
    mod_gzip_can_negotiate                          Yes
    mod_gzip_static_suffix                          .gz
    AddEncoding                     gzip            .gz
    mod_gzip_update_static                          No
    mod_gzip_command_version                        '/mod_gzip_status'
    mod_gzip_keep_workfiles                         No
    mod_gzip_minimum_file_size                      512
    mod_gzip_maximum_file_size                      1048576
    mod_gzip_maximum_inmem_size                     60000
    mod_gzip_min_http                               1000
    mod_gzip_handle_methods                         GET POST
    mod_gzip_item_include           mime            ^text/.*
    mod_gzip_item_include           mime            ^httpd/unix-directory$
    mod_gzip_item_include           file            \.shtml$
    mod_gzip_item_include           file            \.html$
    mod_gzip_item_include           mime            ^application/x-javascript$
    mod_gzip_item_include           mime            ^application/javascript$
    mod_gzip_item_include           file            \.js$
    mod_gzip_item_include           file            \.css$
    mod_gzip_item_include           mime            ^application/x-httpd-php$
    mod_gzip_item_include           file            \.php$
    mod_gzip_item_include           handler         ^cgi-script$
    mod_gzip_dechunk                                Yes
    mod_gzip_item_exclude           mime            ^image/.$
    mod_gzip_item_exclude           mime            ^image/
    mod_gzip_item_exclude           rspheader       Content-Type:image/*

Not in Conversant… Yet

The observant reader will note that this page was not actually served with gzip encoding. That's because this is being served through Conversant, which doesn't yet run under Apache. My next related task is to add a set of gzip compression verbs to Conversant (Frontier). (Longer term, this won't matter so much, but for now this is the quickest way to add this important bandwidth saving system to my favorite web app platform.)

Note, though, that the CSS file used by [tw] _is_ served 'compressed.' Normally it's 20 KB, but it compresses down to just 4.5 KB. (And that's the size reported by Firefox.)


Comments, corrections, and questions are always welcome! Just post a comment, or send me an email. (I prefer comments here on the site.)

Discussion Thread:

There are no replies.


There are no trackbacks.

is Seth Dillingham's
personal web site.
More than the sum of my parts.