From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 01:53:17 -0500 From: "William A. Rowe, Jr." <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org, michaelbecker@example.org Subject: Win32 (OS2) /cgi-bin/ exe and bat experiment and observations [ The following text is in the "iso-8859-1" character set. ] [ Your display is set for the "US-ASCII" character set. ] [ Some characters may be displayed incorrectly. ] I was playing with the script-aliased container. I've removed two odder aspects of the server behavior. One, on Win32/OS2 mod_cgi would look for a file of the requested filename + '.exe' tacked on. That was (as the note said) somewhat bogus, and we have a feature to allow it, so I recently removed it. The second, we can't multimatch just because we have a handler. If they don't ask for a given document (script.pl) we shouldn't serve for the handler (cgi-script). Only negotiation variables should be wildcard matched. Of course, we deal with script.pl.en and script.pl.fr for script.pl just fine today, those are language (content) negotated. So I experimented with the following to prevent the "Do you want to download this Program" message in IE, when requesting a script (thirtytwo.exe). I wanted to grab the script as a multimatch, so here's what I tried; <Directory "/Apache2/cgi-bin"> AllowOverride None Options Multiviews <<< note multiviews Order allow,deny Allow from all AddType text/plain .exe <<< serve up as wildcards AddType text/plain .bat </Directory> Here's what thirtytwo.exe (an html cgi) serves up. HEAD /cgi-bin/thirtytwo HTTP/1.0 HTTP/1.1 200 OK Date: Sat, 01 Sep 2001 06:25:31 GMT Server: Apache/2.0.26-dev (Win32) Content-Location: thirtytwo.exe TCN: choice Vary: negotiate Connection: close Content-Type: text/html; charset=ISO-8859-1 So the test essentially worked. It also sends back text/plain as the default, until the script itself sets the headers via util_script.c parsing. The text/plain .bat file worked as well. Of course, docs/cgi-examples/test-cgi.bat offers another example of using the bogus mime header format to tell IE that text/plain content is to be shown, not run. But notice that even that *must* strip off the extention. So this is a rather cool option, I believe. However, it doesn't quite work at this moment, since the redirect/subreq mechanisms all strip of the query args and POST stuff :( Need to finish thinking through that aspect. Trying the same with the 16 bit .exe, I get this 500 response couldn't create child process: 22505: C:/Apache2/cgi-bin/SIXTEEN.EXE :( So we are obviously not dealing yet with older 16 bit exe's in apr. This may following with the Win9x port in general and cgi's (Win9x .bat files are run by the *16* bit command shell!) Just two cents for the night. Need to get this documented - I'll flag my own message to follow up, and take care of it within two weeks or so if nobody else on docs beats me to it ;) I'll start patching next weekend - this is my weekend to play CATCHUP (and I'm not talking about bugs)! Bill From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 01:06:31 -0600 From: Jerry Baker <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: 2.0.25 Problem with SSI I have a script that does this on Windows 2000: foreach (`tracert -h 20 $ip`) { chomp $_; print $_ . "<br>"; } If I include this in an SSI with "include virtual" Apache inserts the HTTP headers into the SSI document as well. From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 02:07:04 -0500 From: "William A. Rowe, Jr." <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: Re: 2.0.25 Problem with SSI [ The following text is in the "iso-8859-1" character set. ] [ Your display is set for the "US-ASCII" character set. ] [ Some characters may be displayed incorrectly. ] Tonight's tree? Ugh. Send it at the list - I'm sure we are doing -one- too many steps with our security overhaul :( ----- Original Message ----- From: "Jerry Baker" <michaelbecker@example.org> To: <michaelbecker@example.org> Sent: Saturday, September 01, 2001 2:06 AM Subject: 2.0.25 Problem with SSI > I have a script that does this on Windows 2000: > > foreach (`tracert -h 20 $ip`) { > chomp $_; > print $_ . "<br>"; > > } > > If I include this in an SSI with "include virtual" Apache inserts the > HTTP headers into the SSI document as well. > From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 14:47:55 +0200 From: Graham Leggett <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c Bill Stoddard wrote: > Yep, you definitely need CACHE_OUT to be a CONTENT filter in this case since INCLUDES is a > CONTENT filter and you need INCLUDES to be run after CACHE_OUT. I disagree - includes is something that should be cached as it is a performance bottleneck. If mod_includes needs to bypass the cache then Cache-Control directives should be applied. If the user specifies "cache this URL range" it would be wrong to override the user's wishes and not cache something. Regards, Graham -- ----------------------------------------- michaelbecker@example.org "There's a moon over Bourbon Street tonight..." [ Part 2, "S/MIME Cryptographic Signature" ] [ Application/X-PKCS7-SIGNATURE 2.7KB. ] [ Unable to print this part. ] From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 14:51:20 +0200 From: Graham Leggett <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Cc: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c Ryan Bloom wrote: > My own opinion is that the cache should be the last content filter run. Basically, > it should probably be specified as the first HTTP_HEADER filter type. The core question is whether we store data in the cache with transfer encodings already applied. Regards, Graham -- ----------------------------------------- michaelbecker@example.org "There's a moon over Bourbon Street tonight..." [ Part 2, "S/MIME Cryptographic Signature" ] [ Application/X-PKCS7-SIGNATURE 2.7KB. ] [ Unable to print this part. ] From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 14:55:24 +0200 From: Graham Leggett <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Cc: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c Bill Stoddard wrote: > I'm a bit fuzzy on how to run the GZIP filter from CACHE_IN but I assume something clean > can be done. Detecting if the client supports unzip and conditionally installing GUNZIP is > simple and should just work without resorting to anything unnatural. The cache_storage.c was designed to be the place where such optimisations happen. Data is passed to it as a brigade, so it should be relatively easy to pass that brigade through specific filters like GZIP (or GUNZIP) before handing the brigades to the caching subfilters. Regards, Graham -- ----------------------------------------- michaelbecker@example.org "There's a moon over Bourbon Street tonight..." [ Part 2, "S/MIME Cryptographic Signature" ] [ Application/X-PKCS7-SIGNATURE 2.7KB. ] [ Unable to print this part. ] From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 12:17:40 -0400 From: Greg Marr <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c [ The following text is in the "ISO-8859-1" character set. ] [ Your display is set for the "US-ASCII" character set. ] [ Some characters may be displayed incorrectly. ] On Sat, 01 Sep 2001 14:47:55 +0200 Graham Leggett <michaelbecker@example.org> wrote: > Bill Stoddard wrote: > > > Yep, you definitely need CACHE_OUT to be a CONTENT filter in this > > case since INCLUDES is a CONTENT filter and you need INCLUDES to > > be run after CACHE_OUT. > > I disagree - includes is something that should be cached as it is a > performance bottleneck. In Ian's particular case, that is incorrect. The value of his includes vary from request to request, so he needs the cache to be before the includes filter. > If the user specifies "cache this URL range" it would be wrong to > override the user's wishes and not cache something. You're saying that we should override the user's wishes and cache the page with the request-specific information instead of before these are added. From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 09:31:30 -0700 From: Ryan Bloom <michaelbecker@example.org> Reply-To: michaelbecker@example.org, michaelbecker@example.org To: Graham Leggett <michaelbecker@example.org> Cc: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c [ The following text is in the "iso-8859-1" character set. ] [ Your display is set for the "US-ASCII" character set. ] [ Some characters may be displayed incorrectly. ] On Saturday 01 September 2001 05:51, Graham Leggett wrote: > Ryan Bloom wrote: > > My own opinion is that the cache should be the last content filter run. > > Basically, it should probably be specified as the first HTTP_HEADER > > filter type. > > The core question is whether we store data in the cache with transfer > encodings already applied. I would put it before the content encodings, but after the content filters. My reasoning is simple. Content encodings tend to be fast, but content filtering tends to be slow. Cache the slow stuff, ignore the fast stuff. Ryan ______________________________________________________________ Ryan Bloom michaelbecker@example.org Covalent Technologies michaelbecker@example.org -------------------------------------------------------------- From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 20:15:33 +0200 From: Graham Leggett <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c Greg Marr wrote: > In Ian's particular case, that is incorrect. The value of his includes > vary from request to request, so he needs the cache to be before the > includes filter. This isn't necessary - simply use the Cache-Control directives correctly so that the includes content is not cached. Remember that there is already a cache (in the form of an ISP transparent cache) between the browser and server anyway (in most cases), mod_cache should behave exactly like a transparent cache would. > You're saying that we should override the user's wishes and cache the > page with the request-specific information instead of before these are > added. No, I'm saying use the Cache-Control directives correctly exactly as a transparent proxy would. Regards, Graham -- ----------------------------------------- michaelbecker@example.org "There's a moon over Bourbon Street tonight..." [ Part 2, "S/MIME Cryptographic Signature" ] [ Application/X-PKCS7-SIGNATURE 2.7KB. ] [ Unable to print this part. ] From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 20:20:47 +0200 From: Graham Leggett <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Cc: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c Ryan Bloom wrote: > > The core question is whether we store data in the cache with transfer > > encodings already applied. > > I would put it before the content encodings, but after the content filters. My > reasoning is simple. Content encodings tend to be fast, but content filtering > tends to be slow. Cache the slow stuff, ignore the fast stuff. But from a keep-it-simple point of view, if we make the cache the very last filter, and then make it behave exactly like a transparent cache, then we have a good chance that it will "just work", because the HTTP protocol defines the correct behavior already and transparent caches are widely deployed. If we start moving it up and down the filter chain, we're likely to introduce some nasty what-if's that we have not encoutered before. Regards, Graham -- ----------------------------------------- michaelbecker@example.org "There's a moon over Bourbon Street tonight..." [ Part 2, "S/MIME Cryptographic Signature" ] [ Application/X-PKCS7-SIGNATURE 2.7KB. ] [ Unable to print this part. ] From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 14:45:07 -0400 From: Greg Marr <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c [ The following text is in the "ISO-8859-1" character set. ] [ Your display is set for the "US-ASCII" character set. ] [ Some characters may be displayed incorrectly. ] Graham Leggett <michaelbecker@example.org> wrote: > Greg Marr wrote: > > > In Ian's particular case, that is incorrect. The value of his > > includes vary from request to request, so he needs the cache to > > be before the includes filter. > > This isn't necessary - simply use the Cache-Control directives > correctly so that the includes content is not cached. How exactly do you use Cache-Control directives so that the content that is cached is before includes are processed, and that when it is retrieved from the cache, the includes are processed? It just doesn't work that way. In this case you have to put the includes before the cache. > mod_cache should behave exactly like a transparent cache would. In your application, yes, not in the one in question. > No, I'm saying use the Cache-Control directives correctly exactly as > a transparent proxy would. That's not the desired behavior. From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 13:03:08 -0600 From: Jerry Baker <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: Re: 2.0.25 Problem with SSI 2.0.25-alpha from http://dev.apache.org/dist "William A. Rowe, Jr." wrote: > > Tonight's tree? Ugh. Send it at the list - I'm sure we are doing -one- > too many steps with our security overhaul :( > > ----- Original Message ----- > From: "Jerry Baker" <michaelbecker@example.org> > To: <michaelbecker@example.org> > Sent: Saturday, September 01, 2001 2:06 AM > Subject: 2.0.25 Problem with SSI > > > I have a script that does this on Windows 2000: > > > > foreach (`tracert -h 20 $ip`) { > > chomp $_; > > print $_ . "<br>"; > > > > } > > > > If I include this in an SSI with "include virtual" Apache inserts the > > HTTP headers into the SSI document as well. > > -- Jerry Baker PGP Key: http://www.jerrybaker.org/pgp.html LAME MP3 Encoder Binaries: http://www.jerrybaker.org/lame/ Apache 2.0 Web server Installer: http://www.jerrybaker.org/apache/ From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 13:39:40 -0700 From: Ryan Bloom <michaelbecker@example.org> Reply-To: michaelbecker@example.org, michaelbecker@example.org To: Graham Leggett <michaelbecker@example.org> Cc: michaelbecker@example.org Subject: Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c [ The following text is in the "iso-8859-1" character set. ] [ Your display is set for the "US-ASCII" character set. ] [ Some characters may be displayed incorrectly. ] On Saturday 01 September 2001 11:20, Graham Leggett wrote: > Ryan Bloom wrote: > > > The core question is whether we store data in the cache with transfer > > > encodings already applied. > > > > I would put it before the content encodings, but after the content > > filters. My reasoning is simple. Content encodings tend to be fast, but > > content filtering tends to be slow. Cache the slow stuff, ignore the fast > > stuff. > > But from a keep-it-simple point of view, if we make the cache the very > last filter, and then make it behave exactly like a transparent cache, > then we have a good chance that it will "just work", because the HTTP > protocol defines the correct behavior already and transparent caches are > widely deployed. > > If we start moving it up and down the filter chain, we're likely to > introduce some nasty what-if's that we have not encoutered before. But keeping it simple would essentially make the cache less useful. If I request a pdf file using three different browsers, the server will most likely have three different copies of the same file. One with byteranges, one with chunks, and one with neither. That is a waste of space in the server, and chunking and byterange are incredibly cheap operations with bucket brigades. We have very well defined locations for filters, and if somebody tries to modify the content after the start of the HTTP_HEADER_FILTER ftype begins, the cache will be the least of our problems. Ryan ______________________________________________________________ Ryan Bloom michaelbecker@example.org Covalent Technologies michaelbecker@example.org -------------------------------------------------------------- From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 13:40:21 -0700 From: Brian Pane <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: mod_include Re: cvs commit: httpd-2.0/modules/experimental mod_cache.c Graham Leggett wrote: >Bill Stoddard wrote: > >>Yep, you definitely need CACHE_OUT to be a CONTENT filter in this case since INCLUDES is a >>CONTENT filter and you need INCLUDES to be run after CACHE_OUT. >> > >I disagree - includes is something that should be cached as it is a >performance bottleneck. > Yes, include processing is a performance bottleneck. But some of us are trying to fix that. :-) mod_include's slowness is due mostly to implementation details, rather than any fundamental architectural limitations. For example, it's not uncommon for mod_include to spend a third of its execution time setting environment variables that are never read. The two patches that I recently posted (one for the environment variable code, the other for the code that splits buckets if the buffered data is too large) should help streamline mod_include so that it spends most of its time doing what one would expect: scanning the content to look for SSI directives. In addition, Ian is working on improvements to the parser logic itself-- e.g., scanning every 5th character, rather than every character, when looking for a '<!--#' token. Collectively, these changes should help bring the speed of parsed requests closer to the speed of unparsed requests. --Brian From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 16:57:23 -0400 (EDT) From: Cliff Woolley <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: Re: 2.0.26? AFAICT, there is only one remaining showstopper problem with 2.0.26-dev, which is that subrequests for relative paths in other directories (ones that result in r->uri being that "INTERNALLY GENERATED" thing) often cause a segfault (mod_include exhibits this problem when doing includes of files in subdirectories, for example). The hack below disables the "INTERNALLY GENERATED" thing, which eliminates the segfault. I figure that's okay since the block of code in question is already hacked with a bunch of #if 0's pending a re-evaluation of how we handle these code paths for subrequests. With the patch below, the server survives everything I've managed to michaelbecker@example.org. If no one objects, I'll apply this for now so we can at least have something fairly stable to play with and hopefully get 2.0.26 nailed down. Any comments? Are there any other problems of which I'm unaware? Jerry, is that problem you were seeing in 2.0.25 fixed in 2.0.26-dev? --Cliff Index: request.c =================================================================== RCS file: /home/cvs/httpd-2.0/server/request.c,v retrieving revision 1.43 diff -u -d -r1.43 request.c --- request.c 2001/08/31 10:58:47 1.43 +++ request.c 2001/08/31 18:38:32 @@ -1567,9 +1567,11 @@ * either, if the base paths match, we can pick up where we leave off. */ +#if 0 if (strncmp(rnew->filename, fdir, fdirlen) == 0 && rnew->filename[fdirlen] && ap_strchr_c(rnew->filename + fdirlen, '/') == NULL) +#endif { char *udir = ap_make_dirstr_parent(rnew->pool, r->uri); @@ -1646,7 +1648,6 @@ r->filename); res = HTTP_FORBIDDEN; } -#endif } else { /* XXX: @@@: What should be done with the parsed_uri values? */ @@ -1658,7 +1659,7 @@ */ rnew->uri = "INTERNALLY GENERATED file-relative req"; -#if 0 /* XXX When this is reenabled, the cache triggers need to be set to faux + /* XXX When this is reenabled, the cache triggers need to be set to faux * dir_walk/file_walk values. */ -------------------------------------------------------------- Cliff Woolley michaelbecker@example.org Charlottesville, VA From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 01 Sep 2001 14:20:50 -0700 From: Ian Holsman <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: Re: 2.0.26? Cliff Woolley wrote: > AFAICT, there is only one remaining showstopper problem with 2.0.26-dev, > which is that subrequests for relative paths in other directories (ones > that result in r->uri being that "INTERNALLY GENERATED" thing) often cause > a segfault (mod_include exhibits this problem when doing includes of files > in subdirectories, for example). The hack below disables the "INTERNALLY > GENERATED" thing, which eliminates the segfault. I figure that's okay > since the block of code in question is already hacked with a bunch of #if > 0's pending a re-evaluation of how we handle these code paths for > subrequests. With the patch below, the server survives everything I've > managed to michaelbecker@example.org. If no one objects, I'll apply this for now so we > can at least have something fairly stable to play with and hopefully get > 2.0.26 nailed down. Any comments? Are there any other problems of which > I'm unaware? Jerry, is that problem you were seeing in 2.0.25 fixed in > 2.0.26-dev? > the perl-framework is complaining about some CGI tests failing. but I seem to remember that this was a problem with the test itself. > --Cliff > > > Index: request.c > =================================================================== > RCS file: /home/cvs/httpd-2.0/server/request.c,v > retrieving revision 1.43 > diff -u -d -r1.43 request.c > --- request.c 2001/08/31 10:58:47 1.43 > +++ request.c 2001/08/31 18:38:32 > @@ -1567,9 +1567,11 @@ > * either, if the base paths match, we can pick up where we leave off. > */ > > +#if 0 > if (strncmp(rnew->filename, fdir, fdirlen) == 0 > && rnew->filename[fdirlen] > && ap_strchr_c(rnew->filename + fdirlen, '/') == NULL) > +#endif > { > char *udir = ap_make_dirstr_parent(rnew->pool, r->uri); > > @@ -1646,7 +1648,6 @@ > r->filename); > res = HTTP_FORBIDDEN; > } > -#endif > } > else { > /* XXX: @@@: What should be done with the parsed_uri values? */ > @@ -1658,7 +1659,7 @@ > */ > rnew->uri = "INTERNALLY GENERATED file-relative req"; > > -#if 0 /* XXX When this is reenabled, the cache triggers need to be set to faux > + /* XXX When this is reenabled, the cache triggers need to be set to faux > * dir_walk/file_walk values. > */ > > > -------------------------------------------------------------- > Cliff Woolley > michaelbecker@example.org > Charlottesville, VA > > > > From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 14:34:13 -0700 From: Ryan Bloom <michaelbecker@example.org> Reply-To: michaelbecker@example.org, michaelbecker@example.org To: michaelbecker@example.org, Cliff Woolley <michaelbecker@example.org> Subject: Re: 2.0.26? [ The following text is in the "iso-8859-1" character set. ] [ Your display is set for the "US-ASCII" character set. ] [ Some characters may be displayed incorrectly. ] On Saturday 01 September 2001 13:57, Cliff Woolley wrote: I would personally rather see us spend a week or two and get this stuff done right and then roll 2.0.26, than keep hacking this to pieces. I think we are far better off taking a deep breath, finishing this work cleanly, and then moving forward. But, that is just my opinion, and I won't stop a release if it is rolled. I won't however, vote +1 for beta. Ryan > AFAICT, there is only one remaining showstopper problem with 2.0.26-dev, > which is that subrequests for relative paths in other directories (ones > that result in r->uri being that "INTERNALLY GENERATED" thing) often cause > a segfault (mod_include exhibits this problem when doing includes of files > in subdirectories, for example). The hack below disables the "INTERNALLY > GENERATED" thing, which eliminates the segfault. I figure that's okay > since the block of code in question is already hacked with a bunch of #if > 0's pending a re-evaluation of how we handle these code paths for > subrequests. With the patch below, the server survives everything I've > managed to michaelbecker@example.org. If no one objects, I'll apply this for now so we > can at least have something fairly stable to play with and hopefully get > 2.0.26 nailed down. Any comments? Are there any other problems of which > I'm unaware? Jerry, is that problem you were seeing in 2.0.25 fixed in > 2.0.26-dev? > > --Cliff > > > Index: request.c > =================================================================== > RCS file: /home/cvs/httpd-2.0/server/request.c,v > retrieving revision 1.43 > diff -u -d -r1.43 request.c > --- request.c 2001/08/31 10:58:47 1.43 > +++ request.c 2001/08/31 18:38:32 > @@ -1567,9 +1567,11 @@ > * either, if the base paths match, we can pick up where we leave off. > */ > > +#if 0 > if (strncmp(rnew->filename, fdir, fdirlen) == 0 > && rnew->filename[fdirlen] > && ap_strchr_c(rnew->filename + fdirlen, '/') == NULL) > +#endif > { > char *udir = ap_make_dirstr_parent(rnew->pool, r->uri); > > @@ -1646,7 +1648,6 @@ > r->filename); > res = HTTP_FORBIDDEN; > } > -#endif > } > else { > /* XXX: @@@: What should be done with the parsed_uri values? */ > @@ -1658,7 +1659,7 @@ > */ > rnew->uri = "INTERNALLY GENERATED file-relative req"; > > -#if 0 /* XXX When this is reenabled, the cache triggers need to be set to > faux + /* XXX When this is reenabled, the cache triggers need to be > set to faux * dir_walk/file_walk values. > */ > > > -------------------------------------------------------------- > Cliff Woolley > michaelbecker@example.org > Charlottesville, VA -- ______________________________________________________________ Ryan Bloom michaelbecker@example.org Covalent Technologies michaelbecker@example.org -------------------------------------------------------------- From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 14:37:21 -0700 From: Justin Erenkrantz <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org, michaelbecker@example.org Cc: Cliff Woolley <michaelbecker@example.org> Subject: Re: 2.0.26? On Sat, Sep 01, 2001 at 02:34:13PM -0700, Ryan Bloom wrote: > On Saturday 01 September 2001 13:57, Cliff Woolley wrote: > > I would personally rather see us spend a week or two and get this stuff done > right and then roll 2.0.26, than keep hacking this to pieces. I think we are > far better off taking a deep breath, finishing this work cleanly, and then > moving forward. But, that is just my opinion, and I won't stop a release if > it is rolled. I won't however, vote +1 for beta. What Ryan said. Let's fix this first (the right way) and then we can do a release. My $.02. -- justin From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 14:56:45 -0700 From: Justin Erenkrantz <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: [PATCH] Add mod_gz to httpd-2.0 Ian has posted his mod_gz filter before, now I'd like to give it a +1. I told him I'd look at it a while ago, but never got a chance to do so. So, I spent this morning cleaning up the configuration and a bit of the code to fit our style (nothing major). I'd like to add this to the modules/filters directory (which seems like the most appropriate place). Can I get two other +1s? I've reviewed the code and can get it confirmed working with Netscape 4.77 and Mozilla 0.9.3 by adding the following to httpd.conf: <IfModule mod_gz.c> GZFilter On AddOutputFilter GZ html </IfModule> We could remove GZFilter as it really serves no purpose as well as the text/html check in mod_gz. I'd like to commit something that is close to what Ian originally submitted and then tweak it slightly. (Interesting to note that Netscape 4.77 does not allow you to view the source of a gzipped'd entity while Mozilla shows you the decompressed entity. Mozilla is getting cool...) I'm sure we can do more analysis of its performance (what the appropriate deflation settings should be), but I'd really to get this in first. =-) Please test and report back... -- justin Index: config.m4 =================================================================== RCS file: /home/cvs/httpd-2.0/modules/filters/config.m4,v retrieving revision 1.6 diff -u -r1.6 config.m4 --- config.m4 2001/05/12 03:48:31 1.6 +++ config.m4 2001/09/01 21:38:16 @@ -6,6 +6,55 @@ APACHE_MODULE(include, Server Side Includes, , , yes) +APACHE_MODULE(gz, GZip encoding support, "mod_gz.lo", , most, [ + AC_ARG_WITH(z, [ --with-z=DIR use a specific zlib library], + [ + if test "x$withval" != "xyes" && test "x$withval" != "x"; then + ap_zlib_base="$withval" + fi + ]) + if test "x$ap_zlib_base" = "x"; then + AC_MSG_CHECKING([for zlib location]) + AC_CACHE_VAL(ap_cv_zlib,[ + for dir in /usr/local /usr ; do + if test -d $dir && test -f $dir/include/zlib.h; then + ap_cv_zlib=$dir + break + fi + done + ]) + ap_zlib_base=$ap_cv_zlib + if test "x$ap_zlib_base" = "x"; then + enable_gz=no + AC_MSG_RESULT([not found]) + else + AC_MSG_RESULT([$ap_zlib_base]) + fi + fi + if test "$enable_gz" != "no"; then + ap_save_includes=$INCLUDE + ap_save_ldflags=$LDFLAGS + ap_save_libs=$LIBS + if test "$ap_zlib_base" != "/usr"; then + APR_ADDTO(INCLUDES, [-I${ap_zlib_base}/include]) + APR_ADDTO(LDFLAGS, [-L${ap_zlib_base}/lib]) + if test "x$ap_platform_runtime_link_flag" != "x"; then + APR_ADDTO(LDFLAGS, [$ap_platform_runtime_link_flag${ap_zlib_Base}/lib]) + fi + fi + APR_ADDTO(LIBS, [-lz]) + AC_MSG_CHECKING([for zlib library]) + AC_TRY_LINK([#include <zlib.h>], [return Z_OK;], + [AC_MSG_RESULT(found) + AC_CHECK_HEADERS(zutil.h)], + [AC_MSG_RESULT(not found) + enable_gz=no + INCLUDES=$ap_save_includes + LDFLAGS=$ap_save_ldflags + LIBS=$ap_save_libs]) + fi +]) + APR_ADDTO(LT_LDFLAGS,-export-dynamic) APACHE_MODPATH_FINISH Index: mod_gz.c =================================================================== /* ==================================================================== * The Apache Software License, Version 1.1 * * Copyright (c) 2000-2001 The Apache Software Foundation. All rights * reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * 3. The end-user documentation included with the redistribution, * if any, must include the following acknowledgment: * "This product includes software developed by the * Apache Software Foundation (http://www.apache.org/)." * Alternately, this acknowledgment may appear in the software itself, * if and wherever such third-party acknowledgments normally appear. * * 4. The names "Apache" and "Apache Software Foundation" must * not be used to endorse or promote products derived from this * software without prior written permission. For written * permission, please contact michaelbecker@example.org. * * 5. Products derived from this software may not be called "Apache", * nor may "Apache" appear in their name, without prior written * permission of the Apache Software Foundation. * * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE * DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION OR * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * ==================================================================== * * This software consists of voluntary contributions made by many * individuals on behalf of the Apache Software Foundation. For more * information on the Apache Software Foundation, please see * <http://www.apache.org/>. * * Portions of this software are based upon public domain software * (zlib functions gz_open and gzwrite) */ /* * mod_gz.c: GZip's HTML content on the fly * * Written by Ian Holsman (michaelbecker@example.org) * */ #include "httpd.h" #include "http_config.h" #include "http_log.h" #include "apr_strings.h" #include "apr_general.h" #include "util_filter.h" #include "apr_buckets.h" #include "http_request.h" #include "zlib.h" #ifdef HAVE_ZUTIL_H #include "zutil.h" #else /* As part of the encoding process, we must send what our OS_CODE is * (or so it seems based on what I can tell of how gzip encoding works). * * zutil.h is not always included with zlib distributions (it is a private * header), so this is straight from zlib 1.1.3's zutil.h. - JRE */ #ifdef OS2 # define OS_CODE 0x06 #endif #ifdef WIN32 /* Window 95 & Windows NT */ # define OS_CODE 0x0b #endif #if defined(VAXC) || defined(VMS) # define OS_CODE 0x02 #endif #ifdef AMIGA # define OS_CODE 0x01 #endif #if defined(ATARI) || defined(atarist) # define OS_CODE 0x05 #endif #if defined(MACOS) || defined(TARGET_OS_MAC) # define OS_CODE 0x07 #endif #ifdef __50SERIES /* Prime/PRIMOS */ # define OS_CODE 0x0F #endif #ifdef TOPS20 # define OS_CODE 0x0a #endif #ifndef OS_CODE # define OS_CODE 0x03 /* assume Unix */ #endif #endif static const char s_szGZFilterName[] = "GZ"; module AP_MODULE_DECLARE_DATA gz_module; typedef struct { int bEnabled; int windowSize; int memlevel; char*noteName; } GZFilterConfig; /* windowsize is negative to suppress Zlib header */ #define GZFILTER_DEFAULT_WINDOWSIZE -15 #define GZFILTER_DEFAULT_MEMLEVEL 9 #define GZFILTER_BUFFERSIZE 8096 /*#define GZFILTER_COMPRESSION "GZCOMPRESSION" */ typedef struct gz_ctx { z_stream strm; char buffer[GZFILTER_BUFFERSIZE]; unsigned long crc; } gz_ctx; /* Outputs a long in LSB order to the given file * only the bottom 4 bits are required for the GZ file format. */ static void putLong(char *string, unsigned long x) { int n; for (n = 0; n < 4; n++) { string[n] = (int) (x & 0xff); x >>= 8; } } static void *GZFilterCreateServerConfig(apr_pool_t * p, server_rec * s) { GZFilterConfig *pConfig = apr_pcalloc(p, sizeof *pConfig); pConfig->bEnabled = 0; pConfig->memlevel = GZFILTER_DEFAULT_MEMLEVEL; pConfig->windowSize = GZFILTER_DEFAULT_WINDOWSIZE; return pConfig; } static const char *GZSetWindowSize(cmd_parms * cmd, void *dummy, const char* arg) { GZFilterConfig *pConfig = ap_get_module_config(cmd->server->module_config, &gz_module); int iWindowSize; iWindowSize = atoi(arg); if (iWindowSize <1 || iWindowSize >15) return "GZWindowSize must be between 1 and 15"; pConfig->windowSize = iWindowSize *-1; return NULL; } static const char *GZSetNote(cmd_parms * cmd, void *dummy, const char* arg) { GZFilterConfig *pConfig = ap_get_module_config(cmd->server->module_config, &gz_module); pConfig->noteName=apr_pstrdup(cmd->pool,arg); return NULL; } static const char *GZSetMemLevel(cmd_parms * cmd, void *dummy, const char* arg) { GZFilterConfig *pConfig = ap_get_module_config(cmd->server->module_config, &gz_module); int iMemLevel; iMemLevel = atoi(arg); if (iMemLevel <1 || iMemLevel >9) return "GZMemLevel must be between 1 and 9"; pConfig->memlevel = iMemLevel; return NULL; } static const char *GZFilterEnable(cmd_parms * cmd, void *dummy, int arg) { GZFilterConfig *pConfig = ap_get_module_config(cmd->server->module_config, &gz_module); pConfig->bEnabled = arg; return NULL; } static void GZFilterInsertFilter(request_rec * r) { GZFilterConfig *pConfig = ap_get_module_config(r->server->module_config, &gz_module); if (!pConfig->bEnabled) return; ap_add_output_filter(s_szGZFilterName, NULL, r, r->connection); } static int gz_magic[2] = { 0x1f, 0x8b }; /* gzip magic header */ static apr_status_t GZFilterOutFilter(ap_filter_t * f, apr_bucket_brigade * pbbIn) { apr_bucket *pbktIn; apr_bucket_brigade *pbbOut; apr_bucket *pbktOut; char *buf; char *b; const char *accepts; request_rec *r = f->r; gz_ctx *ctx; char *token = NULL; int zRC; GZFilterConfig *pConfig = ap_get_module_config(r->server->module_config, &gz_module); if ( pConfig->bEnabled != 1 ) return ap_pass_brigade(f->next, pbbIn); /* only work on main request/no subrequests */ if (r->main) return ap_pass_brigade(f->next, pbbIn); /* GETs only (for the moment) */ if (r->method_number != M_GET) { return ap_pass_brigade(f->next, pbbIn); } /* only compress text/html files */ if (strncmp(r->content_type, "text/html", 9)) return ap_pass_brigade(f->next, pbbIn); /* some browsers might have problems, so set no-gzip (with browsermatch) * for them */ if (apr_table_get(r->subprocess_env, "no-gzip")) { return ap_pass_brigade(f->next, pbbIn); } /* if they don't have the line, then they can't play */ accepts = apr_table_get(r->headers_in, "Accept-Encoding"); if (accepts == NULL) { return ap_pass_brigade(f->next, pbbIn); } token = ap_get_token(r->pool, &accepts, 0); while (token && token[0] && strcmp(token, "gzip")) { accepts++; /* skip token */ token = ap_get_token(r->pool, &accepts, 0); } if (token == NULL || token[0] == '\0') { return ap_pass_brigade(f->next, pbbIn); } pbbOut = apr_brigade_create(f->r->pool); if (!f->ctx) { f->ctx = apr_pcalloc(f->c->pool, sizeof(*ctx)); ctx = f->ctx; /* ctx->strm.zalloc = (alloc_func) 0; ctx->strm.zfree = (free_func) 0; ctx->strm.opaque = (voidpf) 0; ctx->crc = 0L; */ zRC = deflateInit2(&ctx->strm, Z_BEST_SPEED, Z_DEFLATED, pConfig->windowSize, pConfig->memlevel, Z_DEFAULT_STRATEGY); if (zRC != Z_OK) { f->ctx = NULL; ap_log_rerror(APLOG_MARK, APLOG_ERR | APLOG_NOERRNO, 0, r, "unable to init Zlib: deflateInit2 returned %d: URL %s", zRC, r->uri); return ap_pass_brigade(f->next, pbbIn); } buf = apr_psprintf(r->pool, "%c%c%c%c%c%c%c%c%c%c", gz_magic[0], gz_magic[1], Z_DEFLATED, 0 /*flags */ , 0, 0, 0, 0 /*time */ , 0 /*xflags */ , OS_CODE); pbktOut = apr_bucket_pool_create(buf, 10, r->pool); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); apr_table_setn(r->headers_out, "Content-Encoding", "gzip"); apr_table_setn(r->headers_out, "Vary", "Accept-Encoding"); } else { ctx = f->ctx; } APR_BRIGADE_FOREACH(pbktIn, pbbIn) { const char *data; apr_size_t lenin; unsigned int len; char *buf; char szCRC[4]; char szLen[4]; apr_size_t e_wrt; int done = 0; if (APR_BUCKET_IS_EOS(pbktIn)) { apr_bucket *pbktEOS = apr_bucket_eos_create(); ctx->strm.avail_in = 0; /* should be zero already anyway */ for (;;) { len = GZFILTER_BUFFERSIZE - ctx->strm.avail_out; if (len != 0) { pbktOut = apr_bucket_heap_create(ctx->buffer, len, 1, &e_wrt); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); ctx->strm.next_out = ctx->buffer; ctx->strm.avail_out = GZFILTER_BUFFERSIZE; } if (done) break; zRC = deflate(&ctx->strm, Z_FINISH); if (len == 0 && zRC == Z_BUF_ERROR) zRC = Z_OK; done = (ctx->strm.avail_out != 0 || zRC == Z_STREAM_END); if (zRC != Z_OK && zRC != Z_STREAM_END) break; } putLong(szCRC, ctx->crc); putLong(szLen, ctx->strm.total_in); buf = apr_palloc(r->pool, 8); b=buf; *b++=szCRC[0]; *b++=szCRC[1]; *b++=szCRC[2]; *b++=szCRC[3]; *b++=szLen[0]; *b++=szLen[1]; *b++=szLen[2]; *b++=szLen[3]; pbktOut = apr_bucket_pool_create(buf, 8, r->pool); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); ap_log_rerror(APLOG_MARK, APLOG_DEBUG | APLOG_NOERRNO, 0, r, "Zlib: Compressed %ld to %ld : URL %s", ctx->strm.total_in, ctx->strm.total_out, r->uri); if (pConfig->noteName) { if (ctx->strm.total_in >0) { apr_table_setn(r->notes, pConfig->noteName, apr_itoa(r->pool, (ctx->strm.total_out*100/ctx->strm.total_in))); } else { apr_table_setn(r->notes,pConfig->noteName,"-"); } } deflateEnd(&ctx->strm); f->ctx = NULL; APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktEOS); break; } if (APR_BUCKET_IS_FLUSH(pbktIn)) { /* XXX FIX: do we need the Content-Size set, or can we stream? * we should be able to stream */ /* ignore flush buckets for the moment.. we can't stream as we * need the size ;( */ continue; } /* read */ apr_bucket_read(pbktIn, &data, &lenin, APR_BLOCK_READ); ctx->crc = crc32(ctx->crc, (const Bytef *) data, lenin); /* write */ ctx->strm.next_in = (char *) data; ctx->strm.avail_in = lenin; ctx->strm.next_out = ctx->buffer; ctx->strm.avail_out = GZFILTER_BUFFERSIZE; while (ctx->strm.avail_in != 0) { if (ctx->strm.avail_out == 0) { ctx->strm.next_out = ctx->buffer; len = GZFILTER_BUFFERSIZE - ctx->strm.avail_out; pbktOut = apr_bucket_heap_create(ctx->buffer, len, 1, &e_wrt); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); ctx->strm.avail_out = GZFILTER_BUFFERSIZE; } zRC = deflate(&(ctx->strm), Z_NO_FLUSH); if (zRC != Z_OK) break; } } return ap_pass_brigade(f->next, pbbOut); } static void GZFilterRegisterHooks(apr_pool_t * p) { /* static const char * const aszPost[] = { "CHUNK", NULL }; * ap_hook_insert_filter(GZFilterInsertFilter, NULL, aszPost, APR_HOOK_LAST); */ ap_register_output_filter(s_szGZFilterName, GZFilterOutFilter, AP_FTYPE_HTTP_HEADER); } static const command_rec GZFilterCmds[] = { AP_INIT_FLAG("GZFilter", GZFilterEnable, NULL, RSRC_CONF, "Run a GZ filter on this host"), AP_INIT_TAKE1("GZFilterNote", GZSetNote, NULL, RSRC_CONF, "Set a note to report on compression ratio"), AP_INIT_TAKE1("GZWindowSize", GZSetWindowSize, NULL, RSRC_CONF, "Set the Zlib window size (1-15)"), AP_INIT_TAKE1("GZMemLevel", GZSetMemLevel, NULL, RSRC_CONF, "Set the Zlib Memory Level (1-9)"), {NULL} }; module AP_MODULE_DECLARE_DATA gz_module = { STANDARD20_MODULE_STUFF, NULL, NULL, GZFilterCreateServerConfig, NULL, GZFilterCmds, GZFilterRegisterHooks }; From michaelbecker@example.org Tue Sep 4 18:37:30 2001 Date: Sat, 1 Sep 2001 18:07:58 -0600 From: Peter J. Cranstone <michaelbecker@example.org> Reply-To: michaelbecker@example.org To: michaelbecker@example.org Subject: RE: [PATCH] Add mod_gz to httpd-2.0 Just a heads up for this forum. We have finished porting mod_gzip to Apace 2.0 but are holding off on releasing it until Apache goes solid beta. There are still a lot of unresolved issues and would rather see the 2.x more stable before releasing. Mod_gzip for Apache 2.0 is based on mod_gzip for Apache 1.x and allows those users with existing configs to use them in 2.x Regards Peter J. Cranstone -----Original Message----- From: Justin Erenkrantz [mailto:michaelbecker@example.org] Sent: Saturday, September 01, 2001 3:57 PM To: michaelbecker@example.org Subject: [PATCH] Add mod_gz to httpd-2.0 Ian has posted his mod_gz filter before, now I'd like to give it a +1. I told him I'd look at it a while ago, but never got a chance to do so. So, I spent this morning cleaning up the configuration and a bit of the code to fit our style (nothing major). I'd like to add this to the modules/filters directory (which seems like the most appropriate place). Can I get two other +1s? I've reviewed the code and can get it confirmed working with Netscape 4.77 and Mozilla 0.9.3 by adding the following to httpd.conf: <IfModule mod_gz.c> GZFilter On AddOutputFilter GZ html </IfModule> We could remove GZFilter as it really serves no purpose as well as the text/html check in mod_gz. I'd like to commit something that is close to what Ian originally submitted and then tweak it slightly. (Interesting to note that Netscape 4.77 does not allow you to view the source of a gzipped'd entity while Mozilla shows you the decompressed entity. Mozilla is getting cool...) I'm sure we can do more analysis of its performance (what the appropriate deflation settings should be), but I'd really to get this in first. =-) Please test and report back... -- justin Index: config.m4 =================================================================== RCS file: /home/cvs/httpd-2.0/modules/filters/config.m4,v retrieving revision 1.6 diff -u -r1.6 config.m4 --- config.m4 2001/05/12 03:48:31 1.6 +++ config.m4 2001/09/01 21:38:16 @@ -6,6 +6,55 @@ APACHE_MODULE(include, Server Side Includes, , , yes) +APACHE_MODULE(gz, GZip encoding support, "mod_gz.lo", , most, [ + AC_ARG_WITH(z, [ --with-z=DIR use a specific zlib library], + [ + if test "x$withval" != "xyes" && test "x$withval" != "x"; then + ap_zlib_base="$withval" + fi + ]) + if test "x$ap_zlib_base" = "x"; then + AC_MSG_CHECKING([for zlib location]) + AC_CACHE_VAL(ap_cv_zlib,[ + for dir in /usr/local /usr ; do + if test -d $dir && test -f $dir/include/zlib.h; then + ap_cv_zlib=$dir + break + fi + done + ]) + ap_zlib_base=$ap_cv_zlib + if test "x$ap_zlib_base" = "x"; then + enable_gz=no + AC_MSG_RESULT([not found]) + else + AC_MSG_RESULT([$ap_zlib_base]) + fi + fi + if test "$enable_gz" != "no"; then + ap_save_includes=$INCLUDE + ap_save_ldflags=$LDFLAGS + ap_save_libs=$LIBS + if test "$ap_zlib_base" != "/usr"; then + APR_ADDTO(INCLUDES, [-I${ap_zlib_base}/include]) + APR_ADDTO(LDFLAGS, [-L${ap_zlib_base}/lib]) + if test "x$ap_platform_runtime_link_flag" != "x"; then + APR_ADDTO(LDFLAGS, [$ap_platform_runtime_link_flag${ap_zlib_Base}/lib]) + fi + fi + APR_ADDTO(LIBS, [-lz]) + AC_MSG_CHECKING([for zlib library]) + AC_TRY_LINK([#include <zlib.h>], [return Z_OK;], + [AC_MSG_RESULT(found) + AC_CHECK_HEADERS(zutil.h)], + [AC_MSG_RESULT(not found) + enable_gz=no + INCLUDES=$ap_save_includes + LDFLAGS=$ap_save_ldflags + LIBS=$ap_save_libs]) + fi +]) + APR_ADDTO(LT_LDFLAGS,-export-dynamic) APACHE_MODPATH_FINISH Index: mod_gz.c =================================================================== /* ==================================================================== * The Apache Software License, Version 1.1 * * Copyright (c) 2000-2001 The Apache Software Foundation. All rights * reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * 3. The end-user documentation included with the redistribution, * if any, must include the following acknowledgment: * "This product includes software developed by the * Apache Software Foundation (http://www.apache.org/)." * Alternately, this acknowledgment may appear in the software itself, * if and wherever such third-party acknowledgments normally appear. * * 4. The names "Apache" and "Apache Software Foundation" must * not be used to endorse or promote products derived from this * software without prior written permission. For written * permission, please contact michaelbecker@example.org. * * 5. Products derived from this software may not be called "Apache", * nor may "Apache" appear in their name, without prior written * permission of the Apache Software Foundation. * * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE * DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION OR * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * ==================================================================== * * This software consists of voluntary contributions made by many * individuals on behalf of the Apache Software Foundation. For more * information on the Apache Software Foundation, please see * <http://www.apache.org/>. * * Portions of this software are based upon public domain software * (zlib functions gz_open and gzwrite) */ /* * mod_gz.c: GZip's HTML content on the fly * * Written by Ian Holsman (michaelbecker@example.org) * */ #include "httpd.h" #include "http_config.h" #include "http_log.h" #include "apr_strings.h" #include "apr_general.h" #include "util_filter.h" #include "apr_buckets.h" #include "http_request.h" #include "zlib.h" #ifdef HAVE_ZUTIL_H #include "zutil.h" #else /* As part of the encoding process, we must send what our OS_CODE is * (or so it seems based on what I can tell of how gzip encoding works). * * zutil.h is not always included with zlib distributions (it is a private * header), so this is straight from zlib 1.1.3's zutil.h. - JRE */ #ifdef OS2 # define OS_CODE 0x06 #endif #ifdef WIN32 /* Window 95 & Windows NT */ # define OS_CODE 0x0b #endif #if defined(VAXC) || defined(VMS) # define OS_CODE 0x02 #endif #ifdef AMIGA # define OS_CODE 0x01 #endif #if defined(ATARI) || defined(atarist) # define OS_CODE 0x05 #endif #if defined(MACOS) || defined(TARGET_OS_MAC) # define OS_CODE 0x07 #endif #ifdef __50SERIES /* Prime/PRIMOS */ # define OS_CODE 0x0F #endif #ifdef TOPS20 # define OS_CODE 0x0a #endif #ifndef OS_CODE # define OS_CODE 0x03 /* assume Unix */ #endif #endif static const char s_szGZFilterName[] = "GZ"; module AP_MODULE_DECLARE_DATA gz_module; typedef struct { int bEnabled; int windowSize; int memlevel; char*noteName; } GZFilterConfig; /* windowsize is negative to suppress Zlib header */ #define GZFILTER_DEFAULT_WINDOWSIZE -15 #define GZFILTER_DEFAULT_MEMLEVEL 9 #define GZFILTER_BUFFERSIZE 8096 /*#define GZFILTER_COMPRESSION "GZCOMPRESSION" */ typedef struct gz_ctx { z_stream strm; char buffer[GZFILTER_BUFFERSIZE]; unsigned long crc; } gz_ctx; /* Outputs a long in LSB order to the given file * only the bottom 4 bits are required for the GZ file format. */ static void putLong(char *string, unsigned long x) { int n; for (n = 0; n < 4; n++) { string[n] = (int) (x & 0xff); x >>= 8; } } static void *GZFilterCreateServerConfig(apr_pool_t * p, server_rec * s) { GZFilterConfig *pConfig = apr_pcalloc(p, sizeof *pConfig); pConfig->bEnabled = 0; pConfig->memlevel = GZFILTER_DEFAULT_MEMLEVEL; pConfig->windowSize = GZFILTER_DEFAULT_WINDOWSIZE; return pConfig; } static const char *GZSetWindowSize(cmd_parms * cmd, void *dummy, const char* arg) { GZFilterConfig *pConfig = ap_get_module_config(cmd->server->module_config, &gz_module); int iWindowSize; iWindowSize = atoi(arg); if (iWindowSize <1 || iWindowSize >15) return "GZWindowSize must be between 1 and 15"; pConfig->windowSize = iWindowSize *-1; return NULL; } static const char *GZSetNote(cmd_parms * cmd, void *dummy, const char* arg) { GZFilterConfig *pConfig = ap_get_module_config(cmd->server->module_config, &gz_module); pConfig->noteName=apr_pstrdup(cmd->pool,arg); return NULL; } static const char *GZSetMemLevel(cmd_parms * cmd, void *dummy, const char* arg) { GZFilterConfig *pConfig = ap_get_module_config(cmd->server->module_config, &gz_module); int iMemLevel; iMemLevel = atoi(arg); if (iMemLevel <1 || iMemLevel >9) return "GZMemLevel must be between 1 and 9"; pConfig->memlevel = iMemLevel; return NULL; } static const char *GZFilterEnable(cmd_parms * cmd, void *dummy, int arg) { GZFilterConfig *pConfig = ap_get_module_config(cmd->server->module_config, &gz_module); pConfig->bEnabled = arg; return NULL; } static void GZFilterInsertFilter(request_rec * r) { GZFilterConfig *pConfig = ap_get_module_config(r->server->module_config, &gz_module); if (!pConfig->bEnabled) return; ap_add_output_filter(s_szGZFilterName, NULL, r, r->connection); } static int gz_magic[2] = { 0x1f, 0x8b }; /* gzip magic header */ static apr_status_t GZFilterOutFilter(ap_filter_t * f, apr_bucket_brigade * pbbIn) { apr_bucket *pbktIn; apr_bucket_brigade *pbbOut; apr_bucket *pbktOut; char *buf; char *b; const char *accepts; request_rec *r = f->r; gz_ctx *ctx; char *token = NULL; int zRC; GZFilterConfig *pConfig = ap_get_module_config(r->server->module_config, &gz_module); if ( pConfig->bEnabled != 1 ) return ap_pass_brigade(f->next, pbbIn); /* only work on main request/no subrequests */ if (r->main) return ap_pass_brigade(f->next, pbbIn); /* GETs only (for the moment) */ if (r->method_number != M_GET) { return ap_pass_brigade(f->next, pbbIn); } /* only compress text/html files */ if (strncmp(r->content_type, "text/html", 9)) return ap_pass_brigade(f->next, pbbIn); /* some browsers might have problems, so set no-gzip (with browsermatch) * for them */ if (apr_table_get(r->subprocess_env, "no-gzip")) { return ap_pass_brigade(f->next, pbbIn); } /* if they don't have the line, then they can't play */ accepts = apr_table_get(r->headers_in, "Accept-Encoding"); if (accepts == NULL) { return ap_pass_brigade(f->next, pbbIn); } token = ap_get_token(r->pool, &accepts, 0); while (token && token[0] && strcmp(token, "gzip")) { accepts++; /* skip token */ token = ap_get_token(r->pool, &accepts, 0); } if (token == NULL || token[0] == '\0') { return ap_pass_brigade(f->next, pbbIn); } pbbOut = apr_brigade_create(f->r->pool); if (!f->ctx) { f->ctx = apr_pcalloc(f->c->pool, sizeof(*ctx)); ctx = f->ctx; /* ctx->strm.zalloc = (alloc_func) 0; ctx->strm.zfree = (free_func) 0; ctx->strm.opaque = (voidpf) 0; ctx->crc = 0L; */ zRC = deflateInit2(&ctx->strm, Z_BEST_SPEED, Z_DEFLATED, pConfig->windowSize, pConfig->memlevel, Z_DEFAULT_STRATEGY); if (zRC != Z_OK) { f->ctx = NULL; ap_log_rerror(APLOG_MARK, APLOG_ERR | APLOG_NOERRNO, 0, r, "unable to init Zlib: deflateInit2 returned %d: URL %s", zRC, r->uri); return ap_pass_brigade(f->next, pbbIn); } buf = apr_psprintf(r->pool, "%c%c%c%c%c%c%c%c%c%c", gz_magic[0], gz_magic[1], Z_DEFLATED, 0 /*flags */ , 0, 0, 0, 0 /*time */ , 0 /*xflags */ , OS_CODE); pbktOut = apr_bucket_pool_create(buf, 10, r->pool); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); apr_table_setn(r->headers_out, "Content-Encoding", "gzip"); apr_table_setn(r->headers_out, "Vary", "Accept-Encoding"); } else { ctx = f->ctx; } APR_BRIGADE_FOREACH(pbktIn, pbbIn) { const char *data; apr_size_t lenin; unsigned int len; char *buf; char szCRC[4]; char szLen[4]; apr_size_t e_wrt; int done = 0; if (APR_BUCKET_IS_EOS(pbktIn)) { apr_bucket *pbktEOS = apr_bucket_eos_create(); ctx->strm.avail_in = 0; /* should be zero already anyway */ for (;;) { len = GZFILTER_BUFFERSIZE - ctx->strm.avail_out; if (len != 0) { pbktOut = apr_bucket_heap_create(ctx->buffer, len, 1, &e_wrt); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); ctx->strm.next_out = ctx->buffer; ctx->strm.avail_out = GZFILTER_BUFFERSIZE; } if (done) break; zRC = deflate(&ctx->strm, Z_FINISH); if (len == 0 && zRC == Z_BUF_ERROR) zRC = Z_OK; done = (ctx->strm.avail_out != 0 || zRC == Z_STREAM_END); if (zRC != Z_OK && zRC != Z_STREAM_END) break; } putLong(szCRC, ctx->crc); putLong(szLen, ctx->strm.total_in); buf = apr_palloc(r->pool, 8); b=buf; *b++=szCRC[0]; *b++=szCRC[1]; *b++=szCRC[2]; *b++=szCRC[3]; *b++=szLen[0]; *b++=szLen[1]; *b++=szLen[2]; *b++=szLen[3]; pbktOut = apr_bucket_pool_create(buf, 8, r->pool); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); ap_log_rerror(APLOG_MARK, APLOG_DEBUG | APLOG_NOERRNO, 0, r, "Zlib: Compressed %ld to %ld : URL %s", ctx->strm.total_in, ctx->strm.total_out, r->uri); if (pConfig->noteName) { if (ctx->strm.total_in >0) { apr_table_setn(r->notes, pConfig->noteName, apr_itoa(r->pool, (ctx->strm.total_out*100/ctx->strm.total_in))); } else { apr_table_setn(r->notes,pConfig->noteName,"-"); } } deflateEnd(&ctx->strm); f->ctx = NULL; APR_BRIGADE_INSERT_TAIL(pbbOut, pbktOut); APR_BRIGADE_INSERT_TAIL(pbbOut, pbktEOS); break; } if (APR_BUCKET_IS_FLUSH(pbktIn)) { /* XXX FIX: do we need the Content-Size set, or can we stream? * we should be able to stream */ /* ignore flush buckets for the moment.. we can't stream as we * need the size ;( */ continue; } /* read */ apr_bucket_read(pbktIn, &data, &lenin, APR_BLOCK_READ); ctx->crc = crc32(ctx->crc, (const Bytef *) data, lenin); /* write */ ctx->strm.next_in = (char *) data; ctx->strm.avail_in = lenin; ctx->strm.next_out = ctx->buffer; ctx->strm.avail_out = GZFILTER_BUFFERSIZE; while (ctx->strm.avail_in != 0) { if (ctx->strm.avail_out == 0) { ctx->strm.next_out = ctx->buffer; len
{'source': 'Apache Software Foundation Public Mail Archives'}