nordisch by nature - Home

running 1.5.x - eat your own dog food

2007-04-26T23:28:00Z

now we are running on 1.5.x r1820. :D

As i want to deploy 1.5 on some more important host pretty soon, i migrated my own host to it. Now my fastcgi apps are handled with runit and lighty only gets the sockets configured. I will do a longer article about runit and fastcgi in the next days.

it's done. we have 1.4.14

2007-04-13T17:56:00Z

Yes! My first release is done. 1.4.14 can now be downloaded at www.lighttpd.net. It is mainly a bug fix release, but it has 2 important security fixes for you. :)

Hah! it's not only me!:)

2007-04-10T13:24:00Z

Yes I am advocating to use mod_magnet a lot. So many people might think, i am the only one really using it. But you are wrong. Today Jan found another nice article about mod_magnet and how it can save your life. :)

The folks at sitepoint posted a nice article how to save load time for CSS and JS files. For more see http://www.sitepoint.com/blogs/2007/04/10/faster-page-loads-bundle-your-css-and-javascript/.

I wonder if they will put the script in public. Let's ask them... so stay tuned. :D

you learn something new every day - today lua patterns

2007-02-26T23:25:00Z

Ok, you could have thought that "/drupal-5.1" might cause trouble. Sadly it does. The "-" is a special char for lua patterns. (for more see chapter 5.4.1 of the lua docs)

So i visited my friends at #lua on freenode. After a few minutes of discussion with rici and prec, we got a new drupal.lua. It no longer uses patterns but a simple substring function. And now we happily work with "/drupal-5.1" aswell.

The details

Why did it fail in first place?


-- prefix without the trailing slash
local prefix = "/drupal-5.1"

and later we did:


request_uri = string.gsub(lighty.env["uri.path"], prefix .. "/(.+)", "%1")

We pass an unescaped string to the pattern match. Especially the "-" was causing trouble here. Of course there would have been the option to quote all the special chars properly with "%". But thats more work than what is really needed. Rici and prec suggest do add a removePrefix function based on string.sub


function removePrefix(str, prefix)
  return str:sub(1,#prefix+1) == prefix.."/" and str:sub(#prefix+2)
end
-- calculate the relevant part of the request_uri
request_uri = removePrefix(lighty.env["uri.path"], prefix)

Last but not least rici suggest a small optimization to the query string construction. From the old ugly ...


-- make sure the request_uri is not empty.
if (string.gmatch(request_uri, ".+")) then
  if lighty.env["uri.query"] then
      lighty.env["uri.query"] = lighty.env["uri.query"] .. "&q=" .. request_uri
  else
      lighty.env["uri.query"] = "q=" .. request_uri
  end
end

... we come to the new and shiny ...


local uriquery = lighty.env["uri.query"] or ""
lighty.env["uri.query"] = uriquery .. (uriquery ~= "" and "&" or "") .. "q=" .. request_uri

You learn something new every day.

Drupal on lighttpd with clean urls (updated)

2007-02-06T20:55:00Z

Many people have asked for an example for mod_magnet that doesnt involve Ruby on Rails.

I have choosen drupal. Drupal is a nice little content managment platform written in php. I used it before switching to rails. :) Though lets start.

As already mentioned in the "Dr. Magneto vs Mr 404 handler" article, drupal uses a small nice rewrite rule set:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]

Ok. Lighttpd doesnt support -d/-f. So let’s use mod_magnet. For the example we assume drupal is already installed and accessible under "http://www.example.org/drupal/". The needed drupal.lua can be downloaded here. The script is nearly as simple as the rails example. So is the integration with lighttpd. The configsnippet is:

$HTTP["url"] =~ "^/drupal" {
    # we only need index.php here.
    index-file.names = ( "index.php" )
    # for clean urls
    magnet.attract-physical-path-to = ( "/etc/lighttpd/drupal.lua" )
}

Last but not least you need to configure drupal to use clean urls. You go to "Admin" => "Site configuration" => "Clean urls" (quick link http://www.example.org/drupal/index.php?q=admin/settings/clean-urls ) and click on the "Run the clean URL test" link. If it was successful you can enable clean urls and save the settings.

"But my drupal runs under ‘/someotherpath’, what do i need to change?"

Of course the url conditional in the config snippet above. You also need to change the prefix in the drupal.lua. Do not append a trailing slash there.

Alternatives?

404 handler?

Of course it would work as well, but the 404 handler breaks the 403 return code, which is used by drupal.

url.rewrite-once?

This is a possible solution aswell. But you need to rewrite all static files to itself.

url.rewrite-once = (
    "^(/drupal/files.*)"       => "$1",
    "^(/drupal/themes.*)"      => "$1",
    "^/drupal/([^.?]*)\?(.*)$" => "/drupal/index.php?q=$1&$2",
    "^/drupal/([^.?]*)$"       => "/drupal/index.php?q=$1",
)

Killes mentioned another rewrite solution on #drupal on freenode:

url.rewrite-once = (
    "^/drupal/system/test/(.*)$" => "/drupal/index.php?q=system/test/$1",
    "^/drupal/([^.?]*)\?(.*)$"   => "/drupal/index.php?q=$1&$2",
    "^/drupal/([^.?]*)$"         => "/drupal/index.php?q=$1",
    "^/drupal/search/node/(.*)$" => "/drupal/index.php?q=search/node/$1"
)

Security

A small side note, in SA-2006-006 the drupal team warned about a security hole through carefully named arbitrary scripts in the files directory. The apache workaround of course doesnt work here either. The following block should secure your server with lighttpd. :)

# "Fix" for Drupal SA-2006-006, requires lighttpd 1.4.13
$HTTP["url"] =~ "^/drupal/files" {
    cgi.assign = ()
    fastcgi.server = ()
    scgi.server = ()
}

For versions before 1.4.13 you need to move your cgi/fastcgi/scgi defintions into a conditional that inverts the match as shown in the example above. But i have to admit it complicates the config quite a bit. ;)

Update: the script was broken for prefices containing lua patterns special chars. This has been fixed and a new version was uploaded.

how to use cleanurl.lua?

2007-02-04T23:59:00Z

This is actually a good question. And Jan just reminds me that i never explained that. So here we go: First of all you need a mod_magnet, obviously. =) As second step you need to add mod_magnet to the server.modules array. Make sure mod_magnet is loaded before mod_fastcgi/mod_scgi/mod_cgi/mod_proxy/mod_proxy_core(1.5.x-snapshots). Last but not least you can add this line to your lighttpd.conf:

magnet.attract-physical-path-to = ( "/etc/lighttpd/cleanurl-v5.lua" )

waiting for mod_uploadprogress

2006-12-23T20:59:00Z

Seriously ... i do not understand why everyone cries for this. I can understand the point of showing some progressbar for long running uploads.

But... the browser already has all needed informations. it knows the original size. it knows how much data was sent already. We could do the whole calculation fully client side. Now what are we doing? We waste a 2nd connection to poll the webserver "how much data did i send to you already?". This feels stupid to me.

I am going to file bugs against Opera and Firefox to allow reading the needed informations from javascript.

rewrite + magnet

2006-10-14T21:04:00Z

hmm, ok ... another minor bug. if you used mod_rewrite before mod_magnet you would need another line in the cleanurl.lua.

--- cleanurl.lua 2006-10-08 09:05:19.000000000 +0200
+++ cleanurl.lua        2006-10-09 08:52:32.000000000 +0200
@@ -17,6 +17,7 @@
         -- file still missing. pass it to the fastcgi backend
         lighty.env["uri.path"] = "/dispatch.fcgi"
         lighty.env["physical.rel-path"] = lighty.env["uri.path"]
+        lighty.env["request.orig-uri"]  = lighty.env["request.uri"]
         lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"]
     end
 end

The fixed version is now cleanurl-v5.lua.

ARGH directories

2006-10-08T06:48:00Z

ok .... the cleanurl.lua as shown in "Dr. Magneto vs Mr 404 handler" has a small issue.

Directories

Why directories? My blog decided to create a directory with the same name as one of my controller and suddenly "/controller/" gave me 404. At the first moment i was ... wtf ... I enabled the last line for debugging and got the pointer. So here we are v4 of cleanurl.lua. :)

-- little helper function
function file_exists(path)
  local attr = lighty.stat(path)
  if (attr and attr["is_file"]) then
      return true
  else
      return false
  end
end

-- the magic ;)
if (not file_exists(lighty.env["physical.path"])) then
    -- file does not exist. check if we have a cached version
    lighty.env["physical.path"] = lighty.env["physical.path"] .. ".html"

    if (not file_exists(lighty.env["physical.path"])) then
        -- file still missing. pass it to the fastcgi backend
        lighty.env["uri.path"] = "/dispatch.fcgi"
        lighty.env["physical.rel-path"] = lighty.env["uri.path"]
        lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"]
    end
end
-- fallthrough will put it back into the lighty request loop
-- that means we get the 304 handling for free. ;)

-- debugging code
-- print ("final file is " ..  lighty.env["physical.path"])

download

The little helper saves us a little code duplication.

Dr. Magneto vs Mr 404 handler

2006-10-06T19:25:00Z

Jan has been working hard on lighttpd. With version 1.4.12 he added a new module called mod_magnet.

At first glance, mod_magnet might be mistaken for a newer version of mod_cml - and it may seem a better version. So what makes mod_magnet so cool?

Firstly, you can modify most request parameters
You can avoid creating our own DSL within mod_rewrite to implement -d/-f or even more complex conditionals?
lua is really fast! See this example from jan and his benchmarks.

What is mod_magnet not?

Be Warned:This plugins is not meant for something like "lua server pages". This is because The magnet script is executed within the lighty core, therefore EVERY long-running operation is blocking ALL connections in the server.

For time-consuming or blocking scripts use mod_fastcgi and friends.

the problem

The scenario I use mod_magnet is to provide simple (clean) URLs. With mod_rewrite, you can achieve them rather trivially with:

  # taken from the drupal .htaccess file
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]

So far, lighttpd has lacked this feature....

.... unlike apache mod_rewrite. To quote http://httpd.apache.org/docs/2.2/rewrite/:

`` Despite the tons of examples and docs, mod_rewrite is voodoo. Damned cool voodoo, but still voodoo.''

-- Brian Moore
bem@news.cmc.net

Do we want another voodoo language?

Back to our scenario - with Dr Magneto to the rescue!

Anyone spent any time dealing with rails will be used to:

2006-06-18 21:29:04: (connections.c.1407) Warning: Either the error-handler returned status 404 or the error-handler itself was not found: /dispatch.fcgi
2006-06-18 21:29:04: (connections.c.1409) returning the original status 404
2006-06-18 21:29:04: (connections.c.1411) If this is a rails app: check your production.log

The infamous 404 handler trick rears its ugly head again. So I started with the example shown in the mod_magnet docs. However, like many other users I had problems with luafilesystems. After multiple iterations of the script and a few more changes to mod_magnet, we have a nice and slick version of cleanurl.lua for rails. =)

-- the magic ;)
attr = lighty.stat(lighty.env["physical.path"])
if (not attr) then
  -- file does not exist. check if we have a cached version
  lighty.env["physical.path"] = lighty.env["physical.path"] .. ".html"
  attr = lighty.stat(lighty.env["physical.path"])

  if (not attr or (attr and not attr["is_file"])) then
    -- file still missing. pass it to the fastcgi backend
    lighty.env["uri.path"] = "/dispatch.fcgi"
    lighty.env["physical.rel-path"] = lighty.env["uri.path"]
    lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"]
  end
end
-- fallthrough will put it back into the lighty request loop
-- that means we get the 304 handling for free. ;)

-- debugging code
-- print ("final file is " ..  lighty.env["physical.path"])

download

To use this script, you need a lighttpd build from the 1.4.x branch (in svn). ¹ SuSE users can just grab the lighttpd14-snapshot rpms from the server:http build service project. ²

Jan added lighty.stat() for me. My first 2 versions (v1, v2) cost us 2 uncached stat() calls. The same applies to the complex rewrites example using luafilesystems. lighty.stat() uses the stat-cache of lighttpd. Ideally, the stat() is hit once and than cached. Version two of my script has the advantage of working on plain 1.4.12 already.

The return value of lighty.stat() cache allows to distinguish files from directories. My next project will be a customizable mod_dirlisting replacement in lua - with caching of rendered directory indeces and all those funny icon stuff people want.

As many people complained about the length of the cleanurl.lua script, i add the minimalistic version:

attr = lighty.stat(lighty.env["physical.path"])
if (not attr) then
   lighty.env["uri.path"] = "/dispatch.fcgi"
   lighty.env["physical.rel-path"] = lighty.env["uri.path"]
   lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"]
end

Benchmarks

So how much slower does this make lighttpd? The short answer is not much. I benchmarked my lua script and got about 3200r/s. Without the the mod_magnet script i can get my server to 4200r/s. Does this cost performance? Yes. A whole bunch- but if your server is already pushing over a couple thousand requests a second, you're definitely going to have other issues already. Even the powered by lighttpd list in the wiki lists only 100r/s as top.

mod_magnet can do that without sweating: R.I.P. Mr. 404 handler

To build from SVN

you need at least a checkout of rev 1378 or greater, and lua 5.1.

$ svn co svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x/
$ ./autogen.sh
$ ./configure <your params here>
$ make
$ make install

SuSE notes

The user documentation can be found at the opensuse wiki. Additionally you need the lua51-libs rpm from devel:languages:lua.

Thanks to imajes for making this article a pleasure to read for native english speakers

ok new try on bloggin

2006-10-05T21:04:00Z

yet another try on bloggin, lets see where we get.

Sex basics

2005-03-04T16:27:00Z

i finally found a page that it explains it for geeks like me!

http://www.collegesexadvice.com/sex.shtml