1=========================
2CML (Cache Meta Language)
3=========================
4
5---------------
6Module: mod_cml
7---------------
8
9:Author: Jan Kneschke
10:Date: $Date: 2004/11/03 22:26:05 $
11:Revision: $Revision: 1.2 $
12
13:abstract:
14  CML is a Meta language to describe the dependencies of a page at one side and building a page from its fragments on the other side using LUA.
15
16.. meta::
17  :keywords: lighttpd, cml, lua
18
19.. contents:: Table of Contents
20
21Description
22===========
23
24CML (Cache Meta Language) wants to solves several problems:
25
26 * dynamic content needs caching to perform
27 * checking if the content is dirty inside of the application is usually more expensive than sending out the cached data
28 * a dynamic page is usually fragmented and the fragments have different livetimes
29 * the different fragements can be cached independently
30
31Cache Decision
32--------------
33
34A simple example should show how to a content caching the very simple way in PHP.
35
36jan.kneschke.de has a very simple design:
37
38 * the layout is taken from a template in templates/jk.tmpl
39 * the menu is generated from a menu.csv file
40 * the content is coming from files on the local directory named content-1, content-2 and so on
41
42The page content is static as long non of the those tree items changes. A change in the layout
43is affecting all pages, a change of menu.csv too, a change of content-x file only affects the
44cached page itself.
45
46If we model this in PHP we get: ::
47
48  <?php
49
50  ## ... fetch all content-* files into $content
51  $cachefile = "/cache/dir/to/cached-content";
52
53  function is_cachable($content, $cachefile) {
54    if (!file_exists($cachefile)) {
55      return 0;
56    } else {
57      $cachemtime = filemtime($cachefile);
58    }
59
60    foreach($content as $k => $v) {
61      if (isset($v["file"]) &&
62          filemtime($v["file"]) > $cachemtime) {
63        return 0;
64      }
65    }
66
67    if (filemtime("/menu/menu.csv") > $cachemtime) {
68      return 0;
69    }
70    if (filemtime("/templates/jk.tmpl") > $cachemtime) {
71      return 0;
72    }
73  }
74
75  if (is_cachable(...), $cachefile) {
76    readfile($cachefile);
77    exit();
78  } else {
79    # generate content and write it to $cachefile
80  }
81  ?>
82
83Quite simple. No magic involved. If the one of the files is new than the cached
84content, the content is dirty and has to be regenerated.
85
86Now let take a look at the numbers:
87
88 * 150 req/s for a Cache-Hit
89 * 100 req/s for a Cache-Miss
90
91As you can see the increase is not as good as it could be. The main reason as the overhead
92of the PHP interpreter to start up (a byte-code cache has been used here).
93
94Moving these decisions out of the PHP script into a server module will remove the need
95to start PHP for a cache-hit.
96
97To transform this example into a CML you need 'index.cml' in the list of indexfiles
98and the following index.cml file: ::
99
100  output_contenttype = "text/html"
101
102  b = request["DOCUMENT_ROOT"]
103  cwd = request["CWD"]
104
105  output_include = { b .. "_cache.html" }
106
107  trigger_handler = "index.php"
108
109  if file_mtime(b .. "../lib/php/menu.csv") > file_mtime(cwd .. "_cache.html") or
110     file_mtime(b .. "templates/jk.tmpl")   > file_mtime(cwd .. "_cache.html") or
111     file_mtime(b .. "content.html")        > file_mtime(cwd .. "_cache.html") then
112     return CACHE_MISS
113  else
114     return CACHE_HIT
115  end
116
117Numbers again:
118
119 * 4900 req/s for Cache-Hit
120 *  100 req/s for Cache-Miss
121
122Content Assembling
123------------------
124
125Sometimes the different fragment are already generated externally. You have to cat them together: ::
126
127  <?php
128  readfile("head.html");
129  readfile("menu.html");
130  readfile("spacer.html");
131  readfile("db-content.html");
132  readfile("spacer2.html");
133  readfile("news.html");
134  readfile("footer.html");
135  ?>
136
137We we can do the same several times faster directly in the webserver.
138
139Don't forget: Webserver are built to send out static content, that is what they can do best.
140
141The index.cml for this looks like: ::
142
143  output_contenttype = "text/html"
144
145  cwd = request["CWD"]
146
147  output_include = { cwd .. "head.html",
148                     cwd .. "menu.html",
149                     cwd .. "spacer.html",
150                     cwd .. "db-content.html",
151                     cwd .. "spacer2.html",
152                     cwd .. "news.html",
153                     cwd .. "footer.html" }
154
155  return CACHE_HIT
156
157Now we get about 10000 req/s instead of 600 req/s.
158
159Power Magnet
160------------
161
162Next to all the features about Cache Decisions CML can do more. Starting
163with lighttpd 1.4.9 a power-magnet was added which attracts each request
164and allows you to manipulate the request for your needs.
165
166We want to display a maintainance page by putting a file in a specified
167place:
168
169We enable the power magnet: ::
170
171  cml.power-magnet  = "/home/www/power-magnet.cml"
172
173and create /home/www/power-magnet.cml with: ::
174
175  dr = request["DOCUMENT_ROOT"]
176
177  if file_isreg(dr .. 'maintainance.html') then
178    output_include = { 'maintainance.html' }
179    return CACHE_HIT
180  end
181
182  return CACHE_MISS
183
184For each requested file the /home/www/power-magnet.cml is executed which
185checks if maintainance.html exists in the docroot and displays it
186instead of handling the usual request.
187
188Another example, create thumbnail for requested image and serve it instead
189of sending the big image: ::
190
191  ## image-url is /album/baltic_winter_2005.jpg
192  ## no params -> 640x480 is served
193  ## /album/baltic_winter_2005.jpg/orig for full size
194  ## /album/baltic_winter_2005.jpg/thumb for thumbnail
195
196  dr = request["DOCUMENT_ROOT"]
197  sn = request["SCRIPT_NAME"]
198
199  ## to be continued :) ...
200
201  trigger_handler = '/gen_image.php'
202
203  return CACHE_MISS
204
205
206Installation
207============
208
209You need `lua <http://www.lua.org/>`_ and should install `libmemcache-1.3.x <http://people.freebsd.org/~seanc/libmemcache/>`_ and have to configure lighttpd with: ::
210
211  ./configure ... --with-lua --with-memcache
212
213To use the plugin you have to load it: ::
214
215   server.modules = ( ..., "mod_cml", ... )
216
217Options
218=======
219
220:cml.extension:
221  the file extension that is bound to the cml-module
222:cml.memcache-hosts:
223  hosts for the memcache.* functions
224:cml.memcache-namespace:
225  (not used yet)
226:cml.power-magnet:
227  a cml file that is executed for each request
228
229Language
230========
231
232The language used for CML is provided by `LUA <http://www.lua.org/>`_.
233
234Additionally to the functions provided by lua mod_cml provides: ::
235
236  tables:
237
238  request
239    - REQUEST_URI
240    - SCRIPT_NAME
241    - SCRIPT_FILENAME
242    - DOCUMENT_ROOT
243    - PATH_INFO
244    - CWD
245    - BASEURI
246
247  get
248    - parameters from the query-string
249
250  functions:
251  string md5(string)
252  number file_mtime(string)
253  string memcache_get_string(string)
254  number memcache_get_long(string)
255  boolean memcache_exists(string)
256
257
258What ever your script does, it has to return either CACHE_HIT or CACHE_MISS.
259It case a error occures check the error-log, the user will get a error 500. If you don't like
260the standard error-page use ``server.errorfile-prefix``.
261
262