Figure 1: Website Homepage
The internet is built on open standards that are platform and language neutral. Dynamic, interactive web applications look simple at first sight, but are notoriously hard to build and difficult to adapt to changing requirements. To conquer this complexity, choosing the right tools is paramount. This case study discusses a project to build a dynamic web site and accompanying backoffice web application to maintain it built using non-conventional tools and techniques: Common Lisp and Object Prevalence. Afterwards the application was successfully deployed on an Apple Xserve running Mac OS X Server. These tools and techniques as well as the system software and hardware combination were choosen after careful consideration with the explicit goal of improving productivity. We believe we made the right choices.
The first, visible part of the project was to build the infrastructure for a web site that was dynamic and interactive. The first step in the project was to delimit the scope and put down the requirements. Next a prototype was done by design studio (Aanzet/Making Magazines). We inherited a directory with static html files, images, mp3's and some flash animations - which we placed in CVS. Over a number of weeks we gradually developed all parts of the project: the object model, the business logic, the presentation layer as well as the back end interface - adding dynamic, interactive content to the web site.
The website is that of a concert hall (De Handelsbeurs in Ghent, Belgium), presenting the concerts being organized there. Most of the data being displayed is dynamic, i.e. based on a database: concerts composed of acts, optionally grouped in concert groups. After some initial pages, the main homepage looks like the screenshot in figure 1. The left blue column is the menu, the top logo doubles as a home button. The top section displays today's date and cycles among the news headlines (actually a very simple blog). The first framed picture is some basic info about the concert that is currently in the spotlight. The top right picture is a CD cover. The central white list in the middle is an interal frame of the current concert list, from today on. The two left images below the CD cover are flyers for special concerts or concert series.
The most important information is displayed in the detail page of each concert, like in figure 2. For each concert, often consisting of multiple acts, there is at least an image and an MP3 sample. There is general info like title, genres, ticket prices and a promo copy text. There is a link that brings you to another system where you can buy tickets online, as there might be links to related artist websites.
Some links are active: genre labels for example, bring you to a page to browse concerts by genre. Figure 3 shows the concert browser, browsing concerts by the 'Song & Rock' genre. Other options are to browse by month, season or special series. There is of course an option to search for concerts and acts.
There is also an extensive photo archive of past performances (see figure 4). Once we are past a concert's date, the buy ticket link can be replaced automatically by a link to a page with pictures from the photo archive.
The website also manages a public mailing list: you can subscribe to the mailing list, edit your settings (indicating your genre interests and mail frequency preferences) and of course unsubscribe. You will receive automatic confirmation mails when subscribing or unsubscribing.
The second, invisible part of the project was to build a backoffice web application to maintain the content of the web site. This application is only accessible through HTTPS and is password protected. As can be derived from the backoffice's home page, shown in figure 5, most operations are related to maintaining the database behind the website: concerts (with their acts), concert groups, mailinglist subscriptions, photos, news items and general preferences. For individual acts, images, sound clips and sponsor logos can be uploaded and linked. There is also the option to configure bulk mailings and to look at statistics.
Figure 6 shows the page that is used to edit concerts (and their acts): in this particular case we have a double concert with two acts (the second act is not visible). Information about a concert also contains workflow or process related data such as whether a concert is visible, for sale, sold out or cancelled.
Apart from normal web log analysis, the application itself logs each view of concert details and can report the hits by day in detail, as shown in figure 7. You can clearly see the number of hits running up towards the concert's date and diminishing afterwards. The effect of doing bulk mailings can also be observed this way.
By now you should have an idea about the subject of the application and its relative complexity. It is typical of a small to medium size web project, covering most of the aspects involved: dynamic page generation, forms processing, session tracking, security, uploading files, sending email and managing persistent data.
The success of the Web gave rise to a new kind of application: Web Applications. These are applications that are accessed by their users through an ordinary web browser. The purest kind of web applications are those requiring no special plugins at all: just plain HTML (see The Web Standards Project). The project described above is an example, as are most shopping sites, like Amazon or the Apple Store. Actually, almost all advanced web sites are dynamic websites: they compute their pages from a database, allow searching, allow users to log in, manage preferences, offer mailing lists and forums. Dynamic, interactive web applications look simple at first sight, but are notoriously hard to build and difficult to adapt to changing requirements.
The internet is built on open standards that are platform and language neutral. This allows web applications to be build using almost any technology. And indeed, successful web applications have been build in C, C++, Java, Perl, Python, PHP, and many others on Unix, Linux, Windows and Macintosh using all sorts of frameworks. The great thing is, as a user you can hardly see the difference (apart from the fact that some site are obviously better than others).
As software developers we had the most experience building web applications using Java technology. In the Java world you find all the necessary tools, frameworks, libraries and servers to build almost anything. Java is very good at absorbing all the great abstractions like object technology, garbage collection, model-view-controller, layered systems. However, the Java world is becoming very large and complex. And all things touched by Java (especially the J2EE part) have a tendancy to become overly verbose and cumbersome. This is partly because of limitations in the Java language itself (full static typing, no interactive or dynamic modifications at runtime, little room for data driven programming) and partly it is a community thing (standards reached by consensus and political motivations).
Because we had a past exposure to both Lisp and Smalltalk, we knew that there were very good alternatives to Java. Thanks to Java, a type-safe runtime with automatic memory management and garbage collection became accepted both on the client and server side. People now know that the small price in performance is dwarfed by the advantages in productivity. Lisp and Smalltalk easily match Java feature-wise, but add full dynamic typing, allow interactive and dynamic modifications at runtime and allow for much easier data driven programming. The best part is that current Lisp and Smalltalk implementations often consume less memory and run faster than Java.
Hence we decided to give it a try and implement our next web applications in Common Lisp. Invented more than 40 years ago, Lisp remains in active use today. In 1994 Common Lisp became the first object oriented to get an official ANSI certification (see also the excellent Common Lisp HyperSpec). Today there are many high quality Common Lisp implemetations available on all platforms (for example both Xanalys's LispWorks as well as Franz's Allegro Common Lisp are available for Unix, Linux, Mac OS X and Windows platforms). As far as the ANSI standard is concerned, compatibility between them is very good. There is even some defacto standardization in areas not covered by the standard.
The first step was to select the base on which to build our application. As further detailed in the acknowledgments, we made the following choices (during development):
On top of that we needed a Web Application Framework, supporting the generally accepted presentation-action-model separation (also known as model-view-controller) and offering some of the plumbing needed for all web applications (session and context management, security). Since there was nothing available, we put together our own framework called KPAX (there is a student thesis available that discusses KPAX in some detail - see Nicky Peeters' homepage). KPAX incorporates modified versions of two open source elements: Brendan Burns' session support for allegroserve and John Wiseman's lisp server pages for allegroserve.
With KPAX, a web application is declared using a Lisp form, as shown in listing 1. In Lisp, we don't need to resort to XML files to get data driven programming (but we could if we wanted to do so). One of the key strengths of Lisp is that programs are represented as s-expressions (lists whose elements are either atoms - numbers, strings, symbols - or other lists), and that these s-expression data structures can be used and manipulated both at compile time and at run time. Both constant and computed elements can be used in such declaring forms (for example, the root is defined as the directory of the file this form is in, and the web-user object is actually instantiated on the fly - but the list of users could come from a different source defined somewhere else). A web application context is automatically defined, user sessions are properly managed and the application is secured by a login page.
(defwebapp :factorial
(:root (util:pathname-parent *load-truename*))
(:prefix "factorial")
(:index "index")
(:lsps '("result"))
(:actions '(("compute-factorial" . compute-factorial)))
(:users (list (make-instance 'web-user
:id 101
:short-name "guest"
:full-name "Guest User"
:password "welcome"))))
The example we are describing here is a web application that can be used to compute factorials. When the user logs in, the main index page comes up using the lisp server page (LSP) in listing 2. Like Java server pages or any other kind of server pages, normal HTML markup is extended with a special kind of tag, '<% .. %>', that contains Lisp code that can possibly generate output that is included in the page.
Using Lisp instead of Java here means an easier, more fluid, less verbose style of server pages. A more dynamic language increases a programmer's productivity by requiring less coding. LSP's are actually fun: unlike JSP's, the edit-test cycle is literally instanteneous with no need to copy files, redeploy or wait for the compiler to do its thing.
<html> <head> <title>KPAX Factorial</title></head> <body> <% (standard-header "KPAX Factorial" request entity session) %> <p>Good <% (html (:princ (get-daypart))) %>! Welcome, <% (html (:em (:princ (web-user-short-name (session-attribute session :user))))) %> to the KPAX Factorial example.</p> <p>This web application computes the factorial of a positive integer. Factorial being defined recursively as follows: <ul> <li>the factorial of 0 is 1</li> <li>the factorial of N is N times the factorial of N-1</li> </ul> </p> <p>Compute the factorial of <form action="webaction"> <input type="hidden" name="action" value="compute-factorial"> <input type="text" name="number" value="0"> <input type="submit" value="Go!"> <form> </p> <% (standard-footer request entity session) %> </body> </html>
A standard feature of Lisp is that it has a read-eval-print loop, an interactive evaluator. You can type in any Lisp expression, have it evaluated (executed) and see the result get printed. This means that you can test your program at any point with no need to insert silly print statements and run the whole program over again. Similary, the standard interactive Lisp debugger is just as effective to debug problems handling web requests as it is debugging other code. Debugging web application becomes a lot easier when using Lisp as an implementation language.
LSP's are the central element of the presentation layer, web actions are the core element of the controller layer. Web actions are Lisp functions that get executed when the user clicks a certain links or submits a form. The code in listing 3 is bound to the action named "compute-factorial" and gets executed when the user clicks the 'Go!' button to compute a factorial (submits the form). The code extracts the number parameter from the form, does some validations and optional error handling, and then computes the result (timing it along the way) and forwards the presentation of the result to another LSP, passing some data as request attributes.
(defun compute-factorial (request entity)
(let* ((number-string (request-query-value "number" request))
(number (parse-int number-string nil)))
(cond ((not number)
(display-message request entity
"Error" (format nil "Cannot convert ~a to an integer" number-string)))
((< number 0)
(display-message request entity
"Error" (format nil "Cannot compute factorial of ~d, number must be positive" number)))
((> number 12399)
(display-message request entity
"Sorry" (format nil "Cannot compute factorial of ~d, number to large" number)))
(t
(let* ((start (get-internal-run-time))
(fac (fac number))
(stop (get-internal-run-time)))
(setf (request-attribute request :number) number
(request-attribute request :factorial) fac
(request-attribute request :time) (float (/ (- stop start) internal-time-units-per-second)))
(forward-to-lsp "/result.lsp" request entity))))))
Listing 4 show the page rendering the result computed by the previous web action. It extracts some data as attributes from the request and renders it as HTML. Note how a dynamically typed language is quite elegant and concise.
<html>
<head>
<title>KPAX Factorial Result</title></head>
<body>
<% (standard-header "KPAX Factorial Result" request entity session) %>
<% (let ((number (request-attribute request :number))
(factorial (request-attribute request :factorial))
(time (request-attribute request :time))) %>
<p>The factorial of <% (html (:princ number)) %> is <% (html (:pre (:princ (long-number-to-string factorial)))) %></p>
<p>Computation took <% (html (:princ time)) %>ms.</p>
<p><a href="index.lsp">Compute another factorial</a></p>
<% ) %>
<% (standard-footer request entity session) %>
</body>
</html>
One of the top strengths of Lisp in the application server space is the following. Although Lisp code is written in source files and is typically compiled and written out to object files, the object files can only execute within the context of the lisp runtime environment. Loading object files into the runtime creates the an executable program. To save time, this environment is often saved in what is called an image file. The term Lisp image is used to refer to this file, as well as to the in-memory representation of a running Lisp program.
Ultimately, the Lisp application server is a Lisp image constructed by loading a bunch of compiled source files. This system construction process is of course also written in Lisp (using defsystem, or the more modern ASDF variant). The application server process runs as a background process. With a Lisp application server, the read-eval-print loop including all dynamic Lisp features remains available! This means that we can do the following: login to the server, connect to the read-eval-print loop and inspect all the internal datastructures of the server, while it is running. To update the server to a new version, we connect to the read-eval-print loop and invoke the build process and have it load and install the new code, without taking down the server process - active user sessions and other datastructures (like caches) remain in place and continue to work. Needless to say, this results in serious developer productivity gains and happy customers.
In most non-trivial projects, persistent data storage is needed. Often, an SQL database is used for this purpose. Although there is nothing wrong with this, mapping an object domain model to a relational database is complex, error prone and not very efficient. In most projects, setting up this translation layer takes a lot of developer time. The fact that the object domain model must eventually map to a relational database, limits the designer in using all powerful features that object technology can offer.
Object prevalence is a concept that was developed by Klaus Wuestefeld and some colleagues at Objective Solutions. It basically states the following: most database are small enough to keep them completely in RAM, so the object domain model effectively becomes the database. Querying is ultra fast and very natural: just use the API of the programming language. Modifications are done through transaction objects, serializeable objects that contain all the data they need to apply the change. When executing a transaction, the change is applied to the object model and written out to a transaction log file by serializing it. This operation is fast as well. When the system crashes, the previous state is restored by replaying the transaction log. Multi-threaded access to the system is serialized on a simple lock. Since transactions are fast this is no real problem. Furthermore, there are often more readers than writers, and only a small number of readers need exclusive system access. When the transaction log becomes to big, it is possible to make a snapshot, a serialization of all known objects in the system.
We used our own implementation of object prevalence for Common Lisp (see Common Lisp Prevalence). This particular implementation serializes to XML, which is relatively slow and bulky but robust and portable. In this project we added two extensions: blobs and generic transactions. Blobs are a mechanism to handle larger binary objects in prevalence, keeping the meta information as a normal Lisp object while storing the actual bytes in an automatically managed file. Today we save about 40 Mb worth of data by this mechanism. Generic transactions were introduced because basic object operations like create, change and delete are so common that we became tired of writing unique transactions for each class.
It is our opinion that object prevalence makes a lot of sense in a dynamic, interactive language like Common Lisp (much more sense than it does for Java). A Lisp read-eval-print loop together with Lisp's strong datastructures and manipulation functions makes for a much better alternative to SQL for querying and database maintenance. Without such a capability, using object prevalence will be a lot harder. Also, the serialization to XML helps a lot in making the system maintainable. XML and Common Lisp are much more resilent to ongoing changes in the object model than for example native Java serialization.
The object domain model of our application is shown as a UML-like diagram in figure 8. Listing 5 gives an impression of what it is like to interact with the prevalence layer of the system. First we use telnet to open a connection to the live server, which gives us a fresh read-eval-print loop in its own thread (for security reasons, access is only granted to localhost clients). Next we do a simple query to find all cancelled concerts. Just as an example, we execute a simple transaction so that the last concert is no longer cancelled. Asking for the last transaction, shows us the transaction object itself as it was written to the transaction log. The following query counts the number of instances for each class in the object domain model, we compute a grand total as well. Finally we close the read-eval-print connection with the server, which keeps on running. XML is of course a very verbose format to use for serialization (object slot names as well as value types are repeated over and over again), hence the relatively large sizes of the snapshot and transaction log files shown in the directory listing.
[sven@voyager:~]$ telnet localhost 24365
Connected to localhost.
Escape character is '^]'.
Welcome to OpenMCL Version (Beta: Darwin) 0.13.6!
? (remove-if-not #'get-cancelled-p (find-all-objects *system* 'concert))
(#<CONCERT #655 "The Sexmachines # Swell" 2003/11/02 #x53FF15E>
#<CONCERT #143 "Wannes Van de Velde & groep" 2004/03/31 #x540556E>
#<CONCERT #33 "Ronny Jordan" 2003/10/01 #x540D616>
#<CONCERT #10 "Rory Block" 2003/09/28 #x540EDAE>)
? (execute-transaction (tx-change-object-slots *system* 'concert 10 '((cancelled nil))))
#<CONCERT #10 "Rory Block" 2003/09/28 #x540EDAE>
? (transaction-log-tail *system* 1)
(#<TRANSACTION TX-CHANGE-OBJECT-SLOTS (CONCERT 10 ((CANCELLED NIL)))>)
? (mapcar #'(lambda (c)
(let ((count (length (find-all-objects *system* c))))
(format t "~s has ~d instances~%" c count)
count))
'(concert act concert-group photo news-item mailing-list-subscriber blob))
CONCERT has 80 instances
ACT has 102 instances
CONCERT-GROUP has 6 instances
PHOTO has 135 instances
NEWS-ITEM has 15 instances
MAILING-LIST-SUBSCRIBER has 1765 instances
BLOB has 330 instances
(80 102 6 135 15 1765 330)
? (reduce #'+ *)
2433
? (remote-repl:exit)
Connection closed by foreign host.
[sven@voyager:~]$ ls -la ha-prevalence-system/
total 7320
drwxr-xr-x 5 sven staff 170 Oct 13 15:19 .
drwxr-xr-x 49 sven staff 1666 Oct 22 16:52 ..
-rw-r--r-- 1 sven staff 2972822 Oct 9 11:21 snapshot.xml
-rw-r--r-- 1 sven staff 763984 Oct 23 10:00 transaction-log.xml
Our own conclusion from using object prevalence in a real production system is positive. During the project we learned a lot about the peculiarities of object prevalence, adding debugging and backup tools as well as more abstractions to our original design. We feel confident enough about it, to use it again in future projects.
The Lisp application server running our website and backoffice is self-contained: run it on a standard machine, point your browser at the URL and that's it. Deploying web applications on an server connected to the internet is more difficult. A server connected to the internet must be properly installed, configured, managed and secured. Since dedicated hosting or colocation is more economical than big pipes to your own office, remote management is will also be needed. Furthermore, such servers often serve multiple domains (also known as virutal hosting). There will be a need to monitor what happens on the server and to compile usage statistics.
Our standard architecture for such a server is to use a securily configured Unix machine (typically a Linux machine), using Apache to do the virtual hosting, mod_ssl for HTTPS, with proxy or rewrite rules to pass the proper traffic to the Lisp (or Java) application server, Webalizer for access log analysis and SSH for remote management. So each request arrives at Apache, where the virtual hosting is resolved. Apache then logs the request, figures out which proxing or rewriting to apply and forwards the request to the application server's http server. The application server handles the request and sends the reply to Apache who delivers it to the original requester. For added security (especially in the HTTPS case), the application server should be configured to only allow request from this apache server. We have this configuration working on a website that gets up to 3.5 million hits a day, with peaks of 300.000 hits per hour, transferring up to 12 Gb a day - all this on modest hardware (but using a big pipe).
During development we did test deploys (using CMUCL) on a dedicated server that we were renting at that time from ServerBeach. 24 hours after ordering, we had our own AMD 1Ghz, 512Mmb, 60GB HD, Linux machine with a 400Gb monthly traffic allowance, and all this for $100 a month. In terms of value for money, this offer is hard to beat. However, the quality of the internet connection that came with the server was no that good, and the machine was also slower than we expected (and not upgradeable). Colocation in a datacenter that we could physically access (instead of of one requiring a transatlantic flight), started to sound like a better, although more expensive solution.
We found an excellent colocation solution through a local partner, Rack66, in a Level 3 Communications datacenter in Brussels, Belgium. Not only do they offer high-end service level agreements (SLA's) for network and power, they also back them up with money back guarantees in case of downtime. Next we had to find a good 1U server solution. It is one thing to buy nobrand, white hardware boxes, put Linux on them and use them as workstation or a server in a normal server room - placing them in an environment where you already pay good money for everything else feels odd. Furthermore, good 1U Linux machines are not so cheap as one whould expect. And Linux has one weakness: there are few, if any, big brand manufacturers that will sell you a complete integrated and supported solution.
After careful consideration, looking closely at value for money, we decided to buy 2 Apple Xserves running Mac OS X Server, together with a 3 year AppleCare Premium Service and Support Plan, covering 1 stop hardware and software support and including onsite hardware repairs. We choose one standard Dual Processor Xserve (2 x 1.33 Ghz G4, 2 x 2 Mb Level 3 Cache, 1 Gb RAM) connected with one ethernet port to the internet feed and with another ethernet port directly over a gigabit ethernet to a second cluster node Dual Processor Xserve (identical specs). Figure 13 shows the 2 machines during the week they were being tested and configured in our office.
The hardware is excellent, from the beautiful design over many small details (like useful indicator lights, cable management arm, removable drives) to the very complete package (including all rack mounting gear). A dual processor machine can take serious loads without running out of breath. The software package is also very complete: a solid Unix core including all the necessary open source pieces coupled with GUI based remote management tools.
The website went life September 16th 2003. Figure 14 shows a summary of webalizer's HTTP access log analysis since then. This is by no means a heavy hit web application, but it does receive a decent and constant amount of attention from its target group. In the busiest day so far we got 20.000 hits (5700 hits an hour) and transferred 200 Mb of data. All this was accomplished with the standard allegroserve configuration of 5 worker threads.
Our front Xserve - handling a number of static websites and shuffling traffic for our secondary Xserve - hardly feels this load. Figure 15 is a graph of the CPU load over 3 days. The regular peaks are from the webalizer cron job running every hour. Our current setup, hardware, OS and application server, feels very fast. For example, working with the backoffice over the internet using HTTPS is just as fast as connecting to the application server on your own machine. And given the CPU statistics, we have ample room to grow.
In any project you have a short period in which you lay the foundations and start using exciting new technology. After that period, you have to stop building infrastructure and build the application itself - in order to meet time and budget constraints. The framework that we described so far is the foundation of the current project. During the course of this project, we did think of many possible improvements. Also, not much performance tuning has yet been necessary, but there are certainly many opportunities to speed things up.
The most important thing that I would like to change is this. Building a web application with server pages as presentation layer, actions as control layer and a data model at the bottom is the accepted current practice - in Java, PHP, everywhere. Lisp makes development easier, more productive and more fun. However, after writing many lsps and actions, it became clear that we were doing much of the same coding over and over again. On the presentation side, there is a clear need for a reusable components architecture. On the actions side, there is a need for an infrastructure to ease complex form processing. Also, for components to be any good, they have to come with their own behavior. There is some experimental work in this area, but no clear solution has emerged yet.
Load balancing could be added to improve scalability. Failover support could be added to increase reliability. Both require sharing session related data over multiple application server instances. This has been done before and the same techniques could be implemented in Lisp. The generation of dynamic content could be tuned by more efficient HTML generation code and intelligent caching techniques.
Doing this project, we made a couple of unconventional choices. These were deliberate choices based on sound arguments with the goal of improving our productivity. We feel that we made the right choices. Today, our project (Portable Allegroserve, KPAX web application framework and Common Lisp Prevalence not included) has the following metrics:
The project was successful: time and budget constraints were respected. The website went life September 16th, just in time for the final deadline (the opening concert was September 18th). Actual development of the project started end of July, beginning of August. All work was done by two people (the author, Sven Van Caekenberghe, and Nicky Peeters), for a large part of the time doing pair programming (part of the extreme programming methodoly, XP). We estimate to have worked about 40 days for this project. Another 5 to 10 days of work went into the process between ordering the Xserves and putting them in colocation and production use.
The weeks after the site went life, numerous small additions and small modifications were asked, and a number of bugs had to be fixed (including some really strange and unexpected ones). The ease and fun of doing these additions, modifications and bugfixes surprised us and is certainly another measure of success. The best measure of success however is of course the fact that the customer is happy and that the site is running smoothly.
No piece of software is ever written in a void: we always build on the foundation laid by others. And although it is not possible to list every piece of software that we are using, these projects helped us a lot:
Thank you all - we appreciate your work enormeously.
The best way to say thank you to the open source community is of course to give something back. Parts of the technologies that were used in this project are available as open source code from the authors' home page:
The KPAX web application framework has been packaged as a snapshot, as-is open source release: see the KPAX homepage for more information. There is a student thesis available that discusses KPAX in some detail - see Nicky Peeters' homepage.