A Simple XML Parser for Common Lisp

Introduction

This is an implementation of a simple XML parser for Common Lisp. PLEASE NOTE: This now a project at CL.net as http://common-lisp.net/project/s-xml.

• release 3, Januari 13, 2004: added ASDF systems, optimized print-string-xml
• release 2, June 10, 2003: added echo-xml function, we are no longer taking the car when the last seed is returned from start-parse-xml
• release 1, May 25, 2003: first public release of working code, tested on OpenMCL, rewritten to be event-based, to improve efficiency and to optionally use different DOM representations, more documentation
• release 0, end of 2002: as part of an XML-RPC implementation

This XML parser implementation has the following features:

• It works (handling many common XML usages).
• It is very small (the core is about 400 lines of code, including comments and whitespace).
• It has a core API that is simple, efficient and pure functional, much like that from SSAX (see also http://ssax.sourceforge.net).
• It supports different DOM models: an XSML-based one, an LXML-based one and a classic xml-element struct based one.
• It is reasonably time and space efficient (internally avoiding garbage generatation as much as possible).

This XML parser implementation has the following limitations:

• It does not support CDATA.
• Only supports simple character sets.
• It does not support name spaces
• It does not support any special tags (like processing instructions).
• It is not validating, even skips DTD's all together.

This implementation was written by Sven Van Caekenberghe (http://homepage.mac.com/svc) at the end of 2002, beginning of 2003.

Distribution

You can download this software from my homepage at http://homepage.mac.com/svc where you will find the archive http://homepage.mac.com/svc/xml/xml.tgz. This code is also meant to be included as an example in the OpenMCL distribution.

All software and documentation is Copyright (C) 2002, 2003 Sven Van Caekenberghe. You are granted the rights to distribute and use this software as governed by the terms of the Lisp Lesser GNU Public License (see http://opensource.franz.com/preamble.html), also known as the LLGPL.

The implementation consists of the following files:

• xml.lisp - the main file that contains the parser and the event based API.
• dom.lisp - a generic interface for different DOM representations.
• sxml-dom.lisp - a DOM representation based on XSML.
• lxml-dom.lisp - a DOM representation based on LXML.
• xml-struct-dom.lisp - a DOM representation based on a simple XML-ELEMENT struct.

Installation

Basically, you load the files respecting their dependency order: dom depends on xml, sxml-dom, lxml-dom and xml-struct-dom depend on dom and xml. Two ASDF build files are included, as well as various test code.

Usage

This is open-source software. Have a look at the source code, it is reasonably clean, commented and meant to be read. The test directory contains some unit tests, one test file for each source file: just load them to run the tests. Reading the tests might help you understand how the parser was intended to be used. For the impatient, the XML Parser and DOM API can be found here.

XML Parser

Using a DOM parser is easier, but usually less efficient: see the next sections. To use the event-based API of the parser, you call the function start-parse-xml on a stream, specifying 3 hook functions:

• new-element-hook (name attributes seed) => seed
  Called when the parser enters a new element. The name of the element (tag) and the attributes (an unordered dotted pair list of attribute names as keywords and attribute values as strings) of the element are passed in, as well as the seed from the parent or from the last encountered sibling. The hook must return a new seed value to be used for the next sibling.
• finish-element-hook (name attributes parent-seed seed) => seed
  Called when the parser leaves an element. The name of the element (tag) and the attributes (an unordered dotted pair list of attribute names as keywords and attribute values as strings) of the element are passed in, as well as the parent-seed directly from the parent, as well as the seed from the parent or from the last encountered sibling. The hook must return a new seed value as result to passed to the parent.
• text-hook (string seed) => seed
  Called when the parser finds text as contents. The string of the text encountered is passed in, as well as the seed from the parent or from the last encountered sibling. The hook must return a new seed value to be used for the next sibling or as result passed to the parent.

As an example, consider the next function that will echo an XML input stream to an output stream:

(defun echo-xml (in out)
  (start-parse-xml
   in
   (make-instance 'xml-parser-state
		  :new-element-hook #'(lambda (name attributes seed)
					(declare (ignore seed))
					(format out "<~a~:{ ~a='~a'~}>"
						name
						(mapcar #'(lambda (p) (list (car p) (cdr p)))
							attributes)))
		  :finish-element-hook #'(lambda (name attributes parent-seed seed)
					   (declare (ignore attributes parent-seed seed))
					   (format out "</~a>" name))
		  :text-hook #'(lambda (string seed)
				 (declare (ignore seed))
				 (princ string out)))))

The seed parameters and return values are not used here. As a simplification, we just print text and attribute values, in real code we would have to use print-xml-string to properly escape special characters. Have a look at the implementations of the different DOM representations, as well as the XML-RPC code and the CLOS serialization code elsewhere for more real-world examples.

The parse state can be used to specify the initial seed value (nil by default), and the set of known entities (the 5 standard entities (lt, gt, amp, qout, apos) and nbps by default).

DOM

Using a DOM parser is easier, but usually less efficient. Currently three different DOM's are supported:

• The DOM type :sxml is an XSML-based one
• The DOM type :lxml is an LXML-based one
• The DOM type :xml-struct is a classic xml-element struct based one

There is a generic API that is identical for each type of DOM, with an extra parameter input-type or output-type used to specify the type of DOM. The default DOM type is :lxml. Here are some examples:

? (in-package :xml)
#<Package "XML">

? (setf xml-string "<foo id='top'><bar>text</bar></foo>")
"<foo id='top'><bar>text</bar></foo>"

? (parse-xml-string xml-string)
((:|foo| :|id| "top") (:|bar| "text"))

? (parse-xml-string xml-string :output-type :sxml)
(:|foo| (:@ (:|id| "top")) (:|bar| "text"))

? (parse-xml-string xml-string :output-type :xml-struct)
#S(XML-ELEMENT :NAME :|foo| :ATTRIBUTES ((:|id| . "top"))
               :CHILDREN (#S(XML-ELEMENT :NAME :|bar|
                                         :ATTRIBUTES NIL
                                         :CHILDREN ("text"))))

? (print-xml * :pretty t :input-type :xml-struct)
<foo id="top">
  <bar>text</bar>
</foo>
NIL

? (print-xml '(p "Interesting stuff at " ((a href "http://slashdot.org") "SlashDot")))
<P>Interesting stuff at <A HREF="http://slashdot.org">SlashDot</A></P>
NIL

Tag and attribute names are converted to keywords. Note that XML is case-sensitive, hence the fact that Common Lisp has to resort to the special literal symbol syntax.

XML Parser and DOM API

You can find a full, automatically generated, listing of the XML Parser and DOM API here.

$Id: readme.html,v 1.9 2004/01/13 10:33:17 sven Exp $