Geisha Studios

In these stylesheets, syntax highlighting is performed by the template apply-highlighting, defined in highlighting/common.xsl in the DocBook XSL distribution, to all elements which support support highlighting (programlisting, screen, and synopsis.) It determines the language to be used for highlighting from the element’s language attribute, extracts the content nodes and eventually calls the XPath function highlight with these parameters. The function is looked up, in order, in three different namespaces:

If it fails to determine a language or to look up highlight in these namespaces, it simply copies the contents.

The highlight function returns a list of XML and text nodes. XML nodes with the xslthl namespace prefix represent tokens from the highlighted source code. For instance, there are keyword and comment nodes. Refer to the Processing xslthl results section in the xslthl documentation for more information.

These xslthl nodes are then transformed into proper output format by the format-specific DocBook highlighting stylesheets. Note that these must explicitly be included in the customization layer.

These observations lead to the conclusion that if we want to use a custom highlighting routine then we must re-implement this highlight function and put into any of the mentioned namespace. In Python, we can use the lxml package that provides the same XSLT processing library as xsltproc but that can be extended with custom functions:

            
<<define custom highlight function>>=
xhl = etree.FunctionNamespace('http://net.sf.xslthl/ConnectorXalan')
xhl.prefix = 'xhl'
xhl['highlight'] = html_highlight

          

This adds the namespace http://net.sf.xslthl/ConnectorXalan with the prefix xhl to the global list of functions namespaces maintained by lxml. Then, we add the function html_highlight into the namespace as highlight. Thus the stylesheet can now call the XPath function xhl:highlight.

In order to be able to use etree, first we must import the lxml package.

            
<<modules>>=
from lxml import etree

          

When using literate programming to create the source code blocks to highlight, besides the regular code in the language specified in the language attributes, there are special directives used for tangle programs such as atangle to extract these code blocks and write a complete source code module.

Obviously, these directives aren’t part of the target language, otherwise the tangle program would confuse them for directives and instead of outputting the source code it would try to satisfy the reference. Nevertheless, Pyments is unable to detect that particular use and tries to highlight the directives using the language lexer. Occasionally, that means that the labels gets highlighted as keywords — such as when using for or while inside the directive — or as errors.

Fortunately, Pygments has a method to attach filters to the lexer and perform additional modification to the lexer’s output. Usually this filters are used to complement the lexer by highlighting additional keywords or special strings inside comments, such as TODO, XXX, etc.

In this case I’ll use the filter architecture to detect atangle directives and inform the formatter to render them as labels instead of whatever the lexer believed it to be.

The filter, then, needs to look line by line and check whether a line is either a regular source code or an atangle directive. Given the nature of atangle directives, this can be accomplished using a regular expression.

            
<<atangle regex>>=
self.directive = r'''^\s*<<(\*|[-\w\s\.]+)>>=?\s*$'''

          

To use regular expressions the script needs to import the re module.

            
<<modules>>=
import re

          

This directive matches both the declaration directive, that starts a new atangle code snippet, as well as reference directives. Inside the filter, thus, we need to build an string until we match the end of line, either \n or \r, and then check against this regular expression. If the expression matches, then return the string as a label token. Otherwise, output whatever the lexer gave to the filter.

In some cases, Pygmens passes an string with double end of line characters, such as when it find an invalid syntax. To get these cases into account, I look for a value whose first character is either \n or \r instead of looking at the whole string.

Also, in some languages, notably XML, since the < and > characters are part of the language itself, Pygmens also tends to give the = as a separate value. Then, I also need to check for this character at first value’s position.

            
<<atangle filter>>=
class AtangleFilter(Filter):
    def __init__(self, **options):
        Filter.__init__(self, **options)
        <<atangle regex>>

    def filter(self, lexer, stream):
        lexer_input = []
        line = ""
        for ttype, value in stream:
            if len(value) > 0 and (value[0] == '\n' or value[0] == '\r' or value[0] == '='):
                if re.match(self.directive, line):
                    yield Token.Name.Label, line
                else:
                    for original_ttype, original_value in lexer_input:
                        yield original_ttype, original_value
                # The end of line also needs to be there.
                yield ttype, value
                # start with the next line.
                lexer_input = []
                line = ""
            else:
                lexer_input.append((ttype, value))
                line = line + value;

          

It is also necessary to yield any buffered values from the lexer. This is to avoid losing tokens when there is no line that starts with a newline character and thus the original tokens would never be yielded.

            
<<atangle filter>>=
        for ttype, value in lexer_input:
            yield ttype, value

          

AtangleFilter class derives from Filter which is defined in the Pygments package.

            
<<modules>>=
from pygments.filter import Filter

          

The only thing that remains is to tell the lexer to use this filter. This is done calling add_filter with the lexer.

            
<<add atangle filter to lexer>>=
lexer.add_filter(AtangleFilter())