A significant feature of the internal architecture of Saxon is the Receiver pipeline. A receiver is an object that (rather like a SAX ContentHandler) is called with a sequence of events such as startElement(), characters(), and endElement(); it typically does some processing on these events and then calls similar events on the next Receiver in the pipeline. The mechanism is efficient, because it avoids building a tree in memory, and because it allows much of the conditional logic of the processing (for example, whether or not to validate the document) to be executed at the time the pipeline is constructed, rather than with conditional code executed for every event that occurs.
Receiver pipelines are used throughout Saxon: from doing whitespace stripping on the source document, to serialization of the result tree. The schema validator is implemented as a receiver pipeline, as are operations such as namespace fixup.
But despite the elegance of the design, there have been some perennial problems with the implementation. For example, there are variations on exactly what input different implementations of Receiver will accept: some for example require an open() event while others don't; some accept entire element or document nodes in an append() event while others don't. This limits the ability to construct a pipeline using arbitrary combinations of Receivers, and worse, it's very hard to establish exactly what the permitted combinations are.
This has come to a head recently in trying to get some of the new features in XSLT 3.0 and XQuery 3.1 working reliably and robustly. The straw that broke the camel's back was the innocent-seeming item-separator serialization property. The item-separator is used while doing "sequence normalization" as the first stage of serialization; and the problem was that we didn't really do sequence normalization as a separate step in the processing. The obvious symptom that there are design problems here has been that whenever we get all the XQuery tests working, we find we've broken XSLT; and then when we get all the XSLT tests working, we find XQuery is now failing.
The model according to the specs is that the transformation or query engine produces "raw" results (which can be any sequence of items), and this is then input to the serialization process (or possibly just to sequence normalization, which wraps the results in a document node and then delivers the document). But although Saxon could deliver raw results from an XQuery running in "pull" mode (the XQueryEvaluator.iterate() method) we never really had the capability to produce raw output in push mode: the push code did sequence normalization within the query/transformation logic, rather than leaving it to the serializer. That's for historic reasons, of course: with XSLT 2.0, that's the way it was defined (the result of the transformation was always a document, with optional serialization).
So the first principle to establish in sorting this out is: the interface between the query or transformation engine and the Destination (which may or may not be a Serializer) is a raw sequence, delivered over the Receiver interface.
This requires a definition of exactly how a raw sequence is delivered over this interface: that is, what's the contract between the provider of the Receiver interface and the client (the sender of events). I've created that definition, and I've also written a Receiver implementation which validates that the sequence of events conforms to this definition; we can put this validation step into the pipeline when we feel it useful (for example, when running with assertions enabled). This exercise has revealed quite a few anomalies that should be fixed, for example cases where endDocument() is not being called before calling close().
There are three ways of delivering output from a query or transformation: raw output, document output (the result of sequence normalization), and serialized output. The next question that arises is, who decides which form is delivered. The simplest solution is: this is decided entirely at the API level, and does not depend on anything in the stylesheet or query. (This means that the XSLT build-tree attribute is ignored entirely.) In s9api terms, your choice of Destination object determines which kind of output you get. And at the implementation level, the Destination object always receives raw output; we don't want the transformation engine doing different things depending what kind of Destination has been supplied.
The other related area that needed sorting out was the API interaction with xsl:result-document. We've always had the OutputURIResolver as a callback for determining what should happen to secondary result documents, but this is no longer fit for purpose. It was already a struggle to extend it to handle thread safety when the xsl:result-document instruction became asynchronous; further extending it to work with the s9api Destination framework has never been attempted because it just seemed too difficult. Having made the decision to introduce a dependency on Java 8 for the next major Saxon release, I think we can solve this at the API level with two enhancements:
- on XsltTransformer and Xslt30Transformer, a new method setResultDocumentResolver() which takes as argument an implementation of Function<URI, Destination> - that is a function that accepts an absolute URI as input, and returns a Destination;
- on Destination, a new method onClose() which takes as argument a Consumer<Destination>.
So when xsl:result-document is called, we construct the absolute URI and pass it to the registered result document resolver, and then use the returned Destination to write the result tree. On completion we call any onClose() handler registered with the Destination, which gives the application the opportunity to process the result document (for example, by writing it to a database).
Of course, we have to work out how to implement this while retaining a level of backwards compatibility for applications using the existing OutputURIResolver.
A tricky case with xsl:result-document has been where the href attribute is omitted or empty. I think the cleanest design here is to call the registered result document resolver passing the base output URI as argument, and use the returned Destination in the normal way. The application then has to sort out the fact that the original primary Destination for the transformation is not actually used.
Yet another complication in the design is the rule in XSLT that when xsl:result-document requests schema validation of the output, schema validation is done after sequence normalization and before serialization. This is pretty ugly from a specification point of view: the serialization spec defines serialization as a 6-step process of which sequence normalization is the first; the XSLT spec really has no business inserting an additional step in the middle of this process. When the specification is ugly, the implementation usually ends up being ugly too, and we have to find some way for the transformation engine to inject a validation step into the middle of the pipeline implemented by the Destination, which ought by rights to be completely encapsulated.
Standing back from all this, unlike some refactoring exercises, in this case the basic design of the code proved to be sound, but it needed reinforcement to make the implementation more robust. It needed a clear definition and enforcement of the contract implied by the Receiver interface; it needed a clear separation of concerns between the transformation/query engine and the Destination processing; and it needed a clean API to control it all.