Saxon diaries

Validating Linked Documents

2026-08-13T21:00:00Z

A chance remark at Balisage 2026 (RIP) led me to consider the question: can you write an XSD schema to validate a linked set of XML documents, rather than just a single document?

Here's a use case: the test suite for XQuery/XPath 4.0 consists of a catalog file, which lists 800 or so linked test sets. To make it more specific, the catalog contains many lines rather like this:

Both the top-level catalog and the individual test-set files are governed by XSD schemas. Currently we validate the collection of documents using an XQuery query along the lines of:

//fots:test-set/@file ! (validate{doc(resolve-uri(., base-uri(..)))})

And of course other solutions are possible, for example using XProc. But it occurred to me, can we write a schema for the catalog file that includes an XSD 1.1 assertion that invokes validation on the referenced test-set files? And can we use a different schema to validate the test-set files, different from the one we use for the catalog?

I'll give you the answer first, before describing the journey to get there. I've made it work like this

This won't run out-of-the-box in Saxon 13.0: though most of the ingredients are present I had to make some tweaks to make it all hang together, which I'll describe as I go along. It should work as written in 13.1.

Firstly, the saxon:param element has been available for quite a while as a Saxon extension. I'm using it here not to parameterize the validation, but rather to set up a global variable. The variable is the result of the 4.0 xsd-validator function: a new higher-order function that, given a schema, returns a function that can be used to validate instance documents against this schema. Putting this validator in a global variable means we only need to compile the schema for the test-set documents once.

The expression in the xs:assert/@test attribute contains several features of note:

Firstly, it calls the doc() function to load the test-set document referenced in the @file attribute. We're loading this document because we want to pass it to the validation function. The way xsd-validator works, you can only validate an already-built document tree. We want to do the validation using a different schema, so we can't do this within the doc function itself.
Invoking the validator function (let $validity := $test-set-validator($doc)) returns a record containing potentially three things: a success/failure flag, a list of validation errors, and an annotated document representing the PSVI. We're not interested in the last part, so we didn't ask for it in the options to the xsd:validator call.
We need to know which test-set documents are invalid, so it's helpful to have customized error message if the call on the validator function returns errors. There's an existing Saxon extension, saxon:message, which allows you to produce a custom message when an assertion fails, but unfortunately it's a fixed message, it can't be tailored to the actual data. So I invented a new mechanism. The XSD 1.1 specification says that if the result of the assertion has no effective boolean value, the assertion is taken as failing. So you can now return the sequence (false(), "message") from an assertion. This has no effective boolean value, so it is equivalent to returning false, but it also returns a diagnostic message explaining what is wrong. (The backtick expression is a string template, a new addition in XPath 4.0).
XSD 1.1 expects assertions to use XPath 2.0 syntax but there is a Saxon attribute saxon:extensions="any-xpath-version" to relax this rule. (I found that this was only allowing 3.1 syntax, not 4.0, but this is now fixed).
Finally, XSD 1.1 says that you can't call the doc() function from within an assertion. This is a security mechanism. So I added a new configuration option --assertionsCanSeeResources:on to bypass this mechanism: only set this option if you are using a trusted schema.

So with Saxon 13.1, you will be able to invoke validation of a linked set of documents, using the same schema or different schemas, in a single call of the validation API.

101 Fun things to do with QT4

2026-05-26T18:00:00Z

Here's a fun list of new features in the XPath 4.0, XQuery 4.0, and XSLT 4.0, that are available to play with in Saxon 13.

Select an attribute, using a default value if absent

@discount otherwise 0
Use a conditional with no else branch

if (@discount) { " (reduced!)" }
Shorten function to fn

for-each(//employee, fn($e){$e/@ssn})
Use the context item to simplify arity-1 functions

for-each(//employee, fn{@ssn})
Drop the map keyword from map constructors

let $map := {'x':1, 'y':2}
Include conditional entries in a map constructor

{'x': +@x, 'y': +@y, if (@z) {{'z': +@z}}}
Include multiple entries in a map constructor

{//data ! {@key : string(@value)}}
Build a map from a sequence

map:build(//employee, fn{@ssn})
Select a range of entries from a sequence

$input[1 to 5]
Start talking about atomic items rather than atomic values
Use QName literals

node-name($node) = #xml:space
Use underscore in numeric literals

let $million := 1_000_000
Use hex literals

let $mask := 0xffff
Use alternative names in an element test

//element(chapter|appendix)
Declare namespaces in XPath

declare namespace p = "http:/my.ns"; //p:table
Declare a default namespace in XPath

declare default element namespace "http:/my.ns"; //table
Use ##any to match elements by local name, ignoring namespace

declare default element namespace "##any"; //table
Use string templates

`{$firstname} {$lastname}`
Use × and ÷ for multiplication and division

@price × 1.2
Convert untyped atomic values implicitly to decimal where appropriate

1.1 = 1.1
Write << and >> as precedes and follows

para[. precedes $first-heading]
Compare nodes using is-not

*[. is-not $first-heading]
Use the =!> operator to apply a function to all items in a sequence

tokenize($str) =!> upper-case()
Use the pipeline operator -> to set the context item for an expression

$employee -> `First name: {@first} Last name: {@last}`
Use a multi-character representation of "percent" in a decimal format

percent = "%:pc"
Start talking about coercion rules rather than function conversion rules
Take advantage of down-casting in the coercion rules:

declare function my:f($x as xs:positiveInteger){$x + 1}; my:f(3)
Check that you always have a space after "{#" in a pragma

{# saxon:extension xxx #}
Take advantage of coercion in variable declarations

let $x as xs:decimal := @price
Declare functions with optional parameters

declare function my:f($x as xs:integer := 0)
Use keywords in function calls

substring($value, start := 3)
Declare functions with no namespace prefix

declare function increment($x) {$x+1}; increment(42)
Serialize output as canonical XML

serialize($output, {'method': 'xml', canonical': true()})
Take advantage of the functions in the bin and file namespaces. Previously defined in EXPath, these are now integrated into the specs.
Watch out for decimal=double comparisons. These are now compared as decimals, not as doubles

1.1e0 != 1.1
Convert arbitrary XML to JSON using the element-to-map function

element-to-map(doc('books.xml')/*)
Process CSV input files using the new csv-doc and parse-csv functions

csv-doc('data.csv')
Process HTML input files

parse-html('index.html')
Use higher-order functions knowing they will be there in all implementations. Higher-order functions are no longer an optional feature.
Use the second argument of callback functions to get the position of an item in a sequence

filter($input, fn{$item, $pos){$pos gt 5})
Get the last item of a sequence using foot()

foot($input)
Get selected items in a sequence using slice()

slice($in, start := 5, end := 2, step := -2)
Use the deep-equal() function with options to control the comparison

deep-equal($p1, $p2, {'whitespace': 'normalize'})
Use the all-different() function to check uniqueness of values

all-different(//test-case/@name)
Use the highest() and lowest() functions to find the items with the minimum or maximum value of some property

highest(//employee, fn{@salary})
Use the index-where() function to find the positions of all items with some property

index-where(*, fn{exists(self::h2)})
Use the partition() function to perform positional grouping

partition(*, fn($group, $next){exists($next[self::h2])})
Use the sort-by() function to sort a sequence using multiple sort keys

sort-by(//person, ({'key': fn{@last}}, {'key': fn{@date-of-birth}, 'order': 'descending'}))
Use the sort-with() function to sort using a comparator

sort-with($input, compare#2),
Use take-while to select items from a sequence until some condition is false

take-while(*, fn{not(self::h2)})
Use is-NaN to check whether an input item is NaN

*[not(is-NaN(.))]
Use round() with an explicit rounding mode

round($value, precision := '2', mode := 'half-toward-zero')
Read integers in hexadecimal notation

parse-integer("feff", 16)
Format integers in hexadecimal notation

format-integer($value, '16^xxxxxxxx')
Supply inline options to format-number()

format-number($value, {'decimal-separator': ',', 'grouping-separator': '.'})
Use new mathematical functions math:e(), math:sinh(), etc.

math:cosh(1)
Use the new collation for Unicode case-blind comparison

compare($a, $b, "http://www.w3.org/2005/xpath-functions/collation/unicode-case-insensitive")
Construct a collation with desired properties

compare($a, $b, collation({'lang': 'se', 'numeric': true()}))
Get the character with a given codepoint

char(0xA0)
Get the character with a given HTML entity name

char("nbsp")
Get all the characters in a string

characters($input)
Use lookahead and lookbehind in regular expressions

matches($input, "Chapter(?=\s+[1-9])")
Match word boundaries in a string

matches($input, "\bthe\b")
Compute replacement values for matching substrings

replace($input, "(0-9)+", fn{.+1})
Split a URI into its parts, and recombine them

parse-uri(@href) => map:put('scheme', 'https') => build-uri()
Build a date or time from its components

build-dateTime({'year': $year, 'month': $month, 'day': $day})
Get the civil timezone offset at a given time and place

civil-timezone(current-dateTime(), 'Europe/London')
Use @then and @else on xsl:if
Use @select on xsl:when and xsl:otherwise
Select options using the new xsl:switch instruction
Provide default values for XSLT function parameters
Use XSLT functions in no namespace

...
Use xsl:text in preference to xsl:value-of

{position()} of {last()}
Construct CDATA sections selectively in serialized output
Use capturing accumulators to capture a snapshot of an element as an accumulator value
Use explicit options to control XML parsing in the doc() and document() functions

doc("input.xml", {'dtd-validation': true()})
Build arrays in XSLT using xsl:array and xsl:array-member
Control the handling of duplicate keys in xsl:map with the new duplicates option

...
Stop JSON output displaying / as \/

escape-solidus="no"
Serialize a sequence of maps in json-lines format (one JSON object per line, newline separated)

json-lines="yes"
Use different XSD schemas for validating the input and output of a transformation

,
Write all the template rules for a given mode as children of the xsl:mode element
In an XSLT module, link to the main module to help IDEs locate all the variable and function declarations

...
Use xsl:note elements for structured XSLT documentation

This is a comment
Use fixed-namespaces to reduce the amount of boilerplate on the xsl:transform element

...
Take advantage of new features for simplified stylesheets. The outermost element can now be any instruction, for example xsl:result-document or xsl:map.
Make unprefixed name tests match elements in any namespace

xpath-default-namespace="##any"
Declare named item types
Use types in match patterns

match="~map(xs:string, xs:integer)
Use the separator attribute in xsl:for-each

...
Use record types to declare maps with statically known fields

as record(first as xs:string, middle as xs:string*, last as xs:string)
Declare function parameters using choice types

as (xs:date | xs:time | xs:dateTime)
Declare function parameters using enumeration types

as enum("red", "green", "blue")
Use path expressions to navigate trees of maps and arrays (JNodes)

$map/authors/*[1]/name/first ! string()
Iterate over arrays with an enhanced FLWOR expression

for member $m in $array ... return count($m)
Iterate over maps with an enhanced FLWOR expression

for key $key value $value in $map return `{$key} = {$value}`
Use computed node constructors in XPath

attribute #xlink:href {$target}
Use enhanced for and let expressions in XPath

for $x as xs:string at $pos in $X let $r := string-length($x) + $pos return $r + 1
Use destructuring map assignments

let {$height, $width} := $rectangle return $height × $width
Use destructuring array assignments

let [$first, $second, $third] := $array return ($first, $second, $third)
Read the specifications at http://qt4cg.org/, and provide feedback at https://github.com/qt4cg/qtspecs

Ordered Maps and JNodes

2025-08-19T09:00:00Z

Ordered Maps and JNodes

Since I last wrote about ordered maps, the XPath 4.0 specifications have now introduced the idea of JNodes, which adds a new set of requirements to the specification.

I wrote about JNodes in my paper at Balisage 2025, and I won't repeat the introduction here. Suffice it to say that a JNode can wrap a specific entry in an XDM map, and it allows you (among other things) to:

Establish which of two entries in the same map comes first in document order (or equivalently, in map order);
Get the preceding or following siblings of a map entry (the entries that precede it or follow it in map order).

The prototype implementation of JNodes that I demonstrated at Balisage included support for these operations, but the implementation would be hopelessly inefficient in production. The current implementation of ordered maps on the Saxon 13 development branch uses the LinkedHashMap implementation from the VAVR library, and this doesn't have anything that would support either of these operations.

I'm therefore proposing to use a home-grown map implementation as described below.

First note that map:put and map:remove operations are rare, and it's OK to have a map implementation that allows the map to be constructed by a mutable builder class in one phase of operation, and thereafter to be read-only. If a map:put or map:remove operation is attempted, we can rebuild the map using a different (immutable | persistent | functional) data structure. So let's first consider a structure that doesn't support map:put and map:remove operations directly. I'm calling this a FixedMap.

A FixedMap is underpinned by three Java structures:

keys: An array of keys: type AtomicValue[].
values: An array of values: type GroundedValue[].
index: A map from atomic match keys to integers: type Map>. Note that this map doesn't need to be ordered. The class AtomicMatchKey is a surrogate for an XDM atomic value whose Java equals() method respects the XDM equality semantics: for example, if the key is an xs:string then the AtomicMatchKey is an instance of Saxon's UnicodeString class.

The basic operations on XDM maps are simply and efficiently implemented as follows:

Get the value corresponding to a given key: set i = index.get(key); return if i == null then null else values[i].
Get all keys in map order: iterate over keys.
Get all values in map order: iterate over values.

The new operations required to support JNode navigation can also be efficiently implemented:

Determine which of two entries with keys X and Y is first in document order: compare index.get(X) with index.get(Y) (an integer comparison).
Get preceding or following siblings of an entry with key X: find index.get(X) and then iterate forwards or backwards in the keys and values arrays from this integer offset.

All very straightforward. Now, what about map:put and map:remove?

When one of these operations occurs, I propose to copy the FixedMap to an ExtensibleMap. This will have a very similar structure to the FixedMap, except that:

The arrays of keys and values are replaced by immutable lists of keys and values. I propose to implement these immutable lists using the existing Saxon class ZenoChain. This is described in a previous blog. This uses the same logic as the ZenoString which I introduced in a 2021 Balisage paper. It provides efficient support for getting an entry at a particular integer offset, and for appending individual entries at the end of the list: therefore for map:get and map:put.
The map from AtomicMatchKey to Integer is replaced by an immutable map. There are plenty of implementations available (we could revert to the Michael Froh implementation used in earlier Saxon releases), because there is no need for this structure to support ordering.

The design relies on the fact that adding or removing entries to the map does not change the integer offset of existing entries. That's fine for map:put, which always puts new entries at the end (in terms of map ordering). To support map:remove, I propose to keep a separate list holding the integer offsets of map entries that have been removed; a traversal of the map keys or arrays needs to check this list and skip an entry if it is present. When the number of removed entries exceeds some threshold (a rare event) we can rebuild the map from scratch.

The new design looks as if all the main operations perform in constant or logarithmic time, with traversal of the map in linear time; and it seems to be reasonably economical in terms of memory usage.

The design allows the use of an optimization whereby the index (from atomic match keys to integer offsets) is built lazily, the first time map:get is called. This is useful when processing JSON, because it's likely that many maps constructed during JSON parsing will either be copied unchanged to the transformation output, or will never be referenced at all, so the work needed to construct an index can be saved (it's still necessary to check for duplicate keys, but this can be done cheaply with a simple Bloom filter).

We currently have an optimized implementation of maps for cases where all the keys are known in advance to be instances of xs:string. This can potentially save a bit of space because in place of the StringValue representing each string-valued key, we can simply store the underlying UnicodeString. This saves holding the wrapper object and the type annotation, which can be significant in the case of very large maps.

One of the benefits of this design is that it reduces our dependencies on third-party code, which in turn simplifies porting Saxon across different platforms, and transpiling to C# for the .NET platform. In the current Saxon 13 code base, we're not only dependent on the VAVR library, but we're actually dependent on a fork of that library that uses a different implentation of linked hash maps, to avoid a performance bug.

Ordered Maps

2024-12-29T09:00:00Z

There's been a lot of discussion recently in the QT4 community group about introducing ordered maps: that is, maps in which there is a defined and predictable ordering of entries, typically "last-in, last-out" where the most recently added entry always appears at the end. The main motivation for this is that JSON is designed (like XML) to be human-readable, but JSON content in which the entries appear in random order is anything but: if a phone bill contains a name, address, account number, a summary of charges, and an itemized list of calls, then you don't want the phone number appearing in the middle, sandwiched between the list of calls and the list of charges. Currently when data is serialized as JSON, we provide an option to indent it for readability, but indentation isn't going to make the data readable if it's in random order.

Retaining order is particularly useful for visual inspection of changes: if you write code that modifies one entry in a JSON document, you want to satisfy yourself that the transformation did exactly what you expected, and the best way to convince yourself is by placing the input and output side-by-side and comparing them visually.

There seems to be consensus that support for ordered maps, at least in some circumstances, is desirable. There is debate about whether all maps should be ordered, and about whether ordering should be the default, and about whether ordering should be supported if a map is built incrementally using map:put operations. The answer to those questions depends at least in part on understanding how great the overhead is in retaining order in maps: if the overhead is negligible, then we might as well make all maps ordered.

Normally I'm the first to argue that the language specification should not be driven by performance concerns: we should design a clean language and leave implementors to worry about how to implement it efficiently. But in this case, if we're making a change to the language semantics that affects users whether they want the feature or not, I think we need to understand clearly whether we are asking users to pay a performance price.

Both JavaScript (from ES2015) and Python (from 3.7) have moved in the direction of making all maps (objects/dictionaries) ordered, so we wouldn't be on our own in doing this. However, JavaScript objects and Python dictionaries are mutable, whereas XDM maps are functionally persistent (adding an entry creates a new map, leaving the original unchanged), so the performance constraints are somewhat different.

So let's look now at how Saxon implements maps.

In SaxonJ 12.x there are two main implementations (ignoring special cases such as empty maps and singleton maps). The default implementation is in the class net.sf.saxon.ma.map.HashTrieMap, and this is built using an open source implementation of immutable hash tries written by Michael Froh; it has been in the product since 9.6. In SaxonCS 12.x we replace this with the functionally equivalent Microsoft class System.Collections.Immutable.ImmutableDictionary. Both these library implementations are unordered.

There is a minor tweak that complicates the implementation. In an ideal world, we would create an underlying map of type Map, where AtomicValue is the Saxon class used to hold all atomic values, and GroundedValue is the Saxon class used to hold all sequences other than those that are lazily-evaluated. However, AtomicValue.equals() does not implement the equality semantics defined by XDM for comparing map keys. This is because XPath has different rules for equality comparisons in different circumstances. The Microsoft ImmutableDictionary can take a custom KeyComparer parameter, which would solve this problem, but there is no equivalent in the Froh library that we use in SaxonJ. So instead we implement an underlying map of type Map>, where AtomicMatchKey is a value derived from the AtomicValue that has the correct equality semantics. We need to hold the AtomicValue because in general two atomic values can have the same AtomicMatchKey (for example this is the case when the keys are a mix of different numeric types): and the XPath functionality for maps requires the original key value (including its type annotation) to be retained.

The second implementation of maps found in SaxonJ and SaxonCS is the class net.sf.saxon.ma.map.DictionaryMap. This is implemented over a standard mutable java.util.HashMap> on Java, or System.Collections.Generic.Dictionary on .NET. It is suitable only where the keys are all instances of xs:string (which means we don't need to retain the type annotation), and where no in-situ modification takes place. As soon as an operation such as map:put or map:remove is applied to the map, we make a copy using the more general HashTrieMap implementation. But for many maps, especially those derived from JSON parsing, incremental modification is rare, and the lower-overhead DictionaryMap is perfectly satisfactory.

In Saxon 13 (not yet released), a third map implementation has been introduced: the ShapedMap. This is described in the article Maps and Records, and it is particularly useful in cases where many maps have exactly the same structure. This often happens when parsing CSV or JSON files. A ShapedMap is in two parts: a Shape object which holds a mapping from keys to integer slot numbers, and a simple array of slots holding the values of the fields. The Shape object can be shared between all map instances having a common structure. As with the DictionaryMap, if a ShapedMap is subjected to map:put or map:remove operations, it is immediately copied to a HashTrieMap.

How are these map implementations affected by the requirement to maintain order of entries?

For the ShapedMap, order is already maintained, so it isn't a problem. The only impact is that two maps can only share the same Shape object if their keys are in the same order. There isn't going to be any observable performance regression.

For the DictionaryMap, on the Java platform we can replace the underlying HashMap by a LinkedHashMap. That's easily done, because it supports the same interface. I don't yet know how much overhead it imposes (in space or time); that requires some measurements.

On .NET, unfortunately, there is no equivalent to Java's LinkedHashMap. I have therefore implemented my own: this comprises a Dictionary that maps string-valued keys to integer positions in the sequence, and two lists: a list of AtomicValue for the keys and a list of GroundedValue for the values.

For the HashTrieMap on Java, my plan is to scrap the immutable map implemented by Michael Froh, and substitute it with the io.vavr.collection.LinkedHashMap from the VAVR library, which appears to have the required semantics. Again, there appears to be no direct equivalent on .NET, so a home grown solution is again called for. My current implementation uses the same apprach as for the DictionaryMap: an immutable unordered map from atomic keys to integers, supplemented by ordered immutable lists of AtomicValue for the keys and GroundedValue for the values.

Which brings us to the question, what are the overheads? Answering that question means making some assumptions about the workloads we want to measure. For example, how important are map:put and map:remove operations? Anecdotal evidence suggests these are rather rare, and that most maps are read-only once built. But they might be important to some use cases.

The other complication is that we might be able to mitigate the overheads of making maps ordered by introducing new optimisations. We've already introduced the ShapedMap idea, where ordering hopefully imposes very little overhead. On .NET we could consider taking advantage of the ability to use a custom KeyComparer to avoid the overhead of effectively storing the keys twice.

We could also get smarter about choosing which implementation of maps to use under which circumstances. One change that I'm making is to introduce a MapBuilder class: during the initial construction of a map (for example during JSON parsing or during processing of map:merge or map:build, or during evaluation of a map constructor) we can add entries to a mutable builder object, and this then gives us the opportunity to choose the final map implementation when we know what all the keys and values are. For example, if all the keys have the same type annotation, then in principle we don't need to save the type annotations with every key value. We also know the size of the map at this stage.

We can even go further and avoid indexing the map until the first lookup (or map:get) operation. It might seem surprising, but there are many maps that are never used for lookup. For example, a JSON document might contain thousands of maps that are simply copied unchanged to the output, or that are discarded because they are irrelevant to the particular query. Perhaps the map builder should simply maintain a list of keys and values, and do nothing else until the first map:get? The only complication here is the need to detect duplicate keys, but that could be done cheaply using a Bloom filter.

So we need to do some measurements. But there's a good chance that if it does turn out that ordered maps impose an overhead, we can find compensating optimisations that mean there's no regression on the bottom line.

My first experiments looking at the cost of parsing and re-serializing JSON actually suggest that most of the cost is in the parsing and serializing, and that the choice of data structure for the XDM maps has very little impact on the bottom line. But that's provisional and subject to confirmation.

Maps and Records

2024-08-10T09:00:00Z

Maps have proved to be one of the most powerful new features in the 3.0/3.1 family of standards, and records, which extend the capability will probably prove one of the most powerful in 4.0. Under the name tuple types, the feature has been available as a proprietary Saxon extension since Saxon 9.8, which came out on the same day as the XSLT 3.0 Recommendation in June 2017. The feature is now well established, but the details are still being refined.

A record type is declared like this:

record(longitude as xs:double, latitude as xs:double)

A record type is simply a new way of constraining maps; the instances of the type are maps (in this case a map with two entries, one with key "longitude" and one with key "latitude"). You can use a record type to declare the types of variables and function arguments, but the actual value of the variable is a map, and all the standard map operations are available, such as the lookup operator: $location?longitude.

We're working on an extension that allows named record types to be declared globally:

which would also give you a constructor function: my:location($long, $lat).

The main thing I want to talk about in this article is how records can be efficiently implemented.

Until recently, a record type simply constrained the contents of a map, and had no bearing on the way the map was implemented.

Internally, Saxon represents maps using the interface net.sf.saxon.ma.map.MapItem (actually an abstract class), and there are several implementations of this interface:

EmptyMap for an empty map
SingleEntryMap for a map with one entry, such as the map created by map:entry()
DictionaryMap for a map whose keys are all strings, and that isn't likely to be modified (for example maps derived by parsing JSON, or maps written using literal constructors as option parameter values)
HashTrieMap as the general implementation that handles everything.

For the next release, Saxon 13, we've written a new implementation called a ShapedMap. There are two parts to this: a Shape is a mapping from field names to integer slot numbers, and a ShapedMap is a reference to a Shape, plus an array of slots. So it's great where you have many maps with exactly the same structure, because you only hold the keys once.

So far we're mainly using shaped maps where the structure of the map is defined by the language specification, for example for the key-value pairs returned by map:pairs(), for the results of functions such as parse-csv(), random-number-generator(), and load-xquery-module(), and for the labels attached to values by the new deep-lookup operator (plenty of scope there for future articles). I would love to use them also for the result of parse-json() if we can detect the common case of a JSON file containing thousands of maps (JSON objects) with exactly the same structure. And of course, once we have record constructor functions as described above, they are an obvious candidate for the result of such a function.

Shaped maps immediately give a space saving because the keys and their hash index are shared between instances. The next challenge after that is to make lookup on shaped maps more efficient. Given a lookup expression such as $location?longitude, we ought to be able to extract the corresponding value directly from slot 0 of the ShapedMap object, without the overhead of doing a run-time hash lookup of the string "longitude" in the corresponding Shape in order to establish that this field is always in slot 0.

The obvious, classic way of doing that is through static type inference: if we know the static type of the $location variable, then we can know at compile time what the mapping of field names to slots will be, and can generate an execution plan accordingly.

But I'm becoming a bit disillusioned with relying on static type analysis. Users, in general, are lazy: they want good performance without doing extra work, like declaring the types of all their variables. That's particularly true when you start writing code that relies heavily on higher-order functions, which we want to encourage. So I'm looking increasingly at options that decide the execution plan at run-time, modifying it in the light of actual experience. Given an expression like $location?longitude that is executed repeatedly, the chances are that if $location is a shaped map with longitude in slot 0 on one occasion, then the same will be true next time you execute the same expression.

We've quietly introduced this kind of approach in recent releases, and it's working well. For example, with lazy evaluation of variables and function results we now use a learning approach: we start with lazy evaluation, but if on the first 20 executions the value is immediately read to completion, we switch to eager evaluation. That's because lazy evaluation has a significant set-up overhead to retain the parts of the context on which the expression depends, and there is no benefit in doing this if the caller is going to immediately materialise the value anyway.

The concrete design for shaped record access goes something like this. We augment the MapItem interface with a method map.lookup(key, plan). This method returns the requested value from the map, but also updates the value of plan with information that will be retained the next time the same expression is evaluated. If the map is a shaped map, the returned plan can include the Shape and the slot number; if an incoming request comes with a plan that identifies the same Shape (which it usually will), then we can access the relevant slot number directly, ignoring the value of the key. That only works, of course, for a lookup expression where the key is a literal constant; but that's the normal case when working with records.

If we can make this work (and it seems straightforward), then the same approach might have other applications. For example, can we make path expressions go faster if we optimize for the tree model in use? Or could we get rid of statically-allocated fingerprints (with the inconvenience they cause by not allowing documents and stylesheets to be shared across configurations), and instead have the expression discover the fingerprint and NamePool at execution time?

Saxon is now 25 years old. It seems there are still plenty of exciting ways to make it better.

All Change at Saxonica

2023-10-02T09:00:00Z

Today we are announcing a big change at Saxonica, but it's one that we hope no-one will notice.

Some of you may have noticed that I will be 72 in a couple of weeks, and you may have been wondering how long I intended to continue. Well, I'm still enjoying the challenge, and I'm not giving up just yet, but I have been thinking for some time about how to move the company into a position where it can thrive without me. I decided a while ago that the best approach — in the interests of the staff and the user community — was to move the ownership of the company into an Employee Ownership Trust, and this has now been effected.

What this means is that Saxonica is now owned by a Trust, which has acquired a 100% shareholding from my wife Penny and myself. The Trustees are Norm Tovey-Walsh, Sue Schreiber, and myself, and their task is to appoint directors and approve any big decisions (such as acquisitions). When making such decisions the Trust is obliged to take into consideration the best interests of the staff and the user community, including both paying customers and open-source users.

Indeed, there's a specific clause in the trust deed noting that the company's software is used to deliver open data for the wider benefit of society at large, and the Trustees are required to take this into account.

The most obvious change will be that Norm Tovey-Walsh is taking over as Chief Executive; I will remain on the board (with the title "Director of Innovation"), but Norm is now the boss. I can't think of anyone in the world who is better qualified to steer the ship.

Back at the start of 2020, before the pandemic, I was rather conscious that with several people approaching retirement age, the company was not in a good state to carry on without me; as a result any acquisition could well have led to serious discontinuity for the user community, and indeed for the future of the XML ecosystem of which we have become such a central part. Since then, with Norm joining in a senior technical role, with Ankita Mohinta taking over the marketing and commercial side so effectively, and more recently with Matt Patterson joining O'Neil Delpratt and Debbie Lockett on the development side, we now have a team that has an unmatched depth of talent. This is reflected in excellent financial results over the last couple of years. But it's also true that I know the innards of the software better than anyone, and I don't intend to walk away from my focus on providing technical support to customers, which is probably the part of the job that I enjoy the most.

Penny is taking this opportunity to retire from her role as a director and employee of the company, and this is a good opportunity to thank her for her contribution. She may not have been very visible externally, but she's done a lot of invaluable admin work behind the scenes, and has been a constant source of good advice, most particularly in helping to build a happy and productive team — especially through the pandemic.

I'm confident that through this adminstrative change, the company will retain the core values that have been central to our success: innovation, commitment to standards, outstanding technical support, honesty and integrity, admitting to our mistakes when we make them, and above all being active contributors to the thriving community without which we would not exist. I hope to remain part of that for many years to come.

How Safe is your Schema?

2023-07-22T12:00:00Z

When you validate a document, you expect to set the rules for what it can contain. If you specify that your nesting box can only contain wrens and robins, you don't want any cuckoos in there.

So you write a schema nesting-box.xsd like this:

And you're now comfortable that any that passes validation will look something like this:


   Nice!
   Nicer!
   Nicest!

You're wrong!

The following document also passes validation:


   Nice!
   Nicer!
   Nicest!
   Horrid!

Here cuckoo.xsd is another schema document, like this:

So you thought you were constraining what could appear in the document, and the user found a way past your defenses, submitting a document that your code probably can't handle.

Saxon does allow you to disable use of xsi:schemaLocation, but it's enabled by default. I'm inclined to think the default needs to be changed.

Schema Modularity

2023-06-20T12:00:00Z

Saxon, ever since we first introduced schema awareness back in 2004, has always worked with a single global schema maintained at the level of the Configuration object. This article discusses the advantages and disadvantages of this approach, and looks at possible alternatives.

Let's start by getting some terminology clear. A schema, in the terminology of the XSD specification, is a set of schema components (such as element declarations and type definitions). Don't confuse it with a schema document, which is an XML document rooted at an xs:schema element. A schema is what you get when you compile a collection of schema documents linked to each other using xs:include and xs:import declarations.

Not every set of schema components constitutes a valid schema. Most obviously you can't have two different components (for example, two type definitions) with the same name, unless they are identical. The XSD specification is a bit fuzzy about what it means for two components to be identical.

This means that in general, you can try and combine two schemas into one by taking their union, but the operation won't always succeed because the two schemas may be found to be inconsistent with each other.

The Global Schema

Saxon currently maintains a global schema at the level of the Configuration object. This means that every time you introduce a new schema, for example by compiling a schema-aware query or stylesheet that has an import schema declaration, or by validating a document against a schema loaded using the SchemaManager API, or referenced using xsi:schemaLocation, the schema components from that schema are added to the global pool, provided they are consistent with the declarations already present in the pool.

These consistency checks are of two kinds:

Firstly, the new components must not have names that clash with the old. You can't have two different types with the same name. To test whether two types are the same, Saxon basically checks that they came from the same place: the same line number in the same source schema document.
Secondly, the new components must not change the semantics of existing components, if the existing components have already been used for validation purposes. The test whether components have been used is done at the level of a target namespace: if any declaration or definition in a target namespace has been used for validating a source document, or if it has been used when compiling a stylesheet or query, then the namespace is sealed, which means that its components must not be extended or redefined by any new components. We'll talk later about exactly what this means.

The main benefit of the global schema approach is that you can always be sure that type annotations in validated instance documents are consistent with types that are mentioned (or inferred) in compiled queries and stylesheets. If a query is compiled believing that element E will always be empty, then you can be sure that every validated instance of E will be empty, because no-one is allowed, between compilation and validation, to add a type definition that extends or redefines E allowing it to be non-empty. That's the theory, anyway.

The most obvious disadvantage of the approach is that an application can't work with two different versions of the same schema. If you want to write a stylesheet that transforms input documents from V1 to V2 of the same schema, you can't import both versions into the same stylesheet, one to validate the input and one to validate the output. In fact, you can't even have both versions in the same Configuration — which means you can't process an input collection containing a mix of different versions (or if you do, you have to forgo validation).

There are other less obvious disadvantages. One of them is revealed by a recent embarrassing bug where we discovered that schema compilation isn't thread safe: you can't reliably run two schema compilations within a single Configuration at the same time. We've patched that by adding some locking, but it's an imperfect solution because the lock is rather coarse-grained. We need to find a better solution, and that gives us an opportunity to re-examine the design and see whether we can fix some other long-standing issues at the same time.

Another outstanding issue is a long-standing bug #3531, concerning a situation where two independently-loaded schemas X and Y both extend the same substitution group. This has remained outstanding because we have had no reports of users being affected by it; but it remains an unsatisfactory state of affairs.

The X+Y Problem

Suppose that X and Y are valid schemas. Then we've already seen that their union, which I will call X+Y, is not necessarily a valid schema; their declarations might be inconsistent. Apart from the obvious inconsistencies where X and Y contain different elements or types with the same name, there can be much more subtle inconsistencies:

Y might contain a type that redefines a type in X.

The semantics of xs:redefine as defined in XSD are a bit fuzzy round the edges, but the effect is supposed to be global: if a type is redefined, then all references to that type are affected. Exactly what happens when the same type is redefined more than once isn't made clear.

The introduction of xs:override in XSD 1.1 makes this problem even worse, because xs:override allows existing definitions to be changed in completely arbitrary ways.
Y might contain a type that extends a type in X, by allowing additional child elements or attributes. To be valid, instances that use the extended type have to identify themselves with an xsi:type attribute, but this doesn't alter the fact that adding the extended type to the global schema makes some things valid that were previously invalid. It's bad news if a document that was invalid at breakfast time suddenly becomes valid at tea time because a new schema has been added to the Configuration. It's even worse news if the schema changes during the course of validating an input document.
Y might contain an element declaration that adds to the substitution group of an element defined in X. Again, this means that a document that was invalid when validated against schema X becomes valid when validated against X+Y.
X might contain a wildcard, for example . Suppose an instance document matches this wildcard with an element named W, and is valid (under the rules for lax validation) because the schema contains no element declaration for W. Now schema Y comes along and adds a definition for element W, and suddenly our source document is no longer valid.

It doesn't really help that all these situations are rare. Should the processor simply ignore the problem and hope it doesn't happen? For the first three cases above, Saxon prevents the situation occurring, which imposes an inconvenience on users who are actually doing something completely safe. For the final case (wildcards), Saxon ignores the problem, which creates the theoretical risk that queries and stylesheets are not type-safe: a document that has been validated against a type T might not satisfy all the contraints that the query or stylesheet processor assumes to be true for any valid instance of T.

An Alternative: Modular Schemas

Let's consider an alternative model, where instead of adding all schema components to a single global schema at Configuration level, we keep schemas independent and modular. So two stylesheets that import different schema documents have separate unrelated schemas, and there are no requirements that the two schemas should be consistent with each other.

The challenge now is to ensure that a source document validated against a schema S1 is consistent with a stylesheet that imports schema S2. If the two schemas are identical, there's no problem (and it's not too hard to detect that they are identical, for example if they load the same schema document as their starting point).

But what if S2 is a superset of S1? Suppose the document is validated against a schema with target namespace X, while the stylesheet has two xsl:import-schema declarations, for namespaces X and Y? We're now back with the X+Y problem: a document that is valid against X is not necessarily valid against X+Y.

It gets worse: if we have a pipeline of stylesheets, each of which imports schemas for both its input document and its output document, then the first stylesheet might import schemas for X+Y, and the second for Y+Z, and we need to be sure that when the first stylesheet validates its output against X+Y, the result will also be valid input against Y+Z.

One possible solution here is to keep the imported schemas within a single stylesheet separate. Import one schema for the input, and another for the output, and don't require the two to be consistent. This also solves the problem of transforming from V1 to V2 of the same schema. So in our pipeline, the output of the first stylesheet would be validated not against X+Y, but merely against Y, which is the same schema used for the input of the second stylesheet. This would need language changes: xsl:import-schema declarations would need to identify which schema they belong to, and type names used in type or as attributes would need to qualify the type name with a schema name.

I've started doing work to allow free-standing schemas to be constructed and used for validation, independently of the Configuration. There are clearly cases where this is useful. However, there's a lot more work to be done on ensuring consistency of free-standing schemas, when a validated document is used as input to a schema-aware stylesheet or query. Expect a new class of (initially bewildering) error messages saying that element E is known to be valid against type T in schema X. but it isn't known to be valid against type T in schema Y. Hopefully these will be rare.

What about Wildcards?

I mentioned that there's an open issue with wildcards: if a schema type include a lax wildcard, then an element that's valid against that schema (because there's no element declaration matching the actual element name) can become invalid when more element declarations are added.

This isn't the only issue with wildcards. XSD 1.1 allows you to say, for example notQName="##defined" which means that the name used for an element or attribute must be one that has no global declaration anywhere in the schema. That's another example of how adding new declarations to a schema can make existing content invalid.

I think the answer to this problem is to interpret these definitions in the context of a "schema compilation unit". That is, when you compile a schema, notQName="##defined" is interpreted as meaning "not a name used for a global element/attribute declaration in the that schema"; any names or declarations added later (by merging this schema with others) have no effect on the meaning.

This seems to solve the problem whether using a global schema or local free-standing schemas, and makes the two cases behave more consistently and predictably.

The pattern match="para except appendix//para"

2022-05-26T15:15:00Z

The pattern `match="para except appendix//para"`

If you saw this pattern in an XSLT stylesheet, I can guess your reaction: I haven't seen a pattern like that before. Cool, a neat way of matching paragraphs that aren't in an appendix. Must remember that and use it myself.

Sadly, it doesn't do what you think. Consider this input document:



    
        Ipsum lorem.

You'd probably be as surprised as I was to see that the Ipsum lorem paragraph in this example matches the pattern para except appendix//para.

To see why this is true, go to the spec, section 5.5.3:

An item N matches a pattern P if the following applies, where EE is the equivalent expression to P: N is a node, and the result of evaluating the expression root(.)//(EE) with a singleton focus based on N is a sequence that includes the node N.

So, this is saying that a node matches the pattern if it is selected by the expression root(.)//(para except appendix/para). Assuming that we're in a tree rooted at a document node, that means it must be selected by the expression /descendant-or-self::node()/(para except appendix//para).

Now, in our example document, one of the nodes selected by /descendant-or-self::node() is the section element; and when we evaluate (para except appendix//para) starting at the section element, the first operand (para) selects our paragraph, and the second operand (appendix//para) doesn't select it, so the expression as a whole selects it, and therefore it matches the pattern.

That's totally counter-intuitive, and it's certainly not what the Working Group intended. It's a nasty bug. So the question is, what can we do about it, given that this is a published spec and there are implementations out there, and user applications that depend on it?

Is there anything we can do about it?

Perhaps we should start by asking: what would we like the spec to say, if we had the opportunity to change it?

Given that we already have a special rule for patterns with a top-level union operator (see §6.5 rule 2), we could add a special rule for patterns with a top-level intersect or except operator: a pattern of the form A except B matches an item if pattern A matches the item and pattern B does not. (And analagously for intersect.)

If that's what we think we need to do, that leaves two challenges:

Changing the spec (given there is no longer a Working Group to maintain it).
Changing the Saxon implementation.

Starting with the second point, there are several possibilities:

Just do it, and hope we don't break any existing applications.
Support both the old and new semantics concurrently, with some mechanism for selecting which to use. (Which should be the default? We want new users not to fall into the elephant trap, but we also don't want to break working applications.)
Deprecate the syntax, and provide new syntax for the new semantics (e.g. operators spelled and-also or but-not). Note however, that it's likely most applications currently using except in a pattern are using unproblematic patterns like @* except @code.

The third option seems the most satisfactory. And that suggest a route forward for the spec: in XSLT 4.0, if and when we manage to get it defined, deprecate the except and intersect operators at the top level of a pattern, and replace them with new operators that have the expected intuitive semantics.

XML versus JSON: A Comparison using the XMark Benchmark

2022-05-06T11:00:00Z

For many years we have used the XMark benchmark to check for performance regression between Saxon releases, and to evaluate the impact of internal changes on query performance.

XMark originated with the MonetDB project and is described at https://projects.cwi.nl/xmark/. It consists of a scaleable XML data file (produced using a data generator), and a set of 20 XQuery queries to be run against that data file. We have run the data generator to produce files with nominal sizes of 100Kb, 1Mb, 4Mb, 10Mb, and 100Mb; we use the original queries as published, except for one or two small changes to correct errors in the original publication.

Recently we have converted these data files to JSON, and have produced equivalent XQuery 3.1 queries to deliver the same results as the original. The queries still produce XML rather than JSON output, so that we can compare the results; except in a few cases where large chunks of the original XML are copied to the output, which we cannot reproduce exactly because we don't have the original XML available. The results also differ because JSON maps don't retain order.

In this article I will report first on the data conversion; then on the query conversion; and finally on performance results.

Converting the data

I didn't attempt to use any off-the-shelf XML-to-JSON conversion tools. My instinct is that they wouldn't have done a very good job, and I would have needed an XSLT transformation to refine the output anyway, so I decided to do the whole job using XSLT 3.0.

The conversion stylesheet is not particularly interesting; in fact, it's rather tedious. A few points are worth mentioning:

As mentioned in What should we do about arrays? the XSLT 3.0 spec is weak on capabilities for constructing arrays. This is the only area where we used Saxon extensions. I'm not convinced we yet have a perfect solution to this problem, and I've proposed some new ideas at Constructing Arrays.
The XMark data files are mainly structured data, but there is some use of mixed content for narrative text. Mixed content is bad news for JSON. In real life, I would probably have handled this by embedding XML or HTML fragments within character strings in the JSON structure; but that seemed to be against the spirit of this exercise, so I instead expanded the node structure of the mixed content using JSON maps and arrays. As we'll see later, this had a severe effect on the ease of writing the queries and on their performance.
Most structured data elements in the original file fall into two categories: elements with homogenous element content (multiple children all having the same name), and elements with heterogeous element content (multiple children with different names). These translate very naturally into JSON arrays and maps respectively. A few cases weren't quite so simple: for example the content model for open_auction has the form (initial, bidder*, current, privacy, seller, ...). We handle this as if there were a wrapper element bidders around the sequence of bidder elements, so open_auction translates to a map, and bidders converts to an array. The names of elements within an array are generally dropped.
There are a few attributes in the XML; these posed no particular problem.

The nominal 10Mb file is actually 11,875,066 bytes in its XML form, and 10,464,266 bytes when converted to JSON, a reduction of 13%. Some of this difference (perhaps 200Kb) is due to unnecessary whitespace in the XML; the rest is the overhead of element end tags.

Parsing the XML and building a Saxon TinyTree took 353ms; parsing the JSON and building a structure of XDM maps and arrays took 636ms. I haven't attempted to assess the memory usage of the two data structures, but the maps and arrays are almost certainly larger. This is despite the fact that for maps derived directly from JSON parsing, we use a specialized representation of maps that optimizes for the fact that all keys are instances of xs:string, and therefore don't need to retain a type annotation.

Converting the Queries

The queries were converted by hand. Generally we tried to change the query so it continued to produce the same (XML) output as the original, for ease of comparing results; but for queries whose output copies sizeable chunks of the input XML, we abandoned this principle, instead replicating the intent of the query as far as we could.

In most cases the conversion is very straightforward. For example, this is Q3:


(: Q3. Return the IDs of all open auctions whose current
     increase is at least twice as high as the initial increase. :)

for    $b in /site/open_auctions/open_auction
where  $b/bidder[1]/increase * 2 <= $b/bidder[last()]/increase
return

Which turns into:


(: Q3. Return the IDs of all open auctions whose current
     increase is at least twice as high as the initial increase. :)

for    $b in ?open_auctions?*
where  $b?bidders?*[1]?increase *2 <= $b?bidders?*[last()]?increase
return

Some observations:

We have to use bidders?*[1] rather than bidders?1 because the latter expression throws a dynamic error (rather than returning an empty sequence) for an auction in which there are no bidders.
We use bidders?*[last()] to get the last item in an array because converting the array to a sequence and using a filter is simpler than the alternative of writing bidders?(array:size($b?bidders)).
The element name site is dropped because the JSON file is a map in which open_auctions is a top-level entry.
The element name open_auction is dropped because the open_auctions entry in the JSON contains an array of objects which do not need to be named; the ?* in the JSON query corresponds to the /open_auction in the original.
The JSON form introduces the name bidders as a wrapper for the group of individual bidder elements in the XML (which become anonymous in the JSON).

Some specific difficulties that were encountered in converting other queries:

Query Q4 looks for auctions where one bid differs from a previous bid, and for this purpose it uses the operator << to test the relative order of two nodes in document order. The JSON model offers no equivalent. To solve this I introduced a higher-order function index-where(array, predicate) which returns the index positions in an array of members that satisfy the given predicate; it is then possible to find the required two items and compare their index positions.
For any query using the mixed content description field, I needed to include a recursive function that reconstructs the description as text by flattening the arrays and maps that make it up. This is tedious and expensive. Queries that do a deep dive into the description, like looking for text marked up with keyword tags at any depth, are even more complicated. The map:find function sometimes does what's needed, but not if any context is involved (for example, finding a keyword that's nested within emph markup).
Debugging mistakes can be tricky. The diagnostics you get when you write ?closed_auctions?annotation instead of ?closed_auctions?*?annotation aren't always helpful. I've tried to improve them. I've also proposed a language change so the first expression becomes valid: see Lookup operator on arrays of maps.
It's very easy to forget that if $A is an array, then $A[$index] and $A[condition] are both valid, but neither means what you think, because they treat the array as a single item, not as a collection. With arrays derived from JSON, every member of the array (discounting any nulls) is a singleton, so you can always write $A?*[$index] or $A?*[condition] instead.

Query Performance

For most of the queries, the JSON query was a little slower than the XML version. Queries in this category include:

Query	XML timing (ms)	JSON timing (ms)	Ratio (%)
q1	0.2649	0.6845	258%
q2	0.4861	0.6588	136%
q5	0.2711	0.3190	118%
q8	1.9359	2.3572	122%
q10	11.3329	14.3428	127%
q11	93.5360	144.1105	154%
q16	0.4183	0.8489	203%
q17	0.5964	0.8887	149%
q20	1.2380	2.2084	178%

How do we account for these numbers? My theory (based on gut feeling) is that the XML queries are faster because of the use of integer fingerprints for name matching in the TinyTree. Look at q1, for example, which in the original is:

Q1: for $b in /site/people/person[@id="person0"] return $b/name

(The XMark queries were written by someone who felt that everything ought to be written as a FLWOR expression. It can of course be simplified to a simple XPath. I'm surprised they didn't use a where clause...)

The child and attribute axis steps here (child::people, child::person, attribute::idetc) are implemented in the TinyTree by a sequential search of node entries testing each one for an integer namecode. By contrast the JSON equivalent is:

Q1: for $b in ?people?*[?id="person0"] return $b?name

and this involves string-based lookups in a hash table. Because the fan-out is fairly small, the sequential search wins.

To test this theory, I ran the XML queries using DOM rather than TinyTree as the tree model. Navigation in the DOM uses string matching on element and attribute names. The DOM queries are dramatically slower than the TinyTree: q1: 0.2947 q2: 9.1684 q5: 5.1841 q8: 49.4798 q10: 116.8379 q11: 402.2151 q16: 6.5635 q17: 44.1887 q20: 179.2854.

In the next group of queries, the JSON query is slightly faster:

Query	XML timing (ms)	JSON timing (ms)	Ratio (%)
q3	1.3507	1.2656	94%
q6	0.2870	0.0316	11%
q9	3.2959	2.2320	68%
q12	32.3911	29.2320	90%
q18	0.3134	0.2865	91%
q19	4.9937	4.6699	93%

Query q6 is clearly an outlier. This query counts descendants: the original XML formulation is:

Q6: for $b in /site/regions/* return count ($b//item)

As it happens, item elements cannot appear at any depth, so the return clause could equally have been written count($b/item). In writing the JSON query I took advantage of this knowledge, and wrote the query as:

Q6: map:for-each(?regions, function($k, $v){a:size($v)})

This runs faster firstly because of this simplification, and secondly because the size of a map can be determined in constant time, whereas counting the number of children of an element requires actually scanning them.

For the other queries where there is a small speed-up, the cause is less obvious, but it's usually possible to hazard a guess. Some of them, for example, involve arithmetic and numeric comparisons, and the JSON queries in such cases avoid the overhead of converting strings to numbers on the fly (instead, the conversion is done during JSON parsing). We know from profiling that these conversions, especially if they occur in a filter predicate, can dominate query execution time.

For the final group of queries, the JSON performance is chronically worse:

Query	XML timing (ms)	JSON timing (ms)	Ratio (%)
q7	1.0953	87.4869	7987%
q13	0.3635	15.1646	4171%
q14	12.4252	138.0764	1111%

These three queries all involve access to the description of an item, which in the XML representation is a mixed-content field (text with inline markup). As remarked earlier, this has been represented in JSON by expanding the node tree to a structure of arrays and singleton maps. As a result, a query like this one:

Q14: for $i in /site//item where contains ($i/description,"gold") return ($i/name, $i/description)

becomes thoroughly contorted (and inefficient) in the JSON representation: it is necessary to write a recursive function that assembles the description (sans markup) as a string before executing the contains() function. Even then, the JSON query doesn't faithfully reproduce the original, because it outputs the description as a string, losing the internal markup.

Conclusions

First, if you've got mixed content (text with inline markup) then you probably don't want to be using JSON. If you must use JSON, use XML or HTML within character strings in cases where inline markup is needed.

Secondly, for structured data it's a fairly even match; the differences aren't large enough to be critical for most applications. In Saxon, XML does slightly better on balance. This assumes, however, that for the XML case you are using an efficient model like the Saxon TinyTree, rather than a general-purpose DOM.

We found a few cases where the expressive power of XQuery 3.1 for querying JSON structures has gaps and omissions. Notably, searching for descendants in the tree is difficult; operations based on ordering of items within arrays are also tricky.

What should we do about Arrays?

2021-06-27T15:34:00Z

Arrays were added to the data model for XPath 3.1 (and XQuery 3.1): the main motivation was the need for faithful representation of JSON data structures, while a secondary consideration was the long-standing requirement for "sequences of sequences".

Processing support for arrays in the current languages is rather limited. There's a basic set of functions available, but not much else. Support in XSLT 3.0 is particularly weak, because XSLT 3.0 was primarily designed to work with XPath 3.0 (which didn't have arrays), with 3.1 support added as something of an afterthought.

This note surveys where the gaps are, and how they should be filled.

Many of the complications in processing arrays arise because the members of an array can be arbitrary sequences, not just single items. There were two reasons for this design. One is simply orthogonality: the principle of no unnecessary restrictions. The other was support for the JSON null value, which maps naturally to an empty sequence in XDM, but only if an array is allowed have an empty sequence as one of its members.

Array Construction

XPath 3.1 offers two constructs for creating arrays: the "square" and "curly" constructors. Neither is completely general. The "square" constructor (for example [$X, $Y, $Z]) can construct an array with arbitrary values as its members, but the size of the array needs to be known statically. The "curly" constructor (for example array{$X, $Y, $Z}) can construct an array whose size is decided dynamically, but the members of the array must be singleton items (not arbitrary sequences). The WG failed to come up with a construct for creating an array where both the size of the array and the size of each member are determined dynamically. The only way to achieve this is with a fairly convoluted use of functions such as array:join().

XSLT 3.0 has no mechanism for array construction. An xsl:array instruction has been proposed, and is prototyped as saxon:array in current Saxon releases; but the difficulty is in defining the detail of how it should work. It makes sense for it to enclose a sequence constructor, so instructions like xsl:for-each and xsl:choose can be used when building the content. But sequence constructors deliver sequences of items, not sequences of sequences. So the current proposal for XSLT 4.0 envisages an xsl:array-member instruction that wraps a sequence as a zero-arity function. The problem with this is that the mechanism is transparent yet arbitrary; it looks like (and is) a kludge.

Array Processing

Similarly, there are limited options for processing of arrays. There's no equivalent of the "for" clause in FLWOR expressions that binds a variable to each member of an array in turn. The closest things on offer are the array:filter() and array:for-each() higher order functions – which are more useful in XQuery than in XSLT, because of the difficulty in XSLT of writing an anonymous function that constructs new XML element nodes. XSLT in particular relies heavily (in constructs such as xsl:apply-templates, xsl:for-each, xsl:iterate, and xsl:for-each-group) on binding values implicitly to the context item. But the context item is an item, not an arbitrary value, so binding members of arrays to the context item isn't an option.

Generalizing "." to represent an arbitrary value rather than a single item seems an attractive idea, but it's very hard to do without breaking a lot of existing code.

Iterating over an array and binding each member to a variable works well in XQuery, where adding a "for member" clause to FLWOR expressions works cleanly enough. But there's lots of other functionality for processing sequences that can't be translated easily into equivalent mechanisms for arrays, especially in XSLT.

Parcels

It seems that a solution for both array construction and array processing is to find a way to pack an arbitrary sequence into a single item. We'll refer to a "sequence packed into an item" as a parcel. We can then construct an array from a sequence of parcels, and we can decompose an array into a sequence of parcels, allowing both operations to be implemented using all the existing machinery for handling sequences.

It seems that four operations are sufficient to fill the processing gap:

Wrap a sequence as a parcel
Unwrap a parcel as a sequence
Construct an array from a sequence of parcels
Decompose an array into a sequence of parcels

So four functions should do the job: parcel(item()*) => P, unparcel(P) => item()*, array:of(P*) as array(*), array:members(array(*)) as P*, where P is the item type of a parcel. Of course, we can also add XSLT or XQuery syntactic sugar on top of these building blocks.

We now have to address the question: what kind of item is a parcel? Is it represented using something we already know and love (like an array, or a zero-arity function) or is it something new? How should the type of a parcel be represented in type signatures, and what operations (apart from the above four) should be available on them?

I'm beginning to come to the conclusion that the type safety that comes from treating a parcel as a new kind of item justifies the extra complexity in the type system. If we reuse an existing kind of item (for example, zero-arity functions), then there's always going to be confusion about whether items of that type are to be treated as parcels or as their "ordinary selves".

However, I'm reluctant to add yet another fundamental type. We can't keep adding fundamental types, and new syntax, every time we need something new (cf my Balisage 2020 paper on adding promises). Can't we make the type system more extensible?

Pro tem, I suggest we build on the concept of "extension objects" defined in §25.1.3 of the XSLT specification. These are intended as opaque objects that can be returned by one extension function and supplied to another. This concept should really be defined in XDM rather than in XSLT. We should add that an "extension object" may be an instance of an "extension type", and that extension types are denoted in the ItemType syntax by a QName (that is, the same syntax as atomic types), with the QName being made known to the processor in some implementation-defined way. Then we reserve a namespace URI sys for "built in extension types", and define sys:parcel as such a type.

Saxon-CS says Hello World

2021-03-22T10:34:00Z

The Saxon product on .NET has been living on borrowed time for a while. It's built by converting the Java bytecode of the Java product to the equivalent .NET intermediate language, using the open-source IKVM converter produced by Jeroen Frijters. Jeroen after many years of devoted service decided to give up further development and maintenance of IKVM a few years ago, which didn't immediately matter because the product worked perfectly well. But then Microsoft in 2019 announced that future .NET developments would be based on .NET Core, and IKVM has never supported .NET Core, so we clearly had a problem.

There's a team attempting to produce a fork of IKVM that supports the new .NET, but we've never felt we could put all our eggs in that basket. In any case, we also have performance problems with IKVM that we've never managed to resolve: some applications run 5 times slower than Java, and despite a lot of investigation, we've never worked out why.

So we decided to try a new approach, namely Java-to-C# source code conversion. After a lot of work, we've now achieved successful compilation and execution of a subset of the the code, and for the first time this morning, Saxon-CS successfully ran the minimal "Hello World" query.

We're a long way from having a product we can release, but we can now have confidence that this approach is going to be viable.

How does the conversion work? We looked at some available tools, notably the product from Tangible Solutions, and this gave us many insights into what could be readily converted, and where the remaining difficulties lay; it also convinced us that we'd be better off writing our own converter.

The basic workflow is:

Using the open source JavaParser library, parse the Java code, generate an XML abstract syntax tree for each module, and annotate the syntax tree with type information where needed.
Using XSLT code, do a cross-module analysis to determine which methods override each other, which have covariant return types, etc: information needed when generating the C# code.
Perform an XSLT transformation on each module to generate C# code.

We can't convert everything automatically, so there's a range of strategies we use to deal with the remaining issues:

Some constructs can simply be avoided. We have trouble, for example, converting Java method references like Item::toString, because it needs a fair bit of context information to distinguish the various possible translations. But it's no great hardship to write the Java code a different way, for example as a lambda expression item -> item.toString(). Another example is naming conflicts: C# doesn't allow you, for example, to have a variable with the same name as a method in the containing class. It's no hardship to rename the variables so the problem doesn't arise.
We can use Java annotations to steer the conversion. For example, sometimes we want to generate C# code that's completely unrelated to the Java code. We can move this code into a method of its own, and then add an annotation @CSharpReplaceMethodBody which substitutes different code for the existing method body. The annotation is copied into the XML syntax tree by the JavaParser, and our converter can pick it up from there.
We already have a preprocessor mechanism to mark chunks of code as being excluded from particular variants of the product (such as Saxon-HE or Saxon-PE). We can make further use of this mechanism. However, it's limited by the fact that the code, prior to preprocessing, must be valid Java so that it works in the IDE.

The areas that have caused most trouble in conversion are:

Inner classes. C# has no anonymous inner classes, and its named inner classes correspond only to Java's static inner classes. Guided by the way the Tangible converter handles these, we've found a way of translating them that handles most cases, and we've added Java annotations that provide the converter with extra information where additional complexities arise.
Enumeration types. C#'s enumeration types are much more limited than the equivalent in Java, because enumeration constants can't have custom methods associated with them. We distinguish three kinds of enumeration classes: singleton enumerations (used to implement classes that will only have a single instance); simple enumerations with no custom behaviour, which can be translated to C# enumerations very directly, and more complex enumerations, that result in the generation of two separate C# classes, one to hold the enumeration constants, the other to accommodate the custom methods.
Generics. C# is much stricter about generic types than Java, because the type information is carried through to run-time, whereas in Java it is used only for compile-time type checking, which can be subverted by use of casting. So the rule in C# is, either use generics properly, or don't use them at all. We anticipated some of these issues a year or two ago when we first started thinking about this project: see Java Generics Revisited. The result is that the classes representing XDM sequences and sequence iterators no longer use generics, which has saved a lot of hassle in this conversion. But there are still many problems, notably (a) the type inference needed to support Java's diamond operator (as in new ArrayList<>(), where an explicit type parameter is needed in C#), and (b) the handling of covariant and contravariant wildcards (? extends T, ? super T.)
Iterators and enumerators. A for-each loop in Java (for (X x : collection)) relies on the collection operand implementing the java.lang.Iterable interface. To translate this into a C# for-each loop (foreach (X x in collection)) the collection needs to implement IEnumerable. So we convert all Iterables to IEnumerables, and that means we have to convert Iterators to Enumerators. Unfortunately Java's Iterator interface doesn't lend itself to static translation to a c# IEnumerator: in Java, the hasNext() method is stateless (so you can call it repeatedly), whereas C#'s MoveNext changes the current position (so you can't). We're fortunate that we only make modest use of Java iterators; in most of the code, we use Saxon's SequenceIterator interface in preferance, and this converts without trouble. We examined all the cases where Saxon explicitly uses hasNext() and next(), and made sure these followed the discipline of calling hasNext() exactly once before each call on next(); with this discipline, converting the calls to MoveNext() and Current works without problems.
Lambda expressions and delegates. In Java, lambda expressions can be used where the expected type is a functional interface; a functional interface in other ways is just an ordinary interface, and you can have concrete classes that implement it. So for example the second argument of NodeInfo.iterateAxis(axis, nodeTest) is a NodeTest, for which we can supply either a lambda expression (such as it -> it instanceof XSLExpose), or one of a whole range of implementation classes such as a SchemaElementTest, which tests whether an element belongs to an XSD-defined substitution group. In C#, lambda expressions can only be used when the expected type is a delegate, and if the expected type is a delegate, then (in effect) a lambda expression is the only thing you can supply. The way we've handled this is generally to make the main method (like iterateAxis() expect a non-delegate interface, and then to supply a proxy implementation of this interface that accepts a delegate. It's not a very satisfactory solution, but it works.

One area where we could have had trouble, but avoided it, is in the use of the Java CharSequence class. I wrote about this issue last year at String, CharSequence, IKVM, and .NET. As described in that article, we decided to eliminate our dependence on the CharSequence interface. For a great many internal uses of strings in Saxon, we now use a new interface UnicodeString which as the name implies is much more Unicode-friendly than Java's String and CharSequence. It also reduces memory usage, especially in the TinyTree. But there is a small overhead in the places where we have to convert strings to or from UnicodeStrings, which we can't hide entirely: it represents about 5% on the bottom line. But it does make all this code much easier to port between Java and C#.

What about dependencies? So far we've just been tackling the Saxon-HE code base, and that has very few dependencies that have caused any difficulty. Most of the uses of standard Java library classes (maps, lists, input and output streams, and the like) are handled by the converter, simply translating calls into the nearest C# equivalent. In some cases such as java.util.Properties we've written en emulation of the Java interface (or the parts of it that we actually use). In other cases we've redirected calls to helper methods. For example we don't always have enough type information to know whether Java's List.remove() should be translated to List.Remove() or List.RemoveAt(); so instead we generate a call on a static helper method, which makes the decision at runtime based on the type of the supplied argument.

The only external dependency we've picked up so far is for handling big decimal numbers. We're currently evaluating the BigDecimal library from Singulink, which appears to offer all the required functionality, though its philosophy is sufficiently different from the Java BigDecimal to make conversion non-trivial.

One thing I should stress is that we haven't written a general purpose Java to C# converter. Our converter is designed to handle the Saxon codebase, and nothing else. Some of the conversion rules are specific to particular Saxon classes, and as a general principle, we only convert the subset of the language and of the class library that we actually need. Some of the conversion rules assume that the code is written to the coding conventions that we use in Saxon, but which might not be followed in other projects.

So, Hello World to Saxon-CS. There's still a lot of work to do, but we've reached a significant milestone.

The Zeno Chain: a new data structure for XDM sequences

2021-03-18T15:34:00Z

This article presents the Zeno Chain, a new data structure used to underpin the implementation of XDM sequences in Saxon. It is also designed to be capable of supporting XDM arrays, and might also have potential for holding long character strings.

The current implementation of the Zeno Chain is a mutable list, but the design lends itself easily to creating an immutable variant. It also makes it easy to construct an immutable list from a mutable one, making it efficient to construct a sequence with in-situ modification, and then "freeze" it once construction is complete.

Saxon currently uses a variety of structures for holding sequences and arrays. This variety is a problem in itself. Choosing the right structure for a particular scenario involves somewhat hit-or-miss decision making; it would be better to have a single "all-rounder" structure that performs well in a variety of situations.

There are of course vast numbers of data structures for sequences available in the computer science literature. One promising one, for example, is the "finger tree" which supports a wide range of access patterns efficiently. But it also has drawbacks: any tree structure that requires a node for each item in a list is going to have a large memory overhead when storing a long sequence, and the use of a fine-grained structure like this tends to mean that there is little locality of reference for memory addressing, leading to poor CPU caching performance.

The Zeno chain stores a sequence as a list of lists of items: that is, it is a tree with a constant depth of 2. In the Java implementation, both levels of list are instances of java.util.ArrayList. The key to the performance of the structure is managing the number and size of the second-level lists, which I call segments.

In a list that is constructed by appending individual items on the end (a common scenario), the length of a segment increases the closer it is to the start. For a list of 20,000 items, there are ten segments whose sizes are (8192, 4096, 4096, 2048, 1024, 256, 128, 64, 64, 32). (Now you know why I called it a Zeno chain.) The exact numbers don't matter here: what is important is that the total number of segments increases only logarithmically with the length of the sequence, and that the segments on the right are short, which makes further append operations efficient.

In a list constructed by prepending individual items, the distribution of lengths will be the other way round: shortest segments near the front. In the rare case where both append and prepend operations occur, both ends will have short segments, while longer segments will cluster around the middle.

Here's a summary of the major operations performed on the sequence:

Append an item: if the list is empty, construct a single segment of length 1. Otherwise, if the last segment has length < 32, append to it. If the last segment is already full, coalesce the last segment with the previous segment if the previous segment has sufficient room; if not, work up the list to the start to find adjacent segments that can be merged. A segment is considered to have sufficient room for such expansion if its resulting size would not exceed 2^(N+5) where N is the distance of the segment from the right-hand end of the sequence; it's this formula that ensures that longer segments accumulate at the start of the sequence. If all segments in the sequence are full — that is, if the segment sizes are decreasing powers of two — then add a new segment. Append operations essentially take constant time; 97% of them only affect the final segment.
Prepend an item: simply append in reverse.
Get the Nth item: search the master list of segments examining the sizes of the segments until the right segment is found, then get the item by addressing into the Java ArrayList. This takes logarithmic time. The average access time will be slightly higher in a list built by prepending items, because the chance of finding the required item in the first couple of segments is much lower.
Subsequence: make a new Zeno chain containing whole or part copies of the segments from the original chain that are in the required range.
Iteration: Keep two index positions, the index position in the master list, and the index position in the current segment, and use these indexes to retrieve the next item by calling ArrayList.get() twice.
Sequence concatenation: This is quite a common operation in XSLT and XPath, as it's the basis of the "flattening" operations such as xsl:for-each, xsl:apply-templates, and FLWOR expressions. The most direct approach is simply to concatenate the two master lists, leaving the segments unchanged. This however can lead to fragmentation of the sequence, so we perform a reorganization to reduce the number of short segments. Specifically, working from the right hand end, if any segment is found to be shorter than both its immediate neighbours, we combine it with the left-hand neighbour and reduce the number of segments by one. This has the effect of reducing the incidence of short segments in the middle of the chain.
Insertion, removal, and replacement: these operations are comparatively rare. With the immutable version of the structure, an alteration affecting one of the larger segments will require copying of everything else in that segment. This isn't ideal: but it's better than copying the entire sequence, which is what often happens today. And the use of the Java ArrayList at least means that the copying is very fast.

It's important to note that most operations on sequences don't actually result in a new sequence being constructed. Calling tail(), for example, doesn't copy any data: it delivers an iterator over a portion of the original sequence. The sequence only gets materialized if, for example the result is stored in a variable (and even then, not always).

Saxon's default implementation for a sequence is simply a Java List. Appending an item to a list generally copies the whole list. Where Saxon can detect that this is going to be inefficient, it instead uses a structure called a Chain: this is effectively a tree of segments. But there's little serious attempt to manage the depth of the tree or the size of the segments, and the results in some cases can be rather poor.The Zeno chain offers a signficant improvement; it also looks as if it can be used for arrays as well as sequences.

For managing long strings, I invented a similar structure, which I then discovered already existed in the literature and is known as a Rope: a Rope represents a string as a tree of substrings. The literature on Ropes describes how to keep the tree balanced, but it has nothing to say about how to decide how many substrings to hold, and how long to make them. The Zeno chain might turn out to provide an answer to that question.

Arrow Expressions

2020-11-19T10:20:00Z

Arrow Expressions

When I proposed the arrow operator to the XQuery/XSLT working groups, I thought of it as minor syntactic sugar. It's just a convenience: instead of substring-before(substring-after(X, '['), ']') you can write X => substring-after('[') => substring-before(']') which helps you to avoid going cross-eyed. If you're the kind of person who can play the piano with your hands crossed over, you probably don't need it, but for the rest of us, it makes life just a tiny bit easier.

So I was a bit surprised at XML Prague 2020 that Juri Leino managed to construct an entire presentation around the arrow operator (Shooting Arrows Fast and Accurately). Not only that, he also developed a whole library of functions, called XBow, to increase their power.

Now, XBow actually reveals a bit of a weakness in the construct: you can construct a pipeline of functions, but you can't include arbitrary expressions in the pipeline unless each of the expressions is made available via a function. Moreover, the value output by one step in the pipeline can only be used as the first argument in the next function: you can do X => concat('$') to add a "$" at the end of a string, but there's no simple way of adding a "$" at the front, except by defining a new prepend function that does this for you (or hoping that XBow will have anticipated your requirement).

Now, of course you can do X ! concat('$', .). But that doesn't always fit the bill. Firstly, it only works when you're processing single items (or mapping a sequence to multiple items). Secondly, (to use the current jargon) the optics are wrong: it breaks the pipeline visually.

So my first suggestion is that we allow inline expressions to appear in a pipeline. Something like this: X => {~ + 1}, or X => {concat('$', ~)}. I'm using '~' here as a variable to refer to the implicit argument, that is, the value passed down the pipeline. I would have used '_', as Scala does, but unfortunately '_' is a legal element name so it already has a meaning. And '~' seems to work quite nicely.

The next thing that's been requested is implicit mapping, so you can use something like arrow notation to do X ! substring-after(., '$') ! number(.) => sum(). (Actually, the main obstacle in getting the arrow operator accepted by the XQuery Working Group was that some people wanted it to have this meaning.)

For that I propose we use a "thin arrow": X -> substring-after('$') -> number() => sum(). The effect of the thin arrow is that instead of passing the value of the LHS to the function on the RHS en bloc, we pass it one item at a time. Of course, if the value on the LHS is a single item, then it doesn't matter which kind of arrow we use, both have the same effect.

If you're a fan of map-reduce terminology, then you'll recognize this instantly as a map-reduce pipeline. The -> operations are doing a mapping, and the final => does a reduce. If you're more into functional thinking, you probably think of it more in terms of function composition.

Of course thin arrows can also be used with arbitrary expressions, just like thick arrows: (0 to 3) -> {~ + 1} -> format-integer('a') => string-join('.') returns "a.b.c.d".

And now I'd like to pull one more rabbit out of the hat. What if I want a function that applies the above pipeline to any input sequence. I could write function($x){$x -> {~ + 1} -> format-integer('a') => string-join('.')} but that seems clunky. I'm looking for a nice way to supply functions as arguments to higher-order functions like sort, where other languages have shown that a concise notation for anonymous functions (like a -> a+1 in Javascript) can make code a lot simpler, less verbose, more readable.

So my proposal is this: just remove the left-hand expression, so you have something starting with -> or =>, and use this as an anonymous arity-1 function.

So you can now do: //employee => sort((), ->{~/@salary}) to sort employees by salary, or //employee => sort((), ->{~/@salary}->substring-after('$')->number()) if you need to do a bit more processing.

As another little refinement, in the case of ->, the implicit argument is always a single item, so we can bind it to the context item. So ->{~/@salary} can be simplified to ->{@salary}. Basically, within curly braces on the RHS of ->, . and ~ mean the same thing.

I believe that all these constructs can be added to the grammar without introducing ambiguity or backwards incompatibility, but I haven't proved it conclusively yet.

Postscript

The ~ construct seems to be the missing ingredient to enabling pipelines in XSLT. Consider:

Here "~" is acting as an implicit variable to pass the result of one instruction to be the input for the next: basically eliminating the clunky xsl:variable declarations needed to do this today. The instructions that form the children of the xsl:pipeline element are effectively connected to each other with an implicit => operator.

Draft Proposals for XSLT/XPath/XQuery 4.0

2020-11-14T19:19:00Z

Draft Proposals for XSLT/XPath/XQuery 4.0

I've been working on translating the ideas in my XML Prague 2020 paper, entitled a Proposal for XSLT 4.0 into concrete specifications, and my first attempt at this can be found here:

I'm hoping to gather together a community group of some kind to take this forward; meanwhile I've published a very preliminary set of drafts:

I put these ideas up yesterday on the XML community Slack channel and got some great feedback. Unfortunately Slack isn't really a good vehicle for managing the response to this feedback. I'm going to organise some GitHub space for a more structured discussion, but meanwhile, here are my reactions to the initial comments:

Phil Fearon:

There’s a lot to digest here so hope to provide feedback after I’ve read this more thoroughly. One suggestion (inspired by ReactJS/JSX) is to provide some syntactical sugar for xsl:call-template. So a call to a named template appears more like a literal result element (perhaps with a special namespace), with attributes that correspond to template params. This could also allow the child items of the special-LRE to be passed to the named template, accessed via a special $children param.

Yes, I've been wanting to do something like this for years, and it's really not difficult, so I've added it. If EX is listed in extension-element-prefixes, and if there's a named template name="ex:action", then is interpreted as an xsl:call-template with xsl:wth-param children for parameters a and b.

Liam Quin

first quick note, best NOT to have them say W3C Recommendation on them as this may cause confusion.

Yes, sorry about that, still working my way around the stylesheets that generate the boilerplate text...

2d, cam xsl:text have a select attribute? i don't think value-of can be deprecated :disappointed: but xsl:text select= would be consistent & may help.

It's one of these things that one would like to simplify, but we can only add things not remove them, so that's not easy.

" the tunnel parameters that are implicitly passed in a template call may have names that duplicate the names of non-tunnel parameters that are explicitly passed on the same call." is a major source of difficult debugging if you forget tunnel=yes. Maybe the answer is just a warning from impl'ns.

Yes. How to solve this without breaking compatibility? Perhaps a dynamic error if you declare a non-tunnel parameter, and at run-time there's a tunnel parameter with that name, but no non-tunnel parameter, or vice versa? Or, as you say, just rely on warnings. I agree it's a very common mistake that's hard to debug.

The "at $pos" of XQuery is super useful. position() is tricksy.Maybe for-each at="name" ?

In 3.0 we experimented with replacing some of the context functions with explicit variable bindings and it got a bit messy, but I think it's a shame we didn't persevere. The toughest one is last(), it would be awfully nice if we knew statically whether last() was going to be needed or not, but again, hard to fix without breaking code.

prefix binding didn't make the cut for XPath?

I did a design for this and didn't like it enough to put it in. I'll try again.

item-at() seems not much easier than $xxx ! let $p := position() return $yyy[$p]

I'm toying now with an alternative to item-at() that's much more powerful: slice(sequence, positions) so you can do slice($s, 5) or slice($s, 5 to 10) or slice($s, -1) or slice($s, 1 by 3 to count($s)) or slice($s, -2 by -1 to -count($s)). Here A by B to C is an extension of the current range expression where A to B means A by 1 to B.

replace-with() seems like perl's e flag (JS has one too) but alas no polymorphism so can't write replace(., $expr, myfunc#1m 'e')

However, what about adding a map or an array of matching subgroups? ".{$2 || $1 * 2 || $2}"

Yes, I think it's a really useful capability, but I think it's cleaner to make it a separate function. Have to think about how subgroups might work.

Reece H. Dunn

I like the enum(...) syntax in addition to the union(...) syntax.

I like the extension of element and attribute type tests to be full name tests. The ability to define types for path expressions like (ol|ul) is missing, though.

Yes, I'm in two minds whether union(X, Y, Z) should be restricted to a union of atomic types, or whether it should allow a union of any types including node types. Orthogonality suggests the latter, but I was too timid to propose that.

For named item types, is it possible to make them available as part of the in-scope schema types (renamed to in-scope types that would include the schema and named types?), so you could say person-name instead of item-type(person-name). -- Having to qualify the named item types everywhere could get too verbose, especially if the name is short. _NOTE:_ This is done for MarkLogic types where you can refer to map:map, cts:query, etc.

Interesting idea. There's obviously a need to resolve conflicts but that's not a stopper. I think I was more concerned with the idea that if it's a QName then it must be atomic, and the messy fact that the sets of schema types and item types overlap, and the overlap contains all atomic types and some but not all union types.

Liam Quin

hmm, xsl:sequence could do with an "as" attribute.

Not convinced. You start wanting to put it anywhere e.g. on xsl:if or xsl:apply-templates.

Reece H. Dunn

In https://www.saxonica.com/qt4specs/XP/xpath-40-diff.html#id-itemtype-subtype, rule 2(d) is missing the reference to the EnumerationType symbol ("A is an ," instead of "A is an EnumerationType,").

Stylesheet trouble. The XSLT and XPath spec stylesheets have diverged, the XSLT spec allows and picks up the term from the definition, but the XPath spec requires term. I need to bring them back into line. Applies to your subsequent comments also.

Martin Honnen

I like the separator attribute on xsl:apply-templates and xsl:for-each. I wonder whether it would make sense to add it to xsl:for-each-group as well.

Yes. Also xsl:for-each-member. I'm not sure whether it should be an AVT or a general expression: with a general expression you could insert br or hr separators, especially if we have element constructor functions in XPath (separator="build:element('hr')")

Liam Quin

9 hours ago

i've been wondering abut the possibility of an xsl:uri-resolver for some time.

Not sure what it would do?

Also about xsl:mode elements being able to contain xsl:template elements.

Yes, I've wanted that for a long time. To be honest, it's not in 3.0 because I couldn't convince Sharon.

Reece H. Dunn

The changes for XPath/XQuery look good. I see you changed the syntax for the context item and lambda syntaxes to a unified syntax. I like that the concise and full syntaxes are now consistent.

Given that . is allowed in a ParamList, does that mean I can now define a function that works on the path context item? For example: declare function local:f(.) { xs:integer(.) + 2 }; //values/local:f()

Actually, allowing "." here was an oversight caused by my changing the way the grammar rules worked. But there might be some benefit it keeping it.

... does that mean that the context-dependent functions in F&O should be defined using that syntax. For example: fn:data(.) as xs:anyAtomicType*

I hadn't thought of baking the "implicit . as parameter" convention into the language, but it might make sense if it can be done.

Liam Quin

in XQuery i don't understand "The for evaluation of the function body is absent, unless the signature uses the "." notation, in which case it is evaluated with a singleton focus bound to the supplied argument value."

Markup trouble again.

Martin Honnen

9 hours ago

If xsl:for-each has a separator attribute, wouldn't xsl:for-each-member benefit from it as well?

Yes, see above.

Reece H. Dunn

9 hours ago

For the schema import in XQuery, would it make sense to have: [22] SchemaPrefix ::= ("namespace" NCName "=") | ("default" ("element" | "type") "namespace") now that the element and type namespaces are separate, similar to how DefaultNamespaceDecl has changed.

I thought about this and decided not. If you want finer control, use multiple declarations.

For parameter lists and context items, would it be more useful to have the context item as an optional first parameter? That would mirror the proposed variadic argument syntax (defined for arguments at the end of the parameter list), and would allow arguments to be passed to the function, such as: declare function local:add(., $n) { xs:integer(.) + $n }; //values/local:add(2)

I quite like that in principle. Needs more thought.

Reece H. Dunn 9 hours ago

Is there any description of the arity of context item based functions? -- There should be a note or something similar to say that the context item for a function definition or inline function expression does not count to its arity, so function () and function (.) both have an arity of 0.

I was thinking of them simply as arity-1 functions, suitable for callbacks in things like fn:filter and fn:sort. You're opening up new possibilities which I need to ponder.

line0

Great to see the spec coming to light!

I proposed to add two new signatures to for-each

for-each(item()*, function (item(), xs:positiveInteger) as item()*) as item()*

for-each(item()*, function (item(), xs:positiveInteger, item()*) as item()*) as item()*

I've proposed that the function coercion rules should allow you to supply an arity-1 function where an arity-2 function appears in the signature; so we can extend fn:for-each to take a function(item, integer) as the predicate callback, and you can still supply function(item) if you don't care about the position.

which would bring for-each on par with FLOWR expressions ( for window and for … in … at ).

It would be great to see windowing done with higher-order functions, but it's a significant piece of design and not my top priority - even though it would bring XSLT up to the level of XQuery for this kind of functionality.

Martin Honnen

For fn:transform, Saxon has already added the option source-location as that is needed to use fn:transform with streaming; I think it makes sense to integrate that option into the fn:transform specification of the FO 4 draft.

Good point.

XSLT Update – Some Ideas

2020-10-29T16:19:00Z

XSLT Update – Some Ideas

I can't help feeling that many simple transformations on XML documents could be expressed more simply (an idea I explored with the interactive Gizmo tool available in Saxon 10).

I also think that if we had simpler syntax for simple operations, it might become easier to optimise. In particular, I'd like to be able to do simple operations like adding an attribute to the outermost element of a tree without doing a physical copy of the entire tree. (I wrote about that in an XML Prague paper, but I had to abandon the idea because the code became too complicated. As often happens, the bugbear was namespaces. For example, if you add a namespaced attribute to the outermost element, then the new namespace declaration has to propagate all the way down the tree.)

I'd also like to see transformations on JSON structures (maps and arrays) become much easier.

I've prototyped these ideas in the saxon:deep-update and saxon:update extensions, but I don't think these are the last word on the subject. (Please try them and give feedback.)

A simpler update syntax might also be very useful for updating the HTML page in Saxon-JS.

I think we can pick up ideas from XQuery Update, but without the complications of pending update lists and in-situ modification.

Let's start with:

The idea is that xsl:update is an instruction that returns a deep copy of the context item (or other selected item if there's a select attribute), applying changes defined by the contained rules. In this case there is one rule, to delete elements that match the pattern note.

So it's rather like the copy-modify instruction in XQuery; it makes a copy of a supplied tree, with defined changes.

Other rules that might appear within (for updating XML) might include:







   {@title}

Hopefully the intent is reasonably intuitive. The idea is to base the primitives on those available in XQuery Update. However, I'm not proposing to allow flow-of-control structures such as conditionals and function calls: each invocation of xsl:update will simply process the selected tree recursively, applying matching rules to nodes as they are found, based on pattern matching.

Defining the semantics

We can define the semantics of as being equivalent to using a mode that contains a number of implicit template rules, with a default action of shallow-copy (but extended to handle maps and arrays, see below).

For example, the implicit template rule for the rule might be (roughly):

Now, what if there's a rule to rename an element and another rule to add an attribute to the same element?

The way XQuery Update handles that is to process the rules in a number of phases: for example rename operations are handled in phase 1, delete operations in phase 5.

It's a bit hard to replicate that behaviour using template rules (in fact, this is something users often ask for). We could run a multiphase transformation using multiple modes, but it's not quite the same thing, because the match patterns would apply to the output of the previous phase, not to the original node in the input. And xsl:next-match doesn't do the job either, because we want the effect of the rules to be cumulative.

We could try another approach, which is to have the template rules return functions, so the rule becomes:

so the effect of apply-templates is to return a sequence of functions (in the order determined by the priority attributes) which are then applied to the node in turn.

This still doesn't exactly mirror what XQuery Update does, because after processing a node, it's then going to apply the rules to the new content of the node, not to the old content. But perhaps that actually makes more sense?

Implementation

Part of the aim is not just to have simpler syntax for the user, but also to make the implementation more efficient than the standard transformation approach which always involves physical copying of a tree, no matter how small the changes.

What I want to achieve is to have a data structure, rather like the HashTrie that we use for representing XDM maps, in which changing one entry doesn't involve copying the whole tree, but at the same time leaves the original value intact. The first essential for such a structure is that it doesn't contain parent pointers: instead upwards navigation is achieved by remembering, when we get to a node, how we got there: this means the same node can be reached by multiple routes, allowing subtrees to be shared between different trees.

Suppose we are changing the value of a single attribute. It ought to be possible to achieve this by the following steps:

Find the element we are modifying, remembering the ancestor path of that element.
Create a "virtual copy" of this element (we already have this capability in Saxon)
Modify the virtual copy to add the attribute. Only one element is affected; the descendant tree of the virtual copy is shared with the original tree.
Work back through the ancestors; for each one, create a copy in which the affected child is replaced with the modified child, and all other children are virtual copies of the original.
Return the copied root node.

I'm hoping that it will be a lot easier to achieve this with the new syntax than it is with the current processing model, where we have to deal with all kinds of messiness like namespace inheritance. For example, we can define the new syntax so that it's equivalent to inherit-namespaces="no".

What about JSON?

I would like this mechanism to work just as well with JSON trees (that is, structures of maps and arrays) as with XML trees.

We're starting with some advantages: these structures don't have so much baggage. There's no node identity to worry about, no parent navigation, no namespaces. Also, the implementation data structures that we use for maps and arrays already allow efficient constant-time update.

I've experimented with mechanisms for deep update of a JSON structure with extension functions such as [saxon:pedigree()](https://www.saxonica.com/documentation/index.html#!functions/saxon/with-pedigree). and saxon:with-pedigree(). That's not exactly usable. But it might be the right primitive to implement something more usable.

I've also proposed better pattern syntax for maps and arrays. For example, match="tuple(first, last, *)" matches any map that has entries with keys "first" and "last".

One problem with using the XSLT recursive-descent approach for maps and arrays is that map entries (and indeed array members) aren't actually items. You can match a map as a whole, but it's hard to match one of its entries on its own. Again, I've experimented with various approaches to this. I think the introduction of tuples may help with this: we can define the recursive-descent operation on maps to process (match) each entry in the map in turn, where the entry is presented and matched as a tuple containing key and value. And then we allow syntax such as match="tuple(key: keyPattern, value: valuePattern)" to match these entries.

But perhaps we don't need to expose this. Perhaps we can define a good enough set of primitive actions that match at the level of the map itself, for example:

I think this could fly: but there's a lot of detail to be worked out. Shame we don't have a WG any more to bounce ideas off (and get the bugs out).

String, CharSequence, IKVM, and .NET

2020-07-20T09:06:54Z

A couple of years ago Jeroen Frijters announced that he would no longer be working on new IKVM developments (IKVM is the technology we use to make Saxon, which is written in Java, run on .NET). At one level that's not a problem: the tool works brilliantly and we can continue to use it. However, it doesn't support .NET Core, and Microsoft have announced that .NET 5 will be based on .NET Core, so that creates the risk that Saxon on .NET will hit a brick wall.

Various smart people are working on trying to pick up IKVM where Jeroen left off, but I don't particularly want to bet the business on them being successful. Jeroen produced brilliant software but he left very little in the way of documentation or test material, so it's a hard act to follow.

Meanwhile Microsoft seem to be back-pedalling on their original promise that .NET 5 would support Java interoperability. They've never given any indication of how it would do so, despite much speculation.

So we've been looking at alternative ways of taking Saxon on .NET forward into the future, and one of those is source code conversion. I've been looking at tools such as Tangible, which does a good job to a degree: they don't tackle the difficult parts of the problem where Java and C# are most different, but they give a very good insight into understanding what the difficult parts of the problem are going to be.

And one of those difficult parts, which I'm focussing on at the moment, is the CharSequence problem. CharSequence is a Java interface that we use very extensively, and there's no equivalent on .NET. Unlike other dependencies on Java classes and interfaces, this one is impossible to emulate directly, because java.lang.String implements CharSequence, and there's no way we can make System.String on .NET do the same.

The reason we use CharSequence, as with any interface, is so that we can have multiple implementations with different performance characteristics. To take a simple example, one of our implementations is CompressedWhitespace. A great deal of the text in an XML document is made up of whitespace, which sadly cannot be killed at birth: using a customised representation for strings that contain only whitespace gives a significant space saving. (And space savings also turn into speed improvements, given that execution time these days is dominated by how long it takes to get data in and out of the CPU's internal cache).

Given that CharSequence has no equivalent on .NET, it occurred to me to ask how IKVM deals with it. Although Jeroen never wrote much documentaton, he did write a lot of blog posts about interesting design problems, and sure enough it seems that he gave this a lot of attention back in 2003 (how time flies when you're having fun). I thought that he might use an implementation of CharSequence that wraps a System.String, but it seems he rejected that approach in favour of a mechanism of what he calls "ghost interfaces". There's a lot of detail, but the bottom line seems to be that the code:

CharSequence seq = "foo";
seq.charAt(1);

is compiled to .NET as:

System.Object seq = "foo";
if(seq instanceof System.String)
  ((System.String)seq).charAt(1);
else if(seq instanceof CharSequence)
  ((CharSequence)seq).charAt(1);
else
  throw new IncompatibleClassChangeError()

That looks pretty horrifying, and I've belatedly realised that it could account for a lot of our observations on .NET performance over the years.

When we first built Saxon on .NET, the performance overhead compared with Java was around 30%, which was quite acceptable. In recent years we've seen it getting worse, with some workloads showing a 300% slow-down, and despite considerable effort we've been at a loss to explain why. Synthetic benchmarks on IKVM continued to show a 30% overhead, but for Saxon the figure was far worse. We looked hard without success to find a hot-spot, something we were doing that IKVM handled particularly badly, but the slow-down seemed to be right across the board. I'm now prepared to conjecture that it's all down to our use of CharSequence - because CharSequence.charAt() is something we do very extensively, throughout the product.

When data arrives in Saxon from a SAX parser, the content of text nodes arrives in char[] arrays, while the content of attributes arrives in String objects. And we keep it that way: in the TinyTree, text nodes are effectively slices of a char[] array, and attributes are Strings. All the operations that we perform on text, including performance-critical operations such as equality matching, sorting, and string-to-number conversion, therefore need to work on either representation, and that's essentially why we use CharSequence. In general, we don't want to spend time converting data between different representations so we can perform different operations on it.

(In recent releases, though, we've started using a different representation for operations where we need to count Unicode codepoints rather than UTF-16 chars. For regular expressions, and some other operations such as translate(), we first convert the string to a UnicodeString, which is our own interface that supports direct codepoint addressing, with internal implementations using 8, 16, or 32 bits per character depending on the widest character present in the string).

So if CharSequence is a problem, what should we do instead? Is there any other way we can implement operations such as collation comparison and string-to-number conversion efficiently without first converting the data to a common internal format?

I think part of the solution might be for these operations to be written to use codepoint iterators. Iterating over a string using an IntIterator that delivers codepoints is probably just as efficient as using a for-loop with charAt(), and it's possible to create an IntIterator over any string representation efficiently (meaning, without copying the actual characters).

This suggests the following broad approach:

(a) For attributes, continue to use Strings

(b) For text nodes on the Receiver pipeline and in the TinyTree, use an interface similar to CharSequence - let's call it UniString - that allows multiple implementations, but that doesn't have the magic property that String can be used directly as an implementation. (Instead, there will be an implementation of UniString that wraps a String).

This is a gross simplification: we're dealing with half a million lines of code that's all concerned with string handling, so the detail is horrendous. But having a simplified description of the problem and the solution helps greatly when you're hacking through the jungle.

The Java class hierarchy for XPath type objects

2020-02-10T11:26:47Z

The set of interfaces and classes used in the Java code to represent XSD and XDM types has become something of a nightmare. This article is an attempt to explain it. When you don't understand something well, you can often improve your understanding by trying to explain it to others, so that's what I shall attempt to do.

The first complication is that we have to model schema types and item types, and these are overlapping categories.

Schema types - types as the term is used in XSD - are either simple types or complex types; simple types are either atomic types, union types, or list types. We can forget about complex types for the time being as they are relatively unproblematic.

With simple types, we should mention in passing that one of the problems is that while processing a schema, we don't always immediately know what the variety of a simple type is; if it's derived from a base type and we haven't yet analysed the base type, then we park it as a "SimpleTypeDefinition" to be turned into an AtomicType, UnionType, or ListType later - which means that all references to the type need to be updated.

As well as their use in schema processing, schema types are used as type annotations on nodes in XDM, and they also appear in XPath expressions as the target of a "cast" or "castable" expression.

Item types are purely an XDM concept, and they include atomic types, node types, function types, map types, array types. Item types when combined with an occurrence indicator form a Sequence type. Sequence types are used in XPath in declaring the types of variables, parameters, and function results; they are also used in "instance of" and "treat as" expressions.

Atomic types are both schema types (more specifically, simple types) and item types. Not every schema type is an item type (complex types aren't, list types aren't), and not every item type is a schema type (node types and function types aren't). The categories overlap, so it's not surprising that the Java class hierarchy is complicated.

Union types add another complication. A simple union of atomic types (for example the union of xs:date and xs:dateTime) is useful as an item type, for example to define the type of a function argument or variable. But XSD union types aren't always simple unions of atomic types: they can also include list types, and they can define restrictions beyond those present in the member types. So XDM defines the concept of a "pure union type", which is a simple union of atomic types; pure union types are the only kind that can be used as item types. For convenience it's useful to have a term that embraces atomic types and pure union types: the XDM specifications call these "generalized atomic types", and in Saxon they are referred to as "plain types". Again, these overlapping categories make it very hard to get the Java class hierarchy right.

Simple types form a lattice; at the top of this lattice is the most general type "xs:anySimpleType", and at the bottom is the "void" type "xs:error" (void because it has no instances). These "edge case" types are simple types, but they don't fit cleanly into the classification of union types, list types, and atomic types.

Item types also overlap with XSLT patterns, and with the node tests used in axis steps. Constructs such as element(*) and text() are both node tests (suitable for use in patterns and axis steps) and item types. Not every item type is a node test (for example, array(*) isn't), and not every node test is an item type (for example, *:local isn't). Again, we have two intersecting categories. If we draw the Venn diagram of simple types, item types, and node tests, we find that simple tests don't overlap with node tests, but all other combinations have an intersection.

There's another dimension that we try to capture in the Java class hierarchy: we try to distinguish built-in types from user-defined types. There are built-in atomic types (xs:integer), built-in list types (xs:NMTOKENS), and built-in union types (xs:numeric); and there are also used-defined types in each of the three varieties. Capturing two dimensions of classification in a class hierarchy typically introduces multiple inheritance and complicates the hierarchy.

There's also a lot of complexity concerned with the relationship of schema types to other kinds of schema component. Again at this level we try to distinguish used-defined schema components (those derived from declarations in an XSD source document) from built-in schema components (which include not only simple types, but also complex types such as xs:anyType and xs:untyped). We distinguish "schema components" as defined in the XSD specification (which include not only schema types, but also element declarations, attribute declarations, identity constraints etc) and "schema structures" which are essentially constructs in a source XSD document; but looking at the code, nearly everything you find in a schema seems to be both a "schema component" and a "schema structure" and I'm having trouble seeing exactly what the difference between the two categories is.

The straw that broke the camel's back and made me examine whether refactoring is needed was the introduction of locally-declared union types with the syntax "union(xs:date, xs:time)". These are clearly union types, but they aren't built-in, and they don't correspond to declarations in any source schema, so they don't fit neatly into the existing classification of built-in versus user-defined.

We've got an awful lot of multiple inheritance in this hierarchy, and the accepted wisdom is that if you've got a lot of multiple inheritance, then you need to do some refactoring, and replace some of it with delegation.

We've got a model for that in the way we handle XSLT match patterns. Although node-tests are a subset of patterns, we don't treat node-tests as a subclass of patterns in the Java class hierarchy; rather, the class hierarchy for patterns includes a NodeTestPattern which contains a reference to a NodeTest. Similarly, atomic types are a subset of schema types, but that doesn't mean they need to implement SchemaType in the Java class hierarchy; rather the class hierarchy for SchemaTypes could include an AtomicSchemaType which contains a reference to an AtomicType.

Let's see what we can do.

UPDATE 2020-02-18

Well: I had a good go at refactoring this; but the new scheme was getting just as complex as the old, so I decided to revert all the work.

I tried to split the classes representing simple types into two: the "compile time" information used during XSD schema compilation, and the "executable" types used actively for validation. But I ended up with just as many classes (or more), and just as much multiple inheritance. I did manage to eliminate the messy process whereby a SimpleTypeDefinition is converted to an AtomicType, ListType, or UnionType as soon as we know its variety (i.e. when the reference to its base type is resolved -- it can be a forwards reference), but I found that doesn't open the door to any wider simplification.

Java Generics revisited

2020-01-21T21:14:20Z

In Saxon 9.9 we took considerable pains to adopt Java Generics for processing sequences: in particular the Sequence and SequenceIterator classes, and all their subclasses, became Sequence and SequenceIterator.

I'm now coming to the conclusion that this was a mistake; or at any rate, that we went too far.

What exactly are the benefits of using Generics? It's supposed to improve type safety and reduce the need for casts which, if applied incorrectly, can trigger run-time exceptions. So it's all about detecting more of your errors at compile time.

Well, I don't think we've been seeing those benefits. And the main reason for that is that in most cases, when we're processing sequences, we don't have any static knowledge of the kind of items we are dealing with.

Sure, when we process a particular XPath path expression, we know whether it's going to deliver nodes or atomic values. But when we write the Java code in Saxon to handle path expressions, all we know is that the result will always be a sequence of items.

There are some cases where particular kinds of expression only handle nodes, or only handle atomic values. For example, the input sequences for a union operator will always be sequences of nodes. It would be nice if we didn't have to handle a completely general sequence and cast every item to class NodeInfo. But it's an illusion to think we can get extra type safety that way. The operands of a union are arbitrary expressions, and the iterators returned by the subexpressions are going to be arbitrary iterators; there's no way we can translate the type-safety we are implementing at the XPath level into type-safe evaluators at the Java level.

It's particularly obvious that generics give us no type-safety at the API level. In s9api, XPathSelector.evaluate() returns an XdmValue. That's a lot better than the JAXP equivalent which just returns Object, but the programmer still has to do casting to convert the items in the return XdmValue to nodes, string, integers, or whatever. And there's no way we can change that; the XPath expression is supplied as a string at run-time, so it's only at run-time that we know what type of items it returns. If that's true at the API level, it's equally true internally. Any kind of expression can invoke any other kind of expression (that's what orthogonality in language design is about), which means that the interfaces between an expression and its subexpressions are always going to be general-purpose sequences whose item type is known only at execution time.

There are a couple of aspects of Java generics that cause us real pain.

The first is the XDM rule that every item is itself a Sequence. So if Sequence is a generic type, parameterized by Item type, and Item is a subclass of Sequence, then Item has to be itself a generic type parameterized by its own type. Rather than Item, it has to be Item; or perhaps it should be Item>, and so ad infinitum. And then StringValue extends Item and so on. We found ways around that conundrum, but the complexity is horrendous; it certainly doesn't achieve the goal of making it easier to write correct code.
The second is arrays. Arrays don't play at all well with generics; you can't create an array of a generic type, for example. And yet there are lots of places where it's useful to use arrays, and some where arrays are the only option. VarArgs functions, for example, present their arguments as an array. In some cases we wanted to carry on using arrays (rather than lists) for compatibility, in other cases we want to use them for convenience or for performance. The natural signature for a function call, for example is public Sequence call(Context context, Sequence[] args). There's no way we can refine this in a way that passes static information about the argument types from the caller to the callee, because we're using the same Java signature for all XPath functions.

But having got Generics working, at great effort, in 9.9, should we retain them or drop them?

One reason I'm motivated to drop them is .NET. We have a significant user base on .NET, but we have something of a potential crisis looming in terms of ongoing support for this platform. Microsoft appear to be basing their future strategy around .NET Core, allowing .NET Framework to fade away into the sunset. But the technology we use for bridging to .NET, namely IKVM, only supports .NET Framework and not .NET Core; and Jeroen Frijters who single-handedly developed IKVM and supported it for umpteen years (with no revenue stream to support it) has thrown in the towel and is no longer taking it forward. So we're looking at a number of options for a way forward on .NET. One of these is source code conversion; and to make source code conversion viable without forking the code, we need to minimise our dependencies on Java features that don't translate easily to C#. Notable among those features is generics.

In the short term, I think I'm going to roll back the use of generics in selected areas where they are clearly more trouble than they are worth. That's particularly true of Sequence and its subclasses, including Item. For SequenceIterator it's probably worth keeping generics for the time being, but we'll keep that under review.

Alphacodes for Sequence Types

2019-10-15T14:06:31Z

In the next releases of Saxon and Saxon-JS we have devised a compact notation for representation of SequenceType syntax in the exported SEF file. This note is to document this syntax.

The main aims in devising the syntax were compactness, together with fast generation and fast parsing. In addition it has the benefit that some operations are possible on the raw lexical form without doing a full parse.

The syntax actually handles ItemTypes as well as SequenceTypes; and in addition, it can handle the two examples of NodeTests that are not item types, namely *:local and uri:*. It can therefore be used in the SEF wherever a SequenceType, ItemType, or NodeTest is required.

The first character of an alphacode is the occurrence indicator. This is one of: * (zero or more), + (one or more), ? (zero or one), 0 (exactly zero), 1 (exactly one). If the first character is not one of these, then "1" is assumed; but the occurrence indicator is generally omitted only when representing an item type as distinct from a sequence type.

The occurrence indicator is immediately followed by the "primary alphacode" for the item type. These are chosen so that alphacode(T) is a prefix of alphacode(U) if and only if T is a supertype of U. For example, the primary alphacode for xs:integer is "ADI", and the primary alphacode for xs:decimal is "AD", reflecting the fact that xs:integer is a subtype of xs:decimal. The primary alphacodes are as follows:

"" (zero-length string): item()

A: xs:anyAtomicType
AB: xs:boolean

AS: xs:string
ASN: xs:normalizedString
ASNT: xs:token
ASNTL: xs:language
ASNTM: xs:NMTOKEN
ASNTN: xs:Name
ASNTNC: xs:NCName
ASNTNCI: xs:ID
ASNTNCE: xs:ENTITY
ASNTNCR: xs:IDREF

AQ: xs:QName
AU: xs:anyURI
AA: xs:date
AM: xs:dateTime
AMP: xs:dateTimeStamp
AT: xs:time
AR: xs:duration
ARD: xs:dayTimeDuration
ARY: xs:yearMonthDuration
AG: xs:gYear
AH: xs:gYearMonth
AI: xs:gMonth
AJ: xs:gMonthDay
AK: xs:gDay

AD: xs:decimal
ADI: xs:integer
ADIN: xs:nonPositiveInteger
ADINN: xs:negativeInteger
ADIP: xs:nonNegativeInteger
ADIPP: xs:positiveInteger
ADIPL: xs:unsignedLong
ADIPLI: xs:unsignedInt
ADIPLIS: xs:unsignedShort
ADIPLISB: xs:unsignedByte
ADIL: xs:long
ADILI: xs:int
ADILIS: xs:short
ADILISB: xs:byte

AO: xs:double
AF: xs:float
A2: xs:base64Binary
AX: xs:hexBinary
AZ: xs:untypedAtomic

N: node()
NE: element(*)
NA: attribute(*)
NT: text()
NC: comment()
NP: processing-instruction()
ND: document-node()
NN: namespace-node()

F: function(*)
FM: map(*)
FA: array(*)

E: xs:error

X: external (wrapped) object
XJ: external Java object
XN: external .NET object
XS: external Javascript object

Every item belongs to one or more of these types, and there is always a "most specific" type, which is the one that we choose.

Following the occurrence indicator and primary alphacode are zero or more supplementary codes. Each is preceded by a single space, is identified by a single letter, and is followed by a parameter value. For example the sequence type "element(BOOK)" is coded as "1NE nQ{}BOOK" - here 1 is the occurrence indicator, NE indicates an element node, and nQ{}BOOK is the required element name. The identifying letter here is "n". The supplementary codes (which may appear in any order) are as follows:

n - Name, as a URI-qualified name. Used for node names when the primary alphacode is one of (NE, NA, NP). Also used for the XSD type name when the type is a user-defined atomic or union type: the basic alphacode then represents the lowest common supertype that is a built-in type. (Note: we assume that type names are globally unique. This cannot be guaranteed when deploying a SEF file: the schema at the receiving end might vary from that of the sender.) Also used for the class name in the case of external object types (in this case the namespace part will always be "Q{}"). Note that strictly speaking, the forms *:name and name:* can appear in a NameTest, but never in a SequenceType. However, they can be represented in alphacodes using the syntax "n*:name" and "nQ{uri}*" respectively. The syntax "~localname" is used for a name in the XSD namespace.

c - Node content type (XSD type annotation), as a URI-qualified name optionally followed by "?" to indicate nillable. The syntax "~localname" is used for a name in the XSD namespace. Optionally present when the basic code is (NE, NA); omitted for NE when the content is xs:untyped, and for NA when the content is xs:untypedAtomic. Only relevant for schema-aware code.

k - Key type, present when the basic code is FM (i.e. for maps), omitted if the key type is xs:anyAtomicType. The value is the alphacode of the key type, enclosed in square brackets: it will always start with "1A".

v - Value type, present when when the basic code is (FM, FA) (i.e. for maps and arrays), omitted if the value type is item()*. The value is the alphacode of the value type, enclosed in square brackets. For example the alphacode for array(xs:string+)* is "*FA v[+AS]".

r - Return type, always present for functions. The value is the alphacode of the return type, enclosed in square brackets.

a - Argument types, always present for functions. The value is an array of alphacodes, enclosed in square brackets and separated by commas. For example, the alphacode for the function fn:dateTime#2 (with signature ($arg1 as xs:date?, $arg2 as xs:time?) as xs:dateTime?) is "1F r[?AM] a[?AA,?AT]"

m - Member types of an anonymous union type. The value is an array of alphacodes for the member types (these will always be atomic types), enclosed in square brackets and comma-separated. The basic code in this case will be "A", indicating xs:anyAtomicType. This is not used for the built-in union type xs:numeric, nor for user-defined atomic types defined in a schema; it is used only for anonymous union types defined using the Saxon extension syntax "union(a, b, c)".

e - Element type of a document-node() type, present optionally when the basic code is ND. The value is an alphacode, which will always start with "1NE".

t - Components of a tuple type (Saxon extension). The value is an array of tokens, enclosed in square brackets, where each token comprises the name of the component (an NCName), a colon, and the alphacode of the component type.

i, u, d - Venn type. The item type is the intersection, union, or difference of two item types. The letter "i", "u", or "d" indicates intersection, union, or difference respectively, followed by a list of (currently always two) item types enclosed in square brackets and separated by a comma. The principal type will typically be "N" or "NE". Saxon uses venn types internally to give a more precise inferred type for expressions; it is probably largely unused at run-time, and can therefore be safely ignored when reading a SEF file.

Named union types have a basic alphacode of "A", followed by the name of the union type in the form "A nQ{uri}local". The syntax "~localname" is used for a name in the XSD namespace, so the built-in union types xs:numeric and xs:error are represented as "A n~numeric" and "A n~error" respectively.

TODO: the documentation for union types is not aligned with the current implementation

Examples:

0 - empty-sequence()

1AS - xs:string

1N - node()

1 - item()

* - item()*

1NE nQ{}item - element(item)

1ND e[1NE nQ{}item] - document-node(element(item))

*FM k[1AS] v[?AS] - map(xs:string, xs:string?)*

1F a[?AS,*AO] r[1AB] - function(xs:string?, xs:double*) as xs:boolean

Version: 2019-10-30

A new push event API

2019-05-01T16:34:00Z

For various internal and performance reasons, we're making some changes to Saxon's internal Receiver interface for the next release. This interface is a SAX-like interface for sending an XML document (or in general, any XDM instance) from one processing component to another, as a sequence of events such as startElement(), attributes(), characters(), and so on.

The interface is very widely used within Saxon: it handles communication from the XML parser to the document builder, document validation, serialization, and much else. It also allows instructions to be executed in "push mode", so for example when XSLT constructs a result tree, the tree is never actually constructed in memory, but instead events representing the tree are sent straight from the transformer to the serializer.
I know that although this interface is labelled as internal, some user applications attempt either to implement the interface or to act as a client, sending events to one of Saxon's many implementations of the interface. So in making changes, it seems a good time to recognize that there is a need for an interface at this level, and that existing candidates are really rather clumsy to use.

Among those candidates are the venerable SAX ContentHandler interface, and the newer StAX XMLStreamWriter interface.

There are a number of structural reasons that make the ContentHandler hard to use:

It offers a number of different configuration options for XML parsers, which cause namespace information to be provided in different ways. But the ContentHandler has no way of discovering which of these options the XML parser (or other originator of events) is actually using.
It's not actually one interface but several: some events are sent not to the ContentHandler, but to a LexicalHandler or DTDHandler.
The information available to the ContentHandler doesn't align well with the information defined in the XDM data model; for example, comments are available only to the LexicalHandler, not to the ContentHandler

In addition, the way QNames and namespaces are handled makes life unnecessarily difficult for both sides of the interface.

In some ways the XMLStreamWriter is an improvement, and I've certainly used it in preference when writing an application that has to construct XML documents in this way. But a major problem of the XMLStreamWriter is that it's underspecified, to the extent that there is a separate guidance document from a third-party suggesting how implementations should interpret the spec. Again, the main culprit is namespace.

One of the practical problems with all these event-based interfaces is that debugging can be very difficult. In particular, if you forget to issue an endElement() call, you don't find out until the endDocument() event finds there's a missing end tag somewhere, and tracking down where the unmatched startElement() is in a complex program can be a nightmare. I decided that addressing this problem should be one of the main design aims of a new interface -- and it turns out that it isn't difficult.

Let's show off the new design with an example. Here is some code from Saxon's InvalidityReportGenerator, which generates an XML report of errors found during a schema validation episode, using the XMLStreamWriter interface:

writer.writeStartElement(REPORT_NS, "meta-data");
writer.writeStartElement(REPORT_NS,"validator");
writer.writeAttribute("name", Version.getProductName() + "-" + getConfiguration().getEditionCode());
writer.writeAttribute("version", Version.getProductVersion());
writer.writeEndElement(); //
writer.writeStartElement(REPORT_NS,"results");
writer.writeAttribute("errors", "" + errorCount);
writer.writeAttribute("warnings", "" + warningCount);
writer.writeEndElement(); //
writer.writeStartElement(REPORT_NS,"schema");
if (schemaName != null) {
    writer.writeAttribute("file", schemaName);
}
writer.writeAttribute("xsd-version", xsdversion);
writer.writeEndElement(); //
writer.writeStartElement(REPORT_NS,"run");
writer.writeAttribute("at", DateTimeValue.getCurrentDateTime(null).getStringValue());
writer.writeEndElement(); //
writer.writeEndElement(); //

And here is the equivalent using the new push API:

Push.Element metadata = report.element("meta-data");
metadata.element("validator")
        .attribute("name", Version.getProductName() + "-" + getConfiguration().getEditionCode())
        .attribute("version", Version.getProductVersion());
metadata.element("results")
        .attribute("errors", "" + errorCount)
        .attribute("warnings", "" + warningCount);
metadata.element("schema")
        .attribute("file", schemaName)
        .attribute("xsd-version", xsdversion);
metadata.element("run")
        .attribute("at", DateTimeValue.getCurrentDateTime(null).getStringValue());
metadata.close();

What's different? The most obvious difference is that the method for creating a new element returns an object (a Push.Element) which is used for constructing the attributes and children of the element. This gives it an appearance rather like a tree-building API, but this is an illusion: the objects created are transient. Methods such as attribute() use the "chaining" design - they return the object to which they are applied - making it easy to apply further methods to the same object, without the need to bind variables. The endElement() calls have disappeared - an element is closed automatically when the next child is written to the parent element, which we can do because we know which element the child is being attached to.

There are a few other features of the design worthy of attention:

Names of elements and attributes can be supplied either as a plain local name, or as a QName object. A plain local name is interpreted as being in the default namespace in the case of elements (the default namespace can be set at any level), or as being in no namespace in the case of attributes. For the vast majority of documents, there is never any need to use QNames; very often the only namespace handling is a single call on setDefaultNamespace().
The close() method on elements (which generates the end tag) is optional. If you write another child element, the previous child is closed automatically. If you close a parent element, any unclosed child element is closed automatically. The specimen code above shows one call on close(), which is useful in this case for readability: the reader can see that no further children are going to be added.
The argument of methods such as attribute() and text() that supplies the content may always be null. If the content is null, no attribute or text node is written. This makes it easier to handle optional content without disrupting the method chaining.

I have rewritten several classes that construct content using push APIs to use this interface, and the resulting readability is very encouraging.

Representing namespaces in XDM tree models

2019-02-01T11:59:58Z

Most tree representations of XML, including the Saxon TinyTree and LinkedTree implementation, as well as DOM, represent namespace information by holding a set of namespace declarations and undeclarations on each element node.

I'm considering a change to this representation (for the Saxon implementations) to do something that more closely reflects the way namespaces are actually defined in XDM: each element node has a set of in-scope namespaces (held in a NamespaceMap object) containing all the information about the namespaces that apply to that element.

The obvious objection to this, and the reason I've never done it before, is that it looks at first sight to be very inefficient. But consider:

(a) in the vast majority of documents, there are very few namespace declarations on any element other than the root

(b) if there are no namespace declarations on an element, it can point to the same NamespaceMap object that its parent element points to; in most cases, all elements in the document will point to the same shared NamespaceMap.

(c) having a NamespaceMap object immediately available on every element node means we never need to search up the ancestor axis to resolve namespace prefixes

(d) there are still opportunities for implementations of NamespaceMap that use "deltas" if space-saving in pathological cases is considered necessary.

Note that the NamespaceMap holds prefix=uri pairs, not namespace nodes. Namespace nodes have node identity and parentage, which is what makes them so expensive. prefix-uri pairs are just pairs of strings without such baggage, and they can be freely shared across element nodes.

The current implementation I'm using for NamespaceMap is an immutable map implemented as a pair of String[] arrays, one for prefixes and one for uris. The prefix array is maintained in sorted order so we can use binary search to find a prefix. Insertion of a new prefix/uri mapping is O(n), but this doesn't matter because the number of bindings is usually less than ten, and it's a rare operation anyway that only happens during tree construction.

Because the NamespaceMap is immutable, the system is quite easy to implement in a tree builder that gets notified of namespaces incrementally (for example by a SAX parser). The tree builder maintains a stack of NamespaceMap objects. On a startElement event it allocates to the element the same NamespaceMap object that the parent element is using; when a namespace declaration or undeclaration is encountered, this is replaced with a new NamespaceMap with the required modifications.

The real motivation for the change is in implementing copy operations. In complex multi-phase transformations both deep and shallow element copy operations are very frequent, and copying of the namespace information is a significant cost. The XSLT and XQuery language semantics require that when an element is copied, all its in-scope namespaces are copied, and this requires searching the ancestor axis to find them (we try quite hard to optimize this away, but we're not always successful). If the in-scope namespaces are readily to hand in a simple immutable object, we save this effort and just pass the complete object down the pipeline.

The builder for the tree to which the element is being copied now has to merge this set of namespaces with the existing namespaces inherited from ancestor elements on the receiving tree. It should now be clear why I chose the particular data structure for the NamespaceMap: merging two sets of namespace bindings reduces to merging two sorted arrays, which is quite an efficient operation. It's also easy to optimize for the common case where the in-scope namespaces of the element being copied are exactly the same as the in-scope namespaces of its parent element (typically we'll find that the same NamespaceMap object is in use), in which case the merge becomes a null operation.

Of course, there are many details to work through (not least, how we fit this in with third-party tree models that continue to use declarations and undeclarations). But initial experiments are looking encouraging.

The Receiver Pipeline

2018-06-20T11:07:12Z

A significant feature of the internal architecture of Saxon is the Receiver pipeline. A receiver is an object that (rather like a SAX ContentHandler) is called with a sequence of events such as startElement(), characters(), and endElement(); it typically does some processing on these events and then calls similar events on the next Receiver in the pipeline. The mechanism is efficient, because it avoids building a tree in memory, and because it allows much of the conditional logic of the processing (for example, whether or not to validate the document) to be executed at the time the pipeline is constructed, rather than with conditional code executed for every event that occurs.

Receiver pipelines are used throughout Saxon: from doing whitespace stripping on the source document, to serialization of the result tree. The schema validator is implemented as a receiver pipeline, as are operations such as namespace fixup.

But despite the elegance of the design, there have been some perennial problems with the implementation. For example, there are variations on exactly what input different implementations of Receiver will accept: some for example require an open() event while others don't; some accept entire element or document nodes in an append() event while others don't. This limits the ability to construct a pipeline using arbitrary combinations of Receivers, and worse, it's very hard to establish exactly what the permitted combinations are.

This has come to a head recently in trying to get some of the new features in XSLT 3.0 and XQuery 3.1 working reliably and robustly. The straw that broke the camel's back was the innocent-seeming item-separator serialization property. The item-separator is used while doing "sequence normalization" as the first stage of serialization; and the problem was that we didn't really do sequence normalization as a separate step in the processing. The obvious symptom that there are design problems here has been that whenever we get all the XQuery tests working, we find we've broken XSLT; and then when we get all the XSLT tests working, we find XQuery is now failing.

The model according to the specs is that the transformation or query engine produces "raw" results (which can be any sequence of items), and this is then input to the serialization process (or possibly just to sequence normalization, which wraps the results in a document node and then delivers the document). But although Saxon could deliver raw results from an XQuery running in "pull" mode (the XQueryEvaluator.iterate() method) we never really had the capability to produce raw output in push mode: the push code did sequence normalization within the query/transformation logic, rather than leaving it to the serializer. That's for historic reasons, of course: with XSLT 2.0, that's the way it was defined (the result of the transformation was always a document, with optional serialization).

So the first principle to establish in sorting this out is: the interface between the query or transformation engine and the Destination (which may or may not be a Serializer) is a raw sequence, delivered over the Receiver interface.

This requires a definition of exactly how a raw sequence is delivered over this interface: that is, what's the contract between the provider of the Receiver interface and the client (the sender of events). I've created that definition, and I've also written a Receiver implementation which validates that the sequence of events conforms to this definition; we can put this validation step into the pipeline when we feel it useful (for example, when running with assertions enabled). This exercise has revealed quite a few anomalies that should be fixed, for example cases where endDocument() is not being called before calling close().

There are three ways of delivering output from a query or transformation: raw output, document output (the result of sequence normalization), and serialized output. The next question that arises is, who decides which form is delivered. The simplest solution is: this is decided entirely at the API level, and does not depend on anything in the stylesheet or query. (This means that the XSLT build-tree attribute is ignored entirely.) In s9api terms, your choice of Destination object determines which kind of output you get. And at the implementation level, the Destination object always receives raw output; we don't want the transformation engine doing different things depending what kind of Destination has been supplied.

The other related area that needed sorting out was the API interaction with xsl:result-document. We've always had the OutputURIResolver as a callback for determining what should happen to secondary result documents, but this is no longer fit for purpose. It was already a struggle to extend it to handle thread safety when the xsl:result-document instruction became asynchronous; further extending it to work with the s9api Destination framework has never been attempted because it just seemed too difficult. Having made the decision to introduce a dependency on Java 8 for the next major Saxon release, I think we can solve this at the API level with two enhancements:

on XsltTransformer and Xslt30Transformer, a new method setResultDocumentResolver() which takes as argument an implementation of Function - that is a function that accepts an absolute URI as input, and returns a Destination;
on Destination, a new method onClose() which takes as argument a Consumer.

So when xsl:result-document is called, we construct the absolute URI and pass it to the registered result document resolver, and then use the returned Destination to write the result tree. On completion we call any onClose() handler registered with the Destination, which gives the application the opportunity to process the result document (for example, by writing it to a database).

Of course, we have to work out how to implement this while retaining a level of backwards compatibility for applications using the existing OutputURIResolver.

A tricky case with xsl:result-document has been where the href attribute is omitted or empty. I think the cleanest design here is to call the registered result document resolver passing the base output URI as argument, and use the returned Destination in the normal way. The application then has to sort out the fact that the original primary Destination for the transformation is not actually used.

Yet another complication in the design is the rule in XSLT that when xsl:result-document requests schema validation of the output, schema validation is done after sequence normalization and before serialization. This is pretty ugly from a specification point of view: the serialization spec defines serialization as a 6-step process of which sequence normalization is the first; the XSLT spec really has no business inserting an additional step in the middle of this process. When the specification is ugly, the implementation usually ends up being ugly too, and we have to find some way for the transformation engine to inject a validation step into the middle of the pipeline implemented by the Destination, which ought by rights to be completely encapsulated.

Standing back from all this, unlike some refactoring exercises, in this case the basic design of the code proved to be sound, but it needed reinforcement to make the implementation more robust. It needed a clear definition and enforcement of the contract implied by the Receiver interface; it needed a clear separation of concerns between the transformation/query engine and the Destination processing; and it needed a clean API to control it all.

Navigating XML trees using Java Streams

2018-04-13T08:31:44Z

Navigating XML trees using Java Streams

For the next major Saxon release I am planning an extension to the s9api interface to exploit the facilities of Java 8 streams to allow powerful navigation of XDM trees: the idea is that navigation should be as easy as using XPath, but without the need to drop out of Java into a different programming language. To give a flavour, here is how you might select the elements within a document that have @class='hidden':

doc.select(descendant(isElement())
.where(attribute("class").eq("hidden")))

We'll see how that works in due course.

Why do we need it?

The combination of Java and XML is as powerful and ubiquitous today as it as been for nearly twenty years. Java has moved on considerably (notably, as far as this article is concerned, with the Java 8 Streams API), and the world of XML processing has also made great strides (we now have XSLT 3.0, XPath 3.1, and XQuery 3.1), but for some reason the two have not moved together. The bulk of Java programmers manipulating XML, if we can judge from the questions they ask on forums such as StackOverflow, are still using DOM interfaces, perhaps with a bit of XPath 1.0 thrown in.

DOM shows its age. It was originally designed for HTML, with XML added as an afterthought, and XML namespaces thrown in as a subsequent bolt-on. Its data model predates the XML Infoset and the (XPath-2.0-defined) XDM model. It was designed as a cross-language API and so the designers deliberately eschewed the usual Java conventions and interfaces in areas such as the handling of collections and iterators, not to mention exceptions. It does everything its own way. As a navigational API it carries a lot of baggage because the underlying tree is assumed to be mutable. Many programmers only discover far too late that it's not even thread-safe (even when you confine yourself to retrieval-only operations).

There are better APIs than DOM available (for example JDOM2 and XOM) but they're all ten years old and haven't caught up with the times. There's nothing in the Java world that compares with Linq for C# users, or ElementTree in Python.

The alternative of calling out from Java to execute XPath or XQuery expressions has its own disadvantages. Any crossing of boundaries from one programming language to another involves data conversions and a loss of type safety. Embedding a sublanguage in the form of character strings within a host language (as with SQL and regular expressions) means that the host language compiler can't do any static syntax checking or type checking of the expressions in the sublanguage. Unless users go to some effort to avoid it, it's easy to find that the cost of compiling XPath expressions is incurred on each execution, rather than being incurred once and amortized. And the API for passing context from the host language to the sublanguage can be very messy. It doesn't have to be quite as messy as the JAXP interface used for invoking XPath from Java, but it still has to involve a fair bit of complexity.

Of course, there's the alternative of not using Java (or other general-purpose programming languages) at all: you can write the whole application in XSLT or XQuery. Given the capability that XSLT 3.0 and XQuery 3.1 have acquired, that's a real possibility far more often than most users realise. But it remains true that if only 10% of your application is concerned with processing XML input, and the rest is doing something more interesting, then writing the whole application in XQuery would probably be a poor choice.

Other programming languages have developed better APIs. Javascript has JQuery, C# programmers have Linq, Scala programmers have something very similar, and PHP users have SimpleXML. These APIs all have the characteristic that they are much more deeply integrated into the host language, and in particular they exploit the host language primitives for manipulation of sequences through functional programming constructs, with a reasonable level of type safety given that the actual structure of the XML document is not statically known.

That leads to the question of data binding interfaces: broadly, APIs that exploit static knowledge of the schema of the source document. Such APIs have their place, but I'm not going to consider them any further in this article. In my experience they can work well if the XML schema is very simple and very stable. If the schema is complex or changing, data binding can be a disaster.

The Java 8 Streams API

This is not the place for an extended tutorial on the new Streams API introduced in Java 8. If you haven't come across it, I suggest you find a good tutorial on the web and read it before you go any further.

Java Streams are quite unrelated to XSLT 3.0 streaming. Well, almost unrelated: they share the same high-level objectives of processing large collections of data in a declarative way, making maximum use of lazy evaluation to reduce memory use, and permitting parallel execution. But that's where the similarity ends. Perhaps the biggest difference is that Java 8 streams are designed to process linear data structures (sequences), whereas XSLT 3.0 streaming is designed to process trees.

But just to summarise:

Java 8 introduces a new interface, Stream, representing a linear sequence of items of type X
Like iterators, streams are designed to be used once. Unlike iterators, they are manipulated using functional operations, most notably maps and filters, rather than being processed one item at a time. This makes for less error-prone programming, and allows parallel execution.

The functional nature of the Java 8 Streams API means it has much in common with the processing model of XPath. The basic thrust of the API design presented in this article is therefore to reproduce the primitives of the XPath processing model, re-expressing them in terms of the constructs provided by the Java 8 Streams API.

If the design appears to borrow concepts from other APIs such as LINQ and Scala and SimpleXML, that's not actually because I have a deep familiarity with those APIs: in fact, I have never used them in anger, and I haven't attempted to copy anything across literally. Rather, any similarity is because the functional concepts of XPath processing map so cleanly to this approach.

The Basics of the Saxon s9api API

The Saxon product primarily exists to enable XSLT, XQuery, XPath, and XML Schema processing. Some years ago I decided that the standard APIs (JAXP and XQJ) for invoking such functionality were becoming unfit for purpose. They had grown haphazardly over the years, the various APIs didn't work well together, and they weren't being updated to exploit the newer versions of the W3C specifications. Some appalling design mistakes had been unleashed on the world, and the strict backwards compatibility policy of the JDK meant these could never be corrected.

To take one horrid example: the NamespaceContext interface is used to pass a set of namespace bindings from a Java application to an XPath processor. To implement this interface, you need to implement three methods, of which the XPath processor will only ever use one (getNamespaceURI(prefix)). Yet at the same time, there is no way the XPath processor can extract the full set of bindings defined in the NamespaceContext and copy them into its own data structures.

So I decided some years ago to introduce a proprietary alternative called s9api into the Saxon product (retaining JAXP support alongside), and it has been a considerable success, in that it has withstood the test of time rather well. The changes to XSLT transformation in 3.0 were sufficiently radical that I forked the XsltTransformer interface to create a 3.0 version, but apart from that it has been largely possible to add new features incrementally. That's partly because of a slightly less obsessive attitude to backwards compatibility: if I decide that something was a bad mistake, I'm prepared to change it.

Although s9api is primarily about invoking XSLT, XQuery, and XPath processing, it does include classes that represent objects in the XDM data model, and I will introduce these briefly because the new navigation API relies on these objects as its foundation. The table below lists the main classes.

Class	Description
`XdmValue`	Every value is the XDM model is a sequence of items. The `XdmValue` class is therefore the top of the class hierarchy. Because it's a sequence, it implements `Iterable`, so you can use a Java `foreach` loop to process the items sequentially. In the latest version I have used Java generics to add a type parameter, so `XdmValue` is a sequence of nodes, and `XdmValue` is a sequence of atomic values. As well as an `iterator()` method, it has an `itemAt()` method to get the Nth item, and a `size()` method to count the items. Internally an `XdmValue` might exist as an actual sequence in memory, or as a "promise": sufficient data to enable the items to be materialized when they are needed.
`XdmItem`	This class represents an Item in the XDM model. As such it is both a component of an `XdmValue`, and also an `XdmValue` (of length one) in its own right. It's an abstract class, because every item is actually something more specific (a node, an atomic value, a function). Some of the methods inherited from `XdmValue` become trivial (for example `size()` always returns 1).
`XdmNode`	This is a subclass of `XdmItem` used to represent nodes. Unlike many models of XML, we don't subclass this for different kinds of node: that's mainly because XDM has deliberately aimed at uniformity, with the same accessors available for all node kinds. Many of the methods on `XdmNode`, such as `getNodeName()`, `getStringValue()`, `getTypedValue()`, and `getNodeKind()`, are directly equivalent to accessors defined in the W3C XDM specification. But in addition, `XdmNode` has a method `axisIterator` to navigate the tree using any of the XPath axes, the result being returned as an iterator over the selected nodes.
`XdmAtomicValue`	Another subclass of `XdmItem`, this is used to represent atomic values in the XDM model. As with `XdmNode`, we don't define further subclasses for different atomic types. There are convenience methods to convert `XdmAtomicValue` instances to and from equivalent (or near-equivalent) Java classes such as `String`, `Double`, `BigInteger`, and `Date`.
`XdmFunctionItem`	From XPath 3.0, functions are first-class values alongside nodes and atomic values. These are represented in s9api as instances of `XdmFunctionItem`. Two specific subclasses of function, with their own behaviours, are represented using the subclasses `XdmMap` and `XdmArray`. I won't be saying much about these in this article, because I'm primarily concerned with navigating XML trees.

The new API: Steps and Predicates

The basic concept behind the new extensions to the s9api API is navigation using steps and predicates. I'll introduce these concepts briefly in this section, and then go on to give a more detailed exposition.

The class XdmValue acquires a new method:

XdmStream select(Step step)

The Step here is a function that takes an item of class T as its input, and returns a stream of items. If we consider a very simple Step, namely child(), this takes a node as input and returns a stream of nodes as its result. We can apply this step to an XdmValue consisting entirely of nodes, and it returns the concatenation of the streams of nodes obtained by applying the step to each node in the input value. This operation is equivalent to the "!" operator in XPath 3.0, or to the flatMap() method in many functional programming languages. It's not quite the same as the familiar "/" operator in XPath, because it doesn't eliminate duplicates or sort the result into document order. But for most purposes it does the same job.

There's a class net.sf.saxon.s9api.streams.Steps containing static methods which provide commonly-used steps such as child(). In my examples, I'll assume that the Java application has import net.sf.saxon.s9api.streams.Steps.*; in its header, so it can use these fields and methods without further qualification.

One of the steps defined by this class is net.sf.saxon.s9api.streams.Steps.child(): this step is a function which, given a node, returns its children. There are other similar steps for the other XPath axes. So you can find the children of a node N by writing N.select(child()).

Any two steps S and T can be combined into a single composite step by writing S.then(T): for example Step grandchildren = child().then(child()) gives you a step which can be used in the expression N.select(grandchildren) to select all the grandchildren.

The class Step inherits from the standard Java class Function, so it can be used more generally in any Java context where a Function is required.

Predicate is a standard Java 8 class: it defines a function that can be applied to an object of type T to return true or false. The class net.sf.saxon.s9api.streams.Predicates defines some standard predicates that are useful when processing XML. For example isElement() gives you a predicate that can be applied to any XdmItem to determine if it is an element node.

Given a Step A and a Predicate P, the expression A.where(P) returns a new Step that filters the results of A to include only those items that satisfy the predicate P. So, for example, child().where(isElement()) is a step that selects the element children of a node, so that N.select(child().where(isElement())) selects the element children of N. This is sufficiently common that we provide a shorthand: it can also be written N.select(child(isElement())).

The predicate hasLocalName("foo") matches nodes having a local name of "foo": so N.select(child().where(hasLocalName("foo")) selects the relevant children. Again this is so common that we provide a shorthand: N.select(child("foo")). There is also a two argument version child(ns, "foo") which selects children with a given namespace URI and local name.

Another useful predicate is exists(step) which tests whether the result of applying a given step returns at least one item. So, for example N.select(child().where(exists(attribute("id")))) returns those children of N that have an attribute named "id".

The result of the select() method is always a stream of items, so you can use methods from the Java Stream class such as filter() and flatMap() to process the result. Here are some of the standard things you can do with a stream of items in Java:

You can get the results as an array: N.select(child()).toArray()
Or as a list: N.select(child()).collect(Collectors.toList())
You can apply a function to each item in the stream: N.select(child()).forEach(System.err::println)
You can get the first item in the stream: N.select(child()).findFirst().get()

However, Saxon methods such as select() always return a subclass of Stream called XdmStream, and this offers additional methods. For example:

You can get the results as an XdmValue: N.select(child()).asXdmValue()
A more convenient way to get the results as a Java List: N.select(child()).asList()
If you know that the stream contains a single node (or nothing), you can get this using the methods asNode() or asOptionalNode()
Similarly, if you know that the stream contains a single atomic value (or nothing), you can get this using the methods asAtomic() or asOptionalAtomic()
You can get the last item in the stream: N.select(child("para")).last()

More about Steps

The actual definition of the Step class is:

public abstract class Step implements Function>

What that means is that it's a function that any XdmItem as input, and delivers a stream of U items as its result (where U is XdmItem or some possibly-different subclass). (I experimented by also parameterizing the class on the type of items accepted, but that didn't work out well.)

Because the types are defined, Java can make type inferences: for example it knows that N.select(child()) will return nodes (because child() is a step that returns nodes).

As a user of this API, you can define your own kinds of Step if you want to: but most of the time you will be able to do everything you need with the standard Steps available from the class net.sf.saxon.s9api.stream.Steps. The standard steps include:

The axis steps ancestor(), ancestor-or-self(), attribute(), child(), descendant(), descendantOrSelf(), following(), followingSibling(), namespace(), parent(), preceding(), precedingSibling(), self().
For each axis, three filtered versions: for example child("foo") filters the axis to select elements by local name (ignoring the namespace if any); child(ns, local) filters the axis to select elements by namespace URI and local name, and child(predicate) filters the axis using an arbitrary predicate: this is a shorthand for child().where(predicate).
A composite step can be constructed using the method step1.then(step2). This applies step2 to every item in the result of step1, retaining the order of results and flattening them into a single stream.
A filtered step can be constructed using the method step1.where(predicate1). This selects those items in the result of step1 for which predicate1 returns true.
A path with several steps can be constructed using a call such aspath(child(isElement()), attribute("id")). This returns a step whose effect is to return the id attributes of all the children of the target node.
If the steps are sufficiently simple, a path can also by written means of a simple micro-syntax similar to XPath abbreviated steps. The previous example could also be written path("*", "@id"). Again, this returns a step that can be used like any other step. (In my own applications, I have found myself using this approach very extensively).
The step atomize() extracts the typed values of nodes in the input, following the rules in the XPath specification. The result is a stream of atomic values
The step toString() likewise extracts the string values, while toNumber() has the same effect as the XPath number() function

Last but not least, xpath(path) returns a Step that evaluates an XPath expression. For example, doc.select(xpath("//foo")) has the same effect as doc.select(descendant("foo")). A second argument to the xpath() method may be used to supply a static context for the evaluation. Note that compilation of the XPath expression occurs while the step is being created, not while it is being evaluated; so if you bind the result of xpath("//foo") to a variable, then the expression can be evaluated repeatedly without recompilation.

More about Predicates

The Predicate class is a standard Java 8 interface: it is a function that takes any object as input, and returns a boolean. You can use any predicates you like with this API, but the class net.sf.saxon.s9api.streams.Predicates provides some implementations of Predicate that are particularly useful when navigating XML documents. These include the following:

isElement(), isAttribute(), isText(), isComment(), isDocument(), isProcessingInstruction(), isNamespace() test that the item is a node of a particular kind
hasName("ns", "local"), hasLocalName("n"), and hasNamespaceUri("ns") make tests against the name of the node
hasType(t) tests the type of the item: for example hasType(ItemType.DATE) tests for atomic values of type xs:date
exists(step) tests whether the result of applying the given step is a sequence containing at least one item; conversely empty(step) tests whether the result of the step is empty. For example, exists(CHILD) is true for a node that has children.
some(step, predicate) tests whether at least one item selected by the step satisfies the given predicate. For example, some(CHILD, IS_ELEMENT) tests whether the item is a node with at least one element child. Similarly every(step, predicate) tests whether the predicate is true for every item selected by the step.
eq(string) tests whether the string value of the item is equal to the given string; while eq(double) does a numeric comparison. A two-argument version eq(step, string) is shorthand for some(step, eq(string)). For example, descendant(eq(attribute("id"), "ABC")) finds all descendant elements having an "id" attribute equal to "ABC".
Java provides standard methods for combining predicates using and, or, and not. For example isElement().and(eq("foo")) is a predicate that tests whether an item is an element with string-value "foo".

The XdmStream class

The fact that all this machinery is built on Java 8 streams and functions is something that many users can safely ignore; they are essential foundations, but they are hidden below the surface. At the same time, a user who understands that steps and predicates are Java Functions, and that the result of the select() method is a Java Stream, can take advantage of this knowledge.

One of the key ideas that made this possible was the idea of subclassing Stream with XdmStream. This idea was shamelessly stolen from the open-source StreamEx library by Tagir Valeev (though no StreamEx code is actually used). Subclassing Stream enables additional methods to be provided to handle the results of the stream, avoiding the need for clumsy calls on the generic collect() method. Another motivating factor here is to allow for early exit (short-circuit evaluation) when a result can be delivered without reading the whole stream. Saxon handles this by registering onClose() handlers with the stream pipeline, so that when the consumer of the stream calls the XdmStream.close() method, the underlying supplier of data to the stream is notified that no more data is needed.

Examples

This section provides some examples extracted from an actual program that uses s9api interfaces and does a mixture of Java navigation and XPath and XQuery processing to extract data from an input document.

First, some very simple examples. Constructs like this are not uncommon:

XdmNode testInput = (XdmNode) xpath.evaluateSingle("test", testCase);

This can be replaced with the much simpler and more efficient:

XdmNode testInput = testCase.selectFirst(child("test"));

Similarly, the slightly more complex expression:

XdmNode principalPackage = (XdmNode) xpath.evaluateSingle("package[@role='principal']", testInput);

becomes:

XdmNode principalPackage = testInput.selectFirst(child("package").where(eq(attribute("role"), "principal"));

A more complex example from the same application is this one:

boolean definesError = xpath.evaluate("result//error[starts-with(@code, 'XTSE')]", testCase).size() > 0;

Note here how the processing is split between XPath code and Java code. This is also using an XPath function for which we haven't provided a built-in predicate in s9api. But that's no problem, because we can invoke Java methods as predicates. So this becomes:

boolean definesError = testCase.selectFirst(child("result"), descendant("error").where(
                     some(attribute("code"), (XdmNode n) -> n.getStringValue().startsWith("XTSE"))) != null;

Capturing Accumulators

2018-03-28T08:09:03Z

A recent post on StackOverflow made me realise that streaming accumulators in XSLT 3.0 are much harder to use than they need to be.

A reminder about what accumulators do. The idea is that as you stream your way through a large document, you can have a number of tasks running in the background (called accumulators) which observe the document as it goes past, and accumulate information which is then available to the "main" line of processing in the foreground. For example, you might have an accumulator that simply keeps a note of the most recent section heading in a document; that's useful because the foreground processing can't simply navigate around the document to find the current section heading when it finds that it's needed.

Accumulator rules can fire either on start tags or end tags or both, or they can be associated with text nodes or attributes. But there's a severe limitation: a streaming accumulator must be motionless: that's XSLT 3.0 streaming jargon to say that it can only see what's on the parser's stack at the time the accumulator triggers. This affects both the pattern that controls when the accumulator is triggered, and the action that it can take when the rule fires.

For example, you can't fire a rule with the pattern match="section[title='introduction']" because navigation to child elements (title) is not allowed in a motionless pattern. Similarly, if the rule fires on match="section", then you can't access the title in the rule action (select="title") because the action too must be motionless. In some cases a workaround is to have an accumulator that matches the text nodes (match="section/title/text()[.='introduction']") but that doesn't work if section titles can have mixed content.

It turns out there's a simple fix, which I call a capturing accumulator rule. A capturing accumulator rule is indicated by the extension attribute , which will always be a rule that fires on an end-element tag. For a capturing rule, the background process listens to all the parser events that occur between the start tag and the end tag, and uses these to build a snapshot copy of the node. A snapshot copy is like the result of the fn:snapshot function - it's a deep copy of the matched node, with ancestor elements and their attributes tagged on for good measure. This snapshot copy is then available to the action part of the rule processing the end tag. The match patterns that trigger the accumulator rule still need to be motionless, but the action part now has access to a complete copy of the element (plus its ancestor elements and their attributes).

Here's an example. Suppose you've got a large document like the XSLT specification, and you want to produce a sorted glossary at the end, and you want to do it all in streamed mode. Scattered throughout the document are term definitions like this:

A  stylesheet consists of one or more packages: specifically, one
   top-level package and zero or
   more library packages.

Now we can write an accumulator which simply accumulates these term definitions as they are encountered:

(the select expression here takes the existing value of the accumulator, $value, and appends the snapshot of the current termdef element, which is available as the context item ".")

And now, at the end of the processing, we can output the glossary like this:

The value of the accumulator is a list of snapshots of termdef elements, and because these are snapshots, the processing at this point does not need to be streamable (snapshots are ordinary trees held in memory).

The amount of memory needed to accomplish this is whatever is needed to hold the glossary entries. This follows the design principle behind XSLT 3.0 streaming, which was not to do just those things that required zero working memory, but to enable the programmer to do things that weren't purely streamable, while having control over the amount of memory needed.

I think it's hard to find an easy way to tackle this particular problem without the new feature of capturing accumulator rules, so I hope it will prove a useful extension.

I've implemented this for Saxon 9.9. Interestingly, it only took about 25 lines of code: half a dozen to enable the new extension attribute, half a dozen to allow it to be exported to SEF files and re-imported, two or three to change the streamability analysis, and a few more to invoke the existing streaming implementation of the snapshot function from the accumulator watch code. Testing and documenting the feature was a lot more work than implementing it.

Here's a complete stylesheet that fleshes out the creation of a (skeletal) glossary:

Diagnostics on Type Errors

2018-03-16T15:50:27Z

Providing good diagnostics for programming errors has always been a high priority in Saxon, second only to conformance with the W3C specifications. One important area of diagnostics is reporting on type errors: that is, cases where a particular context requires a value of a given type, and the supplied value is the wrong type. A classic example would be providing a string as the first argument to format-date(), which requires an xs:date to be supplied.

Of course, the more programmers follow the discipline of declaring the expected types of function parameters and variables, the more helpful the compiler can be in diagnosing programming errors caused by supplying the wrong type of value.
Type errors can be detected statically or dynamically. Saxon uses "optimistic type checking".

At compile time, it a value of type R is required in a particular context, and the expression appearing in that context is E, then the compiler attempts to infer the static type of expression E: call this S. Sometimes this is straightforward, for example if E is a call on the node-name() function, then it knows that S is xs:QName. In other case the compiler has to be smarter: for example it knows that the static type of a call on remove() is the same as the static type of the first argument, with an adjustment to the occurrence indicator.

Optimistic type checking reports an error at compile time only if there is nothing in common between the required type R and the inferred static type of E: that is, if there is no overlap between the set of instances of the two types. That would mean that a run-time failure is inevitable (assuming the code actually gets executed), and the W3C specifications allow early reporting of such an error.

There's another interesting case where the types overlap only to the extent that both allow an empty sequence: for example if the required type is (xs:string*) and the supplied type is (xs:integer*). That's almost certainly an error, but W3C doesn't allow an error to be reported here because there is a faint chance that execution could succeed. So Saxon reports this as a warning. With maps and arrays, incidentally, there are analogous situations where the only overlap is an empty map or array, but Saxon isn't yet handling that case specially.

If the types aren't completely disjoint, there are two other possibilities: the required type R might subsume the supplied type S, meaning that no run-time type checking is needed because the call will always succeed. The other possibility is that the types overlap: evaluating the supplied expression E might or might not produce a value that matches the required type R. In this case Saxon generates code to perform run-time type checking. (This is one reason why declaring the types of parameters and variables is such good practice: the code runs faster because there is no unnecessary run-time checking.)

Until recently, the error message for a type error takes the form:

Required item type of CCC is RRR; supplied value has item type SSS

For example:

Required item type of first argument to format-date() is xs:date; supplied value has item type xs:string

which works pretty well in most cases. However, I'm finding that as I write more complex code involving maps and arrays, it's no longer good enough. The problem is that as the types become more complex, simply giving the required and actual types isn't enough to make it clear why they are incompatible. You end up with messages like this one:

Required item type of first argument of local:x() is map(xs:integer, xs:date); supplied value has item type map(xs:anyAtomicType, xs:date).

where an expert user can probably work out that the problem is that the supplied map contains an entry whose key is not an integer; but it doesn't exactly point clearly to the source of the problem.

The problem comes to a head particularly when tuple types are used (see here). If the required type is a tuple type, reporting the supplied type as a map type is particularly unhelpful.

I'm therefore changing the approach: instead of reporting on the supplied type of the value (or the inferred type of the expression, in the case of static errors), I'm reporting an explanation of why it doesn't match. Here's the new version of the message:

The required item type of the first argument of local:x() is map(xs:integer, xs:date); the supplied value map{xs:date("2018-03-16Z"):5, "x":3} does not match. The map contains a key (xs:date("2018-03-16Z")) of type xs:date that is not an instance of the required type xs:integer.

So firstly, I'm outputting the actual value, or an abbreviated form of it, rather than just its type (that only works, of course, for run-time errors). And secondly, I'm highlighting how the type-checker worked out that the value doesn't match the required type: it's saying explicitly which rule was broken.

(Another minor change you can see here is that I'm making more effort to write complete English sentences.)

This doesn't just benefit the new map and array types, you can also see the effect with node types. For example, if the required type is document-node(element(foo)), you might see the message:

The required item type of the first argument of local:x() is document-node(element(Q{}foo)); the supplied value doc() does not match. The supplied document node has an element child () that does not satisfy the element test. The node has the wrong name.

Another change I'm making is to distribute type-checking into a sequence constructor. At present, if a function is defined to return (say) a list of element nodes, and the function body contains a sequence of a dozen instructions, one of which returns a text node, you get a message saying that the type of the function result is wrong, but it doesn't pinpoint exactly why. By distributing the type checking (applying the principle that if the function must return element nodes, then each of the instructions must return element nodes) we can (a) identify the instruction in error much more precisely, and (b) avoid the run-time cost of checking the results of those instructions that we know statically are OK.

Interestingly, all these changes were stimulated by my own recent experience in writing a complex stylesheet. I described the plans for this here and the coding has now been completed (I'll report on the outcome later). It's a classic case of dogfood: if you use your own products in anger, you find ways of improving them that you wouldn't have thought of otherwise, and that users wouldn't have suggested because they don't know what's possible.

Could we write an XSD Schema Processor in XSLT?

2018-02-10T18:58:46Z

Many computing platforms are not well-served by up to date XML technology, and in consequence Saxonica has been slowly increasing its coverage of the major platforms: extending from Java to .NET, C++, PHP, Javascript using a variety of technical approaches. This makes it desirable to implement as much as possible using portable languages, and if we want to minimize our dependence on third-party technologies (IKVMC, for example, is now effectively unsupported) we should be writing in our own languages, notably XSLT.

This note therefore asks the question, could one write an XSD Schema 1.1 processor in XSLT?

In fact a schema processor has two parts, compile time (compiling schema documents into the schema component model and SCM) and run-time (validating an instance document using the SCM).

The first part, compiling, seems to pose no intrinsic difficulty. Some of the rules and constraints that need to be enforced are fairly convoluted, but the only really tricky part is compiling grammars into finite-state-machines, and checking grammars (or the resulting finite-state-machine) for conformance with rules such as the Unique Particle Attribution constraint. But since we already have a tool (written in Java) for compiling schemas into an XML-based SCM file, and since it wouldn't really inconvenience users too much for this tool to be invoked via an HTTP interface, the priority for a portable implementation is really the run-time part of the processor rather than the compile-time part. (Note that this means ignoring xsi:schemaLocation, since that effectively causes the run-time validator to invoke the schema compiler.)

There are two ways one could envisage implementing the run-time part in XSLT: either with a universal stylesheet that takes the SCM and the instance document as inputs, or by generating a custom XSLT stylesheet from the SCM, rather as is done with Schematron. For the moment I'll keep an open mind which of these two approaches is preferable.

Ideally, the XSLT stylesheet would use streaming so the instance document being validated does not need to fit in memory. We'll bear this requirement in mind as we look at the detail.

The XSLT code, of course, cannot rely on any services from a schema processor, so it cannot be schema-aware.

Let's look at the main jobs the validator has to do.

Validating strings against simple types

Validating against a primitive type can be done simply using the XPath castable operator.

Validating against a simple type derived by restriction involves checking the various facets. For the most part, the logic of each facet is easily expressed in XPath. There are a few exceptions:

Patterns (regular expressions). The XPath regular expression syntax is a superset of the XSD syntax. To evaluate XSD regular expressions, we either need some kind of extension to the XPath matches() function, or we need to translate XSD regular expressions into XPath regular expressions. This translation is probably not too difficult. It mainly involves rejecting some disallowed constructs (such as back-references, non-capturing groups, and reluctant quantifiers), and escaping "^" and "$" with a backslash.
Length facets for hexBinary and base64Binary. Base646Binary can be cast to hexBinary, and the length of the value in octets can be computed by converting to string and dividing the string length by 2.

Validating against a list type can be achieved by tokenizing, and testing each token against the item type.

Validating against a union type can be achieved by validating against each member type (and also validing against any constraining facets defined at the level of the union itself).

Validating elements against complex types

The only difficult case here is complex content. It should be possible to achieve this by iterating over the child nodes using xsl:iterate, keeping the current state (in the FSM) as the value of the iteration parameter. On completion the element is valid if the state is a final state. As each element is processed, it needs to be checked against the state of its parent element's FSM, and in addition a new validator is established for validating its children. This is all streamable.

Assertions and Conditional Type Assignment

Evaluating XPath expressions can be achieved using xsl:evaluate. The main difficulty is setting up the node-tree to which xsl:evaluate is applied. This needs to be a copy of the original source subtree, to ensure that the assertion cannot stray outside the relevant subtree. Making this copy consumes the source subtree, which makes streaming tricky: however, the ordinary complex type validation can also happen on the copy, so I think streaming is possible.

Identity constraints (unique, key, keyref)

This is where streaming really gets quite tricky - especially given the complexity of the specification for those rare keyref cases where the key is defined on a different element from the corresponding keyref.

The obvious XSLT mechanism here is accumulators. But accumulator rules are triggered by patterns, and defining the patterns that correspond to the elements involved in a key definition is tricky. For example if sections nest recursively, a uniqueness constraint might say that for every section, its child section elements must have unique @section-number attributes. A corresponding accumulator would have to maintain a stack of sections, with a map of section numbers at each level of the stack, and the accumulator rule for a section would need to check the section number of that section at the current level, and start a new level.

A further complication is that there may be multiple (global and/or local) element declarations with the same name, with different unique / key / keyref constraints. Deciding which of these apply by means of XSLT pattern matching is certainly difficult and may be impossible.

The multiple xs:field elements within a constraint do not have to match components of the key in document order, but a streamed implementation would still be possible using the map constructor, which allows multiple downward selections - provided that the xs:field selector expressions are themselves streamable, which I think is probably always the case.

The problem of streamability could possibly be solved with some kind of dynamic pipelining. The "main" validation process, when it encounters a start tag, is able to establish which element declaration it belongs to, and could in principle spawn another transformation (processing the same input stream) for each key / unique constraint defined in that element declaration: a kind of dynamic xsl:fork.

I think as a first cut it would probably be wise not to attempt streaming in the case of a schema that uses unique / key / keyref constraints. More specifically, if any element has such constraints, it can be deep-copied, and validation can then switch to the in-memory subtree rather than the original stream. After all, we have no immediate plans to implement streaming other than in the Java product, and that will inevitably make an XSLT-based schema processor on other platforms unstreamed anyway.

Outcome of validation

There are two main scenarios we should support: validity checking, and type annotation. With validity checking we want to report many invalidities in a single validation episode, and the main output is the validation report. With type annotation, the main output is a validated version of the instance document, and a single invalidity can cause the process to terminate with a dynamic error.

It is not possible for a non-schema-aware stylesheet to add type annotations to the result tree without some kind of extensions. The XSLT language only allows type annotations to be created as the result of schema validation. So we will need an extension for this purpose: perhaps a saxon:type-annotation="QName" attribute on instructions such as xsl:element, xsl:copy, xsl:attribute.

For reporting validation errors, it's important to report the location of the invalidity. This also requires extensions, such as saxon:line-number().

Conclusion

I don't think there are any serious obstacles to writing a validation engine in XSLT. Making it streamable is harder, especially for integrity constraints. A couple of extensions are needed: the ability to add type annotations to the result tree, and the ability to get line numbers of nodes in the source.

I still have an open mind about whether a universal stylesheet should be used, or a generated stylesheet for a particular schema.

Transforming JSON

2017-11-13T13:02:13Z

In my [conference paper at XML Prague](https://www.saxonica.com/papers/xmlprague-2016mhk.pdf) in 2016 I examined a couple of use cases for transforming JSON structures using XSLT 3.0. The overall conclusion was not particularly encouraging: the easiest way to achieve the desired results was to convert the JSON to XML, transform the XML, and then convert it back to JSON.

Unfortunately this study came too late to get any new features into XSLT 3.0. However, I've been taking another look at the use cases to see whether we could design language extensions to handle them, and this is looking quite encouraging.

Use case 1: bulk update

We start with the JSON document

[ {
  "id": 3, "name": "A blue mouse", "price": 25.50,
  "dimensions": {"length": 3.1, "width": 1.0, "height": 1.0},
  "warehouseLocation": {"latitude": 54.4, "longitude": -32.7 }},
  {
  "id": 2, "name": "An ice sculpture", "price": 12.50,
  "tags": ["cold", "ice"],
  "dimensions": {"length": 7.0, "width": 12.0, "height": 9.5 },
  "warehouseLocation": {"latitude": -78.75, "longitude": 20.4 }
} ]

and the requirement: for all products having the tag "ice", increase the price by 10%, leaving all other data unchanged. I've prototyped a new XSLT instruction that allows this to be done as follows:

How does this work?

First the instruction evaluates the root expression, which in this case returns the map/array representation of the input JSON document. With this root item as context item, it then evaluates the select expression to obtain a sequence of contained maps or arrays to be updated: these can appear at any depth under the root item. With each of these selected maps or arrays as the context item, it then evaluates the action expression, and uses the returned value as a replacement for the selected map or array. This update then percolates back up to the root item, and the result of the instruction is a map or array that is the same as the original except for the replacement of the selected items.

The magic here is in the way that the update is percolated back up to the root. Because maps and arrays are immutable and have no persistent identity, the only way to do this is to keep track of the maps and arrays selected en-route from the root item to the items selected for modification as we do the downward selection, and then modify these maps and arrays in reverse order on the way back up. Moreover we need to keep track of the cases where multiple updates are made to the same containing map or array. All this magic, however, is largely hidden from the user. The only thing the user needs to be aware of is that the select expression is constrained to use a limited set of constructs when making downward selections.

The select expression select="?*[?tags?* = 'ice']" perhaps needs a little bit of explanation. The root of the JSON tree is an array of maps, and the initial ?* turns this into a sequence of maps. We then want to filter this sequence of maps to include only those where the value of the "tags" field is an array containing the string "ice" as one of its members. The easiest way to test this predicate is to convert the value from an array of strings to a sequence of strings (so ?tags?*) and then use the XPath existential "=" operator to compare with the string "ice".

The action expression map:put(., 'price', ?price * 1.1) takes as input the selected map, and replaces it with a map in which the price entry is replaced with a new entry having the key "price" and the associated value computed as the old price multiplied by 1.1.

Use case 2: Hierarchic Inversion

The second use case in the XML Prague 2016 paper was a hierarchic inversion (aka grouping) problem. Specifically: we'll look at a structural transformation changing a JSON structure with information about the students enrolled for each course to its inverse, a structure with information about the courses for which each student is enrolled.

Here is the input dataset:

[
  {
    "faculty": "humanities",
    "courses": [
      {
        "course": "English",
        "students": [
          {
            "first": "Mary",
            "last": "Smith",
            "email": "mary_smith@gmail.com"
          },
          {
            "first": "Ann",
            "last": "Jones",
            "email": "ann_jones@gmail.com"
          }
        ]
      },
      {
        "course": "History",
        "students": [
          {
            "first": "Ann",
            "last": "Jones",
            "email": "ann_jones@gmail.com"
          },
          {
            "first": "John",
            "last": "Taylor",
            "email": "john_taylor@gmail.com"
          }
        ]
      }
    ]
  },
  {
    "faculty": "science",
    "courses": [
      {
        "course": "Physics",
        "students": [
          {
            "first": "Anil",
            "last": "Singh",
            "email": "anil_singh@gmail.com"
          },
          {
            "first": "Amisha",
            "last": "Patel",
            "email": "amisha_patel@gmail.com"
          }
        ]
      },
      {
        "course": "Chemistry",
        "students": [
          {
            "first": "John",
            "last": "Taylor",
            "email": "john_taylor@gmail.com"
          },
          {
            "first": "Anil",
            "last": "Singh",
            "email": "anil_singh@gmail.com"
          }
        ]
      }
    ]
  }
]

The goal is to produce a list of students, sorted by last name then irst name, each containing a list of courses taken by that student, like this:

[
  { "email": "anil_singh@gmail.com",
    "courses": ["Physics", "Chemistry" ]},
  { "email": "john_taylor@gmail.com",
    "courses": ["History", "Chemistry" ]},
  ...
]

The classic way of handling this is in two phases: first reduce the hierarchic input to a flat sequence in which all the required information is contained at one level, and then apply grouping to this flat sequence.

To achieve the flattening we introduce another new XSLT instruction:

Again the root expression delivers a representation of the JSON document as an array of maps. The select expression first selects these maps ("?*"), then for each one it calls map:find() to get an array of maps each representing a student. The result of the instruction is a sequence of maps corresponding to these student maps in the input, where each output map contains not only the fields present in the input (first, last, email), but also fields inherited from parents and ancestors (faculty, course). For good measure it also contains a field _keys containing an array of keys representing the path from root to leaf, but we don't actually use that in this example.

Once we have this flat structure, we can construct a new hierarchy using XSLT grouping:

This can then be serialized using the JSON output method to produce to required output.

Note: the saxon:array and saxon:array-member instructions already exist in Saxon 9.8. They fill an obvious gap in the XSLT 3.0 facilities for handling arrays - a gap that exists largely because the XSL WG was unwilling to create a dependency XPath 3.1.

Use Case 3: conversion to HTML

This use case isn't in the XML Prague paper, but is included here for completeness.

The aim here is to construct an HTML page containing the information from a JSON document, without significant structural alteration. This is a classic use case for the recursive application of template rules, so the aim is to make it easy to traverse the JSON structure using templates with appropriate match patterns.

Unfortunately, although the XSLT 3.0 facilities allow patterns that match maps and arrays, they are cumbersome to use. Firstly, the syntax is awkward:

match=".[. instance of map(...)]"

We can solve this with a Saxon extension allowing the syntax

match="map()"

Secondly, the type of a map isn't enough to distinguish one map from another. To identify a map representing a student, for example, we aren't really interested in knowing that it is a map(xs:string, item()*). What we need to know is that it has fields (email, first, last). Fortunately another Saxon extension comes to our aid: tuple types, described here: https://blog.saxonica.com/mike/2016/09/tuple-types-and-type-aliases.html With tuple types we can change the match pattern to

match="tuple(email, first, last)"

Even better, we can use type aliases:

...

With this extension we can now render this input JSON into HTML using the stylesheet:








{?faculty} Faculty




{?course} Course
List of students:






Name
Email










{?first} {?last}
{?email}

Conclusions

With only the facilities of the published XSLT 3.0 recommendation, the easiest way to transform JSON is often to convert it first to XML node trees, and then use the traditional XSLT techniques to transform the XML, before converting it back to JSON.

With a few judiciously chosen extensions to the language, however, a wide range of JSON transformations can be achieved natively.

Bugs: How well are we doing?

2017-02-05T17:36:01Z

We're about to ship another Saxon 9.7 maintenance release, with another 50 or so bug clearances. The total number of patches we've issued since 9.7 was released in November 2015 has now reached almost 450. The number seems frightening and the pace is relentless. But are we getting it right, or are we getting it badly wrong?

There are frequently-quoted but poorly-sourced numbers you can find on the internet suggesting a norm of 10-25 bugs per thousand lines of code. Saxon is 300,000 lines of (non-comment) code, so that would suggest we can expect a release to have 3000 to 7500 bugs in it. One one measure that suggests we're doing a lot better than the norm. Or it could also mean that most of the bugs haven't been found yet.

I'm very sceptical of such numbers. I remember a mature product in ICL that was been maintained by a sole part-time worker, handling half a dozen bugs a month. When she went on maternity leave, the flow of bugs magically stopped. No-one else could answer the questions, so users stopped sending them in. The same happens with Oracle and Microsoft. I submitted a Java bug once, and got a response 6 years later saying it was being closed with no action. When that happens, you stop sending in bug reports. So in many ways, a high number of bug reports doesn't mean you have a buggy product, it means you have a responsive process for responding to them. I would hate the number of bug reports we get to drop because people don't think there's any point in submitting them.

And of course the definition of what is a bug is completely slippery. Very few of the bug reports we get are completely without merit, in the sense that the product is doing exactly what it says on the tin; at the same time, rather few are incontrovertible bugs either. If diagnostics are unhelpful, is that a bug?

The only important test really is whether our users are satisfied with the reliability of the product. We don't really get enough feedback on that at a high level. Perhaps we should make more effort to find out; but I so intensely hate completing customer satisfaction questionnaires myself that I'm very reluctant to inflict it on our users. Given that open source users outnumber commercial users by probably ten-to-one, and that the satisfaction of our open source users is just as important to us as the satisfaction of our commercial customers (because it's satisfied open source users who do all the sales work for us); and given that we don't actually have any way of "reaching out" to our open source users (how I hate the marketing jargon); and given that we really wouldn't know what to differently if we discovered that 60% of our users were "satisfied or very satisfied": I don't really see very much value in the exercise. But I guess putting a survey form on the web site wouldn't be difficult, some people might interpret it as a signal that we actually care.

With 9.7 there was a bit of a shift in policy towards fixing bugs pro-actively (more marketing speak). In particular, we've been in a phase where the XSLT and XQuery specs were becoming very stable but more test cases were becoming available all the time (many of them, I might add, contributed by Saxonica - often in reaction to queries from our users). So we've continuously been applying new tests to the existing release, which is probably a first. Where a test showed that we were handling edge cases incorrectly, and indeed when the spec was changed in little ways under our feet, we've raised bugs and fixes to keep the conformance level as high as possible (while also maintaining compatibility). So we've shifted the boundary a little between feature changes (which traditionally only come in the next release), and bug fixes, which come in a maintenance release. That shift also helps to explain why the gap between releases is becoming longer - though the biggest factor holding us back, I think, is the ever-increasing amount of testing that we do before a release.

Fixing bugs pro-actively (that is before any user has hit the bug) has the potential to improve user satisfaction if it means that they never do hit the bug. I think it's always as well to remember also that for every user who reports a bug there may be a dozen users who hit it and don't report it. One reason we monitor StackOverflow is that a lot of users feel more confident about reporting a problem there, rather than reporting it directly to us. Users know that their knowledge is limited and they don't want to make fools of themselves, and you need a high level of confidence to tell your software vendor that you think the product is wrong.

On the other hand, destabilisation is a risk. A fix in one place will often expose a bug somewhere else, or re-awaken an old bug that had been laid to rest. As a release becomes more mature, we try to balance the benefits of fixing problems with the risk of de-stabilisation.

So, what about testing? Can we say that because we've fixed 450 bugs, we didn't run enough tests in the first place?

Yes, in a sense that's true, but how many more tests would have had to write in order to catch them? We probably run about a million test cases (say, 100K tests in an average of ten product configurations each) and these days the last couple of months before a major release are devoted exclusively to testing. (I know that means we don't do enough continuous testing. But sorry, it doesn't work for me. If we're doing something radical to the internals of the product then things are going to break in the process, and my style is to get the new design working while it's still fresh in my head, then pick up the broken pieces later. If everything had to work in every nightly build, we would never get the radical things done. That's a personal take, and of course what works with a 3-4 person team doesn't necessarily work with a larger project. We're probably pretty unusual in developing a 300Kloc software package with 3-4 people, so lots of our experience might not extrapolate.)

We've had a significant number of bug reports this time on performance regression. (This is of course another area where it's arguable whether it's a bug or not. Sometimes we will change the design in a way that we know benefits some workloads at the expense of others.) Probably most of these are extreme scenarios, for example compilation time for stylesheets where a single template declares 500 local variables. Should we have run tests to prevent that? Well, perhaps we should have more extreme cases in our test suite: the vast majority of our test cases are trivially small. But the problem is, there will always be users who do things that we would never have imagined. Like the user running an XSD 1.1 schema validation in which tens of thousands of assertions are expected to "fail", because they've written it in such a way that assertion failures aren't really errors, they are just a source of statistics for reporting on the data.

The bugs we hate most (and therefore should to most to prevent) are bugs in bytecode generation, streaming, and multi-threading. The reason we hate them is that they can be a pig to debug, especially when the user-written application is large and complex.

For bytecode generation I think we've actually got pretty good test coverage, because we not only run every test in the QT3 and XSLT3 test suites with bytecode generation enabled, we also artificially complicate the tests to stop queries like 2+5 being evaluated by the compiler before bytecode generation kicks in. We've also got an internal recovery mechanism so if we detect that we've generated bad code, we fall back to interpreted mode and the user never notices (problem with that is of course that we never find out).
Streaming is tricky because the code is so convoluted (writing everything as inverted event-based code can be mind-blowing) and because the effects of getting it wrong often give very little clue as to the cause. But at least the failure is "in your face" for the user, who will therefore report the problem, and it's likely to be reproducible. Another difficulty with streaming is that because not all code is streamable, tests for streaming needed to be written from scratch.
Multi-threading bugs are horrible because they occur unpredictably. If there's a low probability of the problem happening then it can require a great deal of detective work to isolate the circumstances, and this often falls on the user rather than on ourselves. Fortunately we only get a couple of these a year, but they are a nightmare when they come. In 9.7 we changed our Java baseline to Java 6 and were able therefore to replace many of the hand-built multithreading code in Saxon with standard Java libraries, which I think has helped reliability a lot. But there are essentially no tools or techniques to protect you from making simple thread-safety blunders, like setting a property in a shared object without synchronization. Could we do more testing to prevent these bugs? I'm not optimistic, because the bugs we get are so few, and so particular to a specific workload, that searching the haystack just in case it contains a needle is unlikely to be effective.

Summary: Having the product perceived as reliable by our users is more important to us than the actual bug count. Fixing bugs quickly before they affect more users is probably the best way of achieving that. If the bug count is high because we're raising bugs ourselves as a result of our own testing, then that's no bad thing. It hasn't yet got to the level where we can't cope with the volumes, or where we have to filter things through staff who are only employed to do support. If we can do things better, let us know.

Guaranteed Streamability

2016-12-09T21:56:27Z

The XSLT 3.0 specification in its current form provides a set of rules (that can be evaluated statically, purely by inspecting the stylesheet) for determining whether the code is (or is not) guaranteed streamable.

If the code is guaranteed streamable then every processor (if it claims to support streaming at all) must use streaming to evaluate the stylesheet; if it is not guaranteed streamable then the processor can choose whether to use streaming or not.

The tricky bit is that there's a requirement in the spec that if the code isn't guaranteed streamable, then a streaming processor (on request) has to detect this and report it. The status section of the spec says that this requirement is "at risk", meaning it might be removed if it proves too difficult to implement. There are people on the working group who believe passionately that this requirement is really important for interoperability; there are others (including me) who fully understand why users would like to have this, but have been arguing that it is extremely difficult to deliver.

In this article I'm going to try to explain why it's so difficult to achieve this requirement, and to explore possibilities for overcoming these difficulties.

Streamability analysis can't be performed until various other stages of static analysis are complete. It generally requires that names have been resolved (for example, names of modes and names of streamable functions). It also relies on rudimentary type analysis (determining the static type of constructs). For Saxon, this means that streamability analysis is done after parsing, name fixup, type analysis, and rewrite optimization.

When Saxon performs these various stages of analysis, it modifies the expression tree as it goes: not just to record the information obtained from the analysis, but to make use of the information at execution time. It goes without saying that in modifying the expression tree, it's not permitted to replace a streamable construct with a non-streamable one, and that isn't too hard to achieve (though these things are relative...). But the requirement to report departures from guaranteed streamability imposes a second requirement, which is proving much harder. If we are to report any deviations from guaranteed streamability, then up to the point where we do the streamability analysis, we must never replace a non-streamable construct with a streamable one.

There are various points at which we currently replace a non-streamable construct with a streamable one.

Very early in the process, the expression tree that is output by the parsing phase uses the same data structure on the expression tree to represent equivalent constructs in the source. For example, the expression tree produced by will be identical to the expression tree produced by . But streamability analysis makes a distinction between these two constructs. It's not a big distinction (in fact, the only thing it affects is exactly where you are allowed to call the accumulator-after() function) but it's big enough to count.
At any stage in the process, if we spot a constant expression then we're likely to replace it with its value. For example if we see the expression $v+3, and $v is a global variable whose value is 5, we will replace the expression with the literal 8. This won't usually affect streamability one way or the other. However, there are a few cases where it does. The most obvious is where we work out that an expression is void (meaning it always returns an empty sequence). For example, according to the spec, the expression (author[0], author[1]) is not streamable because it makes two downward selections. But Saxon spots that author[0] is void and rewrites the expression as (author[1]), which is streamable. Void expressions often imply some kind of user error, so we often output a warning when this happens, but just because we think the user has written nonsense doesn't absolve us from the conformance requirement to report on guaranteed streamability. Void expressions are particularly likely to be found with schema-aware analysis.
Inlining of calls to user-defined functions will often make a non-streamable expression streamable.
Many other rewrites performed by the optimizer have a similar effect, for example replacing (X|Y) by *[self::X|self::Y].

My first attempt to meet the requirement is therefore (a) to add information to the expression tree where it's needed to maintain a distinction that affects streamability, and (b) to try to avoid those rewrites that turn non-streamable expressions into streamable ones. As a first cut, skipping the optimization phase completely seems an easy way to achieve (b). But it turns out it's not sufficient, firstly because some rewrites are done during the type-checking phase, and secondly because it turns out that without an optimization pass, we actually end up finding that some expressions that should be streamable are not. The most common case for this is sorting into document order. Given the expression A/B, Saxon actually builds an expression in the form sort(A!B) relying on the sort operation to sort nodes into document order and eliminate duplicates. This relies on the subsequent optimization phase to eliminate the sort() operation when it can. If we skip the optimization phase, we are left with an unstreamable expression.

The other issue is that the streamability rules rely on type inferencing rules that are much simpler than the rules Saxon uses. It's only in rare cases that this will make a difference, of course: in fact, it requires considerable ingenuity to come up with such cases. The most obvious case where types make a difference to streamability is with a construct like : this is motionless if $v is a text or attribute node, but consuming if it is a document or element node. If a global variable with private visibility is initialized with select="@price", but has no "as" attribute, Saxon will infer a type of attribute(price) for the variable, but the rules in the spec will infer a type of item()*. So to get the same streamability answer as the spec gives, we need to downgrade the static type inferencing in Saxon.

So I think the changes needed to replicate exactly the streamability rules of the XSLT 3.0 spec are fairly disruptive; moreover, implementing the changes by searching for all the cases that need to change is going to be very difficult to get right (and is very difficult to test unless there is another trustworthy implementation of the rules to test against).

This brings us to Plan B. Plan B is to meet the requirement by writing a completely free-standing tool for streamability analysis that's completely separate from the current static analysis code. One way to do this would be to build on the tool written by John Lumley and demonstrated at Balisage a couple of years ago. Unfortunately that's incomplete and out of date, so it would be a significant effort to finish it. Meeting the requirement in the spec is different here from doing something useful for users: what the spec demands is a yes/no answer as to whether the code is streamable; what users want to know is why, and what they need to change to make the code streamable. The challenge is to do this without users having to understand the difficult abstractions in the spec (posture, sweep, and the rest). John's tool produces an annotated expression tree revealing all the properties: that's great for a user who understands the methodology but probably rather bewildering to the typical end user. Doing the minimum for conformance, a tool that just says yes or no without saying why, involves a lot of work to get a "tick in the box" with a piece of software that no-one will ever use, but would be a lot easier to produce. Conformance has always been a very high priority for Saxonica, but I can't see anyone being happy with this particular solution.

So, assuming the WG maintains its insistence of having this feature (and it seems to me likely that it will), what should we do about it?

One option is simply to declare a non-conformance. Once upon a time, standards conformance was very important to Saxon's reputation in the market, but I doubt that this particular non-conformance would affect our sales.

Another option is to declare conformance, do our best to achieve it using the current analysis technology, and simply log bugs if anyone reports use cases where we get the answer wrong. That seems sloppy and dishonest, and could leave us with a continuing stream of bugs to be fixed or ignored.

Another option is the "minimal Plan B" analyser - a separate tool for streamability analysis, that simply reports a yes/no answer (without explanation). It would be significant piece of work to create this and test it, and it's unclear that anyone would use it, but it's probably the cheapest way of getting the conformance tick-in-the-box.

A final option is to go for a "fully featured" but free-standing streamability analysis tool, one which aims to not only answer the conformance question about guaranteed streamability, but also to provide genuinely useful feedback and advice helping users to create streamable stylesheets. Of course ideally such a tool would be integrated into an IDE rather than being free-standing. I've always argued that there's only a need for one such tool: it's not something that every XSLT 3.0 processor needs to provide. Doing this well would be a large project and involves different skills from those we currently have available.

In the short term, I think the only honest and affordable approach would be the first option: declare a non-conformance. Unfortunately that could threaten the viability of the spec, because we can only get a spec to Recommendation status if all features have been shown to be implementable.

No easy answers.

LATER

I've been thinking about a Plan C which might fly...

The idea here is to try and do the streamability analysis using the current expression tree structure and the current streamability logic, but applying the streamability rules to an expression tree that faithfully represents the stylesheet as parsed, with no modifications from type checking or optimization.

To do this, we need to:

Define a configuration flag --strictStreamability which invokes the following logic.
Fix places where the initial expression tree loses information that's needed for streamability analysis. The two that come to mind are (a) losing the information that something is an instruction rather than an expression (e.g. we lose the distinction between xsl:map-entry and a singleton map expression) - this distinction is needed to assess calls on accumulator-after(); (b) turning path expressions A/B into docSort(A!B). There may be other cases that we will discover along the road (or fail to discover, since we may not have a complete set of test cases...)
Write a new type checker that attaches type information to this tree according to the rules in the XSLT 3.0 spec. This will be much simpler than the existing type checker, partly because the rules are much simpler, but more particularly because the only thing it will do is to assign static types: it will never report any type errors, and it will never inject any code to do run-time type checking or conversion.
Immediately after this type-checking phase, run the existing streamability rules against the expression tree. As far as I'm aware, the streamability rules in Saxon are equivalent to the W3C rules (at any rate, most of the original differences have now been eliminated).

There are then two options. We could stop here: if the user sets the --strictStreamability flag, they get the report on streamability, but they don't get an executable that can actually be run. The alternative would be, if the streamability analysis succeeds, attempt to convert the expression tree into a form that we can actually use, by running the existing simplify / typecheck / optimize phases. The distinctions introduced to the expression tree by the changes described above would be eliminated by the simplify() phase, and we would then proceed along the current lines, probably including a rerun of the streamability analysis against the optimised expression tree (because the posture+sweep annotations are occasionally needed at run-time).

I will do some further exploration to see whether this all looks feasible. It will be very hard to prove that we've got it 100% right. But in a sense that doesn't matter, so long as the design is sound and we're passing known tests then we can report honestly that to the best of our knowledge the requirement is satisfied, which is not the case with the current approach.