A distinct list based on split values - list

Question

My end goal is to have a unique list of ID's I can iterate through. Here goes:
I have an XML of products (Items). In the complete XML there will be +200,000 items. In this example there is two:
<?xml version="1.0" encoding="utf-8"?>
<Export Shop="Demo Webshop" Type="Full" Clean="true" CleanIsolationShopID="SHOP1">
<Items>
<Item ItemNo="1001" ShopID="SHOP1" VariantCode="1616_42.1615_01.ct_HD">
</Item>
<Item ItemNo="1001" ShopID="SHOP1" VariantCode="1616_42.1615_02.ct_HD" >
</Item>
</Items>
The content of attribute VariantCode I need to split. For the first Item that should give me 1616_42 and 1615_01 and ct_HD. The end result is to import it to a table with the composite primary key ItemNo+VariantOption (VariantOption being the split value).
The XSLT furthermore has:
<table tableName="EcomVariantOptionsProductRelation">
<xsl:for-each select="Export/Items/Item">
<xsl:call-template name="split">
<xsl:with-param name="pText" select="#VariantCode"/>
<xsl:with-param name="ProductID" select="concat(#ItemNo,'##',#ShopID)"/>
/xsl:call-template>
</xsl:for-each>
The template being called that performs the actual split:
<xsl:template match="text()" name="split">
<xsl:param name="pText" select="."/>
<xsl:param name= "ProductID" select="." />
<xsl:choose>
<xsl:when test="string-length($pText) > 0">
<xsl:choose>
<xsl:when test="contains($pText, '.')">
<!-- has dot (more than one variantOption) -->
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID">
<xsl:value-of select="substring-before($pText,'.')"/>
</column>
<column columnName="VariantOptionsProductRelationProductID">
<xsl:value-of select="$ProductID"/>
</column>
</item>
</xsl:when>
<xsl:otherwise>
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID">
<xsl:value-of select="$pText"/>
</column>
<column columnName="VariantOptionsProductRelationProductID">
<xsl:value-of select="$ProductID"/>
</column>
</item>
</xsl:otherwise>
</xsl:choose>
<xsl:call-template name="split">
<xsl:with-param name="pText" select="substring-after($pText, '.')"/>
<xsl:with-param name="ProductID" select="$ProductID"/>
</xsl:call-template>
</xsl:when>
<xsl:otherwise>
<!-- empty string (no variants) -->
<xsl:value-of select="$pText"/>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
The problem is that the transformed output, ie
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID"><![CDATA[1616_42]]></column>
<column columnName="VariantOptionsProductRelationProductID"><![CDATA[1001##SHOP1]]></column>
</item>
is repeated, because the "1616_42" (and "ct_HD" also) part exists twice in two different items. And I need for the output to be unique, since it finally goes to a table where this composite key (VariantID+ProductID) is unique.
The desired result for the two should be:
<table tableName="EcomVariantOptionsProductRelation">
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID"><![CDATA[1616_42]]></column>
<column columnName="VariantOptionsProductRelationProductID"><![CDATA[1001##SHOP1]]></column>
</item>
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID"><![CDATA[1615_01]]></column>
<column columnName="VariantOptionsProductRelationProductID"><![CDATA[1001##SHOP1]]></column>
</item>
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID"><![CDATA[ct_HD]]></column>
<column columnName="VariantOptionsProductRelationProductID"><![CDATA[1001##SHOP1]]></column>
</item>
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID"><![CDATA[1615_02]]></column>
<column columnName="VariantOptionsProductRelationProductID"><![CDATA[1001##SHOP1]]></column>
</item>
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID"><![CDATA[1616_50]]></column>
<column columnName="VariantOptionsProductRelationProductID"><![CDATA[1001##SHOP1]]></column>
</item>
<item tableName="EcomVariantOptionsProductRelation">
<column columnName="VariantOptionsProductRelationVariantID"><![CDATA[ct_NHD]]></column>
<column columnName="VariantOptionsProductRelationProductID"><![CDATA[1001##SHOP1]]></column>
</item>
</table>
Point being: no duplicates.
Searching the web I can see the possibility of creating lists with some kind of unique identifier. But I have no clue if it is possible in my scenario, and even if it is, no clue as how to implement.
Ideas? XSLT 1.0 is used.

Answer 1

The only way I can think of doing this (in XSLT 1.0) is by means of a "two-pass" transform. Effectively, you perform two transforms (although this can be done in a single stylesheet, as I am going to demonstrate). The first transform will split your current VariantCode attributes into separate elements, so the result is like so
<Item ProductId="1001##SHOP1">
<Variant>1616_42</Variant>
<Variant>1615_01</Variant>
<Variant>ct_HD</Variant>
</Item>
The second transform can then use a technique called Muenchian Grouping to output the distinct Variant elements you require.
For this to work, the results of the first transform are simply stored in a variable
<xsl:variable name="variantSplit">
<xsl:apply-templates select="//Item" />
</xsl:variable>
So, in this case, you would have a template matching Item to do the copying and splitting required:
<xsl:template match="Item">
<Item ProductID="{#ItemNo}##{#ShopID}">
<xsl:call-template name="VariantCodeSplit" />
</Item>
</xsl:template>
(In case you haven't seen them before, the curly braces in the ProductID attribute are "Attribute Value Templates", and indicate an expresion to be evaluated, rather than output literally).
Now, you have transformed XML in a variable, where each Item element has multiple child Variant elements as shown above.
But wait! This is XSLT 1.0, which means the contents on the variable is actually a "Result Tree Fragment". If you want to start apply templates on it, you need to use an extension function to transform it into a node-set. This depends on what processor you are using, but you are almost certain to have the node-set function available. It is just a case of declaring the correct namespace. (See http://www.xml.com/pub/a/2003/07/16/nodeset.html for more details).
Anyway, the next stage involves the Muenchian Grouping technique. This involves defining a key to match the new Variant elements, by the a combination of the ProductId and the (split) variant code
<xsl:key name="Test" match="Variant" use="concat(../#ProductID, '|', .)" />
Then, to get the distinct Variant elements, you look for the elements that occur first in the xsl:key for their given combination of ProductID and code
<xsl:apply-templates select="msxml:node-set($variantSplit)/Item/Variant
[generate-id() = generate-id(key('Test', concat(../#ProductID, '|', .))[1])]" />
(Note the use of the node-set extension function here. In my case, I am using Microsoft's).
You can then have a template that matches the Variant element, and you know each match will be a distinct occurence, so you can output the product id and code.
Try this XSLT as a starter. Note it doesn't give you the element and attribute names used in your example (I have shortened them for brevity), but it should give you a start, assuming your head hasn't exploded at this point:
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:msxml="urn:schemas-microsoft-com:xslt"
exclude-result-prefixes="msxml">
<xsl:output method="xml" version="1.0" indent="yes" encoding="ISO-8859-1"/>
<xsl:key name="Test" match="Variant" use="concat(../#ProductID, '|', .)" />
<xsl:template match="/">
<xsl:variable name="variantSplit">
<xsl:apply-templates select="//Item" />
</xsl:variable>
<table>
<xsl:apply-templates select="msxml:node-set($variantSplit)/Item/Variant[generate-id() = generate-id(key('Test', concat(../#ProductID, '|', .))[1])]" />
</table>
</xsl:template>
<xsl:template match="Item">
<Item ProductID="{#ItemNo}##{#ShopID}">
<xsl:call-template name="VariantCodeSplit" />
</Item>
</xsl:template>
<xsl:template name="VariantCodeSplit">
<xsl:param name="Code" select="#VariantCode" />
<xsl:choose>
<xsl:when test="contains($Code, '.')">
<Variant>
<xsl:value-of select="substring-before($Code, '.')"/>
</Variant>
<xsl:call-template name="VariantCodeSplit">
<xsl:with-param name="Code" select="substring-after($Code, '.')" />
</xsl:call-template>
</xsl:when>
<xsl:otherwise>
<Variant>
<xsl:value-of select="$Code"/>
</Variant>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
<xsl:template match="Variant">
<Item>
<Column name="Variant">
<xsl:value-of select="."/>
</Column>
<Column name="Product">
<xsl:value-of select="../#ProductID"/>
</Column>
</Item>
</xsl:template>
</xsl:stylesheet>
Of course, if your actual XML has 200000+ elements this may no be particularly fast.