Weka XRFFSaver including missing sparse values - weka

Question

I'm using the XRFFSaver class in the current weka dev version. I'm using xrff rather than arff as I have extremely sparse data and the specs here indicate that sparse instances are handled nicely and efficiently (i.e. not included in output).
However using XRFFSaver they are included in the output like this:
<value index="1" missing="yes"/>
<value index="2" missing="yes"/>
...
Which defeats the purpose of the whole exercise. Anyone know if this is operator error or will I need to write my own saver?
I had a quick look at the source and I could not find any way of toggling this behaviour in either XRFFSaver or XMLInstances, however it was a quick look.
tnx

Answer 1

I quickly hacked up a solution to this:
Note: This is in C# (I use weka through ikvm). However, this should be very simple for anyone to convert to Java.
Note2: The only important line is this one: if (sparse) continue which I also highlighted below with comments. Everything else is a straight copy of the weka source which I found through grepcode and google. Not even sure if its the latest version I copied so please use with discretion.
I also tested to ensure that the standard XRFFLoader handles this correctly and it appears it does.
Tnx
// Usage
var saver = new EfficientXRFFSaver();
saver.setCompressOutput(file.EndsWith(".gz"));
saver.setInstances(Instances);
saver.setFile(new java.io.File(file));
saver.writeBatch();
// Implementation
public class EfficientXRFFSaver : XRFFSaver
{
public override void resetOptions() {
base.resetOptions();
setFileExtension(getCompressOutput() ? XRFFLoader.FILE_EXTENSION_COMPRESSED : XRFFLoader.FILE_EXTENSION);
try { m_XMLInstances = new EfficientXMLInstances(); }
catch { m_XMLInstances = null; }
}
}
public class EfficientXMLInstances : XMLInstances
{
protected override void addInstance(Element parent, Instance inst) {
var node = m_Document.createElement(TAG_INSTANCE);
parent.appendChild(node);
var sparse = inst is SparseInstance;
if (sparse) { node.setAttribute(ATT_TYPE, VAL_SPARSE); }
if (inst.weight() != 1.0) { node.setAttribute(ATT_WEIGHT, Utils.doubleToString(inst.weight(), m_Precision)); }
for (var i = 0; i < inst.numValues(); i++) {
var index = inst.index(i);
var value = m_Document.createElement(TAG_VALUE);
if (inst.isMissing(index)) {
// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// !!!!!!!!!! IMPORTANT !!!!!!!!!!!!!
// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// This line will not add this element if its missing and sparse.
if (sparse) continue;
value.setAttribute(ATT_MISSING, VAL_YES);
} else {
if (inst.attribute(index).isRelationValued()) {
var child = m_Document.createElement(TAG_INSTANCES);
value.appendChild(child);
for (var n = 0; n < inst.relationalValue(i).numInstances(); n++) {
addInstance(child, inst.relationalValue(i).instance(n));
}
} else {
value.appendChild(inst.attribute(index).type() == weka.core.Attribute.NUMERIC ?
m_Document.createTextNode(Utils.doubleToString(inst.value(index), m_Precision)) :
m_Document.createTextNode(validContent(inst.stringValue(index))));
}
}
node.appendChild(value);
if (sparse) { value.setAttribute(ATT_INDEX, "" + (index + 1)); }
}
}
}