Yaml-cpp (new API): Problems mixing maps and scalars in a sequence - c++

I have a very simple problem parsing a yaml file of this form:
- Foo
- Bar:
b1: 5
I would like to parse the top level keys as strings namely "Foo" and "Bar".
As you can see the first entry in the sequence is a scalar and the second is a map containing one key/value pair. Let's say I've loaded this YAML text into a node called config. I iterate over config in the following way:
YAML::Node::const_iterator n_it = config.begin();
for (; n_it != config.end(); n_it++) {
std::string name;
if (n_it->Type() == YAML::NodeType::Scalar)
name = n_it->as<std::string>();
else if (n_it->Type() == YAML::NodeType::Map) {
name = n_it->first.as<std::string>();
}
}
The problem is parsing the second "Bar" entry. I get the following yaml-cpp exception telling me I'm trying to access the key from a sequence iterator n_it.
YAML::InvalidNode: yaml-cpp: error at line 0, column 0: invalid node; this may result from using a map iterator as a sequence iterator, or vice-versa
If I change the access to this:
name = n_it->as<std::string>();
I get a different yaml-cpp exception which I guess is due to the fact that I'm trying to access the whole map as a std::string
YAML::TypedBadConversion<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >: yaml-cpp: error at line 0, column 0: bad conversion
Can somebody please explain to me what's going wrong?
Edit: new problems
I'm still having problems with this api's handling of maps vs sequences. Now say I have the following structure:
foo_map["f1"] = "one";
foo_map["f2"] = "two";
bar_map["b1"] = "one";
bar_map["b2"] = "two";
I want this to be converted to the following YAML file:
Node:
- Foo:
f1 : one
f2 : two
- Bar:
b1 : one
b2 : two
I would do so by doing:
node.push_back("Foo");
node["Foo"]["b1"] = "one";
...
node.push_back("Bar");
However at the last line node has now been converted from a sequence to a map and I get an exception. The only way I can do this is by outputting a map of maps:
Node:
Foo:
f1 : one
f2 : two
Bar:
b1 : one
b2 : two
The problem with this is if I cannot read back such files. If I iterate over Node, I'm unable to even get the type of the node iterator without getting an exception.
YAML::Node::const_iterator n_it = node.begin();
for (; n_it != config.end(); n_it++) {
if (n_it->Type() == YAML::NodeType::Scalar) {
// throws exception
}
}
This should be very simple to handle but has been driving me crazy!

In your expression
name = n_it->first.as<std::string>();
n_it is a sequence iterator (since it's an iterator for your top-level node), which you've just established points to a map. That is,
YAML::Node n = *n_it;
is a map node. This map node (in your example) looks like:
Bar:
b1: 5
In other words, it has a single key/value pair, with the key a string, and the value a map node. It sounds like you want the string key. So:
assert(n.size() == 1); // Verify that there is, in fact, only one key/value pair
YAML::Node::const_iterator sub_it = n.begin(); // This iterator points to
// the single key/value pair
name = sub_it->first.as<std::string>();

Sample.yaml
config:
key1: "SCALER_VAL" # SCALER ITEM
key2: ["val1", "val2"] #SEQUENCE ITEM
key3: # MAP ITEM
nested_key1: "nested_val"
#SAMPLE CODE for Iterate Yaml Node;
YAML::Node internalconfig_yaml = YAML::LoadFile(configFileName);
const YAML::Node &node = internalconfig_yaml["config"];
for(const auto& it : node )
{
std::cout << "\nnested Key: " << it.first.as<std::string>() << "\n";
if (it.second.Type() == YAML::NodeType::Scalar)
{
std::cout << "\nnested value: " << std::to_string(it.second.as<int>()) << "\n";
}
if (it.second.Type() == YAML::NodeType::Sequence)
{
std::vector<std::string> temp_vect;
const YAML::Node &nestd_node2 = it.second;
for(const auto& it2 : nestd_node2)
{
if (*it2)
{
std::cout << "\nnested sequence value: " << it2.as<std::string>() << "\n";
temp_vect.push_back(it2.as<std::string>());
}
}
std::ostringstream oss;
std::copy(temp_vect.begin(), temp_vect.end(),
std::ostream_iterator<std::string>(oss, ","));
std::cout << "\nnested sequence as string: " <<oss.str() << "\n";
}
if (it2.second.Type() == YAML::NodeType::Map)
{
// Iterate Recursively again !!
}
}
Refer here for more details;

This can also be done with the new C++ loop:
std::string name;
for (const auto &entry: node_x) {
assert(name.empty());
name = entry.first.as<std::string>();
}
The assertion will trigger if the node_x is something else than you think. It should be only one entry in this map.

Try something like this:
- Foo: {}
- Bar:
b1: 15

Related

yaml-cpp parsing nested maps and sequences error

I am trying to parse a file with nested maps and sequences which somewhat looks like this
annotations:
- dodge:
type: range based
attributes:
start_frame:
frame_number: 25
time_stamp: 2017-10-14 21:59:43
endframe:
frame_number: 39
time_stamp: 2017-10-14 21:59:45
distances:
- 2
- 5
- 6
I am getting an error saying Ensure the node exists. Below is my sample code.
YAML::Node basenode = YAML::LoadFile(filePath);
const YAML::Node& annotations = basenode["annotations"];
RangeBasedType rm;
for (YAML::const_iterator it = annotations.begin(); it != annotations.end(); ++it)
{
const YAML::Node& gestures = *it;
std::string type = gestures["type"].as<std::string>(); // this works
rm.setGestureName(type);
if (type == "range based")
{
const YAML::Node& attributes = gestures["attributes"];
for (YAML::const_iterator ti = attributes.begin(); ti != attributes.end(); ++ti)
{
const YAML::Node& frame = *ti;
if (frame["start_frame"]) // this fails saying it is not a valid node
{
std::cout << frame["frame_number"].as<int>();
rm.setStartFrame(frame["frame_number"].as<int>());
}
}
}
}
I wish to get the frame_number from nodes start_frame and end_frame. I have checked the YAML format for validity. Any reasons on why this is not working?
This loop:
for (YAML::const_iterator ti = attributes.begin(); ti != attributes.end(); ++ti)
is iterating over a map node. Therefore, the iterator points to a key/value pair. Your next line:
const YAML::Node& frame = *ti;
dereferences it as a node. Instead, you need to look at its key/value nodes:
const YAML::Node& key = ti->first;
const YAML::Node& value = ti->second;
yaml-cpp allows iterators to both point to nodes and key/value pairs because it can be a map or a sequence (or a scalar), and it's implemented as a single C++ type.

yams-cpp nested sequence returning map instead of value

I am trying to parse the following config.yaml file.
config.yaml
foo:
bar:
baz: [1, 2, 3, 4]
bam: "some_string_value"
test.cpp
YAML::Node configObj = YAML::LoadFile("cfig.yaml"); // loads file just fine
YAML::Node fooObj = configObj["foo"]; // this Node object is a Map
// iterate over foo node to get bar node
for( auto it = fooObj.begin(); it != fooObj.end(); ++it){
YAML::Node barMap = it->second; // this Node object is a Map
// iterate over bar node to get bad node
for( auto i = barMap.begin(); i != barMap.end(); ++i){
YAML::Node bazMap = i->second; // this node is a sequence
for( std::size_t i=0; i<bazMap.size(); i++
auto index = bazMap[i].as<int>(); // <<< This is the problem
}
}
}
The problem as far as I can see is that I am expecting index to be an int but bazMap[i].as<int>() I am expecting to be 1 the first loop, 2 the second, etc. What I am getting instead is bazMap[i].as<int>() is of type map. What am I missing in my understanding here?
Thanks,
Bruce
Update The answer was that I stopped early in my nested for loops.
The answer that I found rests in that I was unclear on what I was getting.
In the config.yaml file I was expecting the following:
foo.Type() == Map
bar.Type() == Sequence
baz.Type() == Sequence
but what I was getting was:
foo.Type() == Map
bar.Type() == Map
baz.Type() == Sequence
all I had to do to resolve this issue was to change the config file.
foo:
bar:
- baz: [1, 2, 3, 4]
- bam: "some_string_value"
Then it parsed exactly as I had expected.
So the issue was the format of the config file and not in the parsing logic.

rapidjson - recursively change key value with nested field

I have a Json record with nested object and object arrays, the keys in those field contain spaces, I want to change all spaces to _, so I have to iterate all keys in the json object.
My idea is to write a depth first search to iterate all nested keys using ConstMemberIterator, my question is how can I change the key by given its iterator?
The example below represents my idea:
void DFSReplaceSpaceInKeys(Value::ConstMemberIterator& itr) {
// Replace space in nested key
std::string s(itr->name.GetString());
std::replace(s.begin(), s.end(), ' ', '_');
// set new string to key
itr->name.SetString(s, ?.GetAllocator()); // <----- How can I get the document allocator?
std::cout << "new key: " << itr->name.GetString() << std::endl;
// recursive call in DFS
if (itr->value.GetType() == Type::kObjectType) {
DFSReplaceSpaceInKeys(itr->value.GetObject().MemberBegin());
}
}
A Json record example:
{
"a": {"b": [{"c": [...]}, {...}]
}
You can pass an allocator as parameter. I also think you should better pass Value& to represent a node.
void DFSReplaceSpaceInKeys(Value& value, Value::AllocatorType& allocator) {
if (value.IsObject()) {
for (Value::ConstMemberIterator itr = value.MemberBegin(); itr != MemberEnd(); ++itr)
{
// Modify itr->name here...
DFSReplaceSpaceInKeys(itr->value, allocator);
}
}
else if (value.IsArray()) {
for (Value::ConstIterator itr = value.Begin(); itr != value.End(); ++itr)
DFSReplaceSpaceInKeys(*itr, allocator);
}
}
// ...
Document d;
DFSReplaceSpaceInKeys(d, d.GetAllocator());
If you only need to do the task as mentioned, you may just use the SAX API, which can be easier and faster. Check capitalize example.
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
auto news_obj= news_info["news_feature"].GetObject();
auto title_keyword = news_obj.FindMember ("title_keyword");
if (title_keyword != news_obj.MemberEnd()) {
title_keyword->name.SetString ("title_keywords", allocator);
}

Writing in order to jsoncpp (c++)

Consider the following example for which my source is
Json::Value root;
root["id"]=0;
Json::Value text;
text["first"]="i";
text["second"]="love";
text["third"]="you";
root["text"]=text;
root["type"]="test";
root["begin"]=1;
root["end"]=1;
Json::StyledWriter writer;
string strJson=writer.write(root);
cout<<"JSON WriteTest" << endl << strJson <<endl;
I thought I'd write the json fields in the order of the lines. Instead the result is:
JSON WriteTest
{
"begin" : 1,
"end" : 1,
"id" : 0,
"text" : {
"first" : "i",
"second" : "love",
"third" : "you"
},
"type" : "test"
}
I want json format is
JSON WriteTest
{
"id" : 0,
"text" : {
"first" : "i",
"second" : "love",
"third" : "you"
},
"type" : "test"
"begin" : 1,
"end" : 1,
}
How can I write a Json order?
No, I don't think you can. JsonCpp keeps its values in a std::map<CZString, Value>, which is always sorted by the CZString comparison. So it doesn't know the original order you added items.
This is my workaround to a get an ordered json output from jsoncpp
Json::Value root;
root["*1*id"]=0;
Json::Value text;
text["*1*first"]="i";
text["*2*second"]="love";
text["*3*third"]="you";
root["*2*text"]=text;
root["*3*type"]="test";
root["*4*begin"]=1;
root["*5*end"]=1;
Json::StyledWriter writer;
string resultString=writer.write(root);
resultString=ReplaceAll(resultString,"*1*", "");
resultString=ReplaceAll(resultString,"*2*", "");
resultString=ReplaceAll(resultString,"*3*", "");
resultString=ReplaceAll(resultString,"*4*", "");
resultString=ReplaceAll(resultString,"*5*", "");
cout<<"JSON WriteTest" << endl << resultString <<endl;
with RepleceAll function defined as this
std::string ReplaceAll(std::string str, const std::string& from, const std::string& to) {
size_t start_pos = 0;
while((start_pos = str.find(from, start_pos)) != std::string::npos) {
str.replace(start_pos, from.length(), to);
start_pos += to.length(); // Handles case where 'to' is a substring of 'from'
}
return str;
}
I have a way can solve your problem. Would you like to try? My solution is that you use boost/property_tree/json_parser.hpp, the output is what format you want! About There is my code:
#include <boost/property_tree/json_parser.hpp>
#include <sstream>
#include <iostream>
using namespace std;
int main()
{
boost::property_tree::ptree parser, child;
parser.put("id", 0);
child.put("first", "i");
child.put("second", "love");
child.put("third", "you");
parser.put_child("text", child);
parser.put("type", "test");
parser.put("begin", 1);
parser.put("end", 1);
stringstream ss;
boost::property_tree::json_parser::write_json(ss, parser);
cout << ss.str() << endl;
return 0;
}
Before run the codes, you should install boost 1.57. The codes run well in gcc 4.7, boost 1.57.The output is { "id" : 0, "text" : { "first" : "i", "second" : "love", "third" : "you" }, "type" : "test" "begin" : 1, "end" : 1, }. About boost::property_tree::ptree, you can click here. It used list<pair<key, ptree>> for saving data. So it saved unordered data, unless you called list.sort(). I hope this can help you.
As mentioned by The Dark, JsonCpp keeps its values in a std::map<CZString, Value>, which is always sorted by the CZString comparison, without keeping track neither of the original order in which you added the items nor the desired order in the output.
But you can use this "hidden feature" in your benefit. I mean, you just need that the keys in the desired order follow the "natural" order of CZString. I have a method in my JSONCPP wrapper classes that do this. The quick'n'dirty code, converted to simple function, would be something like this:
std::string sortedStr(Json::Value & value, std::vector<std::string> sortKeys)
{
Json::Value sortedValue; // The JSON object to store the new (sorted) hash
char newKey[60]; // I use C expressions, modify to C++ if you like
// Build new sortedValue
int i = 0;
for (auto & key : sortKeys) {
sprintf(newKey, "SORTEDKEY:%03d-%s", i++, key.c_str());
sortedValue[newKey] = value[key];
}
// Write to string, should be sorted on primary keys
Json::StyledWriter w;
std::string result = w.write(sortedValue);
// Remove aux. tags from primary keys
std::size_t pos = 0;
while ((pos = result.find("SORTEDKEY:", pos)) != std::string::npos) {
result.erase(pos, 14);
}
return result;
}
To use it, just call:
std::string sortedObjStr = sortedValue(myValue, {"first", "second", "third", "fourth"});
Note that:
I use this for relatively small objects (configuration data).
I use the "tag" SORTEDKEY, since this is not going to appear anywhere in my data. Modify according to your needs.
I do not check that the keys used do exist. You can add this check.
You can use this also to generate a restricted, ordered subset of your original object.
The key-value pairs in an object will always be sorted. Json arrays are not sorted, they consists of a series of values without keys.
Objects, as named collections (arrays) of key-value pairs within brackets, in an array, will retain their positions, e.g.
{
"adressen" : [
{
"start" : {
"ID" : 1,
"key" : "2352KJ25",
"lat" : 52.157225922529967,
"lon" : 4.5298663828345527
}
},
{
"eind" : {
"ID" : 2,
"key" : "2352KJ25",
"lat" : 52.157225922529967,
"lon" : 4.5298663828345527
}
}
}
ID, key, lat, lon are sorted, but start and eind are in their original positions.
So, at least your first, second, third could have been
Json::Value text(Json::arrayValue);
text.append("I");
text.append("love");
text.append("you");
No need for the tags first, second and third!
Maybe this helps you to find a workaround.

Creating JSON arrays in Boost using Property Trees

I'm trying to create a JSON array using boost property trees.
The documentation says: "JSON arrays are mapped to nodes. Each element is a child node with an empty name."
So I'd like to create a property tree with empty names, then call write_json(...) to get the array out. However, the documentation doesn't tell me how to create unnamed child nodes. I tried ptree.add_child("", value), but this yields:
Assertion `!p.empty() && "Empty path not allowed for put_child."' failed
The documentation doesn't seem to address this point, at least not in any way I can figure out. Can anyone help?
Simple Array:
#include <boost/property_tree/ptree.hpp>
using boost::property_tree::ptree;
ptree pt;
ptree children;
ptree child1, child2, child3;
child1.put("", 1);
child2.put("", 2);
child3.put("", 3);
children.push_back(std::make_pair("", child1));
children.push_back(std::make_pair("", child2));
children.push_back(std::make_pair("", child3));
pt.add_child("MyArray", children);
write_json("test1.json", pt);
results in:
{
"MyArray":
[
"1",
"2",
"3"
]
}
Array over Objects:
ptree pt;
ptree children;
ptree child1, child2, child3;
child1.put("childkeyA", 1);
child1.put("childkeyB", 2);
child2.put("childkeyA", 3);
child2.put("childkeyB", 4);
child3.put("childkeyA", 5);
child3.put("childkeyB", 6);
children.push_back(std::make_pair("", child1));
children.push_back(std::make_pair("", child2));
children.push_back(std::make_pair("", child3));
pt.put("testkey", "testvalue");
pt.add_child("MyArray", children);
write_json("test2.json", pt);
results in:
{
"testkey": "testvalue",
"MyArray":
[
{
"childkeyA": "1",
"childkeyB": "2"
},
{
"childkeyA": "3",
"childkeyB": "4"
},
{
"childkeyA": "5",
"childkeyB": "6"
}
]
}
What you need to do is this piece of fun. This is from memory, but something like this works for me.
boost::property_tree::ptree root;
boost::property_tree::ptree child1;
boost::property_tree::ptree child2;
// .. fill in children here with what you want
// ...
ptree.push_back( std::make_pair("", child1 ) );
ptree.push_back( std::make_pair("", child2 ) );
But watch out there's several bugs in the json parsing and writing. Several of which I've submitted bug reports for - with no response :(
EDIT: to address concern about it serializing incorrectly as {"":"","":""}
This only happens when the array is the root element. The boost ptree writer treats all root elements as objects - never arrays or values. This is caused by the following line in boost/propert_tree/detail/json_parser_writer.hpp
else if (indent > 0 && pt.count(Str()) == pt.size())
Getting rid of the "indent > 0 &&" will allow it to write arrays correctly.
If you don't like how much space is produced you can use the patch I've provided here
When starting to use Property Tree to represent a JSON structure I encountered similar problems which I did not resolve. Also note that from the documentation, the property tree does not fully support type information:
JSON values are mapped to nodes containing the value. However, all type information is lost; numbers, as well as the literals "null", "true" and "false" are simply mapped to their string form.
After learning this, I switched to the more complete JSON implementation JSON Spirit. This library uses Boost Spirit for the JSON grammar implementation and fully supports JSON including arrays.
I suggest you use an alternative C++ JSON implementation.
In my case I wanted to add an array to a more or less arbitrary location, so, like Michael's answer, create a child tree and populate it with array elements:
using boost::property_tree::ptree;
ptree targetTree;
ptree arrayChild;
ptree arrayElement;
//add array elements as desired, loop, whatever, for example
for(int i = 0; i < 3; i++)
{
arrayElement.put_value(i);
arrayChild.push_back(std::make_pair("",arrayElement))
}
When the child has been populated, use the put_child() or add_child() function to add the entire child tree to the target tree, like this...
targetTree.put_child(ptree::path_type("target.path.to.array"),arrayChild)
the put_child function takes a path and a tree for an argument and will "graft" arrayChild into targetTree
As of boost 1.60.0, problem persists.
Offering a Python 3 workaround (Gist), which can be syscalled just after boost::property_tree::write_json.
#!/usr/bin/env python3
def lex_leaf(lf: str):
if lf.isdecimal():
return int(lf)
elif lf in ['True', 'true']:
return True
elif lf in ['False', 'false']:
return False
else:
try:
return float(lf)
except ValueError:
return lf
def lex_tree(j):
tj = type(j)
if tj == dict:
for k, v in j.items():
j[k] = lex_tree(v)
elif tj == list:
j = [lex_tree(l) for l in j]
elif tj == str:
j = lex_leaf(j)
else:
j = lex_leaf(j)
return j
def lex_file(fn: str):
import json
with open(fn, "r") as fp:
ji = json.load(fp)
jo = lex_tree(ji)
with open(fn, 'w') as fp:
json.dump(jo, fp)
if __name__ == '__main__':
import sys
lex_file(sys.argv[1])
If you want JSON in C++, there's no need for Boost. With this library you can get JSON as a first class data type that behaves like an STL container.
// Create JSON on the fly.
json j2 = {
{"pi", 3.141},
{"happy", true},
{"name", "Niels"},
{"nothing", nullptr},
{"answer", {
{"everything", 42}
}},
{"list", {1, 0, 2}},
{"object", {
{"currency", "USD"},
{"value", 42.99}
}}
};
// Or treat is as an STL container; create an array using push_back
json j;
j.push_back("foo");
j.push_back(1);
j.push_back(true);
// also use emplace_back
j.emplace_back(1.78);
// iterate the array
for (json::iterator it = j.begin(); it != j.end(); ++it) {
std::cout << *it << '\n';
}
Confused with the official document and the above answers.
Below is what I understand.
Property Tree consists of nodes.
Each node is like below
struct ptree
{
map<key_name,value> data;
vector<pair<key_name,ptree>> children;
};
To put 'value' into data with 'put'
To put 'node' into children with 'push_back'\
// Write
bt::ptree root;
bt::ptree active;
bt::ptree requested;
bt::ptree n1, n2, n3;
n1.put("name", "Mark");
n1.put("age", 20);
n1.put("job", "aaa");
n2.put("name", "Rosie");
n2.put("age", "19");
n2.put("job", "bbb");
n3.put("name", "sunwoo");
n3.put("age", "10");
n3.put("job", "ccc");
active.push_back ({ "",l1 });
active.push_back ({ "",l2 });
requested.push_back({ "",l3 });
root.push_back ({"active", active});
root.push_back ({"requested", requested});
bt::write_json("E:\\1.json", root);
// READ
bt::ptree root2;
bt::ptree active2;
bt::ptree requested2;
bt::ptree r1, r2, r3;
bt::read_json("E:\\1.json", root2);
// loop children
for (auto& [k,n] : root.get_child("active"))
{
cout << n.get<string>("name", "unknown");
cout << n.get<int> ("age" , 11);
cout << n.get<string>("job" , "man");
cout << endl << flush;
}