STXXL Dealing with strings - c++

Question

i am dealing with large dataset. May i ask you how it is possible to store strings in the classes i want to use with stxxl? I have read several discussions and everywhere was said that string is not POD type therefore it cannot be stored in the stxxl::vector, but i am not sure,because i tried it and i checked the data and everything seems to be fine. i have also saw an approach here https://github.com/Project-OSRM/osrm-backend/blob/725b93a961625a7b04d54806d7e0f80252a6bcd0/extractor/extraction_containers.hpp and they use stxxl::vector, so maybe the library got updated to support std::string?
class HighWay
{
private:
uint64_t id;
string name;
int speed;
string attributes; //additional attributes of way
string edges; //written uint64_t from,uint64_t to, int distance written as string
string nodes; //vector<uint64_t> written as string
public:
HighWay() = default;
void setId(uint64_t _id) {
id = boost::lexical_cast<string>(_id);
}
void setName(string _name) {
name = _name;
}
void setSpeed(int _speed) {
speed = _speed;
}
void setAttributes(string _attributes) {
attributes = _attributes;
}
void setEdges(string _edges) {
edges = _edges;
}
void setNodes(vector<uint64_t>refs) {
stringstream s;
uint64_t i = 0;
for (; i < refs.size()-1;i++) {
s << boost::lexical_cast<uint64_t>(refs[i]) << " ";
}
s << boost::lexical_cast<uint64_t>(refs[i]);
nodes = s.str();
}
uint64_t getId() {
return id;
}
string getName() {
return name;
}
int getSpeed() {
return speed;
}
string getAttributes() {
return attributes;
}
string getEdges() {
return edges;
}
std::vector<int64_t> getNodes() {
stringstream s(nodes);
uint64_t node;
std::vector<int64_t> result;
while (s >> node) {
result.push_back(static_cast<int64_t>(node));
}
return result;
}
};
I have also created code which stores the strings as POD,storing the string in vector of char and in map remembering lower and upper bound index in the array. But this approach leads to many std::maps used in the application.
//class to store in map
struct TypeName{
uint64_t start;
uint64_t end;
};
std::istream& operator >> (std::istream& i, TypeName& entry)
{
i >> entry.start;
i >> entry.end;
return i;
}
std::ostream& operator << (std::ostream& i, const TypeName& entry)
{
i << entry.start << " ";
i << entry.end;
return i;
}
struct PoiCategories{
uint64_t start;
uint64_t end;
};
std::istream& operator >> (std::istream& i,PoiCategories& entry)
{
i >> entry.start;
i >> entry.end;
return i;
}
std::ostream& operator << (std::ostream& i, const PoiCategories& entry)
{
i << entry.start << " ";
i << entry.end;
return i;
}
//object i want to store
struct Poi {
Poi() = default;
uint64_t id;
char type;
uint64_t id_in_pois; //id in vector pois
void addCategories(
vector<int> &kats, //categories to insert
stxxl::vector<uint64_t> &categories, //vector to store category
std::unordered_map <uint64_t, PoiCategories> &idPoi_categories //index to vector categories to retrieve all categories for Poi
)
{
size_t start = categories.size();
for (auto & kat : kats) {
categories.push_back(kat);
}
size_t end = categories.size() - 1;
idPoi_categories.insert(make_pair(id, PoiCategories{start, end }));
}
vector<int> getCategories(
stxxl::vector<uint64_t> &categories,
std::unordered_map <uint64_t, PoiKategorie> &idPoi_categories
)
{
std::vector<int> result;
PoiCategories bounds = idPoi_categories.find(id)->second;
for (size_t i = bounds.start; i <= bounds.end; i++) {
result.push_back(categories[i]);
}
return result;
}
};
Problem in my application is that i am storing a few string data, which are mainly names of streets and POIs. Maybe i am using wrong library. If so,can you recommend me a better approach to store data while preprocessing?

Answer 1

It's indeed banned, but the symptoms of violating the no-POD rule are generally unpredictable. It may very well appear to work as long as the strings all fit in memory, but in that case you didn't need the STXXL anyway.