strange behaviour of format flag in boost::posix_time - c++

Question

I have a string containing two time stamps. I am trying to convert both the time stamps into boost::posix_time::ptime.
the string format looks like:
"ASTRO20220923.1435+0000-20220923.1440+0000"
the format I use in boost::posix_time::time_input_facet for the second time stamp is ".*+A%Y%m%d.%H%M".
I get correct output but i don’t understand how i get it. On searching the format flags i cannot find any flag as ".*" or "+A". Can someone please help explain why this behaviour is happening?
[LIVE on coliru]
#include <iostream>
#include <boost/date_time/posix_time/posix_time.hpp>
int main()
{
std::string timeStr = "ASTRO20220923.1435+0000-20220923.1440+0000";
std::stringstream ss;
boost::posix_time::ptime pTimeStart;
ss.imbue(std::locale(ss.getloc(), new boost::posix_time::time_input_facet("ASTRO%Y%m%d.%H%M.*")));
ss << timeStr;
ss >> pTimeStart;
std::cout << "START: " << boost::posix_time::to_simple_string(pTimeStart) << std::endl;
boost::posix_time::ptime pTimeEnd;
ss.imbue(std::locale(ss.getloc(), new boost::posix_time::time_input_facet(".*+A%Y%m%d.%H%M")));
ss << timeStr;
ss >> pTimeEnd;
std::cout << "END : " << boost::posix_time::to_simple_string(pTimeEnd) << std::endl;
return 0;
}

Answer 1

Explanation
Boost input facets treat any non-field input character as a wildcard: it matches any input character. This goes for control codes and even embedded NUL characters. None of this is documented, and frankly seems dubious. See this ludicrous test program:
Live On Coliru
Which shows all chosen wildcards succeeding, no matter how far-fetched:
DEBUG : '.....%Y%m%d.%H%M..', '....%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'AAAAA%Y%m%d.%H%MAA', 'AAAA%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'BBBBB%Y%m%d.%H%MBB', 'BBBB%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'CCCCC%Y%m%d.%H%MCC', 'CCCC%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'DDDDD%Y%m%d.%H%MDD', 'DDDD%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'EEEEE%Y%m%d.%H%MEE', 'EEEE%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'fffff%Y%m%d.%H%Mff', 'ffff%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'ggggg%Y%m%d.%H%Mgg', 'gggg%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'hhhhh%Y%m%d.%H%Mhh', 'hhhh%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'iiiii%Y%m%d.%H%Mii', 'iiii%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'jjjjj%Y%m%d.%H%Mjj', 'jjjj%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'kkkkk%Y%m%d.%H%Mkk', 'kkkk%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'lllll%Y%m%d.%H%Mll', 'llll%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'mmmmm%Y%m%d.%H%Mmm', 'mmmm%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : 'nnnnn%Y%m%d.%H%Mnn', 'nnnn%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '11111%Y%m%d.%H%M11', '1111%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '22222%Y%m%d.%H%M22', '2222%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '33333%Y%m%d.%H%M33', '3333%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '44444%Y%m%d.%H%M44', '4444%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '55555%Y%m%d.%H%M55', '5555%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '66666%Y%m%d.%H%M66', '6666%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '77777%Y%m%d.%H%M77', '7777%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '88888%Y%m%d.%H%M88', '8888%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '99999%Y%m%d.%H%M99', '9999%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '00000%Y%m%d.%H%M00', '0000%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '_____%Y%m%d.%H%M__', '____%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '-----%Y%m%d.%H%M--', '----%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '/////%Y%m%d.%H%M//', '////%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '?????%Y%m%d.%H%M??', '????%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '*****%Y%m%d.%H%M**', '****%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '\\\\\\\\\\%Y%m%d.%H%M\\\\', '\\\\\\\\%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '"""""%Y%m%d.%H%M""', '""""%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : "'''''%Y%m%d.%H%M''", "''''%Y%m%d.%H%M" START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '[[[[[%Y%m%d.%H%M[[', '[[[[%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : ']]]]]%Y%m%d.%H%M]]', ']]]]%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '(((((%Y%m%d.%H%M((', '((((%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : ')))))%Y%m%d.%H%M))', '))))%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '&&&&&%Y%m%d.%H%M&&', '&&&&%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '^^^^^%Y%m%d.%H%M^^', '^^^^%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '$$$$$%Y%m%d.%H%M$$', '$$$$%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '#####%Y%m%d.%H%M##', '####%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '#####%Y%m%d.%H%M##', '####%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '!!!!!%Y%m%d.%H%M!!', '!!!!%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : ' %Y%m%d.%H%M ', ' %Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '
%Y%m%d.%H%M
', '
%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '
%Y%m%d.%H%M
', '
%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : ' %Y%m%d.%H%M ', ' %Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
DEBUG : '%Y%m%d.%H%M', '%Y%m%d.%H%M' START: 1400-Jan-01 00:00:00 END: 1400-Jan-01 00:00:00 IGNORED: ASTRO20220923.1435+0000-20220923.1440+0000 (true)
DEBUG : '%Y%m%d.%H%M', '%Y%m%d.%H%M' START: 2022-Sep-23 14:35:00 END: 2022-Sep-23 14:40:00 IGNORED: +0000 (true)
Parsing With Spirit
To parse the start/end period with the given format, I would not use facets in the first place, because none of the format appears locale-aware. The fact that you use ptime (not local_date_time) confirms that the time-zone indication (+0000) is not significant.
So, here's what you could write:
Live On Coliru
// #define BOOST_SPIRIT_X3_DEBUG
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/fusion/adapted/std_tuple.hpp>
#include <boost/spirit/home/x3.hpp>
#include <iostream>
using boost::posix_time::ptime;
namespace Astro {
namespace Parser {
namespace x3 = boost::spirit::x3;
using Part = unsigned short;
using Parts = std::tuple<Part, Part, Part, Part, Part>;
static auto const dig4 = x3::uint_parser<Part, 10, 4, 4>{};
static auto const dig2 = x3::uint_parser<Part, 10, 2, 2>{};
static auto const timeformat
= x3::rule<struct partsformat_, Parts>{"timeformat"}
= dig4 >> dig2 >> dig2 >> '.' >> dig2 >> dig2 >> "+0000";
static auto const period //
= "ASTRO" >> Parser::timeformat >> "-" >> Parser::timeformat >> x3::eoi;
} // namespace Parser
ptime to_ptime(Parser::Parts const& p) {
auto [yyyy, mm, dd, HH, MM] = p;
return {{yyyy, mm, dd}, boost::posix_time::hours(HH) + boost::posix_time::minutes(MM)};
}
struct Period {
ptime start, end;
};
Period parsePeriod(std::string_view input) {
Parser::Parts s, e;
auto both = std::tie(s, e);
if (parse(begin(input), end(input), Parser::period, both))
return {to_ptime(s), to_ptime(e)};
throw std::runtime_error("parsePeriod");
}
}
int main() {
auto [s, e] = Astro::parsePeriod("ASTRO20220923.1435+0000-20220923.1440+0000");
std::cout << "start: " << s << "\n";
std::cout << "end: " << e << "\n";
}
Prints
start: 2022-Sep-23 14:35:00
end: 2022-Sep-23 14:40:00
If you really need stream support, you can build that on top:
Live On Coliru
template <typename It>
bool parsePeriod(It& f, It l, Period& into, bool require_eoi = true) {
Parser::Parts s, e;
auto parts = std::tie(s, e);
bool ok = parse(f, l, Parser::period , parts);
if (ok) {
into = {to_ptime(s), to_ptime(e)};
if (require_eoi)
ok = ok && (f == l);
}
return ok;
}
Period parsePeriod(std::string_view input) {
Period result;
auto f = begin(input), l = end(input);
if (!parsePeriod(f, l, result, true))
throw std::runtime_error("parsePeriod");
return result;
}
std::istream& operator>>(std::istream& is, Period& period) {
auto saved = is.flags();
boost::spirit::istream_iterator f(is >> std::noskipws), l;
if (!parsePeriod(f, l, period , false))
is.setstate(std::ios::failbit);
is.flags(saved);
return is;
}
Which allows you to use it in the same way:
std::istringstream iss("ASTRO20220923.1435+0000-20220923.1440+0000");
Astro::Period p;
if (iss >> p) {
std::cout << "start: " << p.start << "\n";
std::cout << "end: " << p.end << "\n";
} else {
std::cout << "Parse error\n";
}
Printing
start: 2022-Sep-23 14:35:00
end: 2022-Sep-23 14:40:00
BENCHMARKS!
To drive home the point how utterly inefficient the input facet is, I made a benchmark.
It includes the
question code (using boost input facet)
naively using the facet but with some of the issues from the question removed
throwing all the Clever™ tricks to optimize at the boost input facet
my Spirit X3 version
Here's the timings (using Nonius): (click for interactive chart with detail samples)
Benchmark code: Coliru
#include <iostream>
#include <boost/date_time/posix_time/posix_time.hpp>
namespace FacetQuestionCode {
// code from the question, already removing output and one unnecessary allocation (`timeStr` is now a
// string view)
//
// - still has unnecessary facet instantiation/destruction, locale creation/destruction, spurious double
// stream insertion and no error detection
auto parsePeriod(std::string_view timeStr) {
std::stringstream ss;
boost::posix_time::ptime pTimeStart;
ss.imbue(std::locale(ss.getloc(), new boost::posix_time::time_input_facet("ASTRO%Y%m%d.%H%M.*")));
ss << timeStr;
ss >> pTimeStart;
// std::cout << "START: " << boost::posix_time::to_simple_string(pTimeStart) << std::endl;
boost::posix_time::ptime pTimeEnd;
ss.imbue(std::locale(ss.getloc(), new boost::posix_time::time_input_facet(".*+A%Y%m%d.%H%M")));
ss << timeStr;
ss >> pTimeEnd;
// std::cout << "END : " << boost::posix_time::to_simple_string(pTimeEnd) << std::endl;
return std::pair(pTimeStart, pTimeEnd);
}
}
namespace FacetNaive {
// undocumented behaviour, error-prone because not-specifc
auto parsePeriod(std::string_view input) {
std::stringstream ss{std::string(input)};
auto* facet = new boost::posix_time::time_input_facet();
ss.exceptions(std::ios::failbit | std::ios::badbit);
ss.imbue(std::locale(ss.getloc(), facet));
boost::posix_time::ptime s, e;
facet->format("?????%Y%m%d.%H%M??"); ss >> s;
facet->format("????%Y%m%d.%H%M?????"); ss >> e;
return std::pair(s, e);
}
}
namespace FacetClever {
// undocumented behaviour, error-prone because not-specifc
// very dirty optimization tricks
auto parsePeriod(std::string_view input) {
thread_local std::stringstream ss;
thread_local boost::posix_time::time_input_facet* facet = [&] {
auto tmp = new boost::posix_time::time_input_facet();
ss.exceptions(std::ios::failbit | std::ios::badbit);
ss.imbue(std::locale(ss.getloc(), tmp));
return tmp;
}();
ss.clear();
ss.str(std::string(input));
boost::posix_time::ptime s, e;
facet->format("?????%Y%m%d.%H%M??"); ss >> s;
facet->format("????%Y%m%d.%H%M?????"); ss >> e;
return std::pair(s, e);
}
}
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/fusion/adapted/std_tuple.hpp>
#include <boost/spirit/home/x3.hpp>
namespace SpiritX3 {
namespace Parser {
namespace x3 = boost::spirit::x3;
using Part = unsigned short;
using Parts = std::tuple<Part, Part, Part, Part, Part>;
static auto const dig4 = x3::uint_parser<Part, 10, 4, 4>{};
static auto const dig2 = x3::uint_parser<Part, 10, 2, 2>{};
static auto const timeformat
= x3::rule<struct partsformat_, Parts>{"timeformat"}
= dig4 >> dig2 >> dig2 >> '.' >> dig2 >> dig2 >> "+0000";
static auto const period //
= "ASTRO" >> Parser::timeformat >> "-" >> Parser::timeformat >> x3::eoi;
} // namespace Parser
boost::posix_time::ptime to_ptime(Parser::Parts const& p) {
auto [yyyy, mm, dd, HH, MM] = p;
return {{yyyy, mm, dd}, boost::posix_time::hours(HH) + boost::posix_time::minutes(MM)};
}
struct Period {
boost::posix_time::ptime start, end;
};
Period parsePeriod(std::string_view input) {
Parser::Parts s, e;
auto both = std::tie(s, e);
if (parse(begin(input), end(input), Parser::period, both))
return {to_ptime(s), to_ptime(e)};
throw std::runtime_error("parsePeriod");
}
}
#define NONIUS_RUNNER
#include <nonius/benchmark.h++>
#include <nonius/main.h++>
constexpr auto input = "ASTRO20220923.1435+0000-20220923.1440+0000";
NONIUS_BENCHMARK("FacetQuestionCode", []{ FacetQuestionCode::parsePeriod(input); })
NONIUS_BENCHMARK("FacetNaive", []{ FacetNaive::parsePeriod(input); })
NONIUS_BENCHMARK("FacetClever", []{ FacetClever::parsePeriod(input); })
NONIUS_BENCHMARK("SpiritX3", []{ SpiritX3::parsePeriod(input); })
Note how:
the FacetNaive is already 2x faster than the FacetQuestionCode
the Too Clever By Far(TM) optimized version using facets is still 20x slower than Spirit
all the facet versions suffer from lack of specificity and rely on undocumented behavior
The Spirit version is effectively >11,000x faster than the original code, also with the smalles relative deviations
The choice really doesn't seem too hard.