SQL using C++ - Where is the bottleneck? - c++

Question

I'm attempting to read a plain text file line by line, construct an SQL INSERT statement, execute the query, and move on. Presently, I've got a solution that can complete about 200 lines per second on my ~4 year old desktop. However, I've got about 120 million lines to go through and was looking to implement this as a daily task. Taking a few hours to complete it would be fine, but taking nearly a week isn't an option.
The lines will contain one string and anywhere from 5 to 9 integers that range from boolean values (which I've encoded as TINYINT(1)) to microseconds since midnight (BIGINT).
Once read in from the file (via getline()) the lines are tokenized by this function:
#define MAX_TOKENS 10
#define MAX_TOKEN_LENGTH 32
char tokens[MAX_TOKENS][MAX_TOKEN_LENGTH];
//...
void split_line(const string &s)
{
char raw_string[MAX_TOKENS * MAX_TOKEN_LENGTH];
char *rest;
char *token_string;
strcpy(raw_string, s.c_str());
if(tokens[0][0] != '\0')
{
fill(tokens[0], tokens[0]+(MAX_TOKENS*MAX_TOKEN_LENGTH), '\0');
}
for(uint32_t token = 0; token < MAX_TOKENS; token++)
{
if(token == 0) token_string = strtok_r(raw_string, " ", &rest);
else token_string = strtok_r(nullptr, " ", &rest);
if(token_string == nullptr) break;
if(token >= 1)
{
//if it's not a number...
if(token_string[0] < 48 || token_string[0] > 57)
{
if(token_string[0] != 45) //negative numbers are allowed
{
clear_tokens();
break;
}
}
}
strcpy(tokens[token], token_string);
}
}
I had tried a more STL derived version of that tokenizer, but that was proving too slow. It still ranks high in the callgraph, but not as high as it did with proper STL strings.
Anyway, the next step is to build the SQL query. For this, I've tried a few things. One option was stringstreams.
string insert_query = "INSERT INTO data_20170222";
stringstream values;
string query;
while(getline(input_stream, input_stream_line))
{
split_line(input_stream_line);
if(tokens[5][0] != '\0') //the smallest line will have six tokens
{
try
{
query = insert_query;
uint32_t item_type = stoi(tokens[2]);
switch(item_type)
{
case 0: //one type of item
case 1: //another type of item
{
values << " (valueA, valueB, valueC, valueD, valueE, valueF,"
" valueG, valueH) values('"
<< tokens[0] << "', " << tokens[1] << ", "
<< tokens[2] << ", " << tokens[3] << ", "
<< tokens[4] << ", " << tokens[5] << ", "
<< tokens[6] << ", " << tokens[7] << ")";
break;
}
//...
}
query.append(values.str());
values.str(string());
values.clear();
if(mysql_query(conn, query.c_str()))
{
string error(mysql_error(conn));
mysql_close(conn);
throw runtime_error(error);
}
}
catch(exception &ex)
{
cerr << "Error parsing line\n '" << input_stream_line
<< "'\n" << " " << ex.what() << endl;
throw;
}
}
When I run this version, I see 30% of callgrind's samples being measured in std::operator<< within std::basic_ostream.
I originally tried doing this all with strings, ala:
string values;
values = " (valueA, valueB, valueC, valueD, valueE, valueF,"
" valueG, valueH) values('" +
string(tokens[0]) + "', " + tokens[1] + ", "
tokens[2] + ", " + tokens[3] + ", "
tokens[4] + ", " + tokens[5] + ", "
tokens[6] + ", " + tokens[7] + ")";
That proves to be effectively the same speed, but this time with the 30% of samples being allocated to std::operator+ from std::basic_string.
And finally, I switched to straight sprintf().
char values[MAX_TOKENS * MAX_TOKEN_LENGTH];
sprintf(values, " (valueA, valueB, valueC, valueD, valueE, valueF,"
" valueG, valueH) values('%s', %s, %s, %s, %s, %s, %s, %s)",
tokens[0], tokens[1], tokens[2], tokens[3],
tokens[4], tokens[5], tokens[6], tokens[7]);
stringstream was slightly faster than string (though, well within a reasonable margin of error). sprintf() was about 10% faster than both, but that's still not fast enough.
Surely there's a well-established method for accomplishing this task with such large data sets. I'd be grateful for any guidance at this point.
EDIT
Oh wow. I commented out the call to mysql_query() on a whim. Turns out, despite what valgrind says, that's where all my slowdown is. Without that block, it jumps from 200 lines per second to 1.2 million lines per second. That's more like it! Too bad I need the data in a database...
I guess this has become a question about why MariaDB seems to be operating so slowly, now. I've got a good SSD in this system, 16GB RAM, etc. It strikes me as unlikely that my hardware is holding it back.
All the more curious. Thanks in advance for any help!

Answer 1

Batch INSERTing 100 rows per INSERT statement will run 10 times as fast.
Do you need to insert 120M rows per day? That's 1400 per second. Have you calculated how soon before you run out of disk space?
Let's see SHOW CREATE TABLE. Don't use BIGINT (8 bytes) when INT (4 bytes) will suffice. Don't use INT when MEDIUMINT (3 bytes) will do. Etc.
What will you do with a zillion rows of data? Keep in mind that a poorly formed SELECT against a poorly indexed table will take a long time, even with SSDs.
Can the one string be normalized?
Think about packing a bunch of booleans into a SET (1 byte per 8 booleans) or some size int.
Let's see SHOW CREATE TABLE and the main SELECTs.
What is the value of innodb_flush_log_at_trx_commit? Use 2.