Insert data into CiqHistorical master table in Mysql
sql ="""INSERT INTO CiqHistorical(CiqRefID, CoID, GVKEY, IID, GRID, CreateDID, SectorID,
UserID, ClientID, MinPeriodID, MaxPeriodID, MaxPeriodDID, MinAnnualID, MaxAnnualID,
MaxAnnualDID) VALUES(%s,%s,'%s','%s',%s,GetDateID(now()),%s,%s,%s,%s,%s,
GetDateID('%s'),%s,%s,GetDateID('%s'));""" %(ciq_ref_id, coid, gvkey, iid,
grid, sector_id,user_id, client_id, min_period_id,
max_period_id, max_period_did, min_annual_id,
max_annual_id, max_annual_did)
ciq_hist = self.mysql_hermes.execute(sql)
You can insert many records using one INSERT, but you should keep the size of your query not too large. Here's a script that does what you need:
CHUNK_SIZE = 1000
def insert_many(data)
index = 0
while True:
chunk = data[index : index + CHUNK_SIZE]
if not chunk:
break
values_str = ", ".join(
"('{0}', '{1}', '{2}', ...)".format(row['field1'], row['field2'], row['field3'], ...)
for row in chunk
)
sql = "INSERT INTO `your_table` (field1, field2, field3, ...) VALUES {0}".format(values_str)
self.mysql_hermes.execute(sql)
index += CHUNK_SIZE
Related
I want to create a SQL table in QT C++. So I have made this code.
And it is going to create a database for me, where the first argument tableName is the name of the table I want to create. Then the next argument is quite tricky.
Here, columns, specify the column name and it's data type. I think this is a bad way to do. Example
QVector<QPair<QString, QString>> myColumns = new QVector<QPair<QString, QString>>{{"ID", "BIGINT"}, {"logging_id", "INT"}};
Because If i have for example like 50 columns. The myColumns is going to be very long.
My question if QT C++ have some kind of reflections, so I can:
Get the name if every field
Get the data type of every field
If the field is an array, then I'm going to know how many elements there are inside that array
I was planning to have an entity class where I create a class, and use that class to get the information to create each columns in the database.
void Database::createTable(QString tableName, const QVector<QPair<QString, QString>> columns){
QSqlQuery query;
for (int i = 0; i < columns.length(); i++){
/* Get the Qpair values */
QString columnName = columns.at(i).first;
QString dataType = columns.at(i).second;
/* If key is ID, then try to create a new table */
if(columnName == "ID"){
query.exec("CREATE TABLE " + tableName + "(" + columnName + " " + dataType + " NOT NULL AUTO_INCREMENT PRIMARY KEY)");
continue;
}
/* If not, then try append new columns */
query.exec("ALTER TABLE " + tableName + " ADD " + columnName + " " + dataType);
}
}
i have codes to get database records, send test data (not shown in below codes),
and count before/after size.
can i do below assertion to do simple math calculation directly using the printIn result?
assert responseListAfter.size() == responseListBefore.size + results
this is the full codes
//count the DB records which State = New
String dbQuery = "SELECT COUNT(*) as count FROM dispenseorders.dispenseorder where status = 'true'"
List results = CustomKeywords.'swisslog.database.getSQLResults'(GlobalVariable.dbConnString , GlobalVariable.dbUsername , GlobalVariable.dbPassword ,GlobalVariable.dbDriver ,dbQuery )
println results
//Before Refresh. count number of records in Kafka Topic A
def TopicBefore = 'A'
def NewMessageBefore = consumeMessageBefore(TopicBefore)
def responseListBefore = new JsonSlurper().parseText(consumeMessageBefore.getResponseText())
def consumeMessageBefore(def topic) {
WS.sendRequest(findTestObject('Object Repository/Kafka/subscribe_to_topic_A', [('topic_name') : topic, ('KafkaRestName') : GlobalVariable.KafkaRestName
, ('KafkaRestPort') : GlobalVariable.KafkaRestPort]))
return consumeMessageBefore = WS.sendRequestAndVerify(findTestObject('Object Repository/Kafka/consume_message_from_topic_for_DR',
[('KafkaRestName') : GlobalVariable.KafkaRestName, ('KafkaRestPort') : GlobalVariable.KafkaRestPort]))
}
println('before Request Refresh: \n' + responseListBefore.size)
WebUI.delay(10)
//after Refresh. count number of records in Kafka Topic A
def TopicAfter = 'A'
def NewMessageAfter = consumeMessageAfter(TopicAfter)
def responseListAfter = new JsonSlurper().parseText(consumeMessageAfter.getResponseText())
def consumeMessageAfter(def topic) {
WS.sendRequest(findTestObject('Object Repository/Kafka/subscribe_to_topic_for_DR', [('topic_name') : topic, ('KafkaRestName') : GlobalVariable.KafkaRestName
, ('KafkaRestPort') : GlobalVariable.KafkaRestPort]))
return consumeMessageAfter = WS.sendRequestAndVerify(findTestObject('Object Repository/Kafka/consume_message_from_topic_for_DR',
[('KafkaRestName') : GlobalVariable.KafkaRestName, ('KafkaRestPort') : GlobalVariable.KafkaRestPort]))
}
println('after Request Refresh: \n' + responseListAfter.size)
//check total messages. After Refresh count = Before Refresh count + DB records
assert responseListAfter.size() == responseListBefore.size + results
results is a list, and you need a list size (int) to add it to other ints:
assert responseListAfter.size() == responseListBefore.size() + results.size()
MSI has table - 'FeatureComponents' with two columns: 'Feature_' and 'Component_'. What I'm trying to do is to change all values in 'Feature_' column at once.
IntPtr hDb = IntPtr.Zero;
int res = MsiInvoke.MsiOpenDatabase(PathToMsi, MsiInvoke.MSIDBOPEN_TRANSACT, out hDb);
string FFF = "SELECT `Feature_` FROM `FeatureComponents`"; <- sql string
IntPtr hView2 = IntPtr.Zero;
res = MsiInvoke.MsiDatabaseOpenView(hDb, FFF, out hView2);
res = MsiInvoke.MsiViewExecute(hView2, IntPtr.Zero);
IntPtr hRec2 = IntPtr.Zero;
res = MsiInvoke.MsiViewFetch(hView2, out hRec2);
res = MsiInvoke.MsiRecordSetString(hRec2, 1, "DUMMY");
res = MsiInvoke.MsiViewModify(hView2, 4, hRec2);
res = MsiInvoke.MsiViewClose(hView2);
res = MsiInvoke.MsiDatabaseCommit(hDb);
However, this only changes first entry in table. So, I'm wondering, how to iterate over all entries in table and change all column values? As right now, I can only do this to one entry and have no idea how to apply this to all entries.
Basically you just turn that fetching code into a loop, and you keep calling MsiViewFetch until you get the ERROR_NO_MORE_ITEMS result, each call returning the next record until you get that error. Simple old example in C++ but the principle is the same:
while ( (errorI = MsiViewFetch (hViewSELECT, &hRecord)) != ERROR_NO_MORE_ITEMS)
nBuffer = (DWORD)256;
MsiRecordGetString(hRecord, 1, svPropname, &nBuffer);
nBuffer = (DWORD)256;
MsiRecordGetString(hRecord, 2, svPropvalue, nBuffer);
}
I have searched and realized i can use SCOPE but am not sure how to use it. Any help will be appreciated
This is Options insert statement
char sql[256];
sprintf_s(sql, "INSERT INTO Options[Value],[ValuesCorrect],[QuestionId]) VALUES ('%s', '%d', '%d'); "
, choice->getValue()
, choice->getIsAnswer()
, choice->getQuestionId());
pRecordSet->Open(sql, pConnection.GetInterfacePtr(), adOpenForwardOnly, adLockReadOnly, adCmdText);
This is my my Question Table
char sql[256];
"DECLARE #ID = BIGINT";
sprintf_s(sql, "INSERT INTO Questions([Query],[CompetencyLevel],[TopicId]) VALUES('%s', %d, %d); "
,(const char*)question->getQuery()
, question->getCompetencyLevel()
,question->getTopicId());
pRecordSet->Open(sql, pConnection.GetInterfacePtr(), adOpenForwardOnly, adLockReadOnly, adCmdText);
"SELECT#ID = SCOPE_IDENTITY();";
The following query will return the inserted id
INSERT INTO Options (
[Value]
,[ValuesCorrect]
,[QuestionId]
)
OUTPUT inserted.[YourIdColumnName]
VALUES (
'%s'
,'%d'
,'%d'
)
A long time since I used ADO but the code could look something like
pRecordSet->Open(...);
auto id = pRecordSet->Fields->Item[0]->Value;
I've been creating a dictionary file that contains 85000 records with Sqlite and QT. the file of sqlite is too huge, it's 134MB, and another dictionary like MDic DB have same data that created with sqlite and same records is 10 MB!
query.exec("PRAGMA journal_mode = MEMORY");
query.exec("PRAGMA synchronous = OFF");
Dic_entry DictionaryEntry = DictionaryInstance.readEntry();
QString definition, headword, displayedHeadWord;
query.exec("BEGIN Transaction");
int Count = 0;
while(!DictionaryEntry.headword.empty())
{
definition = QString::fromStdString(DictionaryEntry.definition);
definition.replace("'", "''");
headword = QString::fromStdString(DictionaryEntry.headword);
headword.replace("'", "''");
displayedHeadWord = QString::fromStdString(DictionaryEntry.displayedHeadword);
displayedHeadWord.replace("'", "''");
string strQuery = "INSERT INTO Dictionary_Words([Definition], [HeadWord], [DisplayedHeadWord]) "
"values('"
+ definition.toStdString() + "', '"
+ headword.toStdString() + "', '"
+ displayedHeadWord.toStdString()
+ "')";
query.exec(QString::fromStdString(strQuery));
if(Count == 200)
{
query.exec("COMMIT TRANSACTION");
Count = 0;
}
Count++;
DictionaryEntry = DictionaryInstance.readEntry();
}
query.exec("End Transaction");
query.exec("CREATE INDEX HW_idx ON [Dictionary_Words](HeadWord)");
query.exec("CREATE INDEX Def_idx ON [Dictionary_Words](Definition)");
query.exec("CREATE INDEX DHW_idx ON [Dictionary_Words](DisplayedHeadword)");
query.exec("PRAGMA auto_vacuum=FULL");
db.close();
Please help me that how can i reduce my SQlite DB file
I have no way of proving it, but I suspect the indexes are the cause of the trouble. Indexes can take up a huge amount of space, and you've got three of them. Try it without them and see if the access is still acceptably fast; the database size should be much smaller that way.