C++ Slow fetching of Recordset rows - c++

Question

I've got an issue with getting the rows in the Recordset as it is really slow.
We've got an virtual ListCtrl where the data is retrieved and set in the "OnGetdispinfo" method.
This is pretty fast (~2 Seconds for 300k rows on localhost) however if the connection is slow the GUI becomes unrepsonsive and completly unusable until the job is finished.
So I've tried to do the Sql stuff in a different thread and updating the list once all data is fetched.
The issue with the unresponsive GUI is solved with that, but the time it takes to get all the data jumped from 2 seconds to several minutes.
Even if I dont do anything but loop through the rows (just calling MoveNext() in the loop until EOF is reached) it will still take over a minute to complete.
How do I resolve the issue with the freezing GUI without completly destroying the performance here?
I've included the relevant code below
m_pRecordset is a normal Recordset
Old:
void KundenListControlSQLCommand::OnGetdispinfo(NMHDR* pNMHDR, LRESULT* pResult)
{
if (m_pRecordset->IsBOF())
{
*pResult = 0;
return;
}
LV_DISPINFO* pDispInfo = (LV_DISPINFO*)pNMHDR;
LV_ITEM* pItem = &(pDispInfo)->item;
if (pItem->mask & LVIF_TEXT)
{
CString strData;
m_pRecordset->SetAbsolutePosition(pItem->iItem + 1);
if (getStatusRow() != pItem->iSubItem)
{
m_pRecordset->GetFieldValue(short(pItem->iSubItem), strData);
}
::lstrcpy(pItem->pszText, strData);
}
if (pItem->mask & LVIF_IMAGE)
{
int const nIndex = this->GetParent()->SendMessage(OT_VLC_ONGETIMAGEINDEX, pItem->iItem, 0);
if (0 != nIndex)
{
pItem->iImage = nIndex - 1;
}
}
*pResult = 0;
}
void KundenListControlSQLCommand::loadAndDisplayData()
{
ASSERT(!m_strSQLCommand.IsEmpty());
CWaitCursor wc;
try
{
if (!m_pDatabase->IsOpen())
{
CString strSQL = m_pDatabase->getDatabaseInfo().getConnectString();
m_pDatabase->OpenEx(strSQL);
}
// RecordCount ermitteln
m_nRecordCount = m_pRecordset->selectCount(_T("*"), m_strSQLCommand);
if (m_pRecordset->IsOpen())
m_pRecordset->Close();
m_pRecordset->Open(Recordset::snapshot, m_strSQLCommand + m_strSortOrder,
Recordset::executeDirect | Recordset::noDirtyFieldCheck |
Recordset::readOnly | Recordset::useBookmarks);
SetItemCountEx(m_nRecordCount);
}
catch (CDBException* e)
{
e->ReportError();
e->Delete();
}
}
New:
void KundenListControlSQLCommand::loadAndDisplayData()
{
ASSERT(!m_strSQLCommand.IsEmpty());
CWaitCursor wc;
try
{
if (!m_pDatabase->IsOpen())
{
CString strSQL = m_pDatabase->getDatabaseInfo().getConnectString();
m_pDatabase->OpenEx(strSQL);
}
// RecordCount ermitteln
m_nRecordCount = m_pRecordset->selectCount(_T("*"), m_strSQLCommand);
if (m_pRecordset->IsOpen())
m_pRecordset->Close();
m_pRecordset->Open(Recordset::dynaset, m_strSQLCommand + m_strSortOrder,
Recordset::executeDirect | Recordset::noDirtyFieldCheck |
Recordset::readOnly | Recordset::useBookmarks);
m_vResult.clear();
m_vResult.reserve(m_nRecordCount);
int nFieldCount = m_pRecordset->GetODBCFieldCount();
CString strData;
while (!m_pRecordset->IsEOF())
{
for (auto i = 0; i < nFieldCount; i++)
{
m_pRecordset->GetFieldValue(short(i), strData);
m_vResult.push_back(std::move(strData));
}
if (m_bAbort)
{
m_bAbort = false;
return;
}
m_pRecordset->MoveNext();
}
GetParent()->SendMessage(OT_VLC_ON_LIST_DONE, NULL, NULL);
}
catch (CDBException* e)
{
e->ReportError();
e->Delete();
}
}
void KundenListControlSQLCommand::OnGetdispinfo(NMHDR* pNMHDR, LRESULT* pResult)
{
if (m_pRecordset->IsBOF())
{
*pResult = 0;
return;
}
LV_DISPINFO* pDispInfo = (LV_DISPINFO*)pNMHDR;
LV_ITEM* pItem = &(pDispInfo)->item;
UINT nItem = (pItem->iItem * m_pRecordset->GetODBCFieldCount()) + pItem->iSubItem;
if (pItem->mask & LVIF_TEXT && m_vResult.size() >= nItem)
{
::lstrcpy(pItem->pszText, std::move(m_vResult.at(nItem)));
}
if (pItem->mask & LVIF_IMAGE)
{
int const nIndex = this->GetParent()->SendMessage(OT_VLC_ONGETIMAGEINDEX, pItem->iItem, 0);
if (0 != nIndex)
{
pItem->iImage = nIndex - 1;
}
}
*pResult = 0;
}``

Answer 1

As I can see in your code, you read the data and place them into the vector. In such a setting, I think you don't really need a dynaset recordset, which according to the documentation is "A recordset with bi-directional scrolling". It fetches data row-by-row, which may be what makes the process slow. Also, "changes made by other users to the data values are visible following a fetch operation", but I think this is not of critical importance in this case. It would be mostly useful for displaying more "live" data, that are updated often.
Instead, a snapshot, or even forwardOnly recordset would suffice and would be faster. You can also experiment with the CRecordset::useMultiRowFetch option. The documentation says it's faster. It requires some changes to your code (moving next etc). Take a look here: Recordset: Fetching Records in Bulk (ODBC).
An alternative, radically different implementation would be to use bookmarks instead. Loading would be a lot faster, but scrolling somewhat sluggish, as you will have to fetch data in the OnGetdispinfo() function.
Finally a tip, if you are using the MS-SQL server, check the native driver, if you haven't already, many on the i-net claim that it's considerably faster.

Answer 2

I don't know much about ODBC, but suspect that there are better way to get bulk data.
Regardless, you do a lot of unnecessary copying of your vectors. Two easy fixes:
Right after m_vResult.clear();, resize your m_vResult to the number of records.
Instead of m_vResult.push_back(vResult); do m_vResult.push_back(std::move(vResult));, as you don't need your vResult after that.

Answer 3

Another solution is to do a cache list, handling LVN_ODCACHEHINT notification (this example is for CListView but you can adapt it on your CListCtrl:
// header.h
class CYourListView : public CListView
{
// ...
afx_msg void OnLvnOdcachehint(NMHDR* pNMHDR, LRESULT *pResult);
};
and implementation:
// YourListView.cpp
// ...
ON_NOTIFY_REFLECT(LVN_ODCACHEHINT, &CYourListView::OnLvnOdcachehint)
END_MESSAGE_MAP()
void CYourListView::OnLvnOdcachehint(NMHDR* pNMHDR, LRESULT* pResult)
{
LPNMLVCACHEHINT pCacheHint = reinterpret_cast<LPNMLVCACHEHINT>(pNMHDR);
const DWORD dwTo = pCacheHint->iTo;
const DWORD dwFetched = m_vResult.size();
if (dwTo >= dwFetched) // new rows must be fetched
{
const DWORD dwColCount = m_pRecordset->GetColumnCount();
m_vResult.resize(dwTo + 1);
for (DWORD dwRow = dwFetched; dwRow <= dwTo; ++dwRow)
{
CDBRecord* pRecord = new CDBRecord;
pRecord->SetSize(dwColCount);
for (DWORD dwCol = 1; dwCol <= dwColCount; dwCol++)
{
CDBValue* pDBValue = new CDBValue(m_pRecordset, dwCol);
pRecord->SetAt(dwCol - 1, pDBValue);
}
m_vResult.emplace(m_vResult.begin() + dwRow, pRecord);
m_pRecordset->MoveNext();
}
}
*pResult = 0;
}
might be need to adjust some variables / values with your certain situation.