I have a method (from a third party lib)
bool READ_DB(Connection* con, long value);
void TAMLTradeProcessor::CreateThreads() {
long nThreads = 15; // we can configure this
// set up and run the threads
HANDLE* pWaitHandles = new HANDLE[nThreads];
CThreadInfoData* pTid = new CThreadInfoData[nThreads];
UINT nRunningThreads = 0;
long lSharedIndex = -1;
// Initialise data blocks
int i;
for (i = 0; i < nThreads; i++)
{
pTid[i].m_bRunning = true;
pTid[i].m_pnCurrentIndexPosition = &lSharedIndex; // common index
pTid[i].m_pDbConn = new CDatabaseConnection();
pTid[i].m_hThread = (HANDLE )_beginthreadex(NULL,0,ThreadCB,&pTid[i],0,&pTid[i].m_nThreadId);
...
}
It reads data off the database using the connection I pass in and matching the query for that specific value.
I have a huge list of values so I created multiple threads that retrieves values off the list and call the method, in other word I am retrieving the data in parallel using mutiple DB connection.
ThreadCB will call READ_DB.
At the moment I have created the threads myself and I have created 15 of them...just a casual number.
Is there a better way of doing this using Windows ThreadPool API?
In other words if I need to run the same DB query over and over gain for different values (but I can only use one value at time) what is the best approach?
Related
I've implemented a custom CRecordset class, and have code similar to the following:
ASSERT(prs->GetRowsetSize() == 25);
while (!prs->IsEOF())
{
for (int i = 1; i <= prs->GetRowsFetched(); i++)
{
prs->SetRowsetCursorPosition((WORD)i);
// Inspecting data here...
}
prs->MoveNext();
}
prs->Close();
Apparently, when using multi-row fetch, CRecordset does not call my DoFieldExchange override as it does when not using multi-row fetch, and that is by design. And so my data isn't automatically populated. So the question is how do I get the data?
The answer appears to be by calling GetFieldValue(). But I get an Invalid cursor position error when I do! (GetFieldValue() works fine when I'm not using multi-row fetch.)
Below is a streamlined version of my recordset class. In addition, #EylM was good enough to create a sample in the answers below that he says does work for him. However, when I copied his code exactly and just changed what was needed to connect to and query my database, I still get an Invalid cursor position when I call GetFieldValue().
I don't know what else could be different. I see he's using MySQL where I'm using SQL Server. But surely CRecordset works with SQL Server. I've also tried all the available SQL Server ODBC drivers, but the result is always the same.
class CRS : public CRecordset
{
public:
// Data variables
int m_nId;
TCHAR m_szName[CUSTOMER_NAME_MAXLENGTH + 1];
// Bulk data variables
int* m_pnIds;
long* m_pnIdLengths;
LPTSTR m_pszNames;
long* m_pnNameLengths;
// Constructor
CRS(CDatabase* pDatabase = NULL)
: CRecordset(pDatabase)
{
m_nFields = 2;
m_nId = 0;
m_szName[0] = '\0';
m_pnIds = NULL;
m_pnIdLengths = NULL;
m_pszNames = NULL;
m_pnNameLengths = NULL;
}
CString GetDefaultSQL()
{
return CCustomerData::m_szTableName;
}
// This method is never called when
// CRecordset::useMultiRowFetch is specified!
void DoFieldExchange(CFieldExchange* pFX)
{
pFX->SetFieldType(CFieldExchange::outputColumn);
RFX_Int(pFX, _T("Id"), m_nId);
RFX_Text(pFX, _T("Name"), m_szName, CUSTOMER_NAME_MAXLENGTH);
}
// This method is called several times
void DoBulkFieldExchange(CFieldExchange* pFX)
{
pFX->SetFieldType(CFieldExchange::outputColumn);
RFX_Int_Bulk(pFX, _T("Id"), &m_pnIds, &m_pnIdLengths);
RFX_Text_Bulk(pFX, _T("Name"), &m_pszNames, &m_pnNameLengths, (CUSTOMER_NAME_MAXLENGTH + 1) * 2);
}
};
UPDATE:
Spending more time on this, I have been able to write code that reads the data directly from the rowset data (in my case, from m_pnIds, m_pnIdLengths, m_pszNames and m_pnNameLengths). Perhaps that's the approach I need to take.
But the question still stands. Why can't I use GetFieldValue() on a SQL Server database? And what is the point of SetRowsetCursorPosition()?
From documentation of CRecordset::DoFieldExchange:
When bulk row fetching is not implemented, the framework calls this
member function to automatically exchange data between the field data
members of your recordset object and the corresponding columns of the
current record on the data source.
DoFieldExchange is called only if CRecordset::useMultiRowFetch is not specified in the Open function.
Looking at MFC code CRecordset::BindFieldsToColumns, dbcore.cpp using VS 2019 (14.22.27905):
// Binding depends on fetch type
if (m_dwOptions & useMultiRowFetch)
DoBulkFieldExchange(&fx);
else
DoFieldExchange(&fx);
Sounds like that behaviour your are getting is by design.
Edit:
Here is working example for multi row fetch. The thing that did the trick is CRecordset::useExtendedFetch in the opening flags.
Database:
I used MySQL with a simple table with 2 columns. Here is the creation script.
CREATE TABLE `categories` (
`CatID` int(11) NOT NULL,
`Category` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`CatID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
MFC:
CMultiRowSet.h
class CMultiRowSet : public CRecordset
{
public:
CMultiRowSet(CDatabase* pDB);
virtual void DoBulkFieldExchange(CFieldExchange* pFX);
// Field/Param Data
// field data members
long* m_rgID;
LPSTR m_rgName;
// pointers for the lengths
// of the field data
long* m_rgIDLengths;
long* m_rgNameLengths;
};
CMultiRowSet.cpp
void CMultiRowSet::DoBulkFieldExchange(CFieldExchange* pFX)
{
// call the Bulk RFX functions
// for field data members
pFX->SetFieldType(CFieldExchange::outputColumn);
RFX_Long_Bulk(pFX, _T("[CatID]"),
&m_rgID, &m_rgIDLengths);
RFX_Text_Bulk(pFX, _T("[Category]"),
&m_rgName, &m_rgNameLengths, 30);
}
Usage:
CDatabase database;
CString sCatID, sCategory;
TRY
{
CString connStr = (_T("Driver={MySQL ODBC 8.0 Unicode Driver};Server=localhost;Database=XXXX;User=XXX; Password=XXXX; Option = 3;"));
// Open the database
database.OpenEx(connStr,CDatabase::noOdbcDialog);
// Allocate the recordset
CMultiRowSet recset(&database);
// Execute the query
// make sure you use CRecordset::useExtendedFetch.
recset.Open(CRecordset::forwardOnly, _T("SELECT CatID, Category FROM Categories"), CRecordset::readOnly|CRecordset::useMultiRowFetch|CRecordset::useExtendedFetch);
// Loop through each record
while (!recset.IsEOF())
{
// The default `GetRowsetSize` is 25. I have 4 rows in my database.
// GetRowsFetched returns 4 in my case.
for (int rowCount = 1; rowCount <= (int)recset.GetRowsFetched(); rowCount++)
{
recset.SetRowsetCursorPosition(rowCount);
// Copy each column into a variable
recset.GetFieldValue(_T("CatID"), sCatID);
recset.GetFieldValue(_T("Category"), sCategory);
}
// goto next record
recset.MoveNext();
}
recset.Close();
// Close the database
database.Close();
}
CATCH(CDBException, e)
{
// If a database exception occured, show error msg
AfxMessageBox(_T("Database error: ") + e->m_strError);
}
END_CATCH;
External Backups/Snapshots for Google Cloud Spanner recommends to use queries with timestamp bounds to create snapshots for export. On the bottom of the Timestamp Bounds documentation it states:
Cloud Spanner continuously garbage collects deleted and overwritten data in the background to reclaim storage space. This process is known as version GC. By default, version GC reclaims versions after they are one hour old. Because of this, Cloud Spanner cannot perform reads at a read timestamp more than one hour in the past.
So any export would need to complete within an hour. A single reader (i.e. select * from table; using timestamp X) would not be able to export the entire table within an hour.
How can multiple parallel readers be implemented in spanner?
Note: It is mentioned in one of the comments that support for Apache Beam is coming, but it looks like that uses a single reader:
/** A simplest read function implementation. Parallelism support is coming. */
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/NaiveSpannerReadFn.java#L26
Is there a way to do the parallel reader that beam requires today using exising APIs? Or will Beam need to use something that isn't released yet on google spanner?
It is possible to read data in parallel from Cloud Spanner with the BatchClient class. Follow read_data_in_parallel for more information.
If you are looking to export data from Cloud Spanner, I'd recommend you to use Cloud Dataflow (see the integration details here) as it provides higher level abstractions and takes care data processing details, like scaling and failure handling.
Edit 2018-03-30 - The example project has been updated to use the BatchClient offered by Google Cloud Spanner
After the release of the BatchClient for reading/downloading large amounts of data, the example project below has been updated to use the new batch client instead of the standard database client. The basic idea behind the project is still the same: Copy data to/from Cloud Spanner and any other database using standard jdbc functionality. The following code snippet sets the jdbc connection in batch read mode:
if (source.isWrapperFor(ICloudSpannerConnection.class))
{
ICloudSpannerConnection con = source.unwrap(ICloudSpannerConnection.class);
// Make sure no transaction is running
if (!con.isBatchReadOnly())
{
if (con.getAutoCommit())
{
con.setAutoCommit(false);
}
else
{
con.commit();
}
con.setBatchReadOnly(true);
}
}
When the connection is in 'batch read only mode', the connection will use the BatchClient of Google Cloud Spanner instead of the standard database client. When one of the Statement#execute(String) or PreparedStatement#execute() methods are called (as these allow multiple result sets to be returned) the jdbc driver will create a partitioned query instead of a normal query. The results of this partitioned query will be a number of result sets (one per partition) that can be fetched by the Statement#getResultSet() and Statement#getMoreResults(int) methods.
Statement statement = source.createStatement();
boolean hasResults = statement.execute(select);
int workerNumber = 0;
while (hasResults)
{
ResultSet rs = statement.getResultSet();
PartitionWorker worker = new PartitionWorker("PartionWorker-" + workerNumber, config, rs, tableSpec, table, insertCols);
workers.add(worker);
hasResults = statement.getMoreResults(Statement.KEEP_CURRENT_RESULT);
workerNumber++;
}
The result sets that are returned by the Statement#execute(String) are not executed directly, but only after the first call to ResultSet#next(). Passing these result sets to separate worker threads ensures parallel download and copying of the data.
Original answer:
This project was initially created for conversion in the other direction (from a local database to Cloud Spanner), but as it uses JDBC for both source and destination it can also be used the other way around: Converting a Cloud Spanner database to a local PostgreSQL database. Large tables are converted in parallel using a thread pool.
The project uses this open source JDBC driver instead of the JDBC driver supplied by Google. The source Cloud Spanner JDBC connection is set to read-only mode and autocommit=false. This ensures that the connection automatically creates a read-only transaction using the current time as timestamp the first time you execute a query. All subsequent queries within the same (read-only) transaction will use the same timestamp giving you a consistent snapshot of your Google Cloud Spanner database.
It works as follows:
Set the source database to read-only transactional mode.
The convert(String catalog, String schema) method iterates over all
tables in the source database (Cloud Spanner)
For each table the number of records is determined, and depending on the size of the table, the table is copied using either the main thread of the application or by a worker pool.
The class UploadWorker is responsible for the parallel copying. Each worker is assigned a range of records from the table (for example rows 1 to 2,400). The range is selected by a select statement in this format: 'SELECT * FROM $TABLE ORDER BY $PK_COLUMNS LIMIT $BATCH_SIZE OFFSET $CURRENT_OFFSET'
Commit the read-only transaction on the source database after ALL tables have been converted.
Below is a code snippet of the most important parts.
public void convert(String catalog, String schema) throws SQLException
{
int batchSize = config.getBatchSize();
destination.setAutoCommit(false);
// Set the source connection to transaction mode (no autocommit) and read-only
source.setAutoCommit(false);
source.setReadOnly(true);
try (ResultSet tables = destination.getMetaData().getTables(catalog, schema, null, new String[] { "TABLE" }))
{
while (tables.next())
{
String tableSchema = tables.getString("TABLE_SCHEM");
if (!config.getDestinationDatabaseType().isSystemSchema(tableSchema))
{
String table = tables.getString("TABLE_NAME");
// Check whether the destination table is empty.
int destinationRecordCount = getDestinationRecordCount(table);
if (destinationRecordCount == 0 || config.getDataConvertMode() == ConvertMode.DropAndRecreate)
{
if (destinationRecordCount > 0)
{
deleteAll(table);
}
int sourceRecordCount = getSourceRecordCount(getTableSpec(catalog, tableSchema, table));
if (sourceRecordCount > batchSize)
{
convertTableWithWorkers(catalog, tableSchema, table);
}
else
{
convertTable(catalog, tableSchema, table);
}
}
else
{
if (config.getDataConvertMode() == ConvertMode.ThrowExceptionIfExists)
throw new IllegalStateException("Table " + table + " is not empty");
else if (config.getDataConvertMode() == ConvertMode.SkipExisting)
log.info("Skipping data copy for table " + table);
}
}
}
}
source.commit();
}
private void convertTableWithWorkers(String catalog, String schema, String table) throws SQLException
{
String tableSpec = getTableSpec(catalog, schema, table);
Columns insertCols = getColumns(catalog, schema, table, false);
Columns selectCols = getColumns(catalog, schema, table, true);
if (insertCols.primaryKeyCols.isEmpty())
{
log.warning("Table " + tableSpec + " does not have a primary key. No data will be copied.");
return;
}
log.info("About to copy data from table " + tableSpec);
int batchSize = config.getBatchSize();
int totalRecordCount = getSourceRecordCount(tableSpec);
int numberOfWorkers = calculateNumberOfWorkers(totalRecordCount);
int numberOfRecordsPerWorker = totalRecordCount / numberOfWorkers;
if (totalRecordCount % numberOfWorkers > 0)
numberOfRecordsPerWorker++;
int currentOffset = 0;
ExecutorService service = Executors.newFixedThreadPool(numberOfWorkers);
for (int workerNumber = 0; workerNumber < numberOfWorkers; workerNumber++)
{
int workerRecordCount = Math.min(numberOfRecordsPerWorker, totalRecordCount - currentOffset);
UploadWorker worker = new UploadWorker("UploadWorker-" + workerNumber, selectFormat, tableSpec, table,
insertCols, selectCols, currentOffset, workerRecordCount, batchSize, source,
config.getUrlDestination(), config.isUseJdbcBatching());
service.submit(worker);
currentOffset = currentOffset + numberOfRecordsPerWorker;
}
service.shutdown();
try
{
service.awaitTermination(config.getUploadWorkerMaxWaitInMinutes(), TimeUnit.MINUTES);
}
catch (InterruptedException e)
{
log.severe("Error while waiting for workers to finish: " + e.getMessage());
throw new RuntimeException(e);
}
}
public class UploadWorker implements Runnable
{
private static final Logger log = Logger.getLogger(UploadWorker.class.getName());
private final String name;
private String selectFormat;
private String sourceTable;
private String destinationTable;
private Columns insertCols;
private Columns selectCols;
private int beginOffset;
private int numberOfRecordsToCopy;
private int batchSize;
private Connection source;
private String urlDestination;
private boolean useJdbcBatching;
UploadWorker(String name, String selectFormat, String sourceTable, String destinationTable, Columns insertCols,
Columns selectCols, int beginOffset, int numberOfRecordsToCopy, int batchSize, Connection source,
String urlDestination, boolean useJdbcBatching)
{
this.name = name;
this.selectFormat = selectFormat;
this.sourceTable = sourceTable;
this.destinationTable = destinationTable;
this.insertCols = insertCols;
this.selectCols = selectCols;
this.beginOffset = beginOffset;
this.numberOfRecordsToCopy = numberOfRecordsToCopy;
this.batchSize = batchSize;
this.source = source;
this.urlDestination = urlDestination;
this.useJdbcBatching = useJdbcBatching;
}
#Override
public void run()
{
// Connection source = DriverManager.getConnection(urlSource);
try (Connection destination = DriverManager.getConnection(urlDestination))
{
log.info(name + ": " + sourceTable + ": Starting copying " + numberOfRecordsToCopy + " records");
destination.setAutoCommit(false);
String sql = "INSERT INTO " + destinationTable + " (" + insertCols.getColumnNames() + ") VALUES \n";
sql = sql + "(" + insertCols.getColumnParameters() + ")";
PreparedStatement statement = destination.prepareStatement(sql);
int lastRecord = beginOffset + numberOfRecordsToCopy;
int recordCount = 0;
int currentOffset = beginOffset;
while (true)
{
int limit = Math.min(batchSize, lastRecord - currentOffset);
String select = selectFormat.replace("$COLUMNS", selectCols.getColumnNames());
select = select.replace("$TABLE", sourceTable);
select = select.replace("$PRIMARY_KEY", selectCols.getPrimaryKeyColumns());
select = select.replace("$BATCH_SIZE", String.valueOf(limit));
select = select.replace("$OFFSET", String.valueOf(currentOffset));
try (ResultSet rs = source.createStatement().executeQuery(select))
{
while (rs.next())
{
int index = 1;
for (Integer type : insertCols.columnTypes)
{
Object object = rs.getObject(index);
statement.setObject(index, object, type);
index++;
}
if (useJdbcBatching)
statement.addBatch();
else
statement.executeUpdate();
recordCount++;
}
if (useJdbcBatching)
statement.executeBatch();
}
destination.commit();
log.info(name + ": " + sourceTable + ": Records copied so far: " + recordCount + " of "
+ numberOfRecordsToCopy);
currentOffset = currentOffset + batchSize;
if (recordCount >= numberOfRecordsToCopy)
break;
}
}
catch (SQLException e)
{
log.severe("Error during data copy: " + e.getMessage());
throw new RuntimeException(e);
}
log.info(name + ": Finished copying");
}
}
Consider this context:
Having a group of threads doing some work (that work is in a infinite loop, embedded project) where the number of threads (and some parameters) depends from a Database result.
What I need is to remove or create threads from that group when there´s a change in the database.
Here is the code:
for (result::const_iterator pin = pinesBBB.begin(); pin != pinesBBB.end(); ++pin)
{
string pinStr = pin["pin"].as<string>();
boost::thread hiloNuevo(bind(WorkPin, pinStr));
Worker.add_thread(&hiloNuevo);
}
Where result is pqxx::result from pqxx library.
This piece of code iterates a table from an SQL query result and creates a thread for every record found.
After that, there´s this code that checks the same table every a couple of minutes:
`
void ThreadWorker(boost::thread_group *worker, string *pinesLocales)
{
int threadsVivosInt = worker->size();
string *pinesDB;
int contador;
for (;;)
{
contador = 0;
sleep(60);
try
{
result pinesBBB = TraerPines();
for (result::const_iterator pin = pinesBBB.begin(); pin != pinesBBB.end(); ++pin)
{
pinesDB[contador] = pin["pin"].as<string>();
contador++;
}
thread hiloMuerto
}
catch (...)
{
sleep(360);
}
}
}
`
What I want to do is access this thread_group worker and remove one of those threads.
I´ve tryed using an Int index like worker[0] and with thread´s ID boost::thread::id
I can remove a thread using a native_handle and then using an plattform specific like pthread_cancel but I can´t get the thread from the thread group.
Any ideas? Thanks!
boost::thread_group::remove_thread() removes the specified thread from a given thread_group. Once you've done this, you're now responsible for managing the thread.
Im basically facing a blocking problem.
I have my server coded based on C++ Boost.ASIO using 8 threads since the server has 8 logical cores.
My problem is a thread may face 0.2~1.5 seconds of blocking on a MySQL query and I honestly don't know how to go around that since MySQL C++ Connector does not support asynchronous queries, and I don't know how to design the server "correctly" to use multiple threads for doing the queries.
This is where I'm asking for opinions of what to do in this case.
Create 100 threads for async' query sql?
Could I have an opinion from experts about this?
Okay, the proper solution to this would be to extend Asio and write a mysql_service implementation to integrate this. I was almost going to find out how this is done right away, but I wanted to get started using an "emulation".
The idea is to have
your business processes using an io_service (as you are already doing)
a database "facade" interface that dispatches async queries into a different queue (io_service) and posts the completion handler back onto the business_process io_service
A subtle tweak needed here you need to keep the io_service on the business process side from shutting down as soon as it's job queue is empty, since it might still be awaiting a response from the database layer.
So, modeling this into a quick demo:
namespace database
{
// data types
struct sql_statement { std::string dml; };
struct sql_response { std::string echo_dml; }; // TODO cover response codes, resultset data etc.
I hope you will forgive my gross simplifications :/
struct service
{
service(unsigned max_concurrent_requests = 10)
: work(io_service::work(service_)),
latency(mt19937(), uniform_int<int>(200, 1500)) // random 0.2 ~ 1.5s
{
for (unsigned i = 0; i < max_concurrent_requests; ++i)
svc_threads.create_thread(boost::bind(&io_service::run, &service_));
}
friend struct connection;
private:
void async_query(io_service& external, sql_statement query, boost::function<void(sql_response response)> completion_handler)
{
service_.post(bind(&service::do_async_query, this, ref(external), std::move(query), completion_handler));
}
void do_async_query(io_service& external, sql_statement q, boost::function<void(sql_response response)> completion_handler)
{
this_thread::sleep_for(chrono::milliseconds(latency())); // simulate the latency of a db-roundtrip
external.post(bind(completion_handler, sql_response { q.dml }));
}
io_service service_;
thread_group svc_threads; // note the order of declaration
optional<io_service::work> work;
// for random delay
random::variate_generator<mt19937, uniform_int<int> > latency;
};
The service is what coordinates a maximum number of concurrent requests (on the "database io_service" side) and ping/pongs the completion back onto another io_service (the async_query/do_async_query combo). This stub implementation emulates latencies of 0.2~1.5s in the obvious way :)
Now comes the client "facade"
struct connection
{
connection(int connection_id, io_service& external, service& svc)
: connection_id(connection_id),
external_(external),
db_service_(svc)
{ }
void async_query(sql_statement query, boost::function<void(sql_response response)> completion_handler)
{
db_service_.async_query(external_, std::move(query), completion_handler);
}
private:
int connection_id;
io_service& external_;
service& db_service_;
};
connection is really only a convenience so we don't have to explicitly deal with various queues on the calling site.
Now, let's implement a demo business process in good old Asio style:
namespace domain
{
struct business_process : id_generator
{
business_process(io_service& app_service, database::service& db_service_)
: id(generate_id()), phase(0),
in_progress(io_service::work(app_service)),
db(id, app_service, db_service_)
{
app_service.post([=] { start_select(); });
}
private:
int id, phase;
optional<io_service::work> in_progress;
database::connection db;
void start_select() {
db.async_query({ "select * from tasks where completed = false" }, [=] (database::sql_response r) { handle_db_response(r); });
}
void handle_db_response(database::sql_response r) {
if (phase++ < 4)
{
if ((id + phase) % 3 == 0) // vary the behaviour slightly
{
db.async_query({ "insert into tasks (text, completed) values ('hello', false)" }, [=] (database::sql_response r) { handle_db_response(r); });
} else
{
db.async_query({ "update * tasks set text = 'update' where id = 123" }, [=] (database::sql_response r) { handle_db_response(r); });
}
} else
{
in_progress.reset();
lock_guard<mutex> lk(console_mx);
std::cout << "business_process " << id << " has completed its work\n";
}
}
};
}
This business process starts by posting itself on the app service. It then does a number of db queries in succession, and eventually exits (by doing in_progress.reset() the app service is made aware of this).
A demonstration main, starting 10 business processes on a single thread:
int main()
{
io_service app;
database::service db;
ptr_vector<domain::business_process> bps;
for (int i = 0; i < 10; ++i)
{
bps.push_back(new domain::business_process(app, db));
}
app.run();
}
In my sample, business_processes don't do any CPU intensive work, so there's no use in scheduling them across CPU's, but if you wanted you could easily achieve this, by replacing the app.run() line with:
thread_group g;
for (unsigned i = 0; i < thread::hardware_concurrency(); ++i)
g.create_thread(boost::bind(&io_service::run, &app));
g.join_all();
See the demo running Live On Coliru
I'm not a MySQL guru, but the following is generic multithreading advice.
Having NumberOfThreads == NumberOfCores is appropriate when none of the threads ever block and you are just splitting the load over all CPUs.
A common pattern is to have multiple threads per CPU, so one is executing while another is waiting on something.
In your case, I'd be inclined to set NumberOfThreads = n * NumberOfCores where 'n' is read from a config file, a registry entry or some other user-settable value. You can test the system with different values of 'n' to fund the optimum. I'd suggest somewhere around 3 for a first guess.
My question is: How can I order a DBGrid by a calculated field. I am using the C++Builder Starter Editon and do not have a ClientDataSet available in this version to create an Index on the field and order by the index of a column.So this is not an option. (Read this in many threads) I am using an TIBDataSet (ibds below) and I am filtering the data. Works fine....for the DB-columns, not for the calculated ones... Any ideas of how I might get around this problem?
void __fastcall TForm1::DBGrid3TitleClick(TColumn *Column)
{
static cIdx = 0;
static String oby = "ASC";
TBookmark CurrentPosition;
TIBDataSet *ibds = IBDS_accountsDist;
CurrentPosition = ibds->GetBookmark();
if (cIdx != Column->Index) {
oby = "ASC"; // ANOTHER column choosen
} else if (oby == "ASC") {
oby = "DESC";
} else oby = "ASC";
cIdx = Column->Index;
ibds->Filtered = false;
switch (Column->Index){
case 0: ibds->Filter = "ORDER BY SumAj "+oby; break; // SumAj is a calculated field => Does not work
case 1: ibds->Filter = "ORDER BY CSAL_ACCOUNTNAME "+ oby; break; // DB-field WORKS FINE
}
ibds->Filtered = true;
ibds->GotoBookmark(CurrentPosition);
}
You cannot do it. TIBDataSet is a representation of the underlying database. Basically it fetches the records in the order defined in the SQL.
The easiest way is to use TDBClientDataset but it is not included in Starter version of c++ Builder. You can explore other ways, for example pre-loading all records in a std::list and then use the order function to order the records. Finally you can show them using a simple TGrid o TStringGrid.
In any case, I recommend to upgrade C++Builder since TClientDataSet is one of the main pieces in most of data projects, specially when you need to create medium-large projects.
Mixing database specific components like TIBDataSet with the user interface penalizes the scalability and maintenance of the project.