How to execute multiple sql query parallel in java - concurrency

I have 3 method which is returning a List of result and my sql query execute in each method and returning a list of result .I want to execute all 3 method parallel so it will not wait to complete of one and another . I saw one stachoverflow post but its not working.
that link is [How to execute multiple queries in parallel instead of sequentially?
[Execute multiple queries in parallel via Streams
I want to solve using java 8 features.
But the above link how I can call multiple method please tell me.

Execute multiple queries in parallel via Streams works for your task. Here is a sample code which demonstrates it:
public static void main(String[] args) {
// Create Stream of tasks:
Stream<Supplier<List<String>>> tasks = Stream.of(
() -> getServerListFromDB(),
() -> getAppListFromDB(),
() -> getUserFromDB());
List<List<String>> lists = tasks
// Supply all the tasks for execution and collect CompletableFutures
.map(CompletableFuture::supplyAsync).collect(Collectors.toList())
// Join all the CompletableFutures to gather the results
.stream()
.map(CompletableFuture::join).collect(Collectors.toList());
System.out.println(lists);
}
private static List<String> getUserFromDB() {
try {
TimeUnit.SECONDS.sleep((long) (Math.random() * 3));
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName() + " getUser");
return Arrays.asList("User1", "User2", "User3");
}
private static List<String> getAppListFromDB() {
try {
TimeUnit.SECONDS.sleep((long) (Math.random() * 3));
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName() + " getAppList");
return Arrays.asList("App1", "App2", "App3");
}
private static List<String> getServerListFromDB() {
try {
TimeUnit.SECONDS.sleep((long) (Math.random() * 3));
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName() + " getServer");
return Arrays.asList("Server1", "Server2", "Server3");
}
The output is:
ForkJoinPool.commonPool-worker-1 getServer
ForkJoinPool.commonPool-worker-3 getUser
ForkJoinPool.commonPool-worker-2 getAppList
[[Server1, Server2, Server3], [App1, App2, App3], [User1, User2, User3]]
You can see that default ForkJoinPool.commonPool is used and each get* method is executed from separate thread in this pool. You just need to run your SQL queries inside these get* methods

The solution from Alex won't work in an expected way. CompletableFuture executes effects one after another. You will have to run parallel streams for this or use CompletableFuture with parallelStream

Related

BulkProcessor .add() not finishing when number of bulks > concurrentRequests

Here is a sample of the code flow:
Trigger the process with an API specifying bulkSize and totalRecords.
Use those parameters to acquire data from DB
Create a processor with the bulkSize.
Send both the data and processor into a method which:
-iterates over the resultset, assembles a JSON for each result, calls a method if the final JSON is not empty and adds that final JSON to the process using processor.add() method.
This is where the outcome of the code is split
After this, if the concurrentRequest parameter is 0 or 1 or any value < (totalRecords/bulkSize), the processor.add() line is where the code stalls and never continues to the next debug line.
However, when we increase the concurrentRequest parameter to a value > (totalRecords/bulkSize), the code is able to finish the .add() function and move onto the next line.
My reasoning leads me to believe we might be having issues with our BulkProcessListener which is making the .add() no close or finish like it is supposed to. I would really appreciate some more insight about this topic!
Here is the Listener we are using:
private class BulkProcessorListener implements Listener {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
// Some log statements
}
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
// More log statements
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
// Log statements
}
}
Here is the createProcessor():
public synchronized BulkProcessor createProcessor(int bulkActions) {
Builder builder = BulkProcessor.builder((request, bulkListener) -> {
long timeoutMin = 60L;
try {
request.timeout(TimeValue.timeValueMinutes(timeoutMin));
// Log statements
client.bulkAsync(request, RequestOptions.DEFAULT,new ResponseActionListener<BulkResponse>());
}catch(Exception ex) {
ex.printStackTrace();
}finally {
}
}, new BulkProcessorListener());
builder.setBulkActions(bulkActions);
builder.setBulkSize(new ByteSizeValue(buldSize, ByteSizeUnit.MB));
builder.setFlushInterval(TimeValue.timeValueSeconds(5));
builder.setConcurrentRequests(0);
builder.setBackoffPolicy(BackoffPolicy.noBackoff());
return builder.build();
}
Here is the method where we call processor.add():
#SuppressWarnings("deprecation")
private void addData(BulkProcessor processor, String indexName, JSONObject finalDataJSON, Map<String, String> previousUniqueObject) {
// Debug logs
processor.add(new IndexRequest(indexName, INDEX_TYPE,
previousUniqueObject.get(COMBINED_ID)).source(finalDataJSON.toString(), XContentType.JSON));
// Debug logs
}

Is QuestDB (embedded) TableWriter thread-safe?

I am using questdb (embedded) to store a bunch of time series.
I would like to run my storage method inside a parallel stream, but I don't know if TableWriter is thread-safe.
Here is the code:
SqlExecutionContextImpl ctx = new SqlExecutionContextImpl(engine, 1);
try (TableWriter writer = engine.getWriter(ctx.getCairoSecurityContext(), name, "writing")) {
tickerData.stream().parallel().forEach(
r -> {
Instant i = r.getDateTime("DateTime")
.atZone(EST)
.toInstant();
long ts = TimestampFormatUtils.parseTimestamp(i.toString());
TableWriter.Row row = writer.newRow(ts);
row.putDouble(0, r.getDouble("x1"));
row.putDouble(1, r.getDouble("x2"));
row.putDouble(2, r.getDouble("y1"));
row.putDouble(3, r.getDouble("y2"));
row.putDouble(4, r.getDouble("z"));
row.append();
writer.commit();
} catch (NumericException ex) {
log.error("Cannot parse the date {}", r.getDateTime("DateTime"));
} catch (Exception ex) {
log.error("Cannot write to table {}!", name, ex);
}
});
}
This throws all sort of errors, is there a way to make the storage process parallel?
Thanks,
Juan
The short answer is TableWriter is not thread safe. You will be responsible to not use it in parallel threads.
A bit longer answer is that even in stand alone QuestDB parallel writing is restricted. It is only possible from multiple ILP connections at the moment.

Unit testing suspend coroutine

a bit new to Kotlin and testing it... I am trying to test a dao object wrapper with using a suspend method which uses an awaitFirst() for an SQL return object. However, when I wrote the unit test for it, it is just stuck in a loop. And I would think it is due to the awaitFirst() is not in the same scope of the testing
Implementation:
suspend fun queryExecution(querySpec: DatabaseClient.GenericExecuteSpec): OrderDomain {
var result: Map<String, Any>?
try {
result = querySpec.fetch().first().awaitFirst()
} catch (e: Exception) {
if (e is DataAccessResourceFailureException)
throw CommunicationException(
"Cannot connect to " + DatabaseConstants.DB_NAME +
DatabaseConstants.ORDERS_TABLE + " when executing querySelect",
"querySelect",
e
)
throw InternalException("Encountered R2dbcException when executing SQL querySelect", e)
}
if (result == null)
throw ResourceNotFoundException("Resource not found in Aurora DB")
try {
return OrderDomain(result)
} catch (e: Exception) {
throw InternalException("Exception when parsing to OrderDomain entity", e)
} finally {
logger.info("querySelect;stage=end")
}
}
Unit Test:
#Test
fun `get by orderid id, null`() = runBlocking {
// Assign
Mockito.`when`(fetchSpecMock.first()).thenReturn(monoMapMock)
Mockito.`when`(monoMapMock.awaitFirst()).thenReturn(null)
// Act & Assert
val exception = assertThrows<ResourceNotFoundException> {
auroraClientWrapper.queryExecution(
databaseClient.sql("SELECT * FROM orderTable WHERE orderId=:1").bind("1", "123") orderId
)
}
assertEquals("Resource not found in Aurora DB", exception.message)
}
I noticed this issue on https://github.com/Kotlin/kotlinx.coroutines/issues/1204 but none of the work around has worked for me...
Using runBlocking within Unit Test just causes my tests to never complete. Using runBlockingTest explicitly throws an error saying "Job never completed"... Anyone has any idea? Any hack at this point?
Also I fairly understand the point of you should not be using suspend with a block because that kinda defeats the purposes of suspend since it is releasing the thread to continue later versus blocking forces the thread to wait for a result... But then how does this work?
private suspend fun queryExecution(querySpec: DatabaseClient.GenericExecuteSpec): Map {
var result: Map<String, Any>?
try {
result = withContext(Dispatchers.Default) {
querySpec.fetch().first().block()
}
return result
}
Does this mean withContext will utilize a new thread, and re-use the old thread elsewhere? Which then doesnt really optimize anything since I will still have one thread that is being blocked regardless of spawning a new context?
Found the solution.
The monoMapMock is a mock value from Mockito. Seems like the kotlinx-test coroutines can't intercept an async to return a mono. So I forced the method that I can mock, to return a real Mono value instead of a Mocked Mono. To do so, as suggested by Louis. I stop mocking it and return a real value
#Test
fun `get by orderid id, null`() = runBlocking {
// Assign
Mockito.`when`(fetchSpecMock.first()).thenReturn(Mono.empty())
Mockito.`when`(monoMapMock.awaitFirst()).thenReturn(null)
// Act & Assert
val exception = assertThrows<ResourceNotFoundException> {
auroraClientWrapper.queryExecution(
databaseClient.sql("SELECT * FROM orderTable WHERE orderId=:1").bind("1", "123") orderId
)
}
assertEquals("Resource not found in Aurora DB", exception.message)
}

how to apply in DebuggerHiddenAttribute dependent/cascade methods

I'm trying to ignore a exception that happen when run my test Methods.
I`m using a unitTest proyect.
The problem apears when
TestCleanup Method runs. It's a recursive method. This method clean all entities created in DB during the test. This is a recursive method because of dependences.
Anyway this method call to delete generic method in my ORM(Petapoco). It throws an exception if it can't delete the entity. Any problem, it run again with recursive way until delete it.
Now the problem, if i'm debugging VS stop a lot of times in Execute method because of failed deletes. But I can't modify this method to ignore it. I need a way to ignore this stops when i'm debugging tests. A way like DebuggerHiddenAttribute or similar.
Thanks!
I tried to use DebuggerHiddenAttribute, but cannot works in methods called by main method.
[TestCleanup(), DebuggerHidden]
public void CleanData()
{
ErrorDlt = new Dictionary<Guid, object>();
foreach (var entity in TestEntity.CreatedEnt)
{
try
{
CallingTest(entity);
}
catch (Exception e)
{
if (!ErrorDlt.ContainsKey(entity.Key))
ErrorDlt.Add(entity.Key, entity.Value);
}
}
if (ErrorDlt.Count > 0)
{
TestEntity.CreatedEnt = new Dictionary<Guid, object>();
ErrorDlt.ForEach(x => TestEntity.CreatedEnt.Add(x.Key, x.Value));
CleanData();
}
}
public int Execute(string sql, params object[] args)
{
try
{
OpenSharedConnection();
try
{
using (var cmd = CreateCommand(_sharedConnection, sql, args))
{
var retv = cmd.ExecuteNonQuery();
OnExecutedCommand(cmd);
return retv;
}
}
finally
{
CloseSharedConnection();
}
}
catch (Exception x)
{
OnException(x);
throw new DatabaseException(x.Message, LastSQL, LastArgs);
}
}
Error messages are not required.

Apache Beam: Why does it write to Spanner twice on REPORT_FAILURES mode?

I found interesting write operation codes while looking at SpannerIO, and want to understand reasons.
On write(WriteToSpannerFn) and REPORT_FAILURES failure mode, it seems trying to write failed mutations twice.
I think it's for logging each mutation's exceptions. Is it a correct assumption, and is there any workaround?
Below, I removed some lines for simplicity.
public void processElement(ProcessContext c) {
Iterable<MutationGroup> mutations = c.element();
boolean tryIndividual = false;
try {
Iterable<Mutation> batch = Iterables.concat(mutations);
spannerAccessor.getDatabaseClient().writeAtLeastOnce(batch);
} catch (SpannerException e) {
if (failureMode == FailureMode.REPORT_FAILURES) {
tryIndividual = true;
} else {
...
}
}
if (tryIndividual) {
for (MutationGroup mg : mutations) {
try {
spannerAccessor.getDatabaseClient().writeAtLeastOnce(mg);
} catch (SpannerException e) {
LOG.warn("Failed to submit the mutation group", e);
c.output(failedTag, mg);
}
}
}
}
So rather than write each Mutation individually to the database, the SpannerIO.write() connector tries to write a batch of Mutations in a single transaction for efficiency.
If just one of these Mutations in the batch fails, then the whole transaction fails, so in REPORT_FAILURES mode, the mutations are re-tried individually to find which Mutation(s) are the problematic ones...