There is any way to run dependsOn tasks in parallel?
I have many tasks on dependsOn:
task compileTTS32(dependsOn: [compileTTS32Debug, compileTTS32Release])
task compileTTS64(dependsOn: [compileTTS64Debug, compileTTS64Release])
task compileTTS(dependsOn: [compileTTS32, compileTTS64])
Is there any way to run all compile tasks in parallel?
There is no way to do that at present, however if you were able to make TTS32 and TTS64 seperate sub-projects you should be able to get some level of parralellism with http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:parallel_execution
Related
I have a piece of code that creates several threads on a Glue job like this:
threads = []
for data_chunk in data_chunks:
json_data = get_bulk_upload_json(data_chunk)
threads.append(Thread(target=my_func, args=(arg1, arg2)))
for thread in threads:
thread.start()
for thread in threads:
thread.join()
Where data_chunks is a list of dictionaries. Due to the nature of the data, this consumes a lot of memory. The Glue job keeps failing on a memory error, however, after further debugging, it is crashing once it reached the memory limit of just one of the workers. Meaning, it is not using the memory of the other workers at all. Another proof of this, is that not matter how many more workers I add, the same error happens in the same part of the process.
How can I use threads and distribute them between the workers?
It seems that you are misusing AWS Glue.
You shouldn't use Threads, since Glue does the parallelization for you. It is a managed version of Spark. Instead you should use Spark / Glue functions, which will then be executed on the workers.
I am using jmeter tool for load testing. I want to execute all threads are in simultaneously (at the same time).
So I configured
No of Threads:20
Ramp-up period:Empty
Loop Count:1.
Now I run the jmeter tool.
After getting the result, I saw the result in view results in Table.
From this start time is displayed.i.e Threads are executed one by one not simultaneously. I included the image also.
Could you tell me how to run the concurrent threads simultaneously?
Starting Time for threads.
Add Synchronizing Timer in your test plan
Just add a Synchronizing Timer to your Test Plan.
I'm using Celery 3.1. I need to only execute the next task when the last one is finish. How can I assure that there are not two tasks working at the same time? I've read the documentation but it is not clear for me.
I've the following scheme:
Task Main
- Subtask 1
- Subtask 2
I need that when I call "Task Main" the process will run till the end(Subtask 2) without any new "Task Main" starting.
How can I assure this?
One strategy is through the use of locks. The Celery Task Cookbook has an example at http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html.
If I understand you want to execute only MainTask one by one, and you want to call subtasks in your MainTask. Without creating separate queues and at least 2 separate workers this is impossible. Because if you will store in same queue all tasks looks for celery as same tasks.
So solution for is:
map MainTask to main_queue
Start separate worker for this queue like:
celeryd --concurrency=1 --queue=main_queue
map subtasks to sub_queue
Start separate worker for this queue
celeryd --queue=sub_queue
Should work!
But I think this is complecated architecture, may be you can make it much easier if you will redesign your process.
Also you can find this useful (it works for you but it could run parallel MainTask):
You should try to use chains, here is an example on Celery's docs: http://docs.celeryproject.org/en/latest/userguide/tasks.html#avoid-launching-synchronous-subtasks.
I'm trying to run multiple watch tasks with grunt, but it seems to not be able to. I'm using grunt concurrent, but it seems to only run a portion of the tasks I specify, stopping just short.
Here is the snippet of my gruntfile:
concurrent:
compile:
tasks: ["watch:app", "watch:html", "watch:images", "watch:lib", "watch:server"]
options:
logConcurrentOutput: true
When I run this, the output shows:
Running "concurrent:compile" (concurrent) task
Running "watch:images" (watch) task
Waiting...Running "watch:app" (watch) task
Waiting...Running "watch:html" (watch) task
Waiting...Running "watch:lib" (watch) task
Waiting...
For some reason, it's skipping my server task completely. I initially thought the server tasks was not setup correctly, but I can remove a watch task, and it will correctly run the server task along with the other 3, it just cannot run all 5 at once.
Am I implementing this inappropriately?
So from the documentation, simply declare the amount of tasks you want. so in your case:
options:
logConcurrentOutput: true
limit: 5
By default it just uses how many cores your CPU has..
limit
Type: Number Default: Number of CPU cores
(require('os').cpus().length) with a minimum of 2
Limit of how many tasks that are run concurrently.
https://github.com/sindresorhus/grunt-concurrent
I wanted to replace the use of normal threads with the task_group class from ppl, but I ran in to the following problem:
I have a class A with a task_group member,
create 2 different instances of class A,
start a task in the task_group of the first A instance (using run),
after a few seconds start a task in the task_group of the second A instance.
I'm expecting the two tasks to run in parallel but the second task wait for the first task to finish then starts.
This is happening only in my application where the tasks are started from a static function. I did the same scenario in a test application and the tasks are running correctly in parallel.
After spending several hours trying to figure this out I switched back to normal threads.
Does anyone knows why is the concurrency run-time having this behavior, or how I can avoid this?
EDIT
The problem was that it was running on a single core CPU and concurrency run-time looks at throughput. I wonder if microsoft parallel patterns library has the concept of an active object, or something on the lines so that you can specify that the task you are about to lunch is to be executed in parallel with the thread you start it from...
The response can be found here: http://social.msdn.microsoft.com/Forums/en/parallelcppnative/thread/85a84373-4c3d-4862-bff3-9a21ffe82493
For one core machines, this is expected "default" behavior. This can be changed.
By default, number of tasks that can run in parallel = number of hardware threads (num of cores). This improves the raw throughut and efficiency of completing tasks.
However, there are a number of situations where a developer would want many tasks running in parallel, regardless of the number of cores. In this case you have two options:
Oversubsribe locally.
In your example above, you would use
void lengthyTask()
{
Context::Oversubscribe(true)
...do a lengthy task (//OR a blocking task)
Context::Oversubscribe(false)
}
Oversubcribe the scheduler when you start the application.
SchedulerPolicy policy(1, MaxConcurrency, GetProcessorCount() * 2);
SetDefaultSchedulerPolicy(policy);