Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I am trying to send a video stream through a Kafka topic and doing some processing between each messages consumption. This processing takes approximately 20 milliseconds, so after 100 frames, I have a consequent latency. I can detect this latency, however, I need to be able to force my consumer to read from the latest sent frame from the moment the latency is detected. I am aware that this would lead to many frames skipped.
I am using cppkafka which is based on librdkafka.
I can't find a lead that will put me in the right path since most of the answers are either different from my problem or use concepts existing only in other languages APIs.
Any thought on the matter ?
A simple, if kind of brute, way to do it is to check the number of frames in the queue you have. If above a certain upper threshold then do a tight loop where you "consume" and discard (i.e. does no processing at all) of the frames until you reach your lower threshold (in your case one).
Once that tight loop is done you resume normal processing of the queue.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I have to generate & publish data (as protocol buffer messages) with a frequency of 1Hz and I have been thinking of alternatives to just sticking it into a While loop with a clock() function. Maybe I am over thinking the issue but any advice on the matter would be much appreciated thanks
There are many ways to call a function on a regular interval. Which one is best depends on the situation and needs of the program. There's no single "best way".
A few possibilities (there are many more):
Sit in a loop and sleep for some duration every time round the loop. Simple, easy to understand. But rarely the best solution, since nothing else can happen while you sleep.
Sit in a loop waiting for some event to occur. Like a timer event. Process each event as it arrives - if not enough time has passed, ignore the event (maybe).
Set up a timer with a callback function that will call the function at regular intervals.
If your intervals are very tiny, then maybe spin / busy-wait in a loop and check elapsed time each time around the loop and do something when enough time has passed (rarely a good idea since it will burn a lot of CPU time doing nothing, but sometimes it's the right option).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
In a distributed system there is one management instance and multiple processing instances. The management instance sends commands to the different processing instances. So the processing nodes need to have some command parsing logic and more logic to act on these commands.
Throughput is important for the processing instances so the actual data flow should be impacted as little as possible.
Is there a design pattern or programming technique I can use to clearly separate data and control flow in the processing instances while keeping performance of the data flow as high as possible?
Edit:
The general implementation as of now is like follows: There are N processing threads, pooled, and a single control thread. At least virtually, they all have their own private data structures. What the control thread can do is change the actual thread function. I see design patterns as general, high level designs which I don't have to follow closely, but still I am interested if there is such a high level design that minimizes the disturbance of the processing threads.
I'm targeting C++17, if that should be of any concern.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have multiple listeners threads reading a stream of messages (Kafka). Each message has an identifier. The consumers/stream guarantees at-least once consumption. At most of the time, the stream would provide the message exactly once. The count of messages to expect is known beforehand. When all messages are received, I want to shutdown all listener threads. The number of messages can be at most 50 million. What data structure is most suitable for this?
I was thinking of using std::set, std::map and using a mutex at each insertion of the thread. Can a single thread be actually faster in such a use-case? Is there something more optimal?
std::unordered_map would be better. But you should consider using something like HyperLogLog
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I am a hobby programmer interested in dynamic systems like game engines etc. and I had the Idea of a task scheduling system using something like a priority queue to sort different tasks dynamically and maybe include a parallel feature to use multiple cores efficiently. My explicit idea was to use some kind of Task class that itself stores a function pointer and two queue parameters, one being the gravity of the task and one being the time since it was pushed onto the queue, which then would be multiplied to archieve the position in the listing.
Now here comes my question. Would such a system be more efficient in general or at least pay up in any way in comparisation to a hard-coded system (like some 'main loop')?
e.g. is it a better solution / is it faster?
Thanx for the replies.
This is exactly what priority queue's where designed for. Start your design with priority queues and see how well it goes. Then you may want to tweak it if specific issues come up.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
Let's say I have a vector of exact future times of multiple events. I want those events to launch when their respective times have been reached. How do I do this without constantly using if statements to check that the current time is one of those set times? This constant checking, especially if many other things are running in my program (and some of those times may be a bit away from now) will reduce the performance of my program. I'm using c++11, latest version of GCC.
If you're using C++11, it's probably easiest to create a thread, have it sleep until the next alarm time. The most efficient way to store the times for the alarms is probably a priority queue.
If you don't have C++ 11 available, you might consider Boost Threads instead. The standard's threads are based closely on Boost threads, so using them will be fairly similar.
If you don't want to use that, you'll pretty much need to use operating system facilities to do the job. With almost any reasonably modern OS, you can create a thread and have it sleep until the next alarm time, but you'll probably also have something that supports what you want a little more directly (call a function at a specified time). The details of that will be specific to the OS though.