I'm trying to retrieve data from my Kafka 0.8.1 cluster. I have brought into existence an instance of ZookeeperConsumerConnector and then attempt to call createMessageStreams on it. However, no matter what I do, it seems createMessageStreams just hangs and never returns, even if it is the only thing I have done with Kafka.
Reading mailing lists it seems this can sometimes happen for a few reasons, but as far as I can tell I haven't done any of those things.
Further, I'll point out that I'm actually doing this in Clojure using clj-kafka, but I suspect clj-kafka is not the issue because I have the problem even if I run this code:
(.createMessageStreams
(clj-kafka.consumer.zk/consumer {"zookeeper.connect" "127.0.0.1:2181"
"group.id" "my.consumer"
"auto.offset.reset" "smallest"
"auto.commit.enable" "false"})
{"mytopic" (int 1)})
And clj-kafka.consumer.zk/consumer just uses Consumer.createJavaConsumerConnector to create a ZookeeperConsumerConnector without doing anything too fancy.
Also, there are definitely messages in "mytopic" because from the command line I can run the following and get back everything I've already sent to the topic:
% kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic mytopic --from-beginning
So it's also not that the topic is empty.
Feeling stumped at this point. Ideas?
ETA: By "hang" I guess what I really mean is that it seems to spin up a thread and then stay stuck in it never doing anything. If I run this code from the REPL I can get out of it by hitting control-c and then I get this error:
IllegalMonitorStateException java.util.concurrent.locks.ReentrantLock$Sync.tryRelease (ReentrantLock.java:155)
I was experiencing the same issue with the same exception when interrupting the REPL. The reason it hangs is due to the lazy-iterate function in the consumer.zk namespace. The queue from which messages are read is a LinkedBlockingQueue and the call to .hasNext in the lazy-iterate function calls .take on this queue. This creates a read lock on the queue and will block and wait until something is available to take off the queue. This means that the lazy-iterate function will never actually return. lazy-iterate is called by the 'messages' function and if you don't do something like
(take 2 (messages "mytopic" some-consumer))
then the messages function will never return and hang indefinitely. It's my opinion that this is a bug (or design flaw) in clj-kafka. To illustrate that this is indeed what's happening, try setting "consumer.timeout.ms" "0" in your consumer config. It will throw a TimeoutExpection and return control to the REPL.
This further creates a problem with the 'with-resource' macro. The macro takes a binding to a consumer, a shutdown function, and a body; it calls the body and then the shutdown fn. If inside the body, you make a call to 'messages', the body will never return and thus the shutdown function will never be called. The messages function WOULD return if shutdown was called, because shutdown puts a message on the queue that signals the consumer to clean up its resources and threads in preparation for GC. This macro puts the application into a state where the only way to get out of the main loop is to kill the application (or a thread calling it) itself. The library certainly has a ways to go before it's ready for a production environment.
Related
I have an app written with Qt5. There is a parent process which spins off an array of child threads - works fine. When I was prototyping it I installed my own message handler (with qInstallMessageHandler) to customize logging, but I put the call to install the message handler in each thread that was spun off (effectively the threads are little clones of each other, so every thread created had the call to install the message handler). Oddly enough that worked. Even though qInstallMessageHandler should only be called once for an application (only one can have the ring of power), calling it multiple times evidently worked because they were all for the same handler. All the threads sent their qDebug (qWarning, etc) messages to my handler and so did the parent.
Now that I'm finalizing the prototype, I wanted to clean things up so I moved the call for qInstallMessageHandler into the parent (which seems cleaner), but now only the parent uses the message handler but the child threads seem to be oblivious that it is installed. I can't see why this doesn't work. The QMessageHandler is application scope. The actual message handler code is outside of any class (as it's always been).
Can anyone offer any insight as to why the threads can't see my handler?
So here's the deal. My parent process is a daemon (QtService) which it turns out was trying to install a message handler of its own (because I defined debugging for it). When I called the message handler from within my thread the timing was such that I stole the handler from qservice, but when I moved it into the daemon proper, qservice stole the handler from me. In any case, I was just stepping on my own toes as usual. Whoever calls qInstallMessageHandler last wins.
I've read about graceful shutdowns here using the WEBJOBS_SHUTDOWN_FILE and here using Cancellation Tokens, so I understand the premise of graceful shutdowns, however I'm not sure how they will affect WebJobs that are in the middle of processing a queue message.
So here's the scenario:
I have a WebJob with functions listening to queues.
Message is added to Queue and job begins processing.
While processing, someone pushes to develop, triggering a redeploy.
Assuming I have my WebJobs hooked up to deploy on git pushes, this deploy will also trigger the WebJobs to be updated, which (as far as I understand) will kick off some sort of shutdown workflow in the jobs. So I have a few questions stemming from that.
Will jobs in the middle of processing a queue message finish processing the message before the job quits? Or is any shutdown notification essentially treated as "this bitch is about to shutdown. If you don't have anything to handle it, you're SOL."
If we are SOL, is our best option for handling shutdowns essentially to wrap anything you're doing in the equivalent of DB transactions and implement your shutdown handler in such a way that all changes are rolled back on shutdown?
If a queue message is in the middle of being processed and the WebJob shuts down, will that message be requeued? If not, does that mean that my shutdown handler needs to handle requeuing that message?
Is it possible for functions listening to queues to grab any more queue messages after the Job has been notified that it needs to shutdown?
Any guidance here is greatly appreciated! Also, if anyone has any other useful links on how to handle job shutdowns besides the ones I mentioned, it would be great if you could share those.
After no small amount of testing, I think I've found the answers to my questions and I hope someone else can gain some insight from my experience.
NOTE: All of these scenarios were tested using .NET Console Apps and Azure queues, so I'm not sure how blobs or table storage, or different types of Job file types, would handle these different scenarios.
After a Job has been marked to exit, the triggered functions that are running will have the configured amount of time (grace period) (5 seconds by default, but I think that is configurable by using a settings.job file) to finish before they are exited. If they do not finish in the grace period, the function quits. Main() (or whichever file you declared host.RunAndBlock() in), however, will finish running any code after host.RunAndBlock() for up to the amount of time remaining in the grace period (I'm not sure how that would work if you used an infinite loop instead of RunAndBlock). As far as handling the quit in your functions, you can essentially "listen" to the CancellationToken that you can pass in to your triggered functions for IsCancellationRequired and then handle it accordingly. Also, you are not SOL if you don't handle the quits yourself. Huzzah! See point #3.
While you are not SOL if you don't handle the quit (see point #3), I do think it is a good idea to wrap all of your jobs in transactions that you won't commit until you're absolutely sure the job has ran its course. This way if your function exits mid-process, you'll be less likely to have to worry about corrupted data. I can think of a couple scenarios where you might want to commit transactions as they pass (batch jobs, for instance), however you would need to structure your data or logic so that previously processed entities aren't reprocessed after the job restarts.
You are not in trouble if you don't handle job quits yourself. My understanding of what's going on under the covers is virtually non-existent, however I am quite sure of the results. If a function is in the middle of processing a queue message and is forced to quit before it can finish, HAVE NO FEAR! When the job grabs the message to process, it will essentially hide it on the queue for a certain amount of time. If your function quits while processing the message, that message will "become visible" again after x amount of time, and it will be re-grabbed and ran against the potentially updated code that was just deployed.
So I have about 90% confidence in my findings for #4. And I say that because to attempt to test it involved quick-switching between windows while not actually being totally sure what was going on with certain pieces. But here's what I found: on the off chance that a queue has a new message added to it in the grace period b4 a job quits, I THINK one of two things can happen: If the function doesn't poll that queue before the job quits, then the message will stay on the queue and it will be grabbed when the job restarts. However if the function DOES grab the message, it will be treated the same as any other message that was interrupted: it will "become visible" on the queue again and be reran upon the restart of the job.
That pretty much sums it up. I hope other people will find this useful. Let me know if you want any of this expounded on and I'll be happy to try. Or if I'm full of it and you have lots of corrections, those are probably more welcome!
I have a logger class with a tbb::concurrent_queue as a member field. Threads using a logger object call a method to push a message onto this internal queue. The logger has an internal thread which is consuming these messages until it receives a sentinel message and then this internal thread exits. The thing is, if clients of this logger object try to log more messages into this logger (after the call to shutdown has sent the sentinel) the messages are passed onto the queue but never picked up on the other end and silently lost. Ideally I would like to notify when this is the case but using a flag set by the internal consumer thread upon exit to check each time im about to push a new message onto the queue would add a branch cost to what is a critical path for me.
One idea i heard was that maybe the consumer thread..right before exiting..somehow atomically swap out the pointer of the queue so that the next call to it would call something else which could handle the message differently now..
i.e. the call: m_buffer->push(message) where m_buffer is a pointer to a tbb::concurrent_queue should after the consumer thread is done still look like m_buffer->push(message) except it goes somewhere else..a custom handler of mine or so...
How can I do this though? I cant swap m_buffer to point to any other custom class unless I inherit from tbb::concurrent_queue...is there another way to get around this?
Thanks
Sounds more like a design problem to me. What do clients of a logger want? To log a message. They don't care whether it goes onto a queue, or is written out to the screen, or is written onto a piece of paper, inserted into a glass bottle and thrown into the ocean.
So your logger need only expose a log() method, nothing else. Queue swapping would happen internally, by keeping two queues in an array and atomically switching a pointer or index around.
Also, if the speed of your logging code is critical, you might be doing too much of it...
Why would the logging thread have to set the 'no more logging' flag? The 'shutdown' method of the logger should set it to prevent any more messages getting queued up, just before it issues the sentinel that will eventually signal the thread to shut down, so that the sentinel is always the last item queued. What you do with logging requests when the flag is set is up to your logger.
I'm using UI threads and I built one thread with message map and it works fine, the problem is when I'm tring to create another thread from the first one.
When I'm getting to this line:
this->PostThreadMessage(WM_MYTHREADMESSAGE,0,0);
I'm getting the next message:
"No symbols are loaded for any call stack frame. The source code cannot be displayed"
I dont know if its could be the reason for the problem but I have built two message maps, one for each thread, I don't know if its ok to do so.
The question is difficult to understand. I'm assuming that you're stepping through your program in the debugger, and you get to that PostThreadMessage line.
If you choose Step Into, the debugger will try to step into the PostThreadMessage call (or the framework wrapper, depending on the type of this). Since PostThreadMessage is a system call, it's likely you don't have symbols for that code. The debugger will just show you disassembly. You can try to use the Microsoft symbol server, but I don't see much point in trying to trace into PostThreadMessage. If the parameters are right, it's going to post the message to the specified thread's queue. Not much to see there.
Posting message to other threads is tricky business. Most Windows programs, even multithreaded ones, typically keep all the UI work to a single thread. It can be done, but there are a lot of caveats and it's usually not worth the pain.
So there are couple of things:
if you want to notify the UI thread from the worker thread, then you should not use PostThreadMessage, here is why.
When this->PostThreadMessage(...) called in a member function of thread A, the message will be sent to thread A, because this points to CWinThread of A. You have to get a pointer to the other thread to post a message to it.
Finally if you want to notify your UI thread, use PostMessage to send a message to the window created by that thread. Add a corresponding handler to the window message map.
Hope this helps
I'm developing a DLL in C++ which needs to write some data via a (previously established) TCP/IP connection using the write() call. To be precise, the DLL should send a little 'Process 12345 is terminating at 2007-09-27 15:30:42, value of i is 131' message over the wire when the process goes down.
Unfortunately, all the ways I know for detecting that the process is ending are apparently too late for any network calls to succeed. In particular, I tried the following approaches and the write() call returned -1 in every case:
Calling write() from the destructor of a global object.
Calling write() from a callback function registered using atexit().
Calling write() from DllMain (in case the reason argument is DLL_PROCESS_DETACH). I know that this is not a safe thing to do, but I'm getting a bit desperate. :-)
I'm aware that a DLL can't detect any process shutdown (it might have been unloaded long before the process terminates) but since the shutdown data which the DLL needs to send depends on other code in the DLL, that's acceptable. I'm basically looking for the latest moment at which I can safely perform network IO.
Does anybody know how to do this?
Consider monitoring the process from a separate watchdog process.
Determining If a Process Has Exited: http://msdn.microsoft.com/en-us/library/y111seb2(v=VS.71).aspx
Tutorial: Managing a Windows Process: http://msdn.microsoft.com/en-us/library/s9tkk4a3(v=VS.71).aspx
Consider to use Windows Job Objects.
You main program (monitoring program, which will use for example send()) can start child process suspended, place it into a Job and then resume. Then it will run in the job object. You can register notification via SetInformationJobObject with JobObjectAssociateCompletionPortInformation. Then you will be notified if in the job will be created some child process and if some process inside of job will be ended. So you will be able to send all what you need from the monitoring process. If you debug a program in Visual Studio it uses also job objects to have control under your process and all child processes which you start.
I successfully use the technique in C++ and in C#. So if you will have some problem with implementation I could post you a code example.
I suggest taking option 3. Just do your DLL loading/unloading properly and you're fine. Calling write() should work, I can't explain why it's not in your case. Is it possible that the call fails for a different reason that is unrelated?
Does it work if you call your DLL function manually from the host app?
Why? Just close the socket. If that's the only close in the program, which by your description it must be, that tells the other end that this end is exiting, and you can send the process ID information at the beginning instead of the end. You shouldn't do anything time-consuming or potentially blocking in an exit hook or static destructor.
Where is Winsock being shut down using WSACleanup? You need to make sure that your I/O completes before this happens.
You should be able to work out if this is happening by placing a breakpoint on the Win32 call in Winsock2.dll. Unload of DLLs is displayed in the output in the debug window.