I just implemented eventhub with version 5, which has been changed a bit from previous version.
current running code is following:
consumer_client = EventHubConsumerClient.from_connection_string(conn_str=CONNECTION_STR,
consumer_group='fconsumer',
eventhub_name=EVENTHUB_NAME)
consumer_client.receive(on_event=on_event,
partition_id = "0",
track_last_enqueued_event_properties=False,
starting_position="#latest")
By adding an argument for the time duration (or keep_alive ag from prev version), I would make it stop receiving messages and close it after a certain amount of time. Is this possible?
consumer_client.receive(...) will be a blocking call and it won't return on its own. You need to create a thread for consuming events and in the main thread you can get to decide when to close the consumer client. Sample code snippet as below...
thread = threading.Thread(
target=consumer_client.receive,
kwargs={
"on_event": on_event,
"on_partition_initialize": on_partition_initialize,
"on_partition_close": on_partition_close,
"on_error": on_error,
"starting_position": "-1", # "-1" is from the beginning of the partition.
}
)
thread.daemon = True
thread.start()
time.sleep(RECEIVE_DURATION)
consumer_client.close()
thread.join()
Related
I'm trying to put a 1 second delay using setTimeout(()=>{},1000) in the Pre-request Script for a Postman POST call.
var moment = require('moment');
var tap1TimeStr = pm.environment.get("now");
var tap1TimeMoment = moment(tap1TimeStr,"YYYY-MM-DDTHH:mm:ss");
var expTap2Time = tap1TimeMoment.add(2, 'minutes').format("YYYY-MM-DDTHH:mm:ss");
console.log("Tap 2 timestamp should be: " + expTap2Time);
var timestamp;
var timecheck = false;
while(!timecheck)
{
setTimeout(() => {},1000);
timecheck = moment.utc().isSame(expTap2Time);
console.log("timecheck: " + timecheck);
timestamp = moment.utc().format("YYYY-MM-DDTHH:mm:ss");
}
console.log("Timestamp is now: " + timestamp);
pm.environment.set("now", timestamp);
But it doesn't seem to work and I can see that the console.log line is being printed far more frequently than 1sec. And the exercise here is to send the "Tap 2" POST exactly 2mins after the first POST (tracked by the 'now' variable). Also, it seems like Postman takes a fair bit of time before it even starts executing this particular script.
Edit: The main requirement here is to send the "Tap 2" POST request exactly 2mins AFTER the "Tap 1" POST request. HOW best to implement that? Espcially if setTimeout() is non-blocking and thus probably can't be used in a loop.
Anyone has any ideas?
setTimeout() takes a callback function which is executed after the specified delay so only what happens in the callback function will happen after that delay.
setTimeout(() => {
console.log("This will be executed after 1 second");
}, 1000);
console.log("This will immediately be executed");
setTimeout() is asynchronous and non-blocking so JavaScript will call set timeout but not wait for 1 second for it to return and instead immediately move on to the next instruction. Only after that 1 second has passed the callback passed to setTimeout() will be scheduled and executed. Have a look at this YouTube video for a good explanation of what's going on.
We have a ColdFusion edit that checks MX records to soft validate email addresses. We get occasional long responses from the DNS servers and need to restrict how long we wait for a response before giving up.
The component called does the actual lookup. The code reproduced below is part of a larver validation script.
What I want to do is limit how long I will wait for the lookup component to run and then handle the error when it exceeds that time. This code appears correct to me, but it just blows right past the timeout (I have a "sleep" in the component to mimic a slow response.)
try {
requesttimeout="10";
MyNewArray[1] = right(MyEmail, RightChars);
mycnt = new common.functions.mxLookup(MyNewArray);
Caller.EmailReturnCode = iif(mycnt gt 0,'"0"','"2"');
}
catch ("coldfusion.runtime.RequestTimedOutException" a) {
Caller.EmailReturnCode = "2";
}
catch (any e) {
Caller.EmailReturnCode = "2";
}
thread action="run" name="doLookup" {
variables.mycnt = new common.functions.mxLookup(MyNewArray);
}
thread action="join" name="doLookup" timeout="20000";
if(doLookup.Status is "RUNNING"){
thread action="terminate" name="doLookup";
throw("MX Lookup of #MyEmail# timed out", "toException", "MX Lookup exceeded allowed time of 20 seconds at #timeFormat(now(),'HH:mm:sss')#");
}
Caller.EmailReturnCode = iif(mycnt gt 0,'"0"','"2"');
Thanks to Alex
This solution works perfectly.
I am writing a custom plugin for gstreamer 1.0 in C.
This plugin perform some processing on frames and should send an event to the application whenever some conditions are met.
It should not block the pipeline not interfere with it, just a signal so the application can trigger an action unrelated to the pipeline on the side.
The processing is working well but ... i don't know what to do next.
There is a lot of already existing message like EOS or seek but how do i create my own?
The message should contain custom data and therefore i must create one myself that i could send.
Either by sending events or signal i could not find any examples/documentations/explainations on how to handle custom events from a plugin.
I don't even have a sample code to start with.
Any insight would be appreciated.
Take a look at the fpsdisplaysink element:
https://github.com/GStreamer/gst-plugins-bad/blob/master/gst/debugutils/fpsdisplaysink.c
This one emits signals which the application can connect to. Most interesting probably the signal creation:
g_signal_new ("fps-measurements", G_TYPE_FROM_CLASS (klass),
G_SIGNAL_RUN_LAST, 0, NULL, NULL, NULL,
G_TYPE_NONE, 3, G_TYPE_DOUBLE, G_TYPE_DOUBLE, G_TYPE_DOUBLE);
and the periodically triggering of said signal:
g_signal_emit (G_OBJECT (self),
fpsdisplaysink_signals[SIGNAL_FPS_MEASUREMENTS], 0, rr, dr,
average_fps);
Detailed information should be found at the GLib signals documentation:
https://developer.gnome.org/gobject/stable/gobject-Signals.html
#
Alternatively you create your own GstMessage and post it on the bus. See the GstMessage documentation:
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstMessage.html
GstMessage *gst_message_new_application (GstObject *src,
GstStructure *structure);
You can then wrap your data inside the GstStructure. And then post the message to the bus with gst_bus_post().
Thank you Florian for your insight which helped me a lot.
I ended up using gst_message_new and gst_post_bus.
For those who might be interested here is the code in python where i implemented a run loop.
def connect(bus, name):
def _connect(f):
bus.connect(name, f)
return f
return _connect
....
bus = self.pipeline.get_bus()
bus.add_signal_watch()
ret = self.pipeline.set_state(Gst.State.PLAYING)
if ret == Gst.StateChangeReturn.FAILURE:
logger.error("ERROR: Unable to set the pipeline to the playing state")
loop = GObject.MainLoop()
print()
#connect(bus, "message::"+Gst.MessageType.get_name(Gst.MessageType.ERROR))
def on_error(bus, message):
err, dbg = message.parse_error()
print("ERROR:", message.src.get_name().encode('utf-8'), ":", err.message.encode('utf-8'))
if dbg:
print("debugging info:", dbg)
loop.quit()
#connect(bus, "message::"+Gst.MessageType.get_name(Gst.MessageType.EOS))
def on_eos(bus, message):
logger.info("End-Of-Stream reached")
loop.quit()
.... other events
try:
loop.run()
except KeyboardInterrupt:
pass
print("START : Pipeline has stopped")
self.pipeline.set_state(Gst.State.NULL)
What will happen, when I schedule a message in the constructor of my Actor and the actor fails (Exception) before the message was send?
When the actor is resumed or restarted, will the message send like nothing has happened?
When the actor is stopped, will the message send to the dead letter box?
When I start the timer in preStart(), will I have two scheduled message when the actor restarts after a failure?
The answers to your questions are as follows:
1) Yes, the actor will receive the message as long as you used the scheduler scheduleOnce variant that takes an ActorRef as an arg. Because an ActorRef is just a lightweight proxy based on an actor address, it can survive failures of the target actor and still route messages to it as long as it successfully restarts back up and re-binds to the address that the ActorRef represents/
2) Yes, if the ActorRef is for a path that is no longer represented in the ActorSystem then the message will be sent to deadletter instead.
3) Yes you will. If you do it in preStart or in the body of the actor (constructor) and the actor fails and restarts, then the scheduled will now have two jobs to do for the same ActorRef and thus two requests will eventually be received.
A little code to show all of this in action. Consider the following actor:
class TestSchedActor extends Actor{
import context._
override def preStart = {
context.system.scheduler.scheduleOnce(1 second, self, "bar")
}
def receive = {
case "foo" =>
val s:String = null
s.length
case "baz" =>
context stop self
case other =>
println(s"Got $other")
}
}
If you tested it in this way:
val system = ActorSystem("schedtest")
val ref = system.actorOf(Props[TestSchedActor])
ref ! "foo"
Then the output would be:
[ERROR] [04/03/2014 07:58:24.988] [schedtest-akka.actor.default-dispatcher-2] [ akka://schedtest/user/$a] null
java.lang.NullPointerException
at code.TestSchedActor$$anonfun$receive$1.applyOrElse(Asking.scala:27)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Got bar
Got bar
This shows both #1 and #3 as the actor still received the message after failure and actually received 2 as it re-scheduled again when it was restarted.
If you tested the actor like this:
val system = ActorSystem("schedtest")
val ref = system.actorOf(Props[TestSchedActor])
ref ! "baz"
Then you would see the following output:
[INFO] [04/03/2014 08:01:14.199] [schedtest-akka.actor.default-dispatcher-2] [akka://schedtest/user/$a] Message [java.lang.String] from
Actor[akka://schedtest/user/$a#687201705] to Actor[akka://schedtest/user/$a#687201705] was
not delivered. [1] dead letters encountered. This logging can be turned off or adjusted
with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Provided you had not disabled deadletter logging.
I assume that your actor sends a message (using a scheduled task) to itself (using something like system.actorSelection to resolve self address).
Then:
1) Yes;
2) Yes;
3) Yes (moreover, if you start the timer in the constructor you'll get the same behavior).
To avoid all such issues you can start the timer in preStart(), save the received Cancellable into a local variable inside the Actor and then cancel it in postStop(). postStop() / preStart() are called from preRestart() / postRestart(), so your scheduled task will be rescheduled on Actor restarts and cancelled on Actor stop.
I'm trying to write a reusable component in Groovy to easily shoot off emails from some of our Java applications. I would like to pass it a List, where Email is just a POJO(POGO?) with some email info. I'd like it to be multithreaded, at least running all the email logic in a second thread, or make one thread per email.
I am really foggy on multithreading in Java so that probably doesn't help! I've attempted a few different ways, but here is what I have right now:
void sendEmails(List<Email> emails) {
def threads = []
def sendEm = emails.each{ email ->
def th = new Thread({
Random rand = new Random()
def wait = (long)(rand.nextDouble() * 1000)
println "in closure"
this.sleep wait
sendEmail(email)
})
println "putting thread in list"
threads << th
}
threads.each { it.run() }
threads.each { it.join() }
}
I was hoping the sleep would randomly slow some threads down so the console output wouldn't be sequential. Instead, I see this:
putting thread in list
putting thread in list
putting thread in list
putting thread in list
putting thread in list
putting thread in list
putting thread in list
putting thread in list
putting thread in list
putting thread in list
in closure
sending email1
in closure
sending email2
in closure
sending email3
in closure
sending email4
in closure
sending email5
in closure
sending email6
in closure
sending email7
in closure
sending email8
in closure
sending email9
in closure
sending email10
sendEmail basically does what you'd expect, including the println statement, and the client that calls this follows,
void doSomething() {
Mailman emailer = MailmanFactory.getExchangeEmailer()
def to = ["one","two"]
def from = "noreply"
def li = []
def email
(1..10).each {
email = new Email(to,null,from,"email"+it,"hello")
li << email
}
emailer.sendEmails li
}
To get you example above running concurrently you have to replace the line
threads.each { it.run() }
with
threads.each { it.start() }
as run() doesn't start a new thread and thus your code was running sequentially.
There's also a Groovy extension available called GPars. It supports several concurrency techniques like Fork/Join or the Actor model. Using GPars, your code could be simplified to this:
def sendEmails(emails) {
GParsPool.withPool {
emails.eachParallel { email ->
def wait = (long) new Random().nextDouble() * 1000
println "in closure"
this.sleep wait
sendEmail(email)
}
}
}
A couple of Java versions back (1.5) they introduced some new concurrency stuff that makes Java threading (even more) simple. Google for java ThreadExecutor, and you'll find some pages such as:
http://www.deitel.com/articles/java_tutorials/20051126/JavaMultithreading_Tutorial_Part4.html
Whether Groovy makes it even simpler, I can't say, but you might want apply the "new" Java techniques to your Java example first before making the comparison.