I have webjob queue trigger which is responding to queue message and it works fine. However sometimes we push messages manually in queue and if there is manual mistake which causes DecoderFallBackException. But the strange behavior is that looks like webjob keeps trying unlimited times and our AI logs are creating a mess. I tried restarting webjob to see if it clears any internal cache but doesn’t help.
only thing which helps is deleting queue
Ideally any exception beyond deque count should move message to poison queue.
I've tried to reproduce your issue on my side, but it works well. First I create a backend demo to insert invalid byte message in queue that could cause DecoderFallBackException
Encoding ae = Encoding.GetEncoding(
"us-ascii",
new EncoderExceptionFallback(),
new DecoderExceptionFallback());
string inputString = "XYZ";
byte[] encodedBytes = new byte[ae.GetByteCount(inputString)];
ae.GetBytes(inputString, 0, inputString.Length,
encodedBytes, 0);
//make the byte invalid
encodedBytes[0] = 0xFF;
encodedBytes[2] = 0xFF;
CloudQueueMessage message = new CloudQueueMessage(encodedBytes);
queue.AddMessage(message);
Web Job code:
public static void ProcessQueueMessage([QueueTrigger("queue")] string message, TextWriter log)
{
log.WriteLine(message);
}
After 5 times the exception occurs, the message is moved to 'queue-poison'. This is the expected behavior. Check here for details:
maxDequeueCount 5 The number of times to try processing a message before moving it to the poison queue.
You may check if you accidently set "maxDequeueCount" to bigger value. If not, please provide your webjob code and how you find DecoderFallBackException for us to investigate.
Related
I'm trying to send stream of audio from service A to service B with the usage of akka stream refs (akka-streams library version: 2.6.3). Everything is working rather good, except for the fact that once in a month an exception (With daily usage of this service being around 50k calls per day or so) is thrown in the akka stream ref, and I can't find the cause of the problem.
The stacktrace for error is following:
Caused by: java.lang.IllegalStateException: [SourceRef-46] Saw RemoteStreamCompleted(37) while in state UpstreamTerminated(Actor[akka://system-name#serviceA:34363/system/Materializers/StreamSupervisor-3/$$S4-SinkRef-3405#-939568637]), should never happen
at akka.stream.impl.streamref.SourceRefStageImpl$$anon$1.$anonfun$receiveRemoteMessage$1(SourceRefImpl.scala:285)
at akka.stream.impl.streamref.SourceRefStageImpl$$anon$1.$anonfun$receiveRemoteMessage$1$adapted(SourceRefImpl.scala:196)
at akka.stream.stage.GraphStageLogic$StageActor.internalReceive(GraphStage.scala:243)
at akka.stream.stage.GraphStageLogic$StageActor.$anonfun$callback$1(GraphStage.scala:202)
at akka.stream.stage.GraphStageLogic$StageActor.$anonfun$callback$1$adapted(GraphStage.scala:202)
at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:466)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:497)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:599)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:768)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:783)
at akka.actor.Actor.aroundReceive(Actor.scala:534)
at akka.actor.Actor.aroundReceive$(Actor.scala:532)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:690)
... 11 common frames omitted
The code responsible for pushing audio through SourceRef in service A:
Materializer materializer = Materializer.createMaterializer(actorSystem);
AudioExtractor extractor = new AudioExtractorImpl("/path/to/audio/file"); // gets all audio bytes from audio file and puts them into chunks (byte arrays of certain length)
List<AudioChunk> audioChunkList = extractor.getChunkedBytesIntoList();
SourceRef<AudioChunk> sourceRef = Source.from(audioChunkList)
.runWith(StreamRefs.sourceRef(), materializer);
// wrap the sourceRef into msg
serviceBActor.tell(wrappedAudioSourceRefInMsg, getSelf());
Whereas code responsible for accepting audio in service B:
private final List<AudioChunk> audioChunksBuffer = new ArrayList<>();
private final Materializer materializer;
public Receive createReceive() {
return receiveBuilder.match(WrappedAudioSourceRefInMsg.class, response -> {
response.getSourceRef()
.getSource()
.runWith(Sink.forEach(chunk -> audioChunksBuffer.add(chunk)), materializer);
}).build();
}
What I've confirmed is that this error always happens after all audio has been sent from service A, and the stream completed. I can't figure out though why is the SourceRef receiving RemoteStreamCompleted while in state UpstreamTerminated. Especially frustrating is the part of should never happen in the message. :|
Any help with this would be much welcome.
Closing, bug in akka reported here: https://github.com/akka/akka/issues/28852
I am trying fetch messages from FIFO sqs queue. Here is the sample code:
import boto3
sqs_client = boto3.resource(
'sqs',
#aws_access_key_id=AWS_ACCESS_KEY,
#aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
region_name='us-east-2'
)
queue_name = 'test_queue.fifo'
response = sqs_client.create_queue(
QueueName=queue_name,
Attributes={
'FifoQueue': 'true',
'ContentBasedDeduplication': 'true'
}
)
for i in range(0,50):
status = response.send_message(MessageBody = 'This is test message #'+str(i), MessageGroupId='586474de88e03')
while True:
messages = response.receive_messages(MaxNumberOfMessages=10)
if len(messages)>0:
for message in messages:
print message.body
else:
print('Queue is now empty')
break
but what I am getting is only the first 10 messages and then its showing "Queue is now empty" although I can see there are 40 available messages in the queue from AWS console.
So here I want to fetch all the messages from the queue in loop. Any lead would be appreciated. Thanks.
When there is a small number of messages in an SQS queue, especially an extremely small number as in your case, you may not get any messages returned and may need to retry the call:
Short poll is the default behavior where a weighted random set of machines is sampled on a receive-message call. Thus, only the messages on the sampled machines are returned. If the number of messages in the queue is small (fewer than 1,000), you most likely get fewer messages than you requested per receive-message call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular receive-message response. If this happens, repeat the request.
Also, generally speaking, once you receive a set of messages, you process them and then delete the messages that you processed - for testing purpose at least you may want to delete each returned message after each 'print message.body', and before you make the next receive request.
Your Question :I want to fetch all the messages from the queue in loop.............. My answer :(read it completely) for fifo queue . Read that message then send that same message back to that queue and delete it from the queue .... It would be safe only if u do so(by proper exceptions hadlling and Message handler) . Try writing python programs with proper loggers and make it fail safe . Actually ur your is not fail safe .
I have an Akka Stream and I want the stream to send messages down stream approximately every second.
I tried two ways to solve this problem, the first way was to make the producer at the start of the stream only send messages once every second when a Continue messages comes into this actor.
// When receive a Continue message in a ActorPublisher
// do work then...
if (totalDemand > 0) {
import scala.concurrent.duration._
context.system.scheduler.scheduleOnce(1 second, self, Continue)
}
This works for a short while then a flood of Continue messages appear in the ActorPublisher actor, I assume (guess but not sure) from downstream via back-pressure requesting messages as the downstream can consume fast but the upstream is not producing at a fast rate. So this method failed.
The other way I tried was via backpressure control, I used a MaxInFlightRequestStrategy on the ActorSubscriber at the end of the stream to limit the number of messages to 1 per second. This works but messages coming in come in at approximately three or so at a time, not just one at a time. It seems the backpressure control doesn't immediately change the rate of messages coming in OR messages were already queued in the stream and waiting to be processed.
So the problem is, how can I have an Akka Stream which can process one message only per second?
I discovered that MaxInFlightRequestStrategy is a valid way to do it but I should set the batch size to 1, its batch size is default 5, which was causing the problem I found. Also its an over-complicated way to solve the problem now that I am looking at the submitted answer here.
You can either put your elements through the throttling flow, which will back pressure a fast source, or you can use combination of tick and zip.
The first solution would be like this:
val veryFastSource =
Source.fromIterator(() => Iterator.continually(Random.nextLong() % 10000))
val throttlingFlow = Flow[Long].throttle(
// how many elements do you allow
elements = 1,
// in what unit of time
per = 1.second,
maximumBurst = 0,
// you can also set this to Enforcing, but then your
// stream will collapse if exceeding the number of elements / s
mode = ThrottleMode.Shaping
)
veryFastSource.via(throttlingFlow).runWith(Sink.foreach(println))
The second solution would be like this:
val veryFastSource =
Source.fromIterator(() => Iterator.continually(Random.nextLong() % 10000))
val tickingSource = Source.tick(1.second, 1.second, 0)
veryFastSource.zip(tickingSource).map(_._1).runWith(Sink.foreach(println))
I'm having hard time understanding how to work with 0MZ properly. When I'm trying to send a message initialized with size larger than 29, something goes wrong. My code is very simple:
zmq::context_t context (1);
zmq::socket_t req(context,ZMQ_REQ);
req.connect("tcp://localhost:6969");
int msgSize = 100;
zmq::message_t test(msgSize);
snprintf((char*)test.data(),msgSize,"short message");
cout << static_cast<char*>(test.data())<< endl; // this is always fine - 'short message'
so far so good, but after sending this message, if msgSize>29, i can't get the same result again
req.send(test);
cout << static_cast<char*>(test.data())<< endl; // now it's gibberish, like '&?+#'
what's even more puzzling, if my server receives the message it also looks like '&?+#' there, but if it is sending it back simply with PUB socket, I can read it again in my client:
zmq::message_t reply;
req.recv(&reply);
cout << static_cast<char*>(test.data())<< endl; - 'my message' again!
I understand, that there is some 29 bytes limit on short messages, but how can i get around it, without dealing with multipart messages? I literally need like 40 chars....
If the message is > 30 bytes, the memory once occupied by 'test', but then freed, must be being reused by the reply data (obviously by serendipity). Thus, when you look at 'test' again it magically appears to be what you think it should be. This theory should be very simple for you to verify in the debugger by looking at addresses.
Whatever, as Hristo said, sending a message frees it's original contents and shouldn't be used again.
ZeroMQ has an optimisation for small messages where the payload doesn't need to be separately allocated. Again, the fact that you can still see the contents you expect after sending a message is just an artefact; you cannot rely on it.
If you have a requirement to retain the contents of messages after they're sent, take a look at zmq_send_const(), which is new with ZMQ 4.0. I don't know if any bindings make use of it.
As it turned out, I had an error generating piece of code within my server app, after receiving a message I did instant ping pong style reply, like:
zmq::message_t msg(msgSize);
REC.recv(&msg);
//pong
REC.send(msg);
And as above answer points out, sending a message frees it's original contents, leaving me with unwanted gibberish of random bytes.
I have a unit test that tests BufferWithTime. I seem to be getting inconsistent results when values are emitted at the point the buffering will emit a new value.
var scheduler = new TestScheduler();
var source = scheduler.CreateColdObservable(
new Recorded<Notification<int>>(50, new Notification<int>.OnNext(1)),
new Recorded<Notification<int>>(100, new Notification<int>.OnNext(2)),
new Recorded<Notification<int>>(150, new Notification<int>.OnNext(3)),
new Recorded<Notification<int>>(200, new Notification<int>.OnNext(4)),
new Recorded<Notification<int>>(250, new Notification<int>.OnNext(5)),
new Recorded<Notification<int>>(300, new Notification<int>.OnNext(6)),
new Recorded<Notification<int>>(350, new Notification<int>.OnNext(7)),
new Recorded<Notification<int>>(400, new Notification<int>.OnNext(8)),
new Recorded<Notification<int>>(450, new Notification<int>.OnNext(9)),
new Recorded<Notification<int>>(450, new Notification<int>.OnCompleted()));
var results = scheduler.Run(() => source
.BufferWithTime(TimeSpan.FromTicks(150), scheduler));
The results I get back from this are essentially:
results[0] = [1,2]
results[1] = [3,4,5,6]
results[2] = [7,8,9]
My question is, why is there only two items in the first buffer and 4 in the second? I would expect that a source that emits at the same time as buffering is supposed to happen, they either always go in the buffer or are always queued for the next buffer. Have I just stumbled upon a bug?
Based on responses on the MSDN forums this isn't a bug. You can read their answers here.
Basically, when something is scheduled to execute at exactly the same time as something else, it's the order of scheduling that takes precedence i.e. they are queued. When looking at the ordering of the scheduling with the above example you can see why I'm getting the behaviour that I'm getting.
BufferWithTime schedules a window to
open at 0 and close at 150.
The cold Source is then subscribed
to which schedules all other
notifications. At this point, the value to be
emitted at 150 is then queued behind
the closing of the window.
At time 150 the window closes first
(emitting the first buffer of two
values). The next window is opened
and is scheduled to close at 300.
The value that is scheduled to be
emitted at 150 is added to the
second buffer.
At time 300, the value 6 was
scheduled to be emitted first (as it
was scheduled when the source was
subscribed to) so it is added to the
second buffer. BufferWithTime then closes the window (emits the buffer) and opens a new one scheduled to close at 450.
They cycle will then continue consistently.