Alpakka AMQP : How to detect declaration exception? - akka

I have a AMQP Source and AMQP Sink with Declarations:
List<Declaration> declarations = new ArrayList<Declaration>() {{
add(QueueDeclaration.create(sourceExchangeName));
add(BindingDeclaration.create(sourceExchangeName, sourceExchangeName).withRoutingKey(sourceRoutingKey));
}};
amqpSource = AmqpSource
.committableSource(
NamedQueueSourceSettings.create(connectionProvider, sourceExchangeName)
.withDeclarations(declarations),
bufferSize);
AmqpWriteSettings amqpWriteSettings = AmqpWriteSettings.create(connectionProvider)
.withExchange("DEST_XCHANGE")
.withRoutingKey("ROUTE123")
.withDeclaration(ExchangeDeclaration.create(destinationExchangeName,
BuiltinExchangeType.DIRECT.getType()));
amqpSink = AmqpSink.create(amqpWriteSettings);
And then I have a flow..
amqpSource.map(doSomething).async().map(doSomethingElse).async().to(amqpSink)
Now, after i started the app, the messages were sent to source queue was not getting consumed. I later found out that this was due to errors that occurred during declarations. (i.e., it worked fine when I removed the .withDeclarations(..) in the Source and Sink settings.
So my questions:
How to detect if the AMQP Source and Sink are up and running fine?
How to ignore declaration exceptions?
If any exception occurs, how can I know and make the system fail?

To answer 1 and 3, the AmqpSink materializes a CompletionStage<Done> that you would have to keep, and handle (register some callback functions on) to observe failure and completion of the stream. In the docs sample we block on that completion stage which is not good in production code (https://doc.akka.io/docs/alpakka/current/amqp.html#with-sink), that's probably because the sample is included in one of the Alpakka tests. Prefer the usual CompletionStage callback/transformation methods instead (see for example this introduction).
The CompletionStage will fail when an error happens, when the stream is materialized/starting up or during the processing of elements, alternatively complete once the source reaches the end and each element has gone through your flow into the sink. That means that for starting up the stream, if it does not pretty quickly fail it is running.
For question 2 not sure if it is possible to ignore the declaration exceptions, it could be that those always fail the connection.

Related

Live555MediaServer restarts the stream at every new connection. Why setting "reuseSource" to true is not working as expected?

Live555MediaServer can be used to stream video files as rtsp streams. I have 2 clients (vlc) that connect to the server, A and B. I want to see the exact video stream in both the clients. Here is the problem: I connect A and after 10 seconds I connect B. When B is connected the video that I see starts over from the beginning, while A keeps streaming as it was.
I would like the 2 concurrent streams to be synchronized.
The live555 doc says that setting reuseFirstSource to True should work. So I tried to set reuseSource to true at DynamicRTSSPServer:121 but it didn't work. When I connect to the server using client B the video restarts from the beginning.
Boolean const reuseSource = True;
I expect to see the 2 concurrent streams synchronized even if one starts with a delay with respect to the other one.
I finally found a workaround and why there was this 'bug'.
Quick answer: set if condition at line 67 to false, i.e.
if (smsExists && isFirstLookupInSession) {
becomes
if (false) {
Explaination: Every time a new session is starting, the isFirstLookupInSession variable is set to true and the session is removed and recreated.
I wrote to the support of live555 and Finlayson told me and I quote
“LIVE555 Media Server” code was always intended to work this way, and was intended to be a ‘stand-alone appliance’ that does not have its code modified (e.g., by changing the value of “reuseFirstSource”).
Thus the only solution for creating a RTSP server through Live555 is to create your own server starting from the testProgs examples.
The workaround proposed here could generate unwanted behaviors, but for a simple rtsp server with multiple streams it's fine.

Streaming MutationGroups into Spanner

I'm trying to stream MutationGroups into spanner with SpannerIO.
The goal is to write new MuationGroups every 10 seconds, as we will use spanner to query near-time KPI's.
When I don't use any windows, I get the following error:
Exception in thread "main" java.lang.IllegalStateException: GroupByKey cannot be applied to non-bounded PCollection in the GlobalWindow without a trigger. Use a Window.into or Window.triggering transform prior to GroupByKey.
at org.apache.beam.sdk.transforms.GroupByKey.applicableTo(GroupByKey.java:173)
at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:204)
at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:120)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
at org.apache.beam.sdk.transforms.Combine$PerKey.expand(Combine.java:1585)
at org.apache.beam.sdk.transforms.Combine$PerKey.expand(Combine.java:1470)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteGrouped.expand(SpannerIO.java:868)
at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteGrouped.expand(SpannerIO.java:823)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
at quantum.base.transform.entity.spanner.SpannerProtoWrite.expand(SpannerProtoWrite.java:52)
at quantum.base.transform.entity.spanner.SpannerProtoWrite.expand(SpannerProtoWrite.java:20)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
at quantum.entitybuilder.pipeline.EntityBuilderPipeline$Write$SpannerWrite.expand(EntityBuilderPipeline.java:388)
at quantum.entitybuilder.pipeline.EntityBuilderPipeline$Write$SpannerWrite.expand(EntityBuilderPipeline.java:372)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
at quantum.entitybuilder.pipeline.EntityBuilderPipeline.main(EntityBuilderPipeline.java:122)
:entityBuilder FAILED
Because of the error above I assume the input collection needs to be windowed and triggered, as SpannerIO uses a GroupByKey (this is also what I need for my use case):
...
.apply("1-minute windows", Window.<MutationGroup>into(FixedWindows.of(Duration.standardMinutes(1)))
.triggering(Repeatedly.forever(AfterProcessingTime
.pastFirstElementInPane()
.plusDelayOf(Duration.standardSeconds(10))
).orFinally(AfterWatermark.pastEndOfWindow()))
.discardingFiredPanes()
.withAllowedLateness(Duration.ZERO))
.apply(SpannerIO.write()
.withProjectId(entityConfig.getSpannerProject())
.withInstanceId(entityConfig.getSpannerInstance())
.withDatabaseId(entityConfig.getSpannerDb())
.grouped());
When I do this, I get the following exceptions during runtime:
java.lang.IllegalArgumentException: Attempted to get side input window for GlobalWindow from non-global WindowFn
org.apache.beam.sdk.transforms.windowing.PartitioningWindowFn$1.getSideInputWindow(PartitioningWindowFn.java:49)
com.google.cloud.dataflow.worker.StreamingModeExecutionContext$StepContext.issueSideInputFetch(StreamingModeExecutionContext.java:631)
com.google.cloud.dataflow.worker.StreamingModeExecutionContext$UserStepContext.issueSideInputFetch(StreamingModeExecutionContext.java:683)
com.google.cloud.dataflow.worker.StreamingSideInputFetcher.storeIfBlocked(StreamingSideInputFetcher.java:182)
com.google.cloud.dataflow.worker.StreamingSideInputDoFnRunner.processElement(StreamingSideInputDoFnRunner.java:71)
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
org.apache.beam.sdk.values.ValueWithRecordId$StripIdsDoFn.processElement(ValueWithRecordId.java:145)
After investigating further it appears to be due to the .apply(Wait.on(input)) in SpannerIO: It has a global side input which does not seem to work with my fixed windows, as the docs of Wait.java state:
If signal is globally windowed, main input must also be. This typically would be useful
* only in a batch pipeline, because the global window of an infinite PCollection never
* closes, so the wait signal will never be ready.
As a temporary workaround I tried the following:
add a GlobalWindow with triggers instead of fixed windows:
.apply("globalwindow", Window.<MutationGroup>into(new GlobalWindows())
.triggering(Repeatedly.forever(AfterProcessingTime
.pastFirstElementInPane()
.plusDelayOf(Duration.standardSeconds(10))
).orFinally(AfterWatermark.pastEndOfWindow()))
.discardingFiredPanes()
.withAllowedLateness(Duration.ZERO))
This results in writes to spanner only when I drain my pipeline. I have the impression the Wait.on() signal is only triggered when the Global windows closes, and doesn't work with triggers.
Disable the .apply(Wait.on(input)) in SpannerIO:
This results in the pipeline getting stuck on the view creation which
is described in this SO post:
SpannerIO Dataflow 2.3.0 stuck in CreateDataflowView.
When I check the worker logs for clues, I do get the following warnings:
logger: "org.apache.beam.sdk.coders.SerializableCoder"
message: "Can't verify serialized elements of type SpannerSchema have well defined equals method. This may produce incorrect results on some PipelineRunner
logger: "org.apache.beam.sdk.coders.SerializableCoder"
message: "Can't verify serialized elements of type BoundedSource have well defined equals method. This may produce incorrect results on some PipelineRunner"
Note that everything works with the DirectRunner and that I'm trying to use the DataflowRunner.
Does anyone have any other suggestions for things I can try to get this running? I can hardly imagine that I'm the only one trying to stream MutationGroups into spanner.
Thanks in advance!
Currently, SpannerIO connector is not supported with Beam Streaming. Please follow this Pull Request which adds streaming support for spanner IO connector.

How to resolve a failed assert - zmqpp::frame::~frame(): assertion `0 == result' failed?

I am using zeromqpp to send data to another process. I have successfully used zeromqpp with other signals but with my last message added to the producer I frequently get the run time assertion:
zmqpp::frame::~frame()68: assertion "0 == result" failed
The consumer program never gets the error, only the producer and it causes the producer's program to crash. If I remove the code that publishes the message then I don't see the assertion.
The zeromq server doesn't report any errors. GDB doesn't give me a stack trace and the assertion doesn't occur at the same time during multiple program executions, so I am having a hard time tracking down where the problem is. I have started commenting out large portions of code to try to narrow things down, but I am still lost as to the cause of the problem.
Can anyone tell me what this assertion error means so I can narrow down what to look for?
Edit
Here is step 0.
signalProducer_->publish(SIGNAL_NAME, producerData, 1, VALID_SIGNAL_DATA);
Step 0:
post also the very code, that actually started to generate the issuesee your own text
cit.:" If I remove the code that publishes the message then I don't see the assertion."
Step 1:
Review the ZeroMQ low-level API documentation, not the actually used c++ wrapper / binding implementation, for pre-designed details on error-causes recognised & distinguished at the lowest level, so as to realise and understand, what root-causes may appear in particular API-service-calls:
Example: ...ØMQ team is particularly smart & pedantic on this since early releases- kudos
The zmq_send() function shall return number of bytes in the message if successful. Otherwise it shall return -1 and set errno to one of the values defined below.Errors
EAGAIN Non-blocking mode was requested and the message cannot be sent at the moment.ENOTSUP The zmq_send() operation is not supported by this socket type.EFSM The zmq_send() operation cannot be performed on this socket at the moment due to the socket not being in the appropriate state. This error may occur with socket types that switch between several states, such as ZMQ_REP. See the messaging patterns section of zmq_socket(3) for more information.ETERM The ØMQ context associated with the specified socket was terminated.ENOTSOCK The provided socket was invalid.EINTR The operation was interrupted by delivery of a signal before the message was sent.EHOSTUNREACH The message cannot be routed.
/* Send a multi-part
message consisting
of three parts to socket */
rc = zmq_send ( socket, "ABC", 3, ZMQ_SNDMORE ); assert (rc == 3);
rc = zmq_send ( socket, "DEFGH", 5, ZMQ_SNDMORE ); assert (rc == 5);
/* Final part;
no more parts to follow */
rc = zmq_send ( socket, "JK", 2, 0 ); assert (rc == 2);

MATLAB parallel toolbox, remoteParallelFunction : RUNTIME_ERROR during function evaluation

I'm using the parallel computing toolbox (PCT) in combination with the Simbiology toolbox in MATLAB 2012b. I’m receiving an intermittent error message when I run my script with a remote pool of workers, but not with a local pool of workers:
Caught std::exception Exception message is:
vector::_M_range_check
Error using parallel_function (line 589)
Error in remote execution of remoteParallelFunction : RUNTIME_ERROR
Error in PSOFit (line 486)
parfor ns = 1:r.NumSwp
Error in PSOopt_driver (line 209)
PSOFit(ObjFuncName,LB,UB,PSOopts);
The error does not occur when I comment out the call to the function sbiosimulate (a SimBiology function for model evaluation).
I have a couple of ideas:
I’ve introduced some sort of race condition, that causes a problem in accessing the model variables (is this possible in MATLAB?)
Model compilation in simbiology is sometimes but not always compatible with the PCT, and I’ve hit some sort of edge case
Since sbiosimulate evaluates compiled C++ code, for some inputs there might be a bug in the source that generates the exception
I am aware of this.
I'm a developer of SimBiology. I believe this is a bug that was introduced into SimBiology's C++ code in the R2012a release. The bug is triggered when a simulation ends without producing any simulation results. This can sometimes occur when the model is configured to report only particular times (using the OutputTimes options) AND the simulation is configured to end after a particular amount of real time (using the MaximumWallClock option). Basically, the simulation "times out" before it ever gets a chance to log the first output time.
One way to work around this problem is to always include time 0 in the OutputTimes. This time will always get logged before evaluating the MaximumWallClock criterion, preventing the bug from getting triggered. I am also contacting this user directly and will work on fixing the bug in a future release.

HornetQ, consumer can't find queue

I'm trying to use ActiveMQ-CPP with HornetQ. I'm using the ActiveMQ-CPP bundled example, but I'm having a hard time with it.
The producer works like a charm, but the consumer gives me the following message:
* BEGIN SERVER-SIDE STACK TRACE
Message: Queue /queue/exampleQueue does not exist
Exception Class
END SERVER-SIDE STACK TRACE *
FILE: activemq/core/ActiveMQConnection.cpp, LINE: 768
FILE: activemq/core/ActiveMQConnection.cpp, LINE: 774
FILE: activemq/core/ActiveMQSession.cpp, LINE: 350
FILE: activemq/core/ActiveMQSession.cpp, LINE: 281
Time to completion = 0.161 seconds.
The problem is that the queue exists. The code works all right with ActiveMQ+Openwire, but I'm not having the same luck with HornetQ+STOMP.
Any ideas?
Try to set the same Queue's address you defined on Hornetq as the destination.
Probably your queue is defined on HornetQ like this
<queue name="exampleQueue">
<address>jms.queue.exampleQueue</address>
</queue>
So, try to connect to this address via STOMP.
See the following frames according to the protocol:
Subscribing to the queue
SUBSCRIBE
destination:jms.queue.exampleQueue
^#
Sending a message
SEND
destination:jms.queue.exampleQueue
it works
^#
As soon as the message is sent, you'll get the message on the session you subscribed to the queue
MESSAGE
timestamp:1311355464983
redelivered:false
expires:0
subscription:subscription/jms.queue.exampleQueue
priority:0
message-id:523
destination:jms.queue.exampleQueue
it works
-- EDIT
There's one point left I would like to add...
HornetQ doesn't conform to STOMP's naming standarts (see http://community.jboss.org/message/594176 ), so there's a possibility that the activemq-cpp follows the behavior of ativemq-nms, which "normalize" queue's name to the STOMP standart: "/queue/YourQueue" (and causes naming issues).
So, if that's the case, even if you try to change your destination name to 'jms.queue.exampleQueue', activemq-cpp could normalize it to '/queue/jms.queue.exampleQueue'.
In NMS+HornetQ, there's no "out of the box" way of avoiding this. The only choice is to edit NMS's source code and remove the part which normalize queue's names. Maybe it's the same way out on activemq-cpp.
HornetQ doesn't like the "/queue/" prefix for a SUBSCRIBE. I took that out of the ToStomp method in StompHelper and everything worked.