In the new Azure portal I see under "Metrics" section for an EventHub that there were many "Internal Server Error" events on a specific day. Is it possible to find out more about what could have caused it and description about those errors?
As the metrics for Event Hubs state about InternalServerErrors and OtherErrors:
InternalServerErrors: Total number of internal server error exceptions sent back to the sender or receiver while performing run-time operations. This type of error is due to either service-side or network problems.
OtherErrors: These types of errors are due to faults at the sender or receiver side, such as providing bad parameters, not enough credentials, or trying to perform an operation on a nonexistent entity.
I would recommend you log into azure portal, choose your Event hub, click "MONITORING > Diagnostics logs", then turn on diagnostics for collecting logs. For more details, you could refer to Event Hubs diagnostic logs.
Related
I'm running an SAP PI/PO scenario and having a problem with finding the response message (the error one).
Scenario:
PROXY (class) -> SAP PI/PO 7.5 -> SOAP (wsdl)
The error when executing the scenario is:
Transmitting the message using connection
SOAP_http://sap.com/xi/XI/System failed, due to:
com.sap.engine.interfaces.messaging.api.exception.MessagingException:
Error encountered while executing mapping:
com.sap.aii.af.service.mapping.MappingException:
com.sap.aii.utilxi.misc.api.ResourceException: Could not determine
mapping steps for message 3b14b3f8-0860-11ed-a034-000001795062
The problem is that I cannot find the message (3b14b3f8-0860-11ed-a034-000001795062) in SAP PI monitor. It's the response message (who is causing the error), as the request message I did find in the monitor and it's correct.
Where I can find it? Why is not showing?
From what I read, this problem happens with several reasons, I just want to find this message.
The webservice works correctly, I test it in SOAP UI.
To find the response message (which contains the error), we have to first activate the Integrated Configuration log (it's not active by default).
To activate the log, we have to follow these steps (more info: https://blogs.sap.com/2012/11/06/message-staging-and-logging-options-in-advanced-adapter-engine-of-pi-73x/)
Go to "Advanced Settings" in the target Integrated Configuration.
Change the "Use global configuration" radio button to "Use scenario-specific configuration" in both, "Staging" and "Loging".
Change all "Staging" lists to "Store".
Change all "Logging" lists to "Log".
Save and activate.
Now, in the SAP PO monitor (Message monitor), by clicking the "Related Messages" option, we can see the response message error.
In this case, the error was:
<sap:Error xmlns:sap='http://sap.com/xi/XI/Message/30'
SOAP:mustUnderstand='1'>sap:CategoryXIAdapter</sap:Category><sap:Code
area='SOAP'>FAULT</sap:Code>sap:P1http://schemas.microsoft.com/ws/2005/05/addressing/none</sap:P1>sap:P2ActionNotSupported</sap:P2>sap:AdditionalTextThe
message with Action '' cannot be processed at the receiver, due to a
ContractFilter mismatch at the EndpointDispatcher. This may be because
of either a contract mismatch (mismatched Actions between sender and
receiver) or a binding/security mismatch between the sender and the
receiver. Check that sender and receiver have the same contract and
the same binding (including security requirements, e.g. Message,
Transport, None).</sap:AdditionalText><sap:ApplicationFaultMessage
namespace='http://schemas.microsoft.com/ws/2005/05/addressing/none'>ActionNotSupported</sap:ApplicationFaultMessage></sap:Error>
In which the Action was empty.
I'm using Sentry to log my errors, but there are errors I'm not able to fix (or could not be fixed by me) like
OSError (write error)
Or error that come from RQ (each time I deploy my app)
Or client errors (which are client.errors)
I can't just ignore them because I consume all my quota. How I can filter out this errors?
Here some references for interested people.
uwsgi: OSError: write error during GET request
Fixing broken pipe error in uWSGI with Python
https://github.com/unbit/uwsgi/issues/1623
I created a Gist for rate limiting the amount of events that are being send to Sentry:
https://gist.github.com/jurrian/e22f8e724b8499a29c5537e956f0dc7f
It uses ratelimitingfilter which can be configured to set a rate per minute, and additionally add a burst to start rate limiting after a number of events.
I get the same errors, but i never had any problems with my quota. But if you really want to filter it, you can just do it in your sdk:
https://docs.sentry.io/error-reporting/configuration/filtering/?platform=python
But beware, this could hide other errors as mentioned here:
https://github.com/pypa/warehouse/issues/679
To safe yourself some quota, you have two options:
Avoid forwarding events client side, thus preventing events being send to sentry at all. Have a look at the docs for available client-side filters. The drawback with this approach is of course that you need a new code deployment for any adjustment of client-side filters and some clients may not instantly reflect your code changes.
Avoid forwarding events on sentry's side, via inbound filters ([Project] > Project Settings > Inbound Filters). According to the sentry documentation on quota usage, events filtered via inbound filters are not affecting your quota.
Inbound filters include:
Common browser extension errors
Events coming from localhost
Known legacy browsers errors
Known web crawlers
By their error message
From specific release versions of your code
From certain IP addresses
Business plans and above also allow to filter events by error messages.
Is it possible to ensure that a message was successfully delivered to an Event Hub when sending it with the log-to-eventhub policy in API Management?
Edit: In our solution we cannot allow any request to proceed if a message was not delivered to the Event Hub. As far as I can tell the log-to-eventhub policy doesn't check for this.
Welcome to Stackoveflow!
Note: Once the data has been passed to an Event Hub, it is persisted and will wait for Event Hub consumers to process it. The Event Hub does not care how it is processed; it just cares about making sure the message will be successfully delivered.
For more details, refer “Why send to an Azure Event Hub?”.
Hope this helps.
Event Hubs is built on top of Service Bus. According to the Service Bus documentation,
Using any of the supported Service Bus API clients, send operations into Service Bus are always explicitly settled, meaning that the API operation waits for an acceptance result from Service Bus to arrive, and then completes the send operation.
If the message is rejected by Service Bus, the rejection contains an error indicator and text with a "tracking-id" inside of it. The rejection also includes information about whether the operation can be retried with any expectation of success. In the client, this information is turned into an exception and raised to the caller of the send operation. If the message has been accepted, the operation silently completes.
When using the AMQP protocol, which is the exclusive protocol for the .NET Standard client and the Java client and which is an option for the .NET Framework client, message transfers and settlements are pipelined and completely asynchronous, and it is recommended that you use the asynchronous programming model API variants.
A sender can put several messages on the wire in rapid succession without having to wait for each message to be acknowledged, as would otherwise be the case with the SBMP protocol or with HTTP 1.1. Those asynchronous send operations complete as the respective messages are accepted and stored, on partitioned entities or when send operation to different entities overlap. The completions might also occur out of the original send order.
I think this means the SDK is getting a receipt for each message.
This theory is further aided by the RetryPolicy Class used in the ClientEntity.RetryPolicy Property of the EventHubSender Class.
In the API Management section on logging-to-eventhub, there is also a section on retry intervals. Below that are sections on modifying the return response or taking action on certain status codes.
Once the status codes of a failed logging attempt are known, you can modify the policies to take action on failed logging attempts.
We are using Azure Service Bus Topic in workflow manager (approval process). In any way, we don’t want to lose/duplicate messages when we push messages to service bus topic. Now there are two options.
a. Use Retry the only
b. Use Paired service bus only without retry.
As we cannot use both together, let assume during message push, primary service bus is not available then message pus to paired service bus and when primary service bus available then automatically message push to the primary. But if we use retry, retry will try to push message to primary and as primary service bus is not available messages will go to paired service bus also. so there are chances to process duplicate messages.
Which is the best option “a” or “b”, to push message to service bus for the given problem statement?
Both options have their pros and cons.
With Paired Namespaces you get the ability to continue sending messages while your primary namespace is down. But don't get fooled. You only store those messages while the primary namespace is down. They are not retried by the reveiver. Other drawbacks include
No good testability.
Increased cost (you send to the secondary, retrieve back from it to send to the primary).
Failover to the secondary is not very intuitive. You have to manually retry the message after a failure. It is not automatically switches to the secondary namespace.
Have a look at this post for more details.
With retries approach you gain the simplicity. And something you'd need to do anyways. With Azure Service Bus operations can fail with intermittent exceptions and you should retry anyways. The drawback of having only retries - doesn't protect from outages. That's why you could combine it with a secondary namespace using custom implementation, but that's a whole different can of warms. Libraries like NServiceBus provides a custom implementation you can get the idea from.
I am trying to implement the receiver part as per the tutorial
https://azure.microsoft.com/en-us/documentation/articles/event-hubs-java-ephjava-getstarted/
Failure while registering: com.microsoft.azure.eventprocessorhost.EPHConfigurationException:
Encountered error while fetching the list of EventHub PartitionIds
I have 16 partitions in my eventhub. But when I send the data I don't specify any partition. How do I know in which partition my data is sent to? Am I getting the above error because of all the partitions?
Make sure that your consumer sas policies does includes "manage" and not only "listen".
I guess it has to have manage rights to be able to list the partitions.
Ideally - EPH should work with "Listen"-only Claims. Right now we have a bug in EventProcessorHost client-code as a result of which - it needs "Manage" claims. We are working on it.
The error "Encountered error while fetching the list of EventHub PartitionIds" is generic and Thrown at PartitionManager, while querying for Partitions. You ran into one of the exceptions in the below catch block. Please indicate inner-exception for completeness (SEO) & faster resolution.
catch(XPathExpressionException|ParserConfigurationException|IOException|InvalidKeyException|NoSuchAlgorithmException|URISyntaxException|SAXException exception)
{
throw new EPHConfigurationException("Encountered error while fetching the list of EventHub PartitionIds", exception);
}
EDIT:
this issue is fixed in version 0.7.7.
I was working with integrating eventhub with the ELK stack and came across the same error.
To solve this I found that in the settings for the Event Hub Namespace that my event hub was in I had to allow access to the VNET my ELK stack was in.
This can be done by going to the following page: Your EventHubNamespace > Settings - Firewalls and virtual networks.
Either allow access from all networks or add your specific VNET, Subnet, or IP range (for specific machines).
This in addition to setting consumer sas policies to Manage should resolve the "Encountered error while fetching the list of EventHub PartitionIds" error.