I have a continuos running web job which listens to a start queue and a stop queue with their respective functions. There are 3 predefined process in the web job and the start queue contains a message which mentions the process name to be started while the stop queue contains a message which mentions the process name to be stopped.
When ever the webjob stops a process it crashes and restarts with the above exception since last 4 weeks.
Executing: 'Functions.StartProcess' because New queue message detected on 'start'.
Updated the status of Process A to 1
Process A Started
Process A processed 200 records
Process A processed 200 records
Process A processed 200 records
Process A processed 200 records
Process A processed 200 records
Executing: 'Functions.StopProcess' because New queue message detected on 'stop'.
Process A stopped
Disposed Process A
Updated the status of Process A to 3
Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlobSharedImpl.<DeleteBlobImpl>b__1b(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndDelete(IAsyncResult asyncResult)
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Protocols.PersistentQueueWriter`1.<DeleteAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Loggers.CompositeFunctionInstanceLogger.<DeleteLogFunctionStartedAsync>d__e.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueTriggerExecutor.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ProcessMessageAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
The webjob does not even contain any code to delete a blob file.
While debugging, the debugger does not point to any specific line while throwing the exception and hence I have no clue as to what is the actual cause.
PS : It used to run perfectly prior to that and every function's body is enclosed in try catch block. The catch block does not rethrow the exception but just logs the stacktrace and continues.
EDIT
I had to check if the below containers were there
in the webjob's storage and also updated the Microsoft.Web.WebJobs.Publish nuget package. I am really not sure if this fixes the problem for sure though.
It looks like you were right, you do in fact need to manually create the blob containers, or publish the web job, in which case it looks like the environment will create the containers for you.
When I tried it out initially, I did the publish first, and it worked like a charm. Second project, I am doing it differently, so I ran the console app first, and it kept failing with 404.
So, you need to create:
azure-webjobs-dashboard
azure-webjobs-hosts
azure-jobs-hosts-archive
azure-jobs-hosts-output
Hope it helps someone.
I was getting this same error.
What worked for me was updating the WindowsAzure.Storage package in my case from 4.3.0 to 7.1.2, but I would go with latest or whatever your other WebJobs are using (if they are running without issue).
Our WebJob encountered this issue in the middle of a Friday afternoon. Previous deployment was 9 days prior and the WebJob functioned up until it suddenly did not. Not sure why it stopped working.
After much reading I believe this means simply that it cannot connect to the storage account for one reason or another, while initializing the WebJob.
Our solution was to add the AzureWebJobsStorage and AzureWebJobsDashboard connection strings to the connection strings config of the App Service via our ARM Template.
"AzureWebJobsDashboard": {
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccount'), ';AccountKey=', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccount')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value)]",
"type": "Custom"
},
"AzureWebJobsStorage": {
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccount'), ';AccountKey=', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccount')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value)]",
"type": "Custom"
}
These values already existed in the app settings config of the App Service, which must have been how it was functioning prior.
We did update libraries per other suggestions to some other answers to this issue. That alone did not solve it.
We also migrated the WebJob from .Net Framework 4.5 to 4.6 to match the rest of the solution, though I don't think this was really part of the fix, just FYI in case it helps someone.
Related
The new support for Event Hub Riders in 7.0 plus the existing InMemoryRepository backing for Sagas looks like it could provide a straightforward means of creating aggregate states based on a stream of correlated messages, e.g. across all sensors in a Building). In this scenario, the Building's Identifier would be used as the CorrelationId of the Messages, the Saga, and as the PartitionKey of the EventData messages sent to the Event Hub, ensuring the same consuming service instance receives all messages for that device at a given time. Given the way Event Hub's rebalancing works, it can be assumed that at some point while this service is running, the service instance managing messages for a Partition will shift to a new host, which will start reading messages sent by the sensors in the building. At that moment:
The new host does not know anything about the old host's processing. It just knows that it is now receiving messages for the Event Hub partition that includes that Building's messages.
The devices sending the messages do not know anything about the transition in state aggregation responsibility "downstream of them" - they are still happily reporting new measurements as always.
The challenge this creates is: on the new service instance, we need a new Saga to be created to take over for the previous Saga, but the only thing that knows no Saga lives for a given entity is MassTransit: nothing on the new instance knows a sensor reading from Building A is the first one from Building A since this service instance took over tracking the aggregate Building A state. We thought this could be handled by marking the same Message (DataCollected) with both InitiatedBy and Orchestrates:
public class BuildingAggregator:
ISaga,
InitiatedBy<DataCollected>, //init saga on first DataCollected with a given CorrelationId seen
Orchestrates<DataCollected> //then keep handling those in that saga
{
//saga Consume methods
}
However, this throws the following exception when the BuildingAggregator receives its second DataCollected message with a given Guid:
Saga exception on receipt of MassTransitFW_POC.Program+DataCollected: The message cannot be accepted by an existing saga
at MassTransit.Saga.Policies.NewSagaPolicy`2.MassTransit.Saga.ISagaPolicy<TSaga,TMessage>.Existing(SagaConsumeContext`2 context, IPipe`1 next)
at MassTransit.Saga.SendSagaPipe`2.<Send>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at MassTransit.Saga.SendSagaPipe`2.<Send>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MassTransit.Saga.InMemoryRepository.InMemorySagaRepositoryContextFactory`1.<Send>d__4`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
Is there another way of achieving this logic? Is this the "wrong way" to apply Sagas?
As per Chris Patterson's comments on the question above, this is achievable with the state machine syntax:
Initially(
When(DataCollected)
.Then(f => _logger.LogInformation("Initiating Network Manager for Network: {NetworkId}", f.Data.NetworkId))
.TransitionTo(Running));
During(Running,
When(DataCollected)
.Then(f => { // activities and state transitions }),
When(SimulationComplete)
.Then(f => _logger.LogInformation("Network {NetworkId} shutting down.", f.Instance.CorrelationId))
.TransitionTo(Final));
Note how the DataCollected event is handled both in the Initially state transition and in a state transition set by the Initially condition.
I'm trying to add SSE functionality to my server application using Redis PubSub, guided by many articles i.e:
how-to-use-actioncontollerlive-along-with-resque-redis.
The server is hosted in Heroku, thus heartbeating is necessary as well.
...
sse = SSE.new(response.stream)
begin
redis = Redis.new(:url => ENV['REDISCLOUD_URL'])
redis.subscribe(<UUID>, HEARTBEAT_CHANNEL) do |on|
on.message do |channel, data|
begin
if channel == HEARTBEAT_CHANNEL
sse.write('', event: "hb")
else
sse.write(data, event: "offer_update")
end
rescue StandardError => e #I'll call this section- "internal rescue"
puts "Internal: #{$!}"
redis.quit
sse.close
puts "SSE was closed
end
end
end
rescue StandardError => e #I'll call this section- "external rescue"
puts "External: #{$!}"
ensure
redis.quit
sse.close
puts "sse was closed"
end
The questions:
I didn't see the "Internal rescue" in any place over the net talking about SSE. but I don't get who can catch an exception if raised by the sse.write? The common scenario is that HB is sent while client isn't connected anymore which makes this section so critical ("Internal: client disconnected" is appeared). Am I right?
In which case will the "external rescue" be triggered? Does a client disconnection cause sse.write to raise an exception in the "inner block" (inside the on.message body)? because it was never caught by the external rescue when I tried to simulate it tens of times.
This code suffers as well. redis.quit in the internal rescue section raises another exception which is caught by the external rescue statement: External: undefined method 'disconnect' for #<Redis::SubscribedClient:0x007fa8cd54b820>. So - how should it done? how can I recognize ASAP a client disconnection in order to free the memory&socket?
How can it be that exceptions raised by the sse.write have NOT been caught by the external rescue (as it should be from beginning) while another error (described in my 3rd question) has been caught? all of this code (outer+inner sections) is running in the same thread, right? I'd be happy for a deep explaination.
You catch the exception inside the subscribe, and so redis doesn't know about it and will not stop it's inner loop properly. redis.quit will cause it to crash and stop since it cannot keep waiting for a message. This is obviously not a good way to do it.
If your code throws an exception inside the subscribe it will cause redis to gracefully unsubscribe and your exception can be rescued outside, as in your "external rescue".
Another point is that you shouldn't catch exceptions without fully handling them, and you should never catch generic exceptions without re-raising them. In this case you can safely let the ClientDisconnected exception bubble to Rails' code.
Here's how your controller's code should look like:
def subscribe
sse = SSE.new(response.stream)
redis = Redis.new(:url => ENV['REDISCLOUD_URL'])
redis.subscribe(<UUID>, HEARTBEAT_CHANNEL) do |on|
on.message do |channel, data|
if channel == HEARTBEAT_CHANNEL
sse.write('', event: "hb")
else
sse.write(data, event: "offer_update")
end
end
end
ensure
redis.quit if reddis # might not have had a chance to initialize yet
sse.close if sse # might not have had a chance to initialize yet
puts "sse was closed"
end
I'm having an issue getting my nunit tests to run after plugging in Simple Injector to my api project. Any reference to a Service isn't working as if they aren't being loaded properly. I attempted to load my container in my test like explained in this post
Services are still not loading correctly.
I am running the tests in memory and they are coming back with an internal server error as soon as the first Service is referenced. The Services that are failing first are referenced using the GlobalConfiguration.Configuration.DependencyResolver.GetService(typeof(IExampleService)); method and those are all coming back as null. I tried calling the service directly in my test and it is also coming back as null, even after the container is built.
We are building the Simple Injector container using the same method in our web project and the test are having no issue referencing the services which is making me wonder if the in memory tests are the issue.
Update: The full stack trace was requested.
Initially I was getting this Exception:
{"The service type IApiService is not supported.\r\nParameter name: serviceType"}
at System.Web.Http.Services.DefaultServices.GetService(Type serviceType)
at Project.Api.Tests.Helpers.HttpRequestHelper.CreateRequest(String url,
HttpMethod method, String key1, String key2, String secret, String baseUrl)
in ...Project.web\src\Project.Api.Tests\Helpers\HttpRequestHelper.cs:line 537
Then moving this service logic into my test and attempting to create the container in my test I receive this:
System.InvalidOperationException : This method cannot be called during the application's pre-start initialization phase. Please note that the RegisterWebApiControllers(Container, HttpConfiguration) overload makes use of the configured IAssembliesResolver. Web API's default IAssembliesResolver uses the System.Web.Compilation.BuildManager, which can't be used in the pre-start initialization phase or outside the context of ASP.NET (e.g. when running unit tests). Either make sure you call the RegisterWebApiControllers method at a later point in time, register a custom IAssembliesResolver that does not depend on the BuildManager, or supply a list of assemblies manually using the RegisterWebApiControllers(Container, HttpConfiguration, IEnumerable) overload.
And after following the instructions in above mentioned post I received:
{[StackTrace, at Project.Api.Filters.HmacAuthenticationAttribute.
d__1.MoveNext() in ...\Project.web\src\Project.Api\Filters\HmacAuthenticationAttribute.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.AuthenticationFilterResult.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.d__1.MoveNext()]}
And that line reference is from this line in the code:
IExampleService _exampleService = (IExampleService)GlobalConfiguration
.Configuration
.DependencyResolver
.GetService(typeof (IExampleService));
if (_exampleService.CheckValue(value))
We are seeing an issue in the Sitecore log files, the message says - "Cannot use Analytics with a database that has no definition items". We have not enabled Analytics in our Sitecore environment. Is anyone aware of this error ?
Here is the stack trace that I found from Sitecore log file
2764 2015:02:24 14:11:17 ERROR Application error.
Exception: System.InvalidOperationException
Message: Cannot use Analytics with a database that has no definition items
Source: Sitecore.Kernel
at Sitecore.Diagnostics.Assert.ResultNotNull[T](T result, String message)
at Sitecore.Analytics.Commons.LazyReference`1.GetValue(Object mutex, Func`1 initializer)
at Sitecore.Analytics.Data.Items.AnalyticsItems.<get_Profiles>b__7()
at Sitecore.Analytics.Commons.LazyReference`1.GetValue(Object mutex, Func`1 initializer)
at Sitecore.Analytics.Data.TrackingField.NormalizeProfiles(AnalyticsItems analytics)
at Sitecore.Analytics.Data.TrackingField.InitializeProfiles()
at Sitecore.Buckets.Pipelines.UI.DynamicFields.EngagementValue.IsLockedCheck(DynamicFieldsArgs args)
at Sitecore.Buckets.Pipelines.UI.DynamicFields.EngagementValue.Process(DynamicFieldsArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Buckets.Pipelines.UI.DynamicFields.DynamicFieldsPipeline.Run(DynamicFieldsArgs args)
at Sitecore.Buckets.Pipelines.UI.FillItem.SetItemProperties.GetValue(Item innerItem, SitecoreUISearchResultItem sitecoreItem)
at Sitecore.Buckets.Pipelines.UI.FillItem.SetItemProperties.GetQuickActions(FillItemArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Buckets.Pipelines.UI.FillItem.FillItemPipeline.Run(FillItemArgs args)
at ItemBuckets.Services.Search.<ProcessRequestAsync>d__d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Let me know if additional information is needed.
The stack trace shows the following method is being called:
Sitecore.Buckets.Pipelines.UI.DynamicFields.EngagementValue.IsLockedCheck
In that method, a TrackingField object is instantiated which in turn calls it's own method InitializeProfiles (This also appears in the stack trace).
The thing is, that TrackingField object is only created if the value of the static Settings.Analytics.Enabled property is true, and there are 2 elements required for this:
The Analytics.Enabledsetting in the Sitecore.Analytics.config file must be set to true.
The Sitecore licence must include DMS.
Are you sure you don't have analytics enabled, because it seems Sitecore thinks you do.
I originally posted this question asking about logging due to an unexplained error. I have now found the log info and have made a small amount of progress with the error. I am re-writing the question to be more specific to the error.
I am trying to run a .NET (4.0) worker process (reads from an SQS queue and writes the results to an RDS database) on AWS "Elastic Beanstalk". To deploy a worker like this, it has to be built as a Windows service (see previous question: Deploying a .NET worker app with Elastic Beanstalk ). My Windows service starts a new thread which wraps my main processing loop. This works fine on my Win7 PC.
Deployment to Elastic Beanstalk (Win Server 2013) appears to be working fine, but as soon as the service hits my thread method, I get a FileNotFoundException. This was puzzling because the exception appeared to occur as soon as it hit my MainProcessingMethod() and before any diagnostics could be produced. After a bit of web searching, I found that missing packages & assemblies in Windows services can produce a FileNotFoundException, and often without any information as to what the missing assembly is.
I believe I have confirmed this. The following processing method that only consists of diagnostics:
public void MainProcessingMethod()
{
EmailSender.SendExceptionEmail("Main started", null, new Exception("main started"));
EventLog.WriteEntry("WebPlagiarismService", "MainProcessingMethod running", EventLogEntryType.Information);
EmailSender.SendExceptionEmail("Main finishing", null, new Exception("main finished"));
}
Produces the expected diagnostics (two emails and an EventLog entry) and no FileNotFoundException (the exception in the email method is just an dummy object- my email method is intended to automatically email exceptions to my mailbox).
This confirms the missing assembly/package hypothesis - something in my commented code is using a package that is missing. How can I determine what?
How can I determine what assemblies&packages are present?
My application does include quite a few DLLs but it looks like these are being found - the above email method is defined in one of the DLLs.
Debug mode does not add any further information. Here's the exception / stack: (app renamed to MyTestService)
Analysis symbol:
Rechecking for solution: 0
Report Id: 51a8153f-3d92-11e3-9426-22000aeb1e73
Report Status: 4
Hashed bucket:
2013-10-25T16:27:20.000Z Error 100:Application Crashing Events Application Error - Faulting application name: MyTestService.exe, version: 1.0.0.0, time stamp: 0x526a97a6
Faulting module name: KERNELBASE.dll, version: 6.2.9200.16451, time stamp: 0x50988aa6
Exception code: 0xe0434352
Fault offset: 0x000000000003811c
Faulting process id: 0x76c
Faulting application start time: 0x01ced19f12e04665
Faulting application path: c:\Data\TestApp\MyTestService.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 51a8153f-3d92-11e3-9426-22000aeb1e73
Faulting package full name:
Faulting package-relative application ID:
2013-10-25T16:27:19.000Z Error 0:(0) .NET Runtime - Application: MyTestService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileNotFoundException
Stack:
at MyTestService.MyTestService.MainProcessingMethod()
at
System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()
I am also seeing errors 'Deployment package did not contain an iisApp' which makes sense as this is just a service and not a true web app (I see something similar with my Python worker app)
The trick is to add a wrapper method, and enclose the main processing method with a try...catch. Then this catches the assembly load at the beginning of the main processing method, e.g.:
try
{
MainProcessingMethod_internal();
}
catch (Exception e)
{
EmailSender.SendExceptionEmail("Main_intern", null, e);
}
Note that I am using my existing exception email method.
And the result? It is the AWSSDK of all things that is missing!
Exception: Could not load file or assembly 'AWSSDK, Version=1.5.36.0, Culture=neutral, PublicKeyToken=9f476d3089b52be3' or one of its dependencies. The system cannot find the file specified.
at WebPlagiarismService.WebPlagiarismService.MainProcessingMethod_intern()
at WebPlagiarismService.WebPlagiarismService.MainProcessingMethod()
Source: WebPlagiarismService
Target: Void MainProcessingMethod_internal()