I want to be able to test an Azure WebJobs SDK project locally, before I actually publish it to Azure.
If I make a brand new Azure Web Jobs Project, I get some code that looks like this:
Program.cs:
// To learn more about Microsoft Azure WebJobs SDK, please see http://go.microsoft.com/fwlink/?LinkID=320976
class Program
{
// Please set the following connection strings in app.config for this WebJob to run:
// AzureWebJobsDashboard and AzureWebJobsStorage
static void Main()
{
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
}
Functions.cs:
public class Functions
{
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public static void ProcessQueueMessage([QueueTrigger("queue")] string message, TextWriter log)
{
log.WriteLine(message);
}
}
I would like to get around to testing whether or not the QueueTrigger function is working properly, but I can't even get that far, because on host.RunAndBlock(); I get the following exception:
An unhandled exception of type 'System.InvalidOperationException'
occurred in mscorlib.dll
Additional information: Microsoft Azure WebJobs SDK Dashboard
connection string is missing or empty. The Microsoft Azure Storage
account connection string can be set in the following ways:
Set the connection string named 'AzureWebJobsDashboard' in the connectionStrings section of the .config file in the following format
, or
Set the environment variable named 'AzureWebJobsDashboard', or
Set corresponding property of JobHostConfiguration.
I ran the storage emulator, and set the Azure AzureWebJobsDashboard connection string like so:
<add name="AzureWebJobsDashboard" connectionString="UseDevelopmentStorage=true" />
but, when I did that, I'm getting a different error
An unhandled exception of type 'System.InvalidOperationException'
occurred in mscorlib.dll
Additional information: Failed to validate Microsoft Azure WebJobs SDK
Dashboard account. The Microsoft Azure Storage Emulator is not
supported, please use a Microsoft Azure Storage account hosted in
Microsoft Azure.
Is there any way to test my use of the WebJobs SDK locally?
WebJobs 2.0 now works using development storage (I'm using v2.0.0-beta2).
Note that latency in general and Blob triggers in particular are currently far better than you can get in production. Design with care.
If you want to test the WebJobs SDK locally, you need to set up a storage account in Azure. You can't test it against the Azure Emulator. That's what that error is telling you.
Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure.
So to answer your question, you can create a storage account in Azure using the portal, and then set up your connection string in the app.config of your Console Application. Then just drop a message to the queue and run the Console Application locally and it will pick it up (assuming you're trying to interact with the queue obviously).
Make sure that you replace the [QueueTrigger("queue")] "queue" with the name of the queue you want to poll.
Hope this helps
Related
I am trying to setup a new Workflow manager & Service Bus Farm in my local machine. and in the stage of Create a new service bus farm getting error as "Unable to cast object of type 'System.DirectoryServices.AccountManagement.GroupPrincipal' to type 'System.DirectoryServices.AccountManagement.UserPrincipal'" .
Please suggest any resolution.
Specification :-
SQL Server 2012 Enterprise edition,
Workflow manager 1.0 refresh(CU2)+CU5,
Service Bus 1.1 CU1,
Windows fabric 1.0.970.0
Error Screenshot :-
You need to mark sure to run the Workflow Manager with a domain account.
A blog ("Farm Creation" section) lists this similar issue for your reference:
Fun With Installing Service Bus for Windows
Also there is a TechNet blog which lists tips for Successful Installation of Workflow Manager, you can have a look at it.
I have successfully run Firebase emulator:
E:\firebase>firebase emulators:start
i emulators: Starting emulators: functions, firestore
! Your requested "node" version "8" doesn't match your global version "10"
+ functions: Emulator started at http://localhost:5001
! No Firestore rules file specified in firebase.json, using default rules.
i firestore: Serving ALL traffic (including WebChannel) on http://localhost:808
0
! firestore: Support for WebChannel on a separate port (8081) is DEPRECATED and
will go away soon. Please use port above instead.
i firestore: Emulator logging to firestore-debug.log
+ firestore: Emulator started at http://localhost:8080
i firestore: For testing set FIRESTORE_EMULATOR_HOST=localhost:8080
i functions: Watching "E:\firebase\func
tions" for Cloud Functions...
! functions: Your GOOGLE_APPLICATION_CREDENTIALS environment variable points to
E:\firebase\key.json. Non-emulated serv
ices will access production using these credentials. Be careful!
+ functions[notifyNewMessage]: firestore function initialized.
+ All emulators started, it is now safe to connect.
The functions file with notifyNewMessage function is below:
const functions = require('firebase-functions')
const admin = require('firebase-admin')
admin.initializeApp()
exports.notifyNewMessage = functions.firestore
.document('test/{test}')
.onCreate((docSnapshot, context) => {
console.log(docSnapshot.data())
}
When I create a new document manually in my Firebase console, my CLI in Windows does not log anything. How can I fix this so that it logs what the functions says in my CLI?
I was just being an idiot and had not re-compiled by code.
The local emulator doesn't respond to changes in the Firestore database that's hosted in Google's cloud and visible in the console. What it does respond to are changes in the locally emulated Firestore database also running on your machine. If you want your Firestore function to trigger in the local emulator, you will have to instead make a change to the emulated Firestore, as described in the documentation. You might want to go through the provided quickstart to get some experience with this.
If you don't want to use the Firestore emulator and just want to trigger it directly for testing, you can use the Firebase CLI local shell.
My Azure CosmosDB account has a status of 'Creating' and I am unable to delete it. I deployed the account from Visual Studio to my Azure subscription as I have done successfully in the past. However, this time, it got stuck in the 'Creating' status and doesn't seem to be making any progress.
I have tried to delete the account but I get the following error message:
There is already an operation in progress which requires exlusive lock on this service mv-iot-messages. Please retry the operation after sometime.
ActivityId: 6b46b569-661e-4be5-8ccc-90d158a09047, Microsoft.Azure.Documents.Common/2.5.1 (Code: PreconditionFailed)
I have attempted to delete the account from both the Azure Portal as well as from the Microsoft Azure Command prompt and I get the same message.
N/A
I expect the Cosmos DB account to fully deploy, after which, I can delete it or a way to override the 'exclusive lock'.
I registered my task app in Spring Cloud Data Flow, created a definition for it and the status shows 'unknown'. I created the stream and trying to launch the task through task-sink and I get an error:
java.lang.IllegalStateException: failed to resolve MavenResource:
How to launch a task from the task-sink? Am I missing something? Any help is appreciated. Another question I have is how do I access the payload sent via TaskLaunchRequest in my task?
S1 http | step1: transformer-rabbit | log
S2 :S1.step1 > filter --expression=payload.contains('CUSTADDRMODRQ_V15') | task-processor | task-sink
task-sink is launching the task provided by the uri in the TaskLaunchRequest. It is looking for the resource as shown in the log
OUT Using manager EnhancedLocalRepositoryManager with priority 10.0 for /home/vcap/.m2/repository
OUT Using transporter HttpTransporter with priority 5.0 for https://repo.spring.io/libs-snapshot and finally failing.
The task is deployed in our repository and as mentioned I registered and created the definition for it as well.
This one is in cf environment and I am using SCDF server 1.0.0.M4.
In the application.properties for the task-sink i am providing maven.remote.repositories.snapshots.url=**
task create fis-ifx-event-task --definition "fis-event-task"
My goal is launching the task from the stream.
Thanks for the information. I am in fact using the BUILD-SNAPSHOT as I am unable to enable taks in 1.0.0M4 version. Here is the one I am using spring-cloud-dataflow-server-cloudfoundry-1.0.0.BUILD-20160808.144306-116. I am able to register and create task definitions. The status of the task definition is showing as 'unknown' even when I am using the sample task module provided by your team. But when I initiate the flow of the stream and when task-sink tries to launch the task, it is unable to find the maven resource. When I create the task definition, does the task module gets deployed? I don't see any app in Pivotal Apps Manager. As mentioned earlier, I provided maven.remote.repositories.snapshot.url in the application.properties file for the task-sink application. Another thing I observed is when I launch the task manually from dataflow shell it gives an error CF-UnprocessableEntity(10008): The request is semantically invalid: Unknown field(s): 'staging_disk_in_mb', 'staging_memory_in_mb' and also a message saying 'Source is empty'. Presently the task is supposed to print the timestamp and is not dependent on any input.
TaskProcessor code:
#EnableBinding(Processor.class)
#EnableConfigurationProperties(TaskProcessorProperties.class)
public class TaskProcessor {
#Autowired
private TaskProcessorProperties processorProperties;
public TaskProcessor() {
}
#Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)
#ELI(level = "info", eventType = ELIEventType.INBOUND)
public Object setupRequest(String message) {
Map<String, String> properties = new HashMap<String, String>();
properties.put("payload", message);
TaskLaunchRequest request = new TaskLaunchRequest(processorProperties.getUri(), null, properties, null);
return new GenericMessage<>(request);
}
}
TaskSink code:
#SpringBootApplication
#EnableTaskLauncher
#EnableBinding(Sink.class)
#EnableConfigurationProperties(TaskSinkProperties.class)
public class FisIfxEventTaskSinkApplication {
public static void main(String[] args) {
SpringApplication.run(FisIfxEventTaskSinkApplication.class, args);
}
}
I provided the stream I am using earlier in the post. Sink is receiving the TaskLaunchRequest with uri and payload as you can see here and unable to launch the task.
OUT registering [40, java.io.File] with serializer org.springframework.integration.codec.kryo.FileSerializer
2016-08-10T16:08:55.02-0600 [APP/0]
OUT Launching Task for the following resource TaskLaunchRequest{uri='maven://com.xxx:fis.ifx.event-task:jar:1.0-SNAPSHOT', commandlineArguments=[], environmentProperties={payload={"statusCode":0,"fisT
opic":"CustomerDataUpdated","payloadId":"CUSTADDRMODR``Q_V15","customerIds":[1597304]}}, deploymentProperties={}}
Before I begin, you have a number of questions here. In the future, it's better to break them up into multiple questions so that they are easier to find by other users and easier to answer. That being said:
A little context on the current state of things
In order to understand how things will work, it's important to understand the current state of things. The current releases of the software involved are:
Pivotal Cloud Foundry (PCF) - 1.7.12. This version is required for any task support.
Spring Cloud Task (SCT) - 1.0.2.RELEASE
Spring Cloud Data Flow CF (SCDF) - 1.0.0.BUILD-SNAPSHOT (current as of the date of this post).
Currently PCF 1.7.12+ has all the capabilities to run tasks. You can create v3 applications (the type of application used to launch a task), run it as a task, etc. However, the tooling around that functionality is not currently complete. There is no support for v3 applications in Apps Manager or the CLI. There is a plugin for the CLI that is more of a dev tool that can be used to help with some functions (it will show you logs, etc), but it is not fully functional and requires a specific version of the CLI to work [1]. This is one of the reasons that the task functionality within PCF is still considered experimental.
Spring Cloud Task is currently GA and supports all the functionality needed to effectively run tasks on CF. However, it's important to note that SCT doesn't handle orchestration so the actual launching of tasks on CF is the responsibility of either the user, or Spring Cloud Data Flow (the easier route).
Spring Cloud Data Flow's Cloud Foundry server implementation currently has functionality to launch tasks on PCF in the latest snapshots. We have validated this against 1.7.12 as well as the development branch of 1.8.
The task workflow within SCDF
Tasks are fundamentally different from stream applications within the context of SCDF. When you create a stream definition, you are given the option to deploy it. What this does is it actually downloads the Spring Boot über jars and deploys them to PCF as long running processes. If they go down, PCF, will relaunch them as expected, etc.
Tasks on the other hand, are not deployed. They are launched. The difference is that while you create a task definition, there is nothing deployed until you click launch. And when the task completes, the software is shut down and cleaned up. So while a stream definition may have states, it's really a one to one relationship between the definition and the deployed software. Where with a task, you can launch a task definition as many times as you want.
Your issues
Reading through your post, I see a few things that you are struggling with. Let me see if I can help:
Task Definitions within SCDF and launching them via a stream - When launching a task from a stream, the task registry within SCDF is not used. The sink expects the URL for the resource to be within the TaskLauchRequest.
Apps Manager and tasks - As mentioned above, there is no support for v3 applications in Apps Manager yet so you won't be able to see your tasks there.
Viewing the logs - In order to debug what's going wrong with launching your task on CF, you're going to want to view the logs. To do so, use the v3 CLI plugin mentioned above to view them. It's important to note that you can only tail live logs with the plugin, not view logs that have previously been rendered. Because of that, when testing, you'll want to tail the logs as soon as the app is created, before it's launched.
Error in SCDF Shell - The error you received from the SCDF shell (CF-UnprocessableEntity(10008):...) leads me to wonder if you have both the correct version of PCF (1.7.12+) and the correct version of the following other libraries:
spring-cloud-deployer-cloudfoundry - The latest snapshots
cf-java-client - 2.0.0.M10+
reactor-core - 3.0.0.RC1+
I hope this helps!
[1] https://github.com/cloudfoundry/v3-cli-plugin
Task support is not available in 1.0.0.M4 release of SCDF's CF-server. In this release, the task commands/REST-APIs should be disabled - see here. And for that reason, you wouldn't see any docs related to Tasks in the 1.0.0.M4 reference guide.
That said, the Task support is available/enabled in the BUILD-SNAPSHOT release. If you're locally building the CF-server and upon pushing it to CF, you could take advantage the task commands in the shell to create and launch task definitions.
I have created a console app and managed to upload it to the cloud, and I have scheduled it to run every 15 minutes. The console app runs for the first time with success as result and thens fails stating an error in the connection string. Could someone shed light on this please. Would be greatly apprecited.
Thanks
The error message follows:enter image description here
Make sure that you are setting a connection string named AzureJobsRuntime in your Windows Azure Website configuration with a value similar to DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY pointing to the Windows Azure Storage account where the Windows Azure WebJobs Runtime logs are stored.
Please visit the article about configuring connection strings for more information on how you can configure connection strings in your Windows Azure Website.
To clarify a couple of possible gotchas (adding to the accepted answer):
Set these values by going to
App Services -> Your Web App -> Settings / "All Settings" -> Application Settings -> (In page under header) Connection strings
There you'll find Name, Value, and a Type drop down.
Name: Do NOT put your storage account name here! Rather, this is where you put AzureWebJobsDashboard for one connection string and on the next AzureWebJobsStorage. The value for these should look like:
DefaultEndpointsProtocol=https;AccountName=<mysupercoolblobstorageaccountname>;AccountKey=<blahblah==>
-- Old Portal --
I've had problems with this before where it was fixed in the old portal, so for that sake:
Old Portal: Your website -> Configure tab -> under 'connection strings', enter two new values: a) dropdown type CUSTOM, for NAME do NOT enter the name of your storage account! rather Name is: 'AzureWebJobsDashboard' or for the other (enter two entries): 'AzureWebJobsStorage'.
You need to set AzureJobsRuntime as a connection string (for an Azure storage account), you can do that on the Azure portal under: Websites --> Your Website --> CONFIGURE tab --> connection strings.
Web Job is not able to figure out the connectionString value in appsettings.json file. There could be two scenario:
If you are using an emulator, try adding this in your appsettings.json file
{
"ConnectionStrings": {
"AzureWebJobsDashboard": "UseDevelopmentStorage=true",
"AzureWebJobsStorage": "UseDevelopmentStorage=true"
}
}
If you are trying to connect directly to your Azure portal
{
"ConnectionStrings": {
"AzureWebJobsDashboard": "url",
"AzureWebJobsStorage": "url"
}
}