I'm adding to an open source project that uses in this case some Azure cloud functionality, but the same general problem is applicable for any cloud API. I want to write tests for my code, but the results of the test are reliant on something having happened in the cloud service I'm using, and in order for this to happen, I need to supply credentials to the cloud service. In a private project, I could certainly just add my cloud credentials to the testing environment, but for public/open source projects, I can't do this. I can test locally easily enough, but this project uses CI (as do many OSS projects), so this can't really be done.
One approach seems to be using mock or something similar, but that doesn't actually seem to test that things are happening as they should be, and strikes me as a mostly pointless method to achieve 100% coverage.
Are there any 'virtual test cloud' environments that can be spun up to create an identical interface to the cloud service in question, but only for testing? How do these deal with side effects (the code in question creates a DNS entry, and ideally would test for the actual existence of a DNS entry using the system's resolver rather than another cloud call)?
How do people do this kind of testing?
I start with a spike solution to learn how to pass the required credentials. With this knowledge, I can TDD an acceptance test to call a simple API and get a "success" result.
I exclude the credentials from my repository. Instead, I include a template file with instructions.
From there, I drop down to unit tests to TDD sending requests and receiving responses. I don't test actual communication with any service. Instead:
Test the contents of requests.
Create responses and test how they're handled. This makes it really easy to test all sorts of error conditions.
Once I've TDD'd credentials, requests, and responses, I use what I call a spike test to confirm that everything is in fact working. Basically, this uses non-automated confirmation in anything I can quickly hack together.
Related
When I run my Postman monitor it says there are no tests available, even though I have created a single test in the request.
Is this referring to some other type of test?
Have attached both images
Postman monitor error
Developed test
I have the similar issue.
Monitor is an online feature and is only available in workspaces.
I suspect the workspace is in Postman so it has issues to invoke my company's network which is protected with firewall. I tried to turn on and off the proxy but still have the issues.
I feel the newman may work which needs more efforts to implement.
There are a few reasons. The reasons are mentioned well in official doc.
https://learning.postman.com/docs/monitoring-your-api/troubleshooting-monitors/
According to my experience, you should not define "pm.response.json()" command as globally, you need to define it in test function (pm.test()) because saved global variables are not supported in monitors.
I'm starting to teach myself serverless development, using AWS Lambda and the Serverless CLI. So far, all is going great. However, I've got a snag with acceptance testing.
What I'm doing so far is:
Deploy stack to AWS with a generated stage name - I'm using the CI job ID for this
Run all the tests against this deployment
Remove the deployment
Deploy the stack to AWS with the "Dev" stage name
This is fine, until I need some data.
Testing endpoints without data is easy - that's the default state. So I can test that GET /users/badid returns a 404.
What's the typical way of setting up test data for the tests?
In my normal development I do this by running a full stack - UI, services, databases - in a local docker compose stack and the tests can talk to them directly. Is that the process to follow here - Have the tests talk directly to the varied AWS data stores? If so, how do you handle multiple (DynamoDB) tables across different CF stacks, e.g. for testing the UI?
If that's not the normal way to do it, what is?
Also, is there a standard way to clear out data between tests? I can't safely test a search endpoint if the data isn't constant for that test, for example. (If data isn't cleared out then the data in the system will be dependent on the order the tests run in, which is bad)
Cheers
Since, this is about Acceptance tests - those should be designed to care less of the architecture (tech side) and more of the business value. After all, such tests are supposed to be Black box. Speaking from experience with both, SLS or mSOA, the test setup and challenges are quite similar.
What's the typical way of setting up test data for the tests?
There are many ways/patterns to do the job here, depending on your context. The ones that most worked for me are:
Database Sandbox to provide a separate test database for each test run.
Table Truncation Teardown which truncates the tables modified during the test to tear down the fixture.
Fixture Setup Patterns will help you build your prerequisites depending of test run needs
You can look at Fixture Teardown Patterns for a
standard way to clear out data between tests
Maybe, you don't need to
Have the tests talk directly to the varied AWS data stores
as you might create an unrealistic state, if you can just hit the APIs/endpoints to do the job for you. For example, instead of managing multiple DynamoDB instances' PutItem calls - simply hit the register new user API. More info on Back door manipulation layer here.
I am using AWS API Gateway with Lambda/ECS for compute and Cognito for users. But I find it really hard to test such applications. With AWS SAM Local I maybe able to test simple Lambda and API gateway functionality but if I use things like API Gateway authorizers I find it hard to test these end to end.
Looks like to test such applications, I need an entire new setup just for testing? I mean like a separate API Gateway with Lambda/ECS cluster/Cognito user pool just to enable testing? This seems very slow, and I think I will not be able to get things like a code coverage report anymore?
Disclaimer: I'm fairly new to AWS Lambda/ECS/Cognito so take this with a grain of salt.
Unit Tests: SAM Local or some other local docker hosting with a unit testing library (mocha) would be good for this because:
Speed. All your tests should execute quickly against a lambda function
Example : wildrydes with mocha
Integration Tests: Once you stage your changes, there's a bunch of options calling the API. I'd start off with postman to run the API tests and you can chain them together or run them in command line if needed.
End to End (E2E) tests: If the API is your front end then there might not be any difference between E2E and API tests. UI, Voice, Chat front ends differ significantly as do the options so I'll suggest some options:
UI : Selenium (has the most support and options available to you including docker images: Selenium Hub or standalone)
Voice: Suggestions?
Text: Suggestions?
Step functions :
allows you to visualize each step
retries when there are errors
allows you to diagnose and debug problems
X-Ray: collects data about requests that your app serves, and provides tools you can use to view
As for code coverage, I'm not sure how you currently do code coverage. Something like this npm run coverage, maybe?
I am assuming you are using cloudformation for deploying such an extensive stack and the following answer is based on that assumption.
Thus in addition to the #lloyd's answer, I would like to add that you can add custom resources within your cloudformation template for testing each individual lambdas or even api endpoints.
Also for lambda, you can use Deployment Preferences Hooks to test your serverless lambdas before and after moving your lambda to the new version.
https://github.com/awslabs/serverless-application-model/blob/release/v1.8.0/docs/safe_lambda_deployments.rst
I have an application that sends messages to an external web service. I build and deploy this application using MSBuild and Cruisecontrol.NET. As CCNET build and deploys the app it also runs a set of test using NUnit. I'd now like to test the web service communication as well.
My idea is that as part of the build process a web service should be generated (based on the external web services WSDL) and deployed to the build servers local web server. All the web service should do is to receive the message and place it on the file system so I then can check it using ordinary NUnit for example. This would also make development easier as new developers would only have to run the build script and be up and running (not have to spend time to set up a connection to the third party service).
Are there any existing utilities out there that easily mock a web service based on a WSDL? Anyone done something similar using MSBuild?
Are there other ways of testing this scenario?
I just started looking into http://www.soapui.org/ and it seems like it will work nicely for testing web services.
Also, maybe look at adding an abstraction layer in your web service, each service call would directly call a testable method (outside of the web scope)? I just did this with a bigger project I'm working on, and it's testability is working nicely.
In general, a very good way to test things like this is to use mock objects.
At work, we use the product TypeMock to test things like Web Service communication and other outside dependencies. It costs money, so for that reason it may not be suitable for your needs, but I think it's a fantastic product. I can tell you from personal experience that it integrates very well with NUnit and CCNet.
It's got a really simple syntax where you basically say "when this method/property is called, I want you to return this value instead." It's great for testing things like network failures, files not being present, and of course, web services.
Take a look at NMock2. It's a open-source mocking product and allows you to create "virtual" implementations for interfaces that support rich and deep interaction.
For example, if your WS interface is called IService and has a Data GetData() method, you can create a mock that requires the method to be called once and returns a new Data object:
var testService = mockery.NewMock<IService>();
Expect
.Once
.On(testService)
.Method("GetService")
.WithNoArguments()
.Will(
Return.Value(new Data());
At the end of the test, call mockery.VerifyAllExpectationsHaveBeenMet() to assure that the GetData method was actually called.
P.S.: don't confuse the "NMock2" project with the "NMock RC2", which is also called "nmock2" on sourceforge. NMock2-the-project seems to have superseded NMock.
This might also be something - MockingBird. Look useful.
At my work place we are using Typemock and nUnit for our unit testing.
I'm familiar with TDD and use it in both my workplace and my home-brewed web applications. However, every time I have used TDD in a web application, I have had the luxury of having full access to the web server. That means that I can update the server then run my unit tests directly from the server. My question is, if you are using a third party web host, how do you run your unit tests on them?
You could argue that if your app is designed well and your build process is sound and automated, that running unit tests on your production server isn't necessary, but personally I like the peace of mind in knowing that everything is still "green" after a major update.
For everyone who has responded with "just test before you deploy" and "don't you have a staging server?", I understand where you're coming from. I do have a staging server and a CI process set up. My unit tests do run and I make sure they all pass before an an update to production.
I realize that in a perfect world I wouldn't be concerned with this. But I've seen it happen before. If a file is left out of the update or a SQL script isn't run, the effects are immediately apparent when running your unit tests but can go unnoticed for quite some time without them.
What I'm asking here is if there is any way, if only to satisfy my own compulsive desires, to run a unit test on a server that I cannot install applications on or remote into (e.g. one which I will only have FTP access to in order to update files)?
I think I probably would have to argue that running unit tests on your production server isn't really part of TDD because by the time you deploy to your production environment technically speaking, you're past "development".
I'm quite a stickler for TDD, and when I'm preaching the benefits to clients I often find myself saying "you can't half adopt TDD, it's all or nothing"
What you probably should have is some form of automated testing that you perform "after" deployment but these are not part of TDD.
Maybe you should look at your process again.
You could write functional tests in something like WATIR, WATIN or Selenium that test what is returned in the reponse page after posting certain form data or requesting specific URLs.
For clarification: what sort of access do you have to your web server? FTP or WebDAV only? From your question, I'm guessing ssh access isn't available - you're dropping files in a directory to deploy. Is that correct?
If so, the answer for unit testing is likely 'do it before you deploy'. You can set up functional testing driven by an automated tool like Selenium to test your app remotely via the web interface, but that's not really unit testing the sense that you're restricted to testing the system as a whole.
Have you considered setting up a staging server, perhaps as a VMWare instance, that mirrors or at least mimics your deployment environment?
What's preventing you from running unit tests on the server? If you can upload your production code and let it run there, why can't you upload this other code and run it as well?
I've written test tools for sites using python and httplib/urllib2 generally it would have been overkill but it was suitable in these cases. Not sure it's going to be of general use though.