I'm trying to unit test my microservices before deployment, however in the Skaffold pipeline images are tested before deployment. The issue here is that the code in my images are dependent on configs and credentials that are mounted from k8s configmaps and secrets, therefore tests will always fail if unit tests are run before deployment.
How do I run unit tests for microservices with skaffold? How are unit tests normally run for microservices? Looking around the net, no one seems to have a straight answer.
Any guidance will be much appreciated.
There are several strategies I've used or seen people use for end-to-end testing of a deployment:
In-cluster testing, where you deploy one or more jobs that perform tests, and then use the job status to communicate test success/failure. We use this approach to test the container-debug-support images that are used by skaffold debug. We run a little script that builds and deploys a set of images in a integration profile that includes a set of test Jobs (example). The script then waits for these jobs to complete with kubectl wait.
External to the cluster, where you access exposed services: this can be accomplished using after-deploy hooks. You can use this to do some smoke testing or load testing. This approach works if you're planning to keep the app running; I'm not sure that tearing down the app within the hooks is a great idea.
skaffold run and skaffold delete are helpful primitives for staging and tearing down an application.
Related
My question is regarding Helm tests and their basic philosophy.
The simple question is, should the helm test pod be a "full copy" of the production pod?
Usually, in all examples I have seen, the test pod uses a stripped-down Dockerfile with essential things like curl, and the test.yaml file only includes host/port/secrets to a service. The test itself usually only focuses on pinging or connecting to a service.
But is this only what the helm test philosophy is about? Was the idea behind helm tests that the test pod should be a full replica of the production pod and run basically any test you wish?
Hopefully you've run some more targeted tests on your pod before you're building and deploying an image. The sequence I generally use looks something like this:
Outside a container, run the service's unit tests (RSpec/JUnit/PyTest/Jest/catch2/shunit/...).
Once those tests pass, build the Docker image and push it to a repository.
Deploy the candidate image to a test cluster.
Run integration tests making pure HTTP requests, or posting messages to a Kafka or RabbitMQ queue, and observe the results from the outside.
Promote the image to production.
In this sequence, notice that the set of tests that depend on the code proper (the unit tests) run before we build a container at all, and the tests against the deployed system (the integration tests) test the external interface of the container in the deployed environment. In neither case do you need the application code, the tests, and "in a container" all at the same time.
I'd expect helm test tests to be maybe a little more of a smoke test than a full integration test; check that the service is alive and can answer requests with a correct format. It's true that the test is an arbitrary Job and can run an arbitrary container with arbitrary code, and that could include the full application as its image. But running tests on something "next to" the deployed pod isn't the same as running tests on the deployed pod itself.
I'm starting to teach myself serverless development, using AWS Lambda and the Serverless CLI. So far, all is going great. However, I've got a snag with acceptance testing.
What I'm doing so far is:
Deploy stack to AWS with a generated stage name - I'm using the CI job ID for this
Run all the tests against this deployment
Remove the deployment
Deploy the stack to AWS with the "Dev" stage name
This is fine, until I need some data.
Testing endpoints without data is easy - that's the default state. So I can test that GET /users/badid returns a 404.
What's the typical way of setting up test data for the tests?
In my normal development I do this by running a full stack - UI, services, databases - in a local docker compose stack and the tests can talk to them directly. Is that the process to follow here - Have the tests talk directly to the varied AWS data stores? If so, how do you handle multiple (DynamoDB) tables across different CF stacks, e.g. for testing the UI?
If that's not the normal way to do it, what is?
Also, is there a standard way to clear out data between tests? I can't safely test a search endpoint if the data isn't constant for that test, for example. (If data isn't cleared out then the data in the system will be dependent on the order the tests run in, which is bad)
Cheers
Since, this is about Acceptance tests - those should be designed to care less of the architecture (tech side) and more of the business value. After all, such tests are supposed to be Black box. Speaking from experience with both, SLS or mSOA, the test setup and challenges are quite similar.
What's the typical way of setting up test data for the tests?
There are many ways/patterns to do the job here, depending on your context. The ones that most worked for me are:
Database Sandbox to provide a separate test database for each test run.
Table Truncation Teardown which truncates the tables modified during the test to tear down the fixture.
Fixture Setup Patterns will help you build your prerequisites depending of test run needs
You can look at Fixture Teardown Patterns for a
standard way to clear out data between tests
Maybe, you don't need to
Have the tests talk directly to the varied AWS data stores
as you might create an unrealistic state, if you can just hit the APIs/endpoints to do the job for you. For example, instead of managing multiple DynamoDB instances' PutItem calls - simply hit the register new user API. More info on Back door manipulation layer here.
The question is more about the fundamental understanding of a normal/ideal CI flow and understanding the scope of integration testing in it.
As per my understanding, the basic CI CD flow is
UnitTesting --> IntegrationTesting --> Build Artifact --> Deploy to Dev/Sandbox or any other subsequent environments.
So unit tetsing and integration testing collectively decide/make sure if the build is stable and ready to be deployed.
But, recently, we had this discussion in my team where we wanted to run integration tests on deployed instances on Dev/Sandbox etc , so as to verify if the application is working fine after deployment.
And the microsoft's article on Build - Deploy - Test workflows suggests that this could be a possible way.
So , my questions are :-
Are integration tests supposed to test configuration of different environments ?
Are integration tests supposed to be run before packaging application or deploying the application ?
If at all, some automated testing is required to test deployed application functioning on all environments ?
If not integration tests then what could be alternative solutions
You're mixing Integration testing with System testing.
Integration testing checks that some components can work together (can be integrated). You may have integration tests to verify how does the Data layer API operates with a database; or how does the the Web API responds to HTTP calls. You might not have the entire system completely working in order to do integration testing of its components.
Unlike integration tests, the System tests require all the components to be implemented and configured. That is end-to-end testing (e.g. from a web request to a database record). This kind of testing requires the entire system to be deployed which makes them more 'real' but expensive.
I work on a project with a lot of legacy code that has some tests (in JUnit). Those unit tests are pretty slow and running them often slows me down. I plan on refactoring and optimizing them (so they're real unit tests), but before I do this, I'd like to speed them up just for now, by running them in parallel. Of course I could get myself some cluster and run them there, but that's not worth the hassle and the money. Is it possible to do so with some cloud services like e.g. Amazon AWS? Can you recommend me some articles where I could read more about it?
Running unit tests in parallel requires 2 things:
Creating groups of multiple tests using either suites or JUnit categories
A way to launch each group of tests
Once you have #1, an easy way to get #2 is to use Continuous Integration.
Do you already use Continuous Integration ?
If not, it is easy to add. One of the more popular ones is Jenkins
Jenkins also supports distributed builds where a master code launches build jobs on slave nodes. You could use this to run each of your groups in parallel.
Continuous Integration in the Cloud
Once you have Continuous Integration set up, you can move it into the cloud if you wish.
If you want to stick with jenkins, you can use
Cloud Bees which costs money
Red Hat Openshift "Gears" can be used to run Jenkins. I think it is possible to use gears in Amazon WS but I've never done it. You can run a few gears with limited memory usage for free, but if you need more it will cost money.
If you don't want to use Jenkins, there are other cloud based CIs including
Travis-CI which integrates with github. Here is an article about how to speed up builds in Travis
I'm looking at setting up a continuous integration solution for a Django framework project hosted in AWS. Their are client and server tests. The client tests will fire up a headless browser, but it would also be good if it could fire up real browsers like Chrome and Firefox.
Repo hosted on BitBucket. What would be great is if code is committed to a branch, then the CI setup will pull the changes and run the tests in an environment that closely matches production.
Can you help me set up the best possible stack given the above constraints?
I think Fabric would be suitable for this. Fabric allows you to run code on your local machine and the server. Once you learn the basics of Fabric, you will see that you can run a command like 'fab deploy' and Fabric will commit your changes to a branch, then pull the changes onto the server.
You could have steps in between to launch a new server, prepare the environment, then run tests against the code. I have been using Fabric to simplify some deployments and have found it a great experience. Because of the flexibility fabric gives you, you can pretty much deploy however you want, whether it's onto a test server, QA, live etc...