I work on a project with a lot of legacy code that has some tests (in JUnit). Those unit tests are pretty slow and running them often slows me down. I plan on refactoring and optimizing them (so they're real unit tests), but before I do this, I'd like to speed them up just for now, by running them in parallel. Of course I could get myself some cluster and run them there, but that's not worth the hassle and the money. Is it possible to do so with some cloud services like e.g. Amazon AWS? Can you recommend me some articles where I could read more about it?
Running unit tests in parallel requires 2 things:
Creating groups of multiple tests using either suites or JUnit categories
A way to launch each group of tests
Once you have #1, an easy way to get #2 is to use Continuous Integration.
Do you already use Continuous Integration ?
If not, it is easy to add. One of the more popular ones is Jenkins
Jenkins also supports distributed builds where a master code launches build jobs on slave nodes. You could use this to run each of your groups in parallel.
Continuous Integration in the Cloud
Once you have Continuous Integration set up, you can move it into the cloud if you wish.
If you want to stick with jenkins, you can use
Cloud Bees which costs money
Red Hat Openshift "Gears" can be used to run Jenkins. I think it is possible to use gears in Amazon WS but I've never done it. You can run a few gears with limited memory usage for free, but if you need more it will cost money.
If you don't want to use Jenkins, there are other cloud based CIs including
Travis-CI which integrates with github. Here is an article about how to speed up builds in Travis
Related
Hy There!
Please excuse me for not knowing mutch about GitLab, I'll summ up what do I wish for and please tell me if it is possible or not. And if yes, then refer a how to for me please. :slight_smile:
I wish to implement GitLab to store our repos which are mainly Laravel 5 projects. Also I wish to run some tests on them, like PHPUnit test, Behat, etc. For this, I currently use the docer ability of GitLab to build a project. It puts the files into a docker, there I have to run composer install, and a few other things. But this takes soooo long! It just slows down the development.
Is it possible to: Run "composer install" and "npm install" and other things that we need to set up the website ONCE on the repository, and from then on, we can do only the testing.
After you setup docker to cache the dependency downloads, your next step is to move your runners to a new host, or give your current host more RAM.
I'm using GitLab's Omnibus installation and my GitLab instance uses 1.7GB of RAM with very little traffic, and my runners use up to 1GB when running some of my builds and tests. If your GitLab instance and runners have a similar memory footprint, then your machine will start to use the swap memory during tests and that will really slow down your runners.
Also, your runners likely have high CPU usage when running tests, and add on top of that the CPU required when your system is using swap memory, and you start to slow down there too.
I would recommend moving the runners to a different machine, for performance and security reasons. If you can't do that, then at least increase the RAM to 3GB.
I am currently looking for a solution to run arbitrary scripts on a cloud instance (aws, digitalocean, rackspace, I'm not picky). I am not doing something shady I simply want to use it for performance testing and need reproducible results (deploy a service, set specific testdata, run the performance tests, kill everything, repeat if necessary).
Of course I can use the API of these providers and build a custom solution, but I'm wondering if there is a framework or a bunch of tools that will help me with that.
What I need is:
- Only using an instance for the runtime of the script
- possibility to store data outside of the instance for result analysis
There are a lot of tools to automate setting up cloud instances but they all seem targeted for deployment purposes. What I need is a cloud script runner.
From your description is sounds like you might be looking for something like AWS's new(ish) Lambda service.
This allows you define scripts and triggers to run them in he clued without the overhead of spinning up and having to manage cloud compute 'servers'.
More info:
https://aws.amazon.com/lambda/
One thing to be careful of when using the cloud for performance testing - you have no teal control over the actual HW that your code will run on and different runs may run on different HW. This is true even for server or instance based cloud testing.
This may seem like a very broad question, but i am really interested to know about possible approaches. Our team has a Django Web app and we have huge amount of unit tests for our features. Now in github, we have master branch, develop branch, and individual feature/bug branches. Now the problem i want to solve is,
Every time some code is merged into develop branch, i want to run all(or subset) of unit tests against that branch. It would be cool to have it automated, i-e i do not have to trigger the test run.
I have read and heard about Jenkins - http://michal.karzynski.pl/blog/2014/04/19/continuous-integration-server-for-django-using-jenkins/. Currently one of the approaches i am leaning towards.
But i wanted to know if there are better approaches or tools which i can use.
Appreciate all your help.
For what it's worth, you can't really go wrong with Jenkins for the functionality you are looking to achieve.
Although Travis CI may be a better option given that it's meant to work seamlessly with Github and it appears all of your repositories have been moved to Github.
Really depends on your business needs though.
Getting Jenkins up and running, from past experiences, has always gone very smoothly and it gives you the benefit of keeping all data in house as you have the option to host Jenkins on your own private servers but probably doesn't scale or run as efficiently as Travis CI does depending on your setup.
Travis CI will probably allow for an even more seamless approach because it's already being hosted for you and tied directly into Github, but you won't get the privacy as running Jenkins on your own servers. There is a paid option though it appears for Travis CI which again, depending on your business needs, may be a better option.
When developing, my team obviously uses development as our environment.
When we run automated tests, we use testing.
We also have staging and production environments, respectively used for our testers to check out features and the final "live" product.
We're trying to setup an internal CI server to run our automated tests against and to eventually assist with automated deployments.
Since the CI server is really running automated tests, some think it should be run in testing environment. However, in order for the CI server to actually be useful, my thoughts are that it needs to be run in production mode with as close-as-possible a mirror of the actual production environment (without touching the production DB, obviously).
Is there an accepted environment that a CI server should be executed under? production environment (with different DB) seems the only logical answer to me, but I may be missing something...
Running any tests on PROD environment as you said
seems the only logical answer
but is not quite true. There are risks that your tests can seriously damage the actual environment/application to a point where you'll face a recovery option. After all the dark side of testing is to show/find that your software has not only minor bugs and it is working not as expected.
I can think of at least these 'why not test production' considerations:
when the product is launched, the customer rely on it. Expecting that your software is working ()being already tested). Your live environment should do its job and not be loaded with tests. If the product misbehaved (or did not perform), the technical team have to be sent to to cover the damage, fix the gaps and make it run hassle free. Now this not only affected the product cost, but delayed the project deadlines in a major way. This will make a recursive effect at the vendors profits and next few projects.
the production or development team when completes a product development at their end, have to produce this test environment for testing team prior to loading their newly developed product on that environment for testing.
To me, no matter that you
also have staging and production environments
it is essential to use the Test one accordingly. Further more Testing environment should be (configured) as close as it gets to the Production. Also one person could be trying to test while another person breaks the thing that he has been testing. With out the two being separate their is no way to do proper testing.
Just to be full answer, your STAGE environment can have different roles depending on the company.
One is that it can be the QA/STAGE environment that has an exact copy of production which is used for both QA and system testing (testing of the system when a lot of updates/changes or upgrade is going to go into production).
UPDATE:
That was my point too. The QA environment should be a mirror of the PROD. Possible solution about your issue with caching/pre-loading files onto staging/production is creation of pre-/post-steps .bat (let's assume) files.
In our current Test project we use this approach. In pre-steps we set-up files needed for test execution (like removing files from previous runs and downloading latest copies/artifacts). In post-steps we set up reporting files needed.The advantage is that your files will be collected and sync before every execution.
About the
not on the same physical hardware
in my case we support dedicated remote Test server. Advantages are clear, only thing that you need to be considered is that it'll require maintenance (administration).
Within a short-time period I'm going to start a project based on Windows Azure. And I was wondering what are the experiences with testing for Windows Azure projects (in continuous intergration (with a TFS build server))? (Eventual using TDD)
Some things I was wondering:
Do you use mocking (in your own written wrapper class)?
Do you use the storage emulator?
Do you deploy the services to Azure and run the tests from the build server to the cloud? (what about costs)?
Thnaks in advance!
The same good practices for writing unit tests for applications outside of Windows Azure apply. If you have an external dependency to what you are actually testing, that dependency should be mocked and injected for your granular unit test.
For example, when I'm using Windows Azure Storage Queues I will have an interface that I use to interact with the queue itself, so in my code consuming the queue service I can mock the subsystem using the interface and use dependency injection to inject the mock. This removes the necessity to actually deal with the emulator during unit tests. For the most part the actual concrete implementation of the code working with the queue is not much more than a very thin wrapper.
I personally don't shoot for 100% test coverage, so I may not have direct unit tests that utilize the concrete implementation of the wrappers. In many cases I try to have integration tests that will exercise these wrappers and exercise multiple aspects of the system working together. In some cases I can run the integration tests in the emulator (for Storage operations for example), but in some cases they simply have to be run with access to the Windows Azure environment (in the case of usage of ACS or Service Bus).
Ideally you'd like to have a set of scripts that can be run to spin up a minimum set of test servers in Azure, deploy your solution and exercise the integration tests that can't be done on premises. Then get the results of that and have the script shut everything down (or optionally leave it running if you need that). Then run the integration tests suite that utilizes these scripts often enough to detect issues, but you certainly don't need to run them every time you check something in unless you are happy with running the test environment all the time. If you okay with the cost of a semi-permanent test environment running in Azure then just make sure to have the scripts to an update deployment rather than a delete and redeploy to cut down on cost a bit (savings would be relative to how often the deploy occurs).
I believe this question is a very subjective one as you're likely to get several different opinions.