we have quite a lot of RPG-programs here, and we do a lot of automated testing, but we are not very good yet in combining those two. Are there good ways to do automated testing on RPG programs -- or on any other ILE programs for that matter?
I am aware of a project named RPGUnit, but that has it's last update in 2007. However, it seems it is still used, since RPG Next Gen is currently putting some work in including it.
What's you experience with those? Is there something else, that I am missing, like some great sofware tool google fails to find?
I'm interested in unit testing as well as integration testing of complete projects. Anything that integrates with tools like jenkins is welcome. If it involves IBM's Rational Developer or System i Navigator, that's okay, too.
We are in an early phase of creating new testing plans for our RPG development process, and I don't want it, to head in the wrong direction right from the start.
You probably already know how broad a subject 'testing' can be. IBM have a product called Rational Function Tester (I haven't used it) http://www-01.ibm.com/software/awdtools/tester/functional/ I myself use RPGUnit. No, it hasn't been updated recently but it still has all the pieces needed for testing subprocedures in the same way one would test Java methods.
Frankly, that's the easy part. The hard part is creating a test database and keeping it current enough to be representative of the production database. Rodin have some database tooling but I haven't the budget for those, so I roll my own more or less by hand. I use many SQL statements in a CL program to extract production data so I can maintain referential integrity. Then I use some more SQL to add my exceptional test cases - those relationships which aren't present in the production data but need to be tested for. Then I make a complete copy of the test database as a reference point. Then I run my test cases, which will update the test database. I've written a home grown CMPPFM utility that will allow me to compare the reference database against the now modified test database. This will show changes, but it still needs a lot of manual labour to review the comparisons to ensure that the proper rows got the proper updates. I haven't gone the extra mile to automate that yet. One big caveat is there are some columns you don't care about, like a change timestamp.
We went with RPGUNIT, and found it a good base to work from, but ended up extending it a lot to tie in with our Change Management system, and the way we work. I've written about the things we tried here: http://www.littlebluemonkey.com/blog/my-rpg-unit-test-journey
Related
So i have this huge SF2 project, which is luckily pretty 'OK' written. Services are there, background jobs are there, no god classes, it's testable--but, i never gotten any further than just unit-testing stuff, so the question is basically, where do i start taking this further.
The project consists of SF2 and all the yada yada, Doctrine2, Beanstalkd, Gaufrette, some other abstractions--its fine.
The one problem it has is some gluecode in controllers here and there, but i don't see it as a big problem since functional tests are going to me the main focus.
The infrastructure is setup pretty ok as well, its covered by docker so CI is going to work out well also.
But it has basically gotten too large to manually test any longer, so i want full functional coverage on short notice, and let the unit-testing grow over time. (Gonna dive into the isolated objects as they need future adjustments and build test for them in due course)
So i got the unit-testing covered, thats going to need to grow over time, but i want to make some steps towards the functional testing to get some quick gains on the testing dep. YESTERDAY.
My plan as of now is use Behat and Mink for this, the tests are going to be huge, so i might as well want to have it set as stories instead of code. Behat also seem to have a extension for Symfony' BrowserKit.
There are plenty of services and external things happening, but they are all isolated by services, so i can mock them through the test environment service config i guess.
Please some advice here if there is as better way
I'm also going to need fixtures, i'm using Alice for generating some fixtures so far, seems nice together with the doctrine extension, don't think there are "better" options on this one.
How should i test external services? Im mocking things as a Facebook service, but i also want to really test it to some test account, is this advisable? I know that this goes beyond its scope, the service has to be mocked and tested in every way possible to "ensure its working" according to the purist. But in the end of the day it still breaks because of some API key or other problem in the connection, which i cant afford really. So please advice here also
All your suggestions to use other tools are welcome ofcourse, and especially if there is a good book that covers my story.
I'm glad you brought up behat, I was going to suggest the same thing.
I would consider starting with your most business critical pieces; unit test the extremely important business logic and use behat on the rest.
For the most part, I would create stubs for your services that have expected output for expected input. That way you can create failures based on specific input. You can override your services in your test config.
Another approach would be to do very thin functional testing where you make GET requests to all of your endpoints and look for 200's. This is a very quick way to make sure that your pages are at least loading. From there, you can start writing tests for your POST endpoints and expanding your suite further with more detailed test cases.
My Django app is in need of some automated testing.
Many of the views produce tabular data (from a generic list view). I have created fixtures to test some of the more complex cases that have been causing subtle bugs.
What should I be using to test the value in a specific table cell (or column)?
There seems to be a lot of testing tools / libraries out there django-test client, Selenium, Nose. A lot of things seem to be aimed at unit testing (while I am not finding so many bugs at this level). I am looking more to integration testing. Reading all the documentation for all the libraries is going to take a while to find what I want.
So can someone advise what libraries / tools I should use to check the final output values in my list view's tabular output? I would like to give a URL, and confirm that the page returned has a value in particular row / column is equal to my expected value.
There seems to be a lot of testing tools / libraries out there django-test client, Selenium, Nose. A lot of things seem to be aimed at unit testing (while I am not finding so many bugs at this level). I am looking more to integration testing.
It works very well for integration testing. Maybe you would have found out if you had tried ? Maybe it's time for you to (re?) read about the correct hacker attitude
Also, I've been having loads of fun with ghost.py
Reading all the documentation for all the libraries is going to take a while to find what I want.
It's your work to do research as well. Believe it or not it took me like 5 hours to check all solutions yesterday and decide to go with ghost.py and get it to work nicely with django (hence the gist upload !).
But yeah, if you don't want to learn anything new then you're stuck at "Not knowing out to do integration testing". If you want to learn "how to make integration testing" then you have to do research. There's no secret my friend :)
I want to write an algorithm (a bunch of machine learning algorithms) in C/C++ or maybe in Java, possibly in Python. The language doesn't really matter to me - I'm familiar with all of the above.
What matters to me is the testing. I want to train my models using training data. So I have the test input and I know what the output should be and I compare it to the model's output. What kind of test is this? Is it a unit test? How do I approach the problem? I can see that I can write some code to check what I need checking but I want to separate testing from main code. Testing is a well developed field and I've seen this done before but I don't know the name and type of this particular kind of testing so that I can read up on it and not create a mess. I'd be grateful if you could let me know what this testing method is called.
Your best bet is watch the psychology of testing videos from the tetsing God http://misko.hevery.com/
Link of Misko videos:
http://misko.hevery.com/presentations/
And read this Google testing guide http://misko.hevery.com/code-reviewers-guide/
Edited:
Anyone can write tests, they are really simple and there is no magic to write a test, you can simply do something like:
var sut = new MyObject();
var res = sut.IsValid();
if(res != true)
{
throw new ApplicationException("message");
}
That is the theory of course these days we have tools to simplify the tests and we can write something like this:
new MyObject().IsValid().Should().BeTrue();
But what you should do is focus on writing testable code, that's the magic key
Just follow the psychology of testing videos from Misko to get you started
This sounds a lot like Test-Driven Development (TDD), where you create unit-tests ahead of the production code. There are many detailed answers around this site on both topics. I've linked to a couple of pertinent questions to get you started.
If your inputs/outputs are at the external interfaces of your full program, that's black box system testing. If you are going inside your program to zoom in on a particular function, e.g., a search function, providing inputs directly into the function and observing the behavior, that's unit testing. This could be done at function level and/or module level.
If you're writing a machine learning project, the testing and training process isn't really Test-Driven Development. Have you ever heard of co-evolution? You have a set puzzles for your learning system that are, themselves, evolving. Their fitness is determined by how much they confound your cases.
For example, I want to evolve a sorting network. My learning system is the programs that produce networks. My co-evolution system generates inputs that are difficult to sort. The sorting networks are rewarded for producing correct sorts and the co-evolutionary systems are rewarded for how many failures they trigger in the sorting networks.
I've done this with genetic programming projects and it worked quite well.
Probably back testing, which means you have some historical inputs and run your algorithm over them to evaluate the performance of your algorithm. The term you used yourself - training data - is more general and you could search for that to find some useful links.
Its Unit testing. the controllers are tested and the code is checked in and out without really messing up your development code. This process is also called a Test Driven Development(TDD) where your every development cycle is tested before going into the next software iteration or phase.
Although this is a very old post, my 2 cents :)
Once you've decided which algorithmic method to use (your "evaluation protocol", so to say) and tested your algorithm on unitary edge cases, you might be interested in ways to run your algorithm on several datasets and assert that the results are above a certain threshold (individually, or on average, etc.)
This tutorial explains how to do it within the pytest framework, that is the most popular testing framework within python. It is based on an example (comparing polynomial fitting algorithms on several datasets).
(I'm the author, feel free to provide feedback on the github page!)
I am going to pick up a task that no one has ever attempted to try at my workplace. It is a CF app first written using CF 2.0 (Yes, 2.0!) 10 yrs ago with > 10 cfscheduler tasks.. We explored the idea of rewriting the app, but 10 yrs of work simply can't be rewrote in 2-3 months.
What steps shall one take to modernize the app into a maintainable, extendable state? The one that I keep hearing is "write tests", but how can I write tests when it wasn't even in MVC?
Any advice would be appreciated, thanks!
p.s. I should thank Allaire, Macromedia and Adobe for keeping CF so freaking backward compatible all the way back to 2.0!
btw, what's the most modern, maintainable state for a CF app without MVC framework? or should my end goal be ultimately refactoring it into a MVC app?? I can't image how many links I will break if I do... seems impossible... thought?
update: found 2 related Q's...
https://softwareengineering.stackexchange.com/questions/6395/how-do-you-dive-into-large-code-bases
https://softwareengineering.stackexchange.com/questions/29788/how-do-you-dive-into-a-big-ball-of-mud
I am not sure if you need to move the whole site to a MVC application. Recently I did helped with an site that was not MVC, that still had a library with the Models, Services and Assemblers in a clean and organized manor. It worked great, and we didn't need to do anything more than what was necessary.
That being said, my first step would be to organize the spaghetti code into their different purposes. It may be hard to properly create the models, but at the very least you could break out the services like functions from the pages. With that done, it should be a lot cleaner already.
Then, I would try to take the repeated code and put them into custom tags. It will make the code more reusable, and easier to read.
Good Luck!
Consider, whether a full fledged framework is really necessary. In its most basic form a framework is merely highly organized code. So if procedural, that is well organized, works leave it.
Keep in mind something like FW/1 as migration path can be better than say Coldbox if you don't need all the other stuff.
Lastly, consider this I was able to migrate a 4.5 almost 70% of the way to Coldbox (very simple and really more about directory and file organization versus IOC, plugins, modules, etc...) just using a few extra lines per file plus onMissingMethod functions.
Good Luck.
I had to deal with a similar situation for about two years at my last job, however, it wasn't quite as old as yours. I think I was dealing with code from 4.0 on. There's no silver bullet here, and you'll need to be careful that you don't get too caught up in re-factoring the code and costing your company tons of money in the process. If the app works as it is rewriting it would be a pretty big wast of money.
What I did was update small chunks at a time, I wouldn't even refactor whole templates at a time, just small portions of one at a time. If I saw a particular ugly loop, or nested if statements I'd try to clean it up the best I could. If the app can be broken down into smaller modules or areas of functionality and you have the extra time you can try to clean up the code a module at a time.
A good practice I heard from the Hearding Code podcast is create a testing harness template that would use a particular cfm page that has a known output that you can re-run to make sure that it still has the same output once you've done refactoring. Its not nearly as granular as a unit test, but its something and something is almost always better than nothing, right?
I suspect that the reason this app hasn't been touched for years is because for the most part it works. So the old adage "if it ain't broken don't fix it" probably applies; However, code can always be improved :)
The first thing I'd do is switch to Application.cfc and add some good error logging. That way you may find out about things that need to be fixed, and also if you do make changes you're know if they break anything else.
The next thing I'd do is before you change any code is use selenium to create some tests - it can be used as a FireFox plugin and will record what you do. It's really good for testing legacy apps without much work on your part.
Chances are that you won't have much if any protection from SQL injection attacks so you will want to add cfqueryparam to everything!!
After that I'd be looking for duplicated code - eliminating duplicate code is going to make maintenance easier.
Good luck!
Funnily enough, I'm currently involved in converting an old CF app into an MVC3 application.
Now this isn't CF2, it was updated as recently as a year ago so all of this may not apply at all to your scenario, apologies if this is the case.
The main thing I had to do consolidate the mixed up CFQuerys and their calls into logical units of code that I could then start porting in functionality either to C# or JavaScript.
Thankfully this was a very simple application, the majority of the logic was called on a database using the DWR Ajax library; that which wasn't was mostly consolidated in a functions.cfm file.
Obviously a lot of that behavior doesn't need to be replicated as packaging up the separate components of logic (such as they were) in the CF app did map quite neatly to the various Partial Views and Editor Templates that I envisaged in the MVC application.
After that, it was simply a case of, page by page, finding out which logic was called when, what it relied upon that then finally creating a series of UML class and sequence diagrams.
Honestly though, I think I gained the most ground when I simply hit File-New Project and started trying to replicate the behavior of the app from the top of index.cfm.
I would break logical parts of the app into CFC's
Pick a single view, look at the logic within. Move that out to a CFC and invoke it.
Keep doing that you will have something much easier to work with that can be plugged into an MVC later. Its almost no work to do this, just copy and paste sections of code and call them.
You can consider using object factory to layer your application. We have similar situation at work and we started refactoring by putting Lightwire DI framework.
First we migrated all the sql statement into gateways, then we started using services and take a lot of code out of the templates to the services.
The work is not finished yet but the application is looking better already.
For large, really complex applications I'd prefer ColdBox for a re-factor project. However, I just saw a presentation at the D2W Conference on F/W 1 (Framework One), a VERY simple ColdFusion MVC framework. Check out code from the presentation here.
It's 1 (one) CFC file and a set of conventions for organizing your code. I highly recommend evaluating it for your project.
Ok, maybe I'm missing something, but I really don't see the point of Selenium. What is the point of opening the browser using code, clicking buttons using code, and checking for text using code? I read the website and I see how in theory it would be good to automatically unit test your web applications, but in the end doesn't it just take much more time to write all this code rather than just clicking around and visually verifying things work?
I don't get it...
It allows you to write functional tests in your "unit" testing framework (the issue is the naming of the later).
When you are testing your application through the browser you are usually testing the system fully integrated. Consider you already have to test your changes before committing them (smoke tests), you don't want to test it manually over and over.
Something really nice, is that you can automate your smoke tests, and QA can augment those. Pretty effective, as it reduces duplication of efforts and gets the whole team closer.
Ps as any practice that you are using the first time it has a learning curve, so it usually takes longer the first times. I also suggest you look at the Page Object pattern, it helps on keeping the tests clean.
Update 1: Notice that the tests will also run javascript on the pages, which helps testing highly dynamic pages. Also note that you can run it with different browsers, so you can check cross-browser issues(at least on the functional side, as you still need to check the visual).
Also note that as the amount of pages covered by tests builds up, you can create tests with complete cycles of interactions quickly. Using the Page Object pattern they look like:
LastPage aPage = somePage
.SomeAction()
.AnotherActionWithParams("somevalue")
//... other actions
.AnotherOneThatKeepsYouOnthePage();
// add some asserts using methods that give you info
// on LastPage (or that check the info is there).
// you can of course break the statements to add additional
// asserts on the multi-steps story.
It is important to understand that you go gradual about this. If it is an already built system, you add tests for features/changes you are working on. Adding more and more coverage along the way. Going manual instead, usually hides what you missed to test, so if you made a change that affects every single page and you will check a subset (as time doesn't allows), you know which ones you actually tested and QA can work from there (hopefully by adding even more tests).
This is a common thing that is said about unit testing in general. "I need to write twice as much code for testing?" The same principles apply here. The payoff is the ability to change your code and know that you aren't breaking anything.
Because you can repeat the SAME test over and over again.
If your application is even 50+ pages and you need to do frequent builds and test it against X number of major browsers it makes a lot of sense.
Imagine you have 50 pages, all with 10 links each, and some with multi-stage forms that require you to go through the forms, putting in about 100 different sets of information to verify that they work properly with all credit card numbers, all addresses in all countries, etc.
That's virtually impossible to test manually. It becomes so prone to human error that you can't guarantee the testing was done right, never mind what the testing proved about the thing being tested.
Moreover, if you follow a modern development model, with many developers all working on the same site in a disconnected, distributed fashion (some working on the site from their laptop while on a plane, for instance), then the human testers won't even be able to access it, much less have the patience to re-test every time a single developer tries something new.
On any decent size of website, tests HAVE to be automated.
The point is the same as for any kind of automated testing: writing the code may take more time than "just clicking around and visually verifying things work", maybe 10 or even 50 times more.
But any nontrivial application will have to be tested far more than 50 times eventually, and manual tests are an annoying chore that will likely be omitted or done shoddily under pressure, which results in bugs remaining undiscovered until just bfore (or after) important deadlines, which results in stressful all-night coding sessions or even outright monetary loss due to contract penalties.
Selenium (along with similar tools, like Watir) lets you run tests against the user interface of your Web app in ways that computers are good at: thousands of times overnight, or within seconds after every source checkin. (Note that there are plenty of other UI testing pieces that humans are much better at, such as noticing that some odd thing not directly related to the test is amiss.)
There are other ways to involve the whole stack of your app by looking at the generated HTML rather than launching a browser to render it, such as Webrat and Mechanize. Most of these don't have a way to interact with JavaScript-heavy UIs; Selenium has you somewhat covered here.
Selenium will record and re-run all of the manual clicking and typing you do to test your web application. Over and over.
Over time studies of myself have shown me that I tend to do fewer tests and start skipping some, or forgetting about them.
Selenium will instead take each test, run it, if it doesn't return what you expect it, it can let you know.
There is an upfront cost of time to record all these tests. I would recommend it like unit tests -- if you don't have it already, start using it with the most complex, touchy, or most updated parts of your code.
And if you save those tests as JUnit classes you can rerun them at your leisure, as part of your automated build, or in a poor man's load test using JMeter.
In a past job we used to unit test our web-app. If the web-app changes its look the tests don't need to be re-written. Record-and-replay type tests would all need to be re-done.
Why do you need Selenium? Because testers are human beings. They go home every day, can't always work weekends, take sickies, take public holidays, go on vacation every now and then, get bored doing repetitive tasks and can't always rely on them being around when you need them.
I'm not saying you should get rid of testers, but an automated UI testing tool complements system testers.
The point is the ability to automate what was before a manual and time consuming test. Yes, it takes time to write the tests, but once written, they can be run as often as the team wishes. Each time they are run, they are verifying that behavior of the web application is consistent. Selenium is not a perfect product, but it is very good at automating realistic user interaction with a browser.
If you do not like the Selenium approach, you can try HtmlUnit, I find it more useful and easy to integrate into existing unit tests.
For applications with rich web interfaces (like many GWT projects) Selenium/Windmill/WebDriver/etc is the way to create acceptance tests. In case of GWT/GXT, the final user interface code is in JavaScript so creating acceptance tests using normal junit test cases is basically out of question. With Selenium you can create test scenarios matching real user actions and expected results.
Based on my experience with Selenium it can reveal bugs in the application logic and user interface (in case your test cases are well written). Dealing with AJAX front ends requires some extra effort but it is still feasible.
I use it to test multi page forms as this takes the burden out of typing the same thing over and over again. And having the ability to check if certain elements are present is great. Again, using the form as an example your final selenium test could check if something like say "Thanks Mr. Rogers for ordering..." appears at the end of the ordering process.