In my understanding, design specifications help to formulate unit test cases which make use of internal knowledge (white-box techniques), while if we only have functional requirements, the black-box techniques are more suited.
What happens if we don't have design specifications, and the requirements are often vague or do not have defined boundaries? How will it affect the unit test process? And how do you compensate for it? Do you use your experience, or a specific practice/technique, to fill in the gaps?
Would developing first the functionality be better suited? As you keep developing and gain more knowledge of how it works internally, you can at some point use either white or black-box techniques. You use your previous experience on similar situations to complete the functionality, which means you may have as well written the functional requirement yourself. At this point, do you go for white-box? black-box? Or it depends on the risk/importance of the new functionality?
I'll guess that you're talking about TDD
Would developing first the functionality be better suited? As you keep developing and gain more knowledge of how it works internally, you can at some point use either white or black-box techniques.
or it'll be difficult to create unit tests for functionality that is missing. If we assume that their main idea is to exercise the
the smallest testable part of an application
and mainly are created by the developers. If the case is
don't have design specifications, and the requirements are often vague or do not have defined boundaries
than the code design and implementation are very likely to be with the same characteristics. And the unit tests wont be so much of a help. IMHO it's crucial to know what are building before testing it, even for TDD.
Does anyone know if there's standardized process for unit testing a new language.
I mean any new language will have basic flow control like IF, CASE etc.
How does one normally test the language itself?
Unit testing is one strategy to achieve a goal: verify that a piece of software meets a stated specification. Let's assume you are more interested in the goal, instead of exclusively using unit testing to achieve it.
The question of verifying that a language meets a specification or exhibits specific desirable qualities is profound. The earliest work led to type theory, in which one usually extends the language with new syntax and rules to allow one to talk about well-typed programs: programs that obey these new rules.
Hand-in-hand with these extensions are mathematical proofs demonstrating that any well-typed program will exhibit various desired qualities. For example, perhaps a well-typed program will never attempt to perform integer arithmetic on a string, or try to access an out-of-bounds element of an array.
By requiring programs to be well-typed before allowing them to execute, one can effectively extend these guarantees from well-typed programs to the language itself.
Type systems can be classified by the kinds of rules they include, which in turn determine their expressive power. For example, most typed languages in common use can verify my first case above, but not the second. With the added power comes greater complexity: their type verification algorithms are correspondingly harder to write, reason about, etc.
If you want to learn more, I suggest you read this book, which will take you from the foundations of functional programming up through the common type system families.
You could lookup what other languages do for testing. When I was developing a language I was thinking about doing something like Python. They have tests written in python itself.
You could lookup their tests. These are some of then: grammar, types, exceptions and so on.
Offcourse, there is a lot of useful stuff there if you are looking for examples, so I recomend that you dig in :).
As part of my lecture in C++ the students will have to solve assignments. Each solution shall implement the same functions with the same functionality and the same parameters (function name, return value, passing parameters). Only the code inside is different.
So i'm thinking about a way to test all solutions (around 30) in an effective way. Maybe the best way is to write a unit test as well as a shell script (or something similar) that compiles each solution once with the unit test and runs it.
But maybe there is a different and much better solution to this problem.
The reason why unit tests are one of the most efficient types of automated testing is because the return of investment is relatively small (compared to other types of testing), so it makes perfect sense to me to write a verification suite of tests.
You might even go so far as to give the students the test suite instead of a specification written in prose. This could introduce them to the concept of Test-Driven Development (although we normally tend to write the tests iteratively, and not in batches).
yes, unit tests are the obvious solution for most cases.
compiler warnings and static analysis is also useful.
timing the program's execution given a set of parameters is another fairly automated option -- depends on what you are interested in evaluating.
creating base classes with good diagnostics (which you can swap out the implementation for your evaluation if you prefer) is another option. you can also provide interfaces they must use, and hold two implementations. then exercise the programs as usual using the diagnostic implementation. it depends on what you are looking for.
Maybe I'm missing something obvious, but wouldn't it be sufficient just to run the code several times with boundary-testing parameter values?
I have asked a related question, but I did not get a satisfactory answer. So, perhaps I should ask it a different way.
How do large-scale C projects, like Perl or Ruby or even the Linux kernel, handle unit-testings? Or even in any functional language?
I am familiar with Dependency Injection and Abstract Factory for testings in OOP, but I don't see a scalable and manageable equivalence in non OOP. In C or Haskell, for example, there would be layers over layers of functions, higher ones implicitly calling the lower ones. How do I find seams to test just an unit of code instead of all its dependencies?
One way to avoid the need for seams all together is to keep the depths of call-dependency graph very low. Code horizontally rather than vertically, so to speak. Keep as much of the application logic as possible in the "leaf" functions; and make sure the "node" functions do no work other than plumbing the data to other node/leaf functions. Then, test only the "leaf" functions; leave the "node" functions out to integration tests. Is that approach effective?
The largest software today are still written in procedural languages. There must be some methodologies being employed that work. Could someone with experience with large-scale software in procedural languages with good unit-testings comment?
Functional languages have other constructs (than objects) for modularity, such as ML functors. "Dependency injection" is basically a glorified name for "abstracting over things" and has been used for ages in functional languages.
Testing, in all paradigms, should follow the specification boundaries. If you have an idea what a given piece of code (function, method, object...) is supposed to do, you should test against this specification. For leaf functions, this will be unit testing, for "node" function this can be considered "integration testing" if you like, but it's really the same activity.
I think you will find that the same methodologies essentially applies to functional programming, with essentially the same results; in particular, (re)designing code to be easy to test also improves its modularity and maintainability.
designing code to be easy to test also improves its modularity and maintainability.
Not true. Designing code to be easy to test improves the ease of writing test code.
Natural divisions of functional modules usually aren't the most convenient for unit testing. That's why so much Java code these days is broken into tiny, scattered pieces, with each piece mostly not useful by itself.
I've started to look into the whole unit testing/test-driven development idea, and the more I think about it, the more it seems to fill a similar role to static type checking. Both techniques provide a compile-time, rapid-response check for certain kinds of errors in your program. However, correct me if I'm wrong, but it seems that a unit test suite with full coverage would test everything static type checking would test, and then some. Or phrased another way, static type checks only go part of the way to "prove" that your program is correct, whereas unit tests will let you "prove" as much as you want (to a certain extent).
So, is there any reason to use a language with static type checking if you're using unit testing as well? A somewhat similar question was asked here, but I'd like to get into more detail. What specific advantages, if any, does static type checking have over unit tests? A few issues like compiler optimizations and intellisense come to mind, but are there other solutions for those problems? Are there other advantages/disadvantages I haven't thought of?
There is one immutable fact about software quality.
If it can't compile, it can't ship
In this rule, statically typed languages will win over dynamically typed languages.
Ok, yes this rule is not immutable. Web Apps can ship without compiling (I've deployed many test web apps that didn't compile). But what is fundamentally true is
The sooner you catch an error, the cheaper it is to fix
A statically typed language will prevent real errors from happening at one of the earliest possible moments in the software development cycle. A dynamic language will not. Unit Testing, if you are thorough to a super human level can take the place of a statically typed language.
However why bother? There are a lot of incredibly smart people out there writing an entire error checking system for you in the form of a Compiler. If you're concerned about getting errors sooner use a statically typed language.
Please do not take this post as a bashing of dynamic languages. I use dynamic languages daily and love them. They are incredibly expressive and flexible and allow for incredibly fanscinating program.s However in the case of early error reporting they do lose to statically typed languages.
For any reasonably sized project, you just cannot account for all situations with unit tests only.
So my answer is "no", and even if you manage to account for all situations, you've thereby defeated the whole purpose of using a dynamic language in the first place.
If you want to program type-safe, better use a type-safe language.
I would think that automated unit testing will be important to dynamic typed languages, but that doesn't mean it would replace static type checking in the context that you apply. In fact, some of those who use dynamic typing might actually be using it because they do not want the hassles of constant type safety checks.
The advantages dynamically typed languages offer over static typed languages go far beyond testing, and type safety is merely one aspect. Programming styles and design differences over dynamic and static typed languages also vary greatly.
Besides, unit tests that are written too vigorously enforce type safety would mean that the software shouldn't be dynamically typed after all, or the design being applied should be written in a statically typed language, not a dynamic one.
Having 100% code coverage doesn't mean you have fully tested your application. Consider the following code:
if (qty > 3)
{
applyShippingDiscount();
}
else
{
chargeFullAmountForShipping();
}
I can get 100% code coverage if I pump in values of qty = 1 and qty = 4.
Now imagine my business condition was that "...for orders of 3 or more items I am to apply a discount to the shipping costs..". Then I would need to be writing tests that worked on the boundaries. So I would design tests where qty was 2,3 and 4. I still have 100% coverage but more importantly I found a bug in my logic.
And that is the problem that I have with focusing on code coverage alone. I think that at best you end up with a situation where the developer creates some initial tests based on the business rules. Then in order to drive up the coverage number they reference their code when design new test cases.
Manifest typing (which I suppose you mean) is a form of specification, unit testing is much weaker since it only provides examples. The important difference is that a specification declares what has to hold in any case, while a test only covers examples. You can never be sure that your tests cover all boundary conditions.
People also tend to forget the value of declared types as documentation. For example if a Java method returns a List<String>, then I instantly know what I get, no need to read documentation, test cases or even the method code itself. Similarly for parameters: if the type is declared then I know what the method expects.
The value of declaring the type of local variables is much lower since in well-written code the scope of the variable's existence should be small. You can still use static typing, though: instead of declaring the type you let the compiler infer it. Languages like Scala or even C# allow you to do just this.
Some styles of testing get closer to a specification, e.g. QuickCheck or it's Scala variant ScalaCheck generate tests based on specifications, trying to guess the important boundaries.
I would word it a different way--if you don't have a statically-typed language, you had better have very thorough unit tests if you plan on doing anything "real" with that code.
That said, static typing (or rather, explicit typing) has some significant benefits over unit tests that make me prefer it generally. It creates much more understandable APIs and allows for quick viewing of the "skeleton" of an application (i.e. the entry points to each module or section of code) in a way that is much more difficult with a dynamically-typed language.
To sum up: in my opinion, given solid, thorough unit tests, the choice between a dynamically-typed language and a statically-typed language is mostly one of taste. Some people prefer one; others prefer the other. Use the right tool for the job. But this doesn't mean they're identical--statically-typed languages will always have an edge in certain ways, and dynamically-typed languages will always have an edge in certain different ways. Unit tests go a long way towards minimizing the disadvantages of dynamically-typed languages, but they do not eliminate them completely.
No.
But that's not the most important question, the most important question is: does it matter that it can't?
Consider the purpose of static type checking: avoiding a class of code defects (bugs). However, this has to be weighed in the context of the larger domain of all code defects. What matters most is not a comparison along a narrow sliver but a comparison across the depth and breadth of code quality, ease of writing correct code, etc. If you can come up with a development style / process which enables your team to produce higher quality code more efficiently without static type checking, then it's worth it. This is true even in the case where you have holes in your testing that static type checking would catch.
I suppose it could if you are very thorough. But why bother? If the language is already checking to ensure static types are correct, there would be no point in testing them (since you get it for free).
Also, if you are using static typed languages with an IDE, the IDE can provide you with errors and warnings, even before compiling to test. I am not certain there are any automated unit testing applications that can do the same.
Given all the benefits of dynamic, late-binding languages, I suppose that's one of the values offered by Unit Tests. You'll still need to code carefully and intentionally, but that's the #1 requirement for any kind of coding IMHO. Being able to write clear and simple tests helps prove the clarity and simplicity of your design and your implementation. It also provides useful clues for those who see your code later. But I don't think I'd count on it to detect mismatched types. But in practice I don't find that type-checking really catches many real errors anyway. It's just not a type of error I find occurring in real code, if you have a clear and simple coding style in the first place.
For javascript, I would expect that jsLint will find almost all type-checking issues. primarily by suggesting alternate coding styles to decrease your exposure.
Type checking helps enforce contracts between components in a system. Unit testing (as the name implies) verifies the internal logic of components.
For a single unit of code, I think unit testing really can make static type checking unnecessary. But in a complex system, automated tests cannot verify all the multitude ways that different components of the system might interact. For this, the use of interfaces (which are, in a sense, a kind of "contract" between components) becomes a useful tool for reducing potential errors. And interfaces require compile-time type checking.
I really enjoy programming in dynamic languages, so I'm certainly not bashing dynamic typing. This is just a problem that recently occurred to me. Unfortunately I don't really have any experience in using a dynamic language for a large and complex system, so I'd be interested to hear from other people whether this problem is real, or merely theoretical.