Is there any way to write something like a "unit test" which makes sure some code does not compile?
Why would I want such a thing? Two reasons.
1) Check the type safety of my API. I'd like a way to make sure if someone passes in a bad value, you get a compiler error, not just a runtime error. Obviously, I can just run the compiler and check for the error, but having it formalized in a unit test is good for avoiding a regression & also for documentation.
Eg., consider this test. There is some commented out code which I had used to check type-safety:
https://github.com/squito/boxwood/blob/master/core/src/test/scala/com/quantifind/boxwood/EnumUnionTest.scala#L42
(lines 42 & 48 -- on line 34 I call a different API which has a runtime exception, which I can check)
It actually took me a while to get the type-safety right, so those were important checks. Now if I go and modify the underlying implementation, I can't just run my test suite -- I've got to also remember to uncomment those lines and check for a compiler error.
2) Testing error handling of macros. If a macro has some bad input, it should result in a compiler error. Same issues here, same desire to have it in a easy-to-run test-suite.
I use ScalaTest, but I'm happy to here a solution with any unit-testing framework.
As I note in a comment above, Shapeless 2.0 (not yet released but currently available as a milestone) has a very nice implementation of the functionality you're looking for, based on a solution by Stefan Zeiger. I've added a demo to your project here (note that I've had to update to Scala 2.10, since this solution uses a macro). It works like this:
import shapeless.test.illTyped
//this version won't even compile
illTyped("getIdx(C.Ooga)")
//We can have multiple enum unions exist side by side
import Union_B_C._
B.values().foreach {b => Union_B_C.getIdx(b) should be (b.ordinal())}
C.values().foreach {c => Union_B_C.getIdx(c) should be (c.ordinal() + 2)}
//Though A exists in some union type, Union_B_C still doesn't know about it,
// so this won't compile
illTyped("""
A.values().foreach {a => Union_B_C.getIdx(a) should be (a.ordinal())}
""")
If we were to change the code in the second call to illTyped to something that will compile:
B.values().foreach {a => Union_B_C.getIdx(a) should be (a.ordinal())}
We'd get the following compilation error:
[error] .../EnumUnionTest.scala:56: Type-checking succeeded unexpectedly.
[error] Expected some error.
[error] illTyped("""
[error] ^
[error] one error found
[error] (core/test:compile) Compilation failed
If you'd prefer a failed test, you could pretty easily adapt the implementation in Shapeless. See Miles's answer to my previous question for some addition discussion.
Scalatest can also do this.
Checking that a snippet of code does not compile
Often when creating libraries you may wish to ensure that certain
arrangements of code that represent potential “user errors” do not
compile, so that your library is more error resistant. ScalaTest
Matchers trait includes the following syntax for that purpose:
"val a: String = 1" shouldNot compile
If you want to ensure that a
snippet of code does not compile because of a type error (as opposed
to a syntax error), use:
"val a: String = 1" shouldNot typeCheck
Note that the shouldNot typeCheck syntax will only succeed if the given snippet of code does
not compile because of a type error. A syntax error will still result
on a thrown TestFailedException.
If you want to state that a snippet of code does compile, you can make
that more obvious with:
"val a: Int = 1" should compile
Although the previous three constructs
are implemented with macros that determine at compile time whether the
snippet of code represented by the string does or does not compile,
errors are reported as test failures at runtime.
Travis Brown's answer is absolutely correct. Just in the interest of completeness, I want to add that this also works for testing macros, as demonstrated here.
One minor point: the illTyped check doesn't seem to work in the repl. It never throws an error, even if the given expression does type check. But don't let that fool you, it does work well.
> test:console
Welcome to Scala version 2.10.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_65).
Type in expressions to have them evaluated.
Type :help for more information.
scala> def foo(s: String) = s
foo: (s: String)String
scala> import shapeless.test.illTyped
import shapeless.test.illTyped
scala> foo(1)
<console>:10: error: type mismatch;
found : Int(1)
required: String
foo(1)
^
scala> illTyped("""foo(1)""")
scala> illTyped("""foo("hi there")""") // <--- works, but shouldn't!
Related
I am not sure if this is possible or if there's a better architecture for this. I wrote a function that does some tests:
fun validate(a: Any?, b: Any?){
a shouldBe b
}
My function is obviously more complex than that, but now I would like to test the test itself. For example, I want to pass a=1 and b=null and make sure it fails. Is there a way to test for it to fail or succeed using Kotest?
If not, what is a proper way to architect this?
I am trying to rely on this test often and in many places of my service, so I would like it to be as reliable as possible.
Short answer: use shouldFail { ... }.
This is possible because failing tests in Kotest (and many other frameworks) will throw an AssertionError. You can check for that exception to see whether the test failed. So to check for a failed assertion, you can write:
shouldThrow<AssertionError> { /* your failing test here */ }
As you'd probably expect, shouldThrow will catch the exception so that it doesn't propagate. So in effect it will turn a failing test into a passing test, and vice versa.
To make it a bit more convenient and readable, you could write a couple of extension functions, like this:
fun shouldFail(block: () -> Unit) = shouldThrow<AssertionError>(block)
fun shouldNotFail(block: () -> Unit) = shouldNotThrow<AssertionError>(block)
In fact Kotest includes the shouldFail function in the standard assertions, so you don't even need to define it yourself. Their implementation is the same as what I have suggested here. They don't include a shouldNotFail, though, presumably because the usefulness is a bit questionable. Wrapping a test in "shouldNotFail" makes it fail if the assertion fails and pass if the assertion passes, which is... exactly what it would have done if you didn't wrap it in anything at all.
To test your own assertion functions, you can use it like this:
shouldFail { 1 shouldBe 2 }
If you look at the source code for Kotest itself, you'll find many unit tests for the built-in assertion functions using exactly this approach.
I developed some ANTLR + LLVM parser code in spring. Since it is only a recreational project, I did not touch the code or see whether it compiles in the meantime. During that time, several system updates took place, which I assume caused my current problems:
When trying to compile the code today (with clang++), I suddenly got several error messages. Initially the std::any class was not found at all. I then played with "std=c++17" and "std=c++20" options. Now the main error (as I understand it) seems to be
error: no member named 'as' in 'std::any'
The error occurs whenever i do something like for instance
args.push_back(visit(*it).as<sarg>());
or, more stripped-down:
antlrcpp::Any myvar;
//...
myvar.as<some_type>();
I picked this up from Antlr tutorial codes where this seems to be the standard idiom to cast an antlrcpp::Any object to the type desired by the calling function.
I noticed that antlrcpp::Any apparently is merely a wrapper for std::any, which apparently really does not support this "as" method.
What can i do to make my code work again?
Google Test assertions ASSERT_* are supposed to be used in the form of ASSERT_EQ(expected, actual) where the 1st parameter is expected value and the 2nd is actual value. But very often I see in existing codebase that these parameters are reversed like in this code:
TEST(test, test)
{
ASSERT_EQ(foo(), 1);
}
This is almost ok, but it produces a bit weird error message in case of test failure like: "the result of foo() was expected but actually it was 1". This seems to be a minor issue but is there any way to force correct order of expected and actual at compile time?
You can use Hamcrest matchers from google mock:
ASSERT_THAT(foo(), Eq(1) );
This improve readability and force order of parameters.
For me the best way avoiding wired messages is, doing it correctly, from the beginning!
Okay, that doesn't help for existing test wrongly written, but in all unit-testing Frameworks I know (C#, Java, C++) it is always the same:
ASSERT(EXPECTED, ACTUAL)
And if another developer is reading your tests he should rely on you that you've done it correctly in the past.
I'm currently trying to compile a python project (5files # total 1200 lines of code) with shedskin.
I tried shedskin Version 0.9.3 and 0.9.2 both result in the same errors.
This is the first error I encounter:
mmain.cpp: In function ‘__shedskin__::list<__shedskin__::list<int>*>* __mmain__::list_comp_3(__shedskin__::__ss_int)’:
mmain.cpp:133: error: no matching function for call to ‘__shedskin__::list<__shedskin__::list<int>*>::append(__shedskin__::list<double>*)’
Moreover, I after running shedskin (i.e. before typing "make") I receive many warnings - all related to dynamic types:
*WARNING* mmain.py: expression has dynamic (sub)type: {float, int, list}
However, shedskin seems to work flawlessly with the provided examples since I can compile and execute them without any errors.
Do you have an idea where to look for the error or what the error is related to?
mmain.cpp:133: error: no matching function for call to ‘__shedskin__::list<__shedskin__::list<int>*>::append(__shedskin__::list<double>*)’
This error means that you've got a Python object that shedskin has inferred as a list of lists of ints, but now you're trying to append something that it's inferred as a list of floats. You can get that by, for example, doing this:
a = [[1], [2]]
b = 1.0
a.append([b])
However, from the line above it, the function name is list_comp_3. Unless you've actually named a function list_comp_3 (which you haven't), this is a list comprehension. So, you may be doing something like this:
a = [1, 2, 3.0]
b = [[i] for i in a]
You may be wondering why it let you get away with a but failed on b. Well, first of all, it probably didn't really let you get away with it, if you've got dozens of warnings you haven't dealt with. But second, as the documentation says:
Integers and floats can often be mixed, but it is better to avoid this where possible, as it may confuse Shed Skin:
a = [1.0]
a = 1 # wrong - use a float here, too
As for the warnings, they can mean anything from "you got away with it this time, but don't expect to always do so" to "an error is coming up related to this" to "this will compile, but to something less efficient than the original Python code rather than more" to "this will compile, but to something incorrect".
More generally, it sounds like your program just can't be statically typed by shedskin's inference engine. Without actually seeing your code, it's impossible to tell you what you're doing wrong, but if you re-read the Typing Restrictions and Python Subset Restrictions sections of the docs, that should give you ideas of what kinds of things are and aren't appropriate.
to avoid confusion, please note that both code snippets provided by 'abartert' compile and run fine when compiled separately (shedskin 0.9.3). my guess is also that the problem should disappear after resolving the dynamic typing warnings. if not, I'd be very interested in seeing the program you are trying to compile, or at least enough of it to reproduce the problem.
update: btw, as of 0.9.1 or so, shedskin should be smarter about int and float mixing. if it encounters something that would lead to broken or inefficient c++ code (because of necessary run-time conversion of sorts), it should now usually complain with an 'incompatible types' warning. so perhaps it's time to update this part of the documentation slightly for 0.9.3.
I'm trying to write a test to verify a compiling error. It's about assigning a number to a String type property. But since it's a compiling error, the unit test code won't even be compiled at the first place. So anyone have any suggestion on how to do this?
I'm thinking maybe I can assign the number in runtime and check to see if certain exception got thrown? But I'm not sure how exactly to do that.
Thanks in advance!
If I understand you correctly, you have some piece of code that may not compile and you want to write the unit test that fails if the code really doesn't compile. If that is the case, then you should not write any unit tests. You should understand that you should write unit tests only for your code, not the code that somebody else wrote.
You didn't specify the programming language, so I will do some pseudo-code. Say you are writing a function to add two numbers:
function add(a, b) {
return a + b;
}
Very simple. You should unit test this, e.g. by making tests like the below:
function testAdd() {
assertEquals(4, add(2, 2));
assertEquals(46, add(12, 34));
}
However, you should not write a test that tests whether + operator is working fine. This is the job of whoever wrote the library that implements how + operator really works.
So, if that's the case, don't write any unit tests. Compiling your code is job of your compiler. The compiler should report that there is a compilation error in a proper way. You should not test whether compiler does its job correctly - testing that is the job of the people who are writing the compiler.
You did not specify the language
Ruby, Python -- dynamic & strong typesystem. This means that the types deduced during runtime (dynamic), but implicit conversions between types are prohibited
Js, Perl -- dynamic & weak typesystem.
C++ -- static and strong typesystem.
Let's assume we are talking about C++. Moreover I can create more really example. Imagine that you implement static assert for your project which doesn't not use c++11 compiler
template <bool>
struct my_static_assert;
template <>
struct my_static_assert<true> {};
If you want to check that such mechanism works ok. You should create unittest function which do the follow steps:
Create some file to compiler
Create external process of the compiler and pass test compile unit to it
Wait for compiler process completion
Retrive return code from the compiler process
Your function check return code from 4.
I checked google-test guide but it seems that they doesn't support such concept https://github.com/google/googletest/blob/master/docs/advanced.md