I'm trying out writing some elisp for the first time. I'd like to be able to run unit tests on it.
I have a buffer with the following in it:
(ert-deftest addition-test ()
(should (= (+ 1 2) 4)))
And I ran M-x eval-buffer to install the ert-deftest definition into my emacs. When I run M-x ert RET t RET, I see:
Selector: t
Passed: 0
Failed: 0
Total: 0/1
Started at: 2015-11-16 00:27:46-0800
Aborted.
Aborted at: 2015-11-16 00:27:46-0800
A
A addition-test
aborted
And I see the message "Test failed: ...." in the minibuffer at the bottom of the screen. Why is the failure reported there rather in the ERT buffer?
EDIT:
It appears that the "aborted" message occurs after a failing test is hit, then any remaining tests aren't run. Example:
(ert-deftest test1 ()
(should 1))
(ert-deftest test2 ()
(should nil))
(ert-deftest test3 ()
(should 1))
M-x eval-buffer RET M-x ert RET opens up a buffer named *ert* that says:
Selector: t
Passed: 1
Failed: 0
Total: 1/3
Started at: 2015-11-16 02:06:57-0800
Aborted.
Aborted at: 2015-11-16 02:06:57-0800
.A-
A test2
aborted
And in *Messages* I see:
Aborted: Ran 3 tests, 1 results were as expected
ert-fail: Test failed: ((should nil) :form nil :value nil)
If I go in the *ert* and press d over test2, I get the following in *Messages*:
Running test test2...ABORTED
ert-fail: Test failed: ((should nil) :form nil :value nil)
I can convince it to run test3 if I hit enter on the - in .A-, expand test3, then hit d on it. That looks like:
Selector: t
Passed: 2
Failed: 0
Total: 2/3
Started at: 2015-11-16 02:08:40-0800
Aborted.
Aborted at: 2015-11-16 02:08:40-0800
.A.
A test2
aborted
. test3
passed
In terms of getting a backtrace, hitting b over test2 gives me:
ert-results-pop-to-backtrace-for-test-at-point: cl-etypecase failed: [cl-struct-ert-test-aborted-with-non-local-exit (((should nil) :form nil :value nil))], (ert-test-passed ert-test-result-with-condition)
I don't understand the output of the backtrace at all. This behavior doesn't seem to match what the ERT documentation says so I'm really puzzled. What's going on here?
The code you have posted works in a clean Emacs started with emacs -Q. Most likely, it is caused by some library that has been loaded somehow.
Related
In package.json I have 2 script commands:
"test:unit": "jest --watch --testNamePattern='^(?!\\[functional\\]).+$'",
"test:functional": "jest --watch --testNamePattern='\\[functional\\]'",
copying ^(?!\\[functional\\]).+$ into https://regex101.com/, it does not match the test string below inside argument 1 of describe()
describe("[functional] live tests", () => {
When changed to ([functional]).+$, the pattern does match. I have to remove a pair of \ on each end to remove escapes for .json files (I think).
Here is what I see when running npm run test:unit in my project root:
// the functional test runs (not desired)
$ npm run test:unit
functions/src/classes/__tests__/Functional.test.ts:30:47 - error TS2339: Property 'submit' does not exist on type 'Element'.
30 await emailForm.evaluate(form => form.submit());
~~~~~~
RUNS ...s/__tests__/Functional.test.ts
Test Suites: 1 failed, 1 skipped, 3 passed, 4 of 5 total
Tests: 2 skipped, 16 passed, 18 total
Snapshots: 0 total
Time: 8.965s, estimated 27s
Ran all test suites with tests matching "^(?!\[functional\]).+$".
Active Filters: test name /^(?!\[functional\]).+$/
The functional tests are not built out which explains the syntax error, it's not important here. The key issue, is why the tests were not skipped.
I believe the problem has to do with the regex negative matcher. The positive matcher without the ! only matches tests that have, or are nested in a describe block with [functional]
$ npm run test:functional
Test Suites: 1 failed, 4 skipped, 1 of 5 total
Active Filters: test name /\[functional\]/
Anyone know why the negative regex pattern is failing during npm run test:unit ?
Instead of a regex fix I changed the flag on the unit testing script to an ignore, then copying the matching pattern for [functional]:
"test:unit": "jest --watch --testIgnorePattern='\\[functional\\]'",
"test:functional": "jest --watch --testNamePattern='\\[functional\\]'",
I'm trying to parse some logs using split and regexes in powershell
Here's my code :
$string = "Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code. Test 229: Passed Test 260: Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1"
$string -Split '(Test (\d)+:)'
Result :
Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Test 229:
9
Passed
Test 260:
0
Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1
Expected result:
Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Test 229:
Passed
Test 260:
Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1
On this site : https://regexr.com/3c0lf I tried this regex and the groups captured were : Test 260: and Test 229: (which is exactly what I want)
I do not understand where the 0 and the 9 comes from.
Thanks a lot
Those are the last digits of the number. 0 from 26*0* and 9 from 22*9*.
You are seeing those because you've created an additional capturing group by putting parentheses around the digits. Just remove them like so:
$string -Split '(Test \d+:)
You probably don't even need those parentheses either, leaving just
$string -Split 'Test \d+:
There's something fundamental I'm not getting here. I expected the following test to pass. But the 2nd test case "staging"/"staging" fails. Its as if with-redefs-fn is failing to advance through the test-case instances. But the logging says everything is fine. This is confusing.
(deftest test-bad-derive-s3-environment
(testing "variants of props environments"
(doseq [test-case [{:env "qa1" :expect "qa1"}
{:env "dev" :expect "qa1"}
{:env "staging" :expect "staging"}]]
(log/infof "test-case %s" test-case)
(with-redefs-fn {#'config/environment (fn [] (:env test-case))}
(let [actual (fs/derive-s3-environment (config/environment))
_ (log/infof "within redefs :env %s :expect %s" (:env test-case) (:expect test-case))]
#(is (= actual (:expect test-case))))))))
...
lein test com.climate.test.mapbook.filestore
2016-05-03 16:16:29,353 INFO filestore:288 - test-case {:env "qa1", :expect "qa1"}
2016-05-03 16:16:29,355 INFO EnvConfig:98 - Loading config properties from /export/disk0/wb/etc/env.properties
2016-05-03 16:16:29,357 INFO EnvConfig:98 - Loading config properties from /export/disk0/wb/etc/local.properties
2016-05-03 16:16:29,358 INFO filestore:288 - within redefs :env qa1 :expect qa1
2016-05-03 16:16:29,359 INFO filestore:288 - test-case {:env "staging", :expect "staging"}
2016-05-03 16:16:29,359 INFO filestore:288 - within redefs :env staging :expect staging
lein test :only com.climate.test.mapbook.filestore/test-bad-derive-s3-environment
FAIL in (test-bad-derive-s3-environment) (filestore.clj:29)
variants of props environments
expected: (= actual (:expect test-case))
actual: (not (= "qa1" "staging"))
2016-05-03 16:16:29,364 INFO filestore:288 - test-case {:env "dev", :expect "qa1"}
2016-05-03 16:16:29,364 INFO filestore:288 - within redefs :env dev :expect qa1
Why does my with-redefs-fn fail to redefine the config/environment function in terms of the current test-case?
First of all, notice that your final test instance has an :expect of "qa1" – the same as the first test instance – so it should actually fail if the code worked as you intended it to; its passing is a symptom of the same problem as the second instance's failing.
Now for the fix – there are two options:
Just use with-redefs instead of with-redefs-fn:
(with-redefs [config/environment (fn [] (:env test-case))]
…)
Most of the time this is what you want to do and you can consider with-redefs-fn to be an implementation detail behind with-redefs – although strictly speaking it does have some utility of its own in that it can redefine dynamically constructed collections of Vars.
Use with-redefs-fn, but move the inner let form inside the anonymous function:
(with-redefs-fn {…}
#(let […]
(is …)))
Finally, the reason these work and the version from the question text does not:
with-redefs-fn is a function, so at runtime its arguments will be evaluated before it is actually invoked with their runtime values passed in. In particular, the let expression that you pass in as the second argument will be evaluated before the redefinition takes place, and so the local called actual will get the result of evaluating (config/environment) before the redefinition as its value, and that value will be installed in the anonymous closure created in the let's body. That closure, however, will then be called with the redefinition in place, and so it will take its notion of the "actual" value from before the redefinition and compare it with the expectation set after the redefinition, resulting in the observed behaviour.
Moving the let inside the closure, as in the second approach above, fixes this mismatch problem – the let local's value is computed with the redefinition in place and all is well. The first approach using with-redefs expands to the second approach.
The log printouts are fine, because they are only concerned with the doseq local and never examine any Vars. If they did, they would only see the pre-redefinition values.
My t.ms file contains:
interface(prettyprint=0): kernelopts(assertlevel=1):
ASSERT(1<1):
ASSERT(2<2):
When I run:
maple -q t.ms; echo $?
I get:
Error, assertion failed
Error, assertion failed
0
When I run:
maple -e 2 -q t.ms; echo $?
I get:
Error, assertion failed
0
I want to get:
Error, assertion failed
4
That is, I want Maple to exit with a nonzero exit status upon the first failing assertion. (I don't care if the exit code is 1 or anything else as long as it's nonzero. I've got the number 4 from the documentation, related to errorbreak) How do I get that?
The documentation doesn't make it very clear that one must use,
`quit`(n)
with name-quotes.
interface(prettyprint=0):
handler:=proc(e::uneval)
local failed;
printf("entered\n"); # remove this when satisfied
failed:=false;
try
if evalb(eval(e)) <> true then
error;
end if;;
catch:
failed:=true;
printf("Error, assertion failed\n");
finally;
if failed then
`quit`(5);
end if;
end try;
true;
end proc:
ASSERT( handler( 1<1 )):
ASSERT( handler( 2<2 )):
Now, saving this as file uh.mpl then using Maple 18.01 for Linux I see,
$ maple18.01 -q -A 2 ~/uh.mpl ; echo $?
entered
Error, assertion failed
5
And if run without the -A 2 then it doesn't run the asserted checks.
[edited] Here below is a slight modification, to process additional arguments as part of the printing.
handler:=proc(e::uneval)
local failed;
printf("entered\n"); # remove this when satisfied
failed:=false;
try
if evalb(eval(e)) <> true then
error;
end if;;
catch:
failed:=true;
printf("Error, assertion failed, %q\n", _rest);
finally;
if failed then
`quit`(5);
end if;
end try;
true;
end proc:
I'm a Ruby on Rails newbie and writing tests. Some of these generate exceptions; I would like the "rake test" output to give me the exception error message but not the whole backtrace. (I'd like to write tests which exercise unimplemented functionality, which I'll then fill in.)
For example, actual output:
Started
E
Finished in 0.081054 seconds.
1) Error:
test_should_fail(VersioningTest):
ActiveRecord::StatementInvalid: PGError: ERROR: null value in column "client_ip" violates not-null constraint
: INSERT INTO "revisions" ("created_at", "id") VALUES ('2011-02-03 20:14:17', 980190962)
/Users/rpriedhorsky/.rvm/gems/ruby-1.9.2-p136/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract_adapter.rb:202:in `rescue in log'
/Users/rpriedhorsky/.rvm/gems/ruby-1.9.2-p136/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract_adapter.rb:194:in `log'
/Users/rpriedhorsky/.rvm/gems/ruby-1.9.2-p136/gems/activerecord-3.0.3/lib/active_record/connection_adapters/postgresql_adapter.rb:496:in `execute'
[... etc. etc. etc. ...]
1 tests, 0 assertions, 0 failures, 1 errors, 0 skips
Desired output:
Started
E
Finished in 0.081054 seconds.
1) Error:
test_should_fail(VersioningTest):
ActiveRecord::StatementInvalid: PGError: ERROR: null value in column "client_ip" violates not-null constraint
1 tests, 0 assertions, 0 failures, 1 errors, 0 skips
I found info (e.g.) on the opposite direction, but not on suppressing stack traces.
Edit:
It would be nice to turn them on and off easily; as pointed out below, sometimes they are useful for tracking down bugs.
You could take a look at "backtrace silencers" - for me (Rails 2.3.8), this is the file config/initializers/backtrace_silencers.rb:
# Be sure to restart your server when you modify this file.
# You can add backtrace silencers for libraries that you're using but
# don't wish to see in your backtraces.
# Rails.backtrace_cleaner.add_silencer { |line| line =~ /my_noisy_library/ }
# You can also remove all the silencers if you're trying do debug a
# problem that might steem from framework code.
# Rails.backtrace_cleaner.remove_silencers!
Rails.backtrace_cleaner.add_silencer {|line| line =~ /gems/}
Rails.backtrace_cleaner.add_silencer {|line| line =~ /passenger/}
It looks like you should be able to put a line like
Rails.backtrace_cleaner.add_silencer {|line| true}
In your config/environments/test.rb file, and that would wipe your backtraces clean away (though it might just apply to the logger - I'm not very familiar with the method).
But ask yourself - do you really want to do away with backtraces entirely? They can be pretty useful for tracking down bugs...