Running “cabal test” passes although there are test failures - unit-testing

Question

I have a Cabal test target:
test-suite Tests
type: exitcode-stdio-1.0
main-is: Main.hs
hs-source-dirs: test, src
build-depends: base, …
default-language: Haskell2010
And a simple testing Main.hs:
import Test.HUnit
testSanity = TestCase $ assertEqual "Should fail" 2 1
main = runTestTT testSanity
Now running cabal test passes:
Test suite Tests: RUNNING...
Test suite Tests: PASS
Test suite logged to: dist/test/Project-0.1.0-Tests.log
1 of 1 test suites (1 of 1 test cases) passed.
Even though there are failures correctly logged in the test suite log:
Test suite Tests: RUNNING...
Cases: 1 Tried: 0 Errors: 0 Failures: 0
### Failure:
Should fail
expected: 2
but got: 1
Cases: 1 Tried: 1 Errors: 0 Failures: 1
Test suite Tests: PASS
Test suite logged to: dist/test/Project-0.1.0-Tests.log
What am I doing wrong?

Answer 1

main has to be of type IO a where a can be anything, and that value is not tied in any way to the POSIX exit status of the program. You need to look at the Counts returned by runTest and explicitly choose a successful or failed output code.
import System.Exit
main :: IO ()
main = do
cs#(Counts _ _ errs fails) <- runTestTT testSanity
putStrLn (showCounts cs)
if (errs > 0 || fails > 0)
then exitFailure
else exitSuccess

Answer 2

I believe exitcode-stdio-1.0 expects the test to communicate failure via the program's exit code. However, if you run the test program manually from the shell (look in the dist/build directory) I think you'll find that the exit code is always 0.
Here is a related SO question which suggests that your test should call exitFailure if your suite doesn't pass:
QuickCheck exit status on failures, and cabal integration
FWIW, it works correctly when using stack:
$ mkdir new-dir
$ cd new-dir
$ stack new
(edit new-template.cabal to add HUnit as a test dependency)
(add test case to test/Spec.hs)
$ stack test
Perhaps stack is also scanning the test output.