with-redefs-fn fails to pick up binding from do-seq? - clojure

Question

There's something fundamental I'm not getting here. I expected the following test to pass. But the 2nd test case "staging"/"staging" fails. Its as if with-redefs-fn is failing to advance through the test-case instances. But the logging says everything is fine. This is confusing.
(deftest test-bad-derive-s3-environment
(testing "variants of props environments"
(doseq [test-case [{:env "qa1" :expect "qa1"}
{:env "dev" :expect "qa1"}
{:env "staging" :expect "staging"}]]
(log/infof "test-case %s" test-case)
(with-redefs-fn {#'config/environment (fn [] (:env test-case))}
(let [actual (fs/derive-s3-environment (config/environment))
_ (log/infof "within redefs :env %s :expect %s" (:env test-case) (:expect test-case))]
#(is (= actual (:expect test-case))))))))
...
lein test com.climate.test.mapbook.filestore
2016-05-03 16:16:29,353 INFO filestore:288 - test-case {:env "qa1", :expect "qa1"}
2016-05-03 16:16:29,355 INFO EnvConfig:98 - Loading config properties from /export/disk0/wb/etc/env.properties
2016-05-03 16:16:29,357 INFO EnvConfig:98 - Loading config properties from /export/disk0/wb/etc/local.properties
2016-05-03 16:16:29,358 INFO filestore:288 - within redefs :env qa1 :expect qa1
2016-05-03 16:16:29,359 INFO filestore:288 - test-case {:env "staging", :expect "staging"}
2016-05-03 16:16:29,359 INFO filestore:288 - within redefs :env staging :expect staging
lein test :only com.climate.test.mapbook.filestore/test-bad-derive-s3-environment
FAIL in (test-bad-derive-s3-environment) (filestore.clj:29)
variants of props environments
expected: (= actual (:expect test-case))
actual: (not (= "qa1" "staging"))
2016-05-03 16:16:29,364 INFO filestore:288 - test-case {:env "dev", :expect "qa1"}
2016-05-03 16:16:29,364 INFO filestore:288 - within redefs :env dev :expect qa1
Why does my with-redefs-fn fail to redefine the config/environment function in terms of the current test-case?

Answer 1

First of all, notice that your final test instance has an :expect of "qa1" – the same as the first test instance – so it should actually fail if the code worked as you intended it to; its passing is a symptom of the same problem as the second instance's failing.
Now for the fix – there are two options:
Just use with-redefs instead of with-redefs-fn:
(with-redefs [config/environment (fn [] (:env test-case))]
…)
Most of the time this is what you want to do and you can consider with-redefs-fn to be an implementation detail behind with-redefs – although strictly speaking it does have some utility of its own in that it can redefine dynamically constructed collections of Vars.
Use with-redefs-fn, but move the inner let form inside the anonymous function:
(with-redefs-fn {…}
#(let […]
(is …)))
Finally, the reason these work and the version from the question text does not:
with-redefs-fn is a function, so at runtime its arguments will be evaluated before it is actually invoked with their runtime values passed in. In particular, the let expression that you pass in as the second argument will be evaluated before the redefinition takes place, and so the local called actual will get the result of evaluating (config/environment) before the redefinition as its value, and that value will be installed in the anonymous closure created in the let's body. That closure, however, will then be called with the redefinition in place, and so it will take its notion of the "actual" value from before the redefinition and compare it with the expectation set after the redefinition, resulting in the observed behaviour.
Moving the let inside the closure, as in the second approach above, fixes this mismatch problem – the let local's value is computed with the redefinition in place and all is well. The first approach using with-redefs expands to the second approach.
The log printouts are fine, because they are only concerned with the doseq local and never examine any Vars. If they did, they would only see the pre-redefinition values.