Why does my code generate a list of empty lists? - clojure

Question

I have following data structure: There are scenes which can be parts of sequences.
For example, let's say we have scene sc026:
{:sceneId "sc026"}
It is part of seq07:
(def seq07
{
:SeqId "seq07"
:Desc "Sequence name"
:Scenes [sc026]
:Comments []
}
)
Given a list of scenes, I want to create a list which for every scene contains a list of IDs of sequences that a particular scene is part of.
Example
Let's assume there is a list of two scenes sc026 and sc027. sc026 is part of seq07, sc027 is not part of any sequence.
The result I want to achieve is this: [["seq07"], []].
What I tried to implement
I have a function generate-scene-overview which, among others, needs to create that list. It has following signature:
(defn- generate-scene-overview
[scene-list time-info seqs]
scene-list is the collection of scenes (result of (filter some? my-scene-list) where my-scene-list is a list of scenes that contains nil elements).
seqs is a list of sequences.
Sequences in seqs can be structured and unstructured. The unstructured ones have a non-empty list in :Scenes field.
Therefore, in generate-scene-overview I first extract the unstructured scenes from seqs:
unstructured-seqs (filter
(fn [cur-seq]
(let
[scenes (get cur-seq :Scenes)]
(not (empty? scenes))
)
)
seqs)
Next I need to convert the unstructured sequences into a collection of scene-sequence tuples:
unstructured-seq-tuples (compose-unstructured-tuple-list unstructured-seqs)
compose-unstructured-tuple-list is defined as follows.
(defn- compose-unstructured-tuple-list
[unstructured-seqs]
(into []
(map
(fn [cur-seq]
(let
[
scenes (get cur-seq :Scenes)
seqId (get cur-seq :SeqId)
scene-seq-tuples (into []
(map
(fn [cur-scene]
(let [scene-id (get cur-scene :sceneId)]
{
:SceneId scene-id
:SeqId seqId
}
)
)
scenes
)
)
]
scene-seq-tuples
)
)
)
unstructured-seqs
)
)
Next, I need to combine the tuples for structured sequences with those from unstructured ones:
seq-tuples (set/union unstructured-seq-tuples structured-seq-tuples)
Finally, seq-tuples are converted into a list of sequence IDs for each scene:
scene-seqs (compose-scene-seqs scene-list seq-tuples)
compose-scene-seqs is defined as follows:
(defn compose-scene-seqs
[scene-list seq-tuples]
(into [] (map (fn [cur-scene]
(let
[scene-id (get cur-scene :sceneId)]
(findSeqIdsBySceneId scene-id seq-tuples)
)
)
scene-list
)
)
)
findSeqIdsBySceneId looks like this:
(defn findSeqIdsBySceneId
[scene-id seq-tuples]
(let
[
scene-tuples (filter (fn [cur-tuple]
(let [cur-tuple-scene-id (get cur-tuple :SceneId)]
(= scene-id cur-tuple-scene-id))
)
seq-tuples
)
seqs (map (fn [cur-tuple]
(get cur-tuple :SeqId)
)
scene-tuples
)
]
seqs
)
)
My problem
When I run the above code in debugger, scene-seqs only contains empty collections.
It should contain exactly one non-empty collection for scene sc026 (with string seq07 inside it).
How I tried to diagnose the problem
I tried to reproduce the problem in automated tests.
First attempt -- findSeqIdsBySceneId:
(deftest findSeqIdsBySceneId-test
(is (= ["seq07"]
(findSeqIdsBySceneId "sc026" [{
:SceneId "sc026"
:SeqId "seq07"
}])
)
)
(is (= ["seq07", "seq06"]
(findSeqIdsBySceneId "sc026" [{
:SceneId "sc026"
:SeqId "seq07"
}
{
:SceneId "sc026"
:SeqId "seq06"
}
])
)
)
)
Those tests run, so I wrote a couple of tests for compose-scene-seqs:
(deftest compose-scene-seqs-test
(is (= [["seq07"]]
(let
[
scene-list [{:sceneId "sc026"}]
seq-tuples [
{
:SceneId "sc026"
:SeqId "seq07"
}
]
]
(compose-scene-seqs scene-list seq-tuples)
)
))
)
(deftest compose-scene-seqs-test2
(is (= [["seq07"] []]
(let
[
scene-list [
{:sceneId "sc026"}
{:sceneId "sc027"}
]
seq-tuples [
{
:SceneId "sc026"
:SeqId "seq07"
}
]
]
(compose-scene-seqs scene-list seq-tuples)
)
))
)
(deftest compose-scene-seqs-test3
(is (= [[] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] ["seq07"] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] []]
(let
[
scene-list my-scene-list
seq-tuples [
{
:SceneId "sc026"
:SeqId "seq07"
}
]
]
(compose-scene-seqs scene-list seq-tuples)
)
))
)
All of them run.
If I replace
scene-list my-scene-list
with
scene-list (filter some? перечень-сцен2)
I get the following assertion error, but even then there is one non-empty collection:
Question
What else can I do to diagnose and fix the error?
Update 1:
Full code is available in this GitHub gist.
I managed to reproduce the error in the following test:
(deftest compose-scene-seqs-test4
(is (= [[] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] ["seq07"] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] []]
(let
[
scene-list (filter some? перечень-сцен2)
unstructured-seqs [seq07]
unstructured-seq-tuples (compose-unstructured-tuple-list unstructured-seqs)
seq-tuples (set/union unstructured-seq-tuples [])
]
(compose-scene-seqs scene-list seq-tuples)
)
)
)
)

Answer 1

Here is a complete working solution for the stated task:
(defn- contains-scene? [seq scene-id]
(some #(= scene-id (:sceneId %)) (:Scenes seq)))
(defn- seq-ids-containing-scene [seqs scene-id]
(keep #(and (contains-scene? % scene-id) (:SeqId %)) seqs))
(defn seq-ids-containing-scenes [seqs scenes]
(map #(seq-ids-containing-scene seqs (:sceneId %)) scenes))
Test case:
(def sc026 {:sceneId "sc026"})
(def sc027 {:sceneId "sc027"})
(def seq07 {:SeqId "seq07"
:Desc "Sequence name"
:Scenes [sc026]
:Comments []})
(seq-ids-containing-scenes [seq07] [sc026 sc027]) ;; => (("seq07") ())
I couldn't follow the logic of the attempted solution. It introduces a concept "unstructured" (not the same as Clojure's destructuring) which doesn't seem to add value. I tried re-creating the issue but found the presented code was incomplete, so I can't offer any help on why it fails.
Here is a second alternative solution which builds a map scene-map in a single pass over the collection of sequences. scene-map has scene id key and a collection of sequence ids as the corresponding value:
(defn seq-ids-containing-scenes* [seqs scenes]
(let [maps (for [seq seqs
scene (:Scenes seq)]
{(:sceneId scene) [(:SeqId seq)]})
scene-map (apply merge-with into maps)]
(map #(get scene-map (:sceneId %) []) scenes)))
Update:
I found the bug in the original code presented in the question. In function compose-unstructured-tuple-list, replace the first map with mapcat.

Answer 2

I'd probably first refactor this to be a bit smaller
I think you could do this in one pass with reduce by creating a new data structure where the scene name is a key and a list of seqid information build those list up with conj as you iterate through the old data with reduce
In terms of debugging intellji has a step debugger which you can use to observe the list building and an expression window which you can use to run commands like in the repl but in the context of your break point that should hopefully give you enough observability to understand your problem

Answer 3

This is a very interesting problem as it uses data (seqid) conbined with deeper data (secnes). Conceptually that seems to suggest using a zipper but those are hard to deal and only "worth it" if you are manipulating data. If you have a lot of data like this to handle then specter might be worth a look.
Stephans solution is shorter than this attempt but I agree with #arcanine that a single pass solution. Scenes and sequences suggest that multiple passes might not be a problem but here is a single pass effort anyway:
(defn collect-seqs [seqs scenes]
(let [scene->seqs (->> seqs
(mapcat (fn seq-sc-pairs [sq]
(->> sq
:Scenes
(keep (comp (set scenes) :sceneId))
(map (partial vector (:SeqId sq))))))
;; group collection of ([seqid sceneid], ...)
(group-by second))]
(for [sc scenes]
(map first (get scene->seqs sc)))))
Note that this could be a little shorter if a map were an accepable solution. that might also mean that a reduction function would work