Here is a parsing example using Enlive. Would there be differences with Enliven?
(ns parse.enlive
(:require [net.cgrand.enlive-html :as html]))
(def ^:dynamic *base-url* "https://news.ycombinator.com/")
(defn fetch-url [url]
(html/html-resource (java.net.URL. url)))
(defn hn-headlines []
(map html/text (html/select (fetch-url *base-url*) [:td.title :a])))
(defn hn-points []
(map html/text (html/select (fetch-url *base-url*) [:td.subtext html/first-child])))
(defn print-headlines-and-points []
(doseq [line (map #(str %1 " (" %2 ")") (hn-headlines) (hn-points))]
(println line)))
If there are not much differences in that simple example, where would Enliven be different when doing some web scraping?
Thanks!
Related
Is there anyway to include clojure.spec'd functions in a generalized test suite? I know we can register specs and directly spec functions.
(ns foo
(:require [clojure.spec :as s]
[clojure.spec.test :as stest]))
(defn average [list-sum list-count]
(/ list-sum list-count))
(s/fdef average
:args (s/and (s/cat :list-sum float? :list-count integer?)
#(not (zero? (:list-count %))))
:ret number?)
And later, if I want to run generative tests against that spec'd function, I can use stest/check.
=> (stest/check `average)
({:spec #object[clojure.spec$fspec_impl$reify__14282 0x68e9f37c "clojure.spec$fspec_impl$reify__14282#68e9f37c"], :clojure.spec.test.check/ret {:result true, :num-tests 1000, :seed 1479587517232}, :sym edgar.core.analysis.lagging/average})
But i) is there anyway to include these test runs in my general test suite? I'm thinking of the kind of clojure.test integration that test.check has. The closest thing that I can see ii) is the stest/instrument (see here) function. But that seems to just let us turn on checking at the repl. Not quite what I want. Also, iii) are function specs registered?
(defspec foo-test
100
;; NOT this
#_(prop/for-all [v ...]
(= v ...))
;; but THIS
(stest/some-unknown-spec-fn foo))
Ok, solved this one. Turns out there's no solution out of the box. But some people on the clojure-spec slack channel have put together a defspec-test solution for clojure.spec.test and clojure.test.
So given the code in the question. You can A) define the defspec-test macro that takes your test name and a list of spec'd functions. You can then B) use it in your test suite.
Thanks Clojure community!! And hopefully such a utility function makes it into the core library.
A)
(ns foo.test
(:require [clojure.test :as t]
[clojure.string :as str]))
(defmacro defspec-test
([name sym-or-syms] `(defspec-test ~name ~sym-or-syms nil))
([name sym-or-syms opts]
(when t/*load-tests*
`(def ~(vary-meta name assoc
:test `(fn []
(let [check-results# (clojure.spec.test/check ~sym-or-syms ~opts)
checks-passed?# (every? nil? (map :failure check-results#))]
(if checks-passed?#
(t/do-report {:type :pass
:message (str "Generative tests pass for "
(str/join ", " (map :sym check-results#)))})
(doseq [failed-check# (filter :failure check-results#)
:let [r# (clojure.spec.test/abbrev-result failed-check#)
failure# (:failure r#)]]
(t/do-report
{:type :fail
:message (with-out-str (clojure.spec/explain-out failure#))
:expected (->> r# :spec rest (apply hash-map) :ret)
:actual (if (instance? Throwable failure#)
failure#
(:clojure.spec.test/val failure#))})))
checks-passed?#)))
(fn [] (t/test-var (var ~name)))))))
B)
(ns foo-test
(:require [foo.test :refer [defspec-test]]
[foo]))
(defspec-test test-average [foo/average])
The above example can fail in the case where :failure is false due to how stest/abbrev-result tests for failure. See CLJ-2246 for more details. You can work around this by defining your own version of abbrev-result. Also, the formatting of failure data has changed.
(require
'[clojure.string :as str]
'[clojure.test :as test]
'[clojure.spec.alpha :as s]
'[clojure.spec.test.alpha :as stest])
;; extracted from clojure.spec.test.alpha
(defn failure-type [x] (::s/failure (ex-data x)))
(defn unwrap-failure [x] (if (failure-type x) (ex-data x) x))
(defn failure? [{:keys [:failure]}] (not (or (true? failure) (nil? failure))))
;; modified from clojure.spec.test.alpha
(defn abbrev-result [x]
(let [failure (:failure x)]
(if (failure? x)
(-> (dissoc x ::stc/ret)
(update :spec s/describe)
(update :failure unwrap-failure))
(dissoc x :spec ::stc/ret))))
(defn throwable? [x]
(instance? Throwable x))
(defn failure-report [failure]
(let [expected (->> (abbrev-result failure) :spec rest (apply hash-map) :ret)]
(if (throwable? failure)
{:type :error
:message "Exception thrown in check"
:expected expected
:actual failure}
(let [data (ex-data (get-in failure
[::stc/ret
:result-data
:clojure.test.check.properties/error]))]
{:type :fail
:message (with-out-str (s/explain-out data))
:expected expected
:actual (::s/value data)}))))
(defn check?
[msg [_ body :as form]]
`(let [results# ~body
failures# (filter failure? results#)]
(if (empty? failures#)
[{:type :pass
:message (str "Generative tests pass for "
(str/join ", " (map :sym results#)))}]
(map failure-report failures#))))
(defmethod test/assert-expr 'check?
[msg form]
`(dorun (map test/do-report ~(check? msg form))))
Here's a slightly modified version of grzm's excellent answer that works with [org.clojure/test.check "0.10.0-alpha4"]. It uses the new :pass? key that comes from this PR: https://github.com/clojure/test.check/commit/09927b64a60c8bfbffe2e4a88d76ee4046eef1bc#diff-5eb045ad9cf20dd057f8344a877abd89R1184.
(:require [clojure.test :as t]
[clojure.string :as str]
[clojure.spec.alpha :as s]
[clojure.spec.test.alpha :as stest])
(alias 'stc 'clojure.spec.test.check)
;; extracted from clojure.spec.test.alpha
(defn failure-type [x] (::s/failure (ex-data x)))
(defn unwrap-failure [x] (if (failure-type x) (ex-data x) x))
;; modified from clojure.spec.test.alpha
(defn abbrev-result [x]
(if (-> x :stc/ret :pass?)
(dissoc x :spec ::stc/ret)
(-> (dissoc x ::stc/ret)
(update :spec s/describe)
(update :failure unwrap-failure))))
(defn throwable? [x]
(instance? Throwable x))
(defn failure-report [failure]
(let [abbrev (abbrev-result failure)
expected (->> abbrev :spec rest (apply hash-map) :ret)
reason (:failure abbrev)]
(if (throwable? reason)
{:type :error
:message "Exception thrown in check"
:expected expected
:actual reason}
(let [data (ex-data (get-in failure
[::stc/ret
:shrunk
:result-data
:clojure.test.check.properties/error]))]
{:type :fail
:message (with-out-str (s/explain-out data))
:expected expected
:actual (::s/value data)}))))
(defn check?
[msg [_ body :as form]]
`(let [results# ~body
failures# (remove (comp :pass? ::stc/ret) results#)]
(if (empty? failures#)
[{:type :pass
:message (str "Generative tests pass for "
(str/join ", " (map :sym results#)))}]
(map failure-report failures#))))
(defmethod t/assert-expr 'check?
[msg form]
`(dorun (map t/do-report ~(check? msg form))))
Usage:
(deftest whatever-test
(is (check? (stest/check `whatever
;; optional
{:clojure.spec.test.check/opts {:num-tests 10000}})))
I'm trying to parse HTML with CSS into Hiccup in a Reagent project. I am using Hickory. When I parse HTML with inline CSS, React throws an exception.
(map
as-hiccup (parse-fragment "<div style='color:red'>test</div>")
)
The above generates [:div {:style color:red} "test"] & Reactjs returns exception from Reactjs:
Violation: The style prop expects a mapping from style properties to values, not a string.
I believe [:div {:style {"color" "red"}} "test"] must be returned instead.
Here is the code view:
(ns main.views.job
(:require [reagent.core :as reagent :refer [atom]]
[hickory.core :refer [as-hiccup parse parse-fragment]]))
(enable-console-print!)
(defn some-view [uid]
[:div
(map as-hiccup (parse-fragment "<div style='color:red'>test</div>"))
])
The whole repo is here and it works. I added the parsing from style tag to a map for React in the core.cljs file:
(ns hickory-stack.core
(:require [clojure.string :as s]
[clojure.walk :as w]
[reagent.core :as reagent :refer [atom]]
[hickory.core :as h]))
(enable-console-print!)
(defn string->tokens
"Takes a string with syles and parses it into properties and value tokens"
[style]
{:pre [(string? style)]
:post [(even? (count %))]}
(->> (s/split style #";")
(mapcat #(s/split % #":"))
(map s/trim)))
(defn tokens->map
"Takes a seq of tokens with the properties (even) and their values (odd)
and returns a map of {properties values}"
[tokens]
{:pre [(even? (count tokens))]
:post [(map? %)]}
(zipmap (keep-indexed #(if (even? %1) %2) tokens)
(keep-indexed #(if (odd? %1) %2) tokens)))
(defn style->map
"Takes an inline style attribute stirng and converts it to a React Style map"
[style]
(tokens->map (string->tokens style)))
(defn hiccup->sablono
"Transforms a style inline attribute into a style map for React"
[coll]
(w/postwalk
(fn [x]
(if (map? x)
(update-in x [:style] style->map)
x))
coll))
;; Test Data
(def good-style "color:red;background:black; font-style: normal ;font-size : 20px")
(def html-fragment
(str "<div style='" good-style "'><div id='a' class='btn' style='font-size:30px;color:white'>test1</div>test2</div>"))
;; Rendering
(defn some-view []
[:div (hiccup->sablono
(first (map h/as-hiccup (h/parse-fragment html-fragment))))])
(reagent/render-component [some-view]
(. js/document (getElementById "app")))
Now can use compojure this way:
(GET ["/uri"] [para1 para2]
）
Para1 and para2 are all of type String.
I would like to let it know the type correcttly,like this:
(GET ["/uri"] [^String para1 ^Integer para2]
）
It can convert para1 to be Sting and para2 to Integer.
Is there some library or good way to do this?
This is possible as of Compojure 1.4.0 using the syntax [x :<< as-int]
This is not currently possible with only Compojure.
You could use Prismatic schema coercion.
(require '[schema.core :as s])
(require '[schema.coerce :as c])
(require '[compojure.core :refer :all])
(require '[ring.middleware.params :as rparams])
(def data {:para1 s/Str :para2 s/Int s/Any s/Any})
(def data-coercer (c/coercer data c/string-coercion-matcher ))
(def get-uri
(GET "/uri" r
(let [{:keys [para1 para2]} (data-coercer (:params r))]
(pr-str {:k1 para1 :k2 (inc para2)}))))
(def get-uri-wrapped
(let [keywordizer (fn [h]
(fn [r]
(h (update-in r [:params] #(clojure.walk/keywordize-keys %)))))]
(-> get-uri keywordizer rparams/wrap-params)))
Here is a sample run:
(get-uri-wrapped {:uri "/uri" :query-string "para1=a¶2=3" :request-method :get})
{:status 200,
:headers {"Content-Type" "text/html; charset=utf-8"},
:body "{:k1 \"a\", :k2 4}"}
This function takes a list of a files and is supposed to return a list of artists:
(defn get-artists [files]
(map #(.get-artist (->Mp3 %)) files))
Here the rest of the code:
(ns musicdb.filesystem
(:use [green-tags.core]))
(import '(java.io.File) '(java.net.url) '(java.io))
(require '[clojure.string :as str])
(defn get-files [search-path]
(let [directory (clojure.java.io/file search-path)
files (file-seq directory)
fonly (filter #(.isFile %) files)]
(map #(last (str/split (.toString %) #"/")) fonly)))
(defprotocol MusicFile
(get-artist [this])
(get-song [this])
(get-album [this]))
(defrecord Mp3 [filename]
MusicFile
(get-artist [this]
(:artist (get-all-info filename)))
(get-song [this]
(:title (get-all-info filename)))
(get-album [this]
(:album (get-all-info filename))))
And here are my tests:
(ns musicdb.core-test
(:require [clojure.test :refer :all]
[musicdb.core :refer :all]
[musicdb.filesystem :refer :all]
[clojure.pprint :refer :all]
))
(deftest test_0
(testing "getFiles returns valid result"
(is (> (count (get-files "/home/ls/books/books")) 50))))
(deftest test_1
(testing "check for file included"
(is (some #{"02 Backlit.mp3"} (get-files "/home/ls/Musik")))))
(deftest test_2
(testing "creating music file record"
(let [myfile (->Mp3 "/home/ls/Musik/Panopticon/02 Backlit.mp3")]
(is (= "Isis" (.get-artist myfile)))
(is (= "Backlit" (.get-song myfile))))))
(deftest test_3
(testing "testing get-artists"
(let [artists (get-artists (get-files "/home/ls/Musik"))
]
(is (> (count artists) 10)))))
(deftest test_4
(testing "testing get-artists check for artist"
(let [artists (get-artists (get-files "/home/ls/Musik"))
]
(is (some #{"Isis"} artists))))) ;artists is [nil nil nil ...]
From this tests only the last fails, which returns a list of nils.
If you want to reproduce ths be sure to include the green-tags dependency in your leiningen project.clj:
[green-tags "0.3.0-alpha"]
Your get-files function doesn't return the full path of the file so get-all-info just returns nil (https://github.com/DanPallas/green-tags/blob/master/src/green_tags/core.clj#L59 in combination with https://github.com/DanPallas/green-tags/blob/master/src/green_tags/core.clj#L120).
Here is a simple example that works:
(map (comp :artist get-all-info)
(filter #(.isFile %)
(file-seq (java.io.File. "/home/vema/Downloads/mp3"))))
;=> ("Yo Yo Honey Singh (DJJOhAL.Com)")
(Humoristic?) disclaimer: The MP3 should not be taken as an example of my musical taste, it was just the first free MP3 I found online.
two files
types.clj:
(ns test.types)
(defrecord Price [date price])
(defrecord ProductPrice [name prices])
core.clj (It's OK)
(ns test.core
(:use [test.types])
(:use [clojure.string :only (split)]))
(defn read-data [file]
(let [name (subs (.getName file) 0 4)]
(with-open [rdr (clojure.java.io/reader file)]
(doall (map #(apply ->Price (split % #"\t")) (drop 2 (line-seq rdr)))))))
core.clj (java.lang.IllegalArgumentException: Unable to resolve classname: ProductPrice)
(ns test.core
(:use [test.types])
(:use [clojure.string :only (split)]))
(defn read-data [file]
(let [name (subs (.getName file) 0 4)]
(with-open [rdr (clojure.java.io/reader file)]
(ProductPrice. name (doall (map #(apply ->Price (split % #"\t")) (drop 2 (line-seq rdr))))))))
core.clj (It's OK)
(ns test.core
(:use [test.types])
(:use [clojure.string :only (split)]))
(defrecord tProductPrice [name prices])
(defn read-data [file]
(let [name (subs (.getName file) 0 4)]
(with-open [rdr (clojure.java.io/reader file)]
(tProductPrice. name (doall (map #(apply ->Price (split % #"\t")) (drop 2 (line-seq rdr)))))))
core.clj (java.lang.IllegalStateException: ->ProductPrice already refers to: #'test.types/->ProductPrice in namespace: test.core)
(ns test.core
(:use [test.types])
(:use [clojure.string :only (split)]))
(defrecord ProductPrice [name prices])
(defn read-data [file]
(let [name (subs (.getName file) 0 4)]
(with-open [rdr (clojure.java.io/reader file)]
(ProductPrice. name (doall (map #(apply ->Price (split % #"\t")) (drop 2 (line-seq rdr)))))))
I totally confused about these exceptions. And I can't find any more usage about 'record' except some simplest examples from clojure.org and books.
Any help, Thank you very much!
defrecord creates a java class in the package named after the current namespace. (ProductPrice. ...) is a call to the constructor of that type; this is java interop - not a plain function call.
You cannot refer to a class defined outside of java.lang or the current namespace unless you explicitly import it or specify the full package name. This includes calling its constructor.
So, to fix the problem you need to import Price and ProductPrice.
(ns test.core (:import [test.types Price]))
(Price. ...)
or call the full class+package name:
(test.types.Price. ...)