Clojure EDN as a Code Generation Metadata source - templates

New to Clojure and wondering how to use it to make other languages I program in easier.
One thing I would like to be able to do is use Clojure for code generation.
For example, given the input from a data file (EDN format) how should I (1) walk this structure or (2) push the data into an existing templating mechanism?
Data below would be for defining simple REST API so that you could generate Clients from it. Generate clients in multiple languages by using different templates.
(:restcall "GetAccountBalance"
{:method "GET" :path "account/balance"}
{:id int})
(:restcall "GetLastTransactions"
{:method "GET" :path "account/transactions"}
{:page int})
resulting code would be something like
public void GetAccountBalance(int id)
{
var input = new { id = id };
callIntoRestLibrary("GET", "account/balance", input);
}
public void GetLastTransactions(int page)
{
var input = new { page = page };
callIntoRestLibrary("GET", "account/transactions", input);
}
Note: my end goal would having these as System.Net.Http.HttpClient calls via C#, but also be able to translate these into JavaScript/Ajax calls also

You have several choices for templating with Clojure. One place to look is the Clojure Toolbox.
Here is an example with clostache, a small library (358 loc) implementation of mustache.
(ns so.core
(:require [clostache.parser :refer (render)]))
(def template "
public void {{restcall}}({{id}} id)
{
var input = new { id = id };
callIntoRestLibrary(\"{{method}}\", \"{{path}}\", input);
}")
(def data
{:restcall "GetAccountBalance"
:method "GET" :path "account/balance" :id "int"})
(print (render template data))
Output:
public void GetAccountBalance(int id)
{
var input = new { id = id };
callIntoRestLibrary("GET", "account/balance", input);
}
To clear up confusion about what it means to read EDN.
(spit "foo.txt" (prn-str data))
Now the file foo.txt contains a textual representation of data, presumably your starting point.
(def data2 (with-open [r (java.io.PushbackReader. (java.io.FileReader. "foo.txt"))]
(clojure.edn/read r)))
(= data data2) ;=> true
So, read doesn't just pull in text, but also parses it into its data representation.

Related

How to Output a Set as Simple Strings?

I need to give an output like this:
/Fotos/Azoren-2018/buffet
/Fotos/Azoren-2018/restaurant/dc
/Fotos/Azoren-2018/restaurant/dc-41.jpg
/Fotos/Azoren-2018/restaurant/dc-42.jpg
/Fotos/Azoren-2018/restaurant/dc-43.jpg
/Fotos/Mallorca-2017/dc-10.jpg
/Fotos/Mallorca-2017/dc-11.jpg
/Fotos/Mallorca-2017/dc-19.jpg
But I think my output is a set:
#{"/Fotos/Mallorca-2017/dc-10.jpg" "/Fotos/Azoren-2018/buffet" "/Fotos/Azoren-2018/restaurant/dc-43.jpg" "/Fotos/Azoren-2018/restaurant/dc-42.jpg" "/Fotos/Mallorca-2017/dc-19.jpg" "/Fotos/Azoren-2018/restaurant/dc-41.jpg" "/Fotos/Mallorca-2017/dc-11.jpg" "/Fotos/Azoren-2018/restaurant/dc"}
Any ideas how can I convert that set into this output?
Use sort with clojure.string/join:
(->> #{"/Fotos/Mallorca-2017/dc-10.jpg" "/Fotos/Azoren-2018/buffet" "/Fotos/Azoren-2018/restaurant/dc-43.jpg" "/Fotos/Azoren-2018/restaurant/dc-42.jpg" "/Fotos/Mallorca-2017/dc-19.jpg" "/Fotos/Azoren-2018/restaurant/dc-41.jpg" "/Fotos/Mallorca-2017/dc-11.jpg" "/Fotos/Azoren-2018/restaurant/dc"}
sort
(clojure.string/join "\n")
println)
Output:
/Fotos/Azoren-2018/buffet
/Fotos/Azoren-2018/restaurant/dc
/Fotos/Azoren-2018/restaurant/dc-41.jpg
/Fotos/Azoren-2018/restaurant/dc-42.jpg
/Fotos/Azoren-2018/restaurant/dc-43.jpg
/Fotos/Mallorca-2017/dc-10.jpg
/Fotos/Mallorca-2017/dc-11.jpg
/Fotos/Mallorca-2017/dc-19.jpg
=> nil

Kadena Chainweaver Deploy Contract Failure: Tx Failed: Keyset failure (keys-any)

I get this error when I Deploy a Hello World contract with Kadena Chainweaver on VM (https://chainweaver-builds.s3.amazonaws.com/2.2/kadena-chainweaver-vm-2.2.0.1.ova):
Failure: Tx Failed: Keyset failure (keys-any)
This is the contract I'm trying to Deploy(It's the default hello world that exists when you create a chainweaver account):
;;
;; "Hello, world!" smart contract/module
;;
;; To try it out, click "Load into REPL" and type into the repl:
;; (free.hello-world.set-message "universe")
;; (free.hello-world.greet)
;;
;; Check lines 21 and 34 to play with Formal Verification
;;
(namespace "free")
;; Define the module.
(module hello-world MODULE_ADMIN
"A smart contract to greet the world."
; no-op module admin for example purposes.
; in a real contract this could enforce a keyset, or
; tally votes, etc.
(defcap MODULE_ADMIN () true)
(defschema message-schema
#doc "Message schema"
#model [(invariant (!= msg ""))]
msg:string)
(deftable
message:{message-schema})
(defun set-message
(
m:string
)
"Set the message that will be used next"
; uncomment the following to make the model happy!
;(enforce (!= m "") "set-message: must not be empty")
(write message "0" {"msg": m})
)
(defun greet ()
"Do the hello-world dance"
(with-default-read message "0" { "msg": "" } { "msg":= msg }
(format "Hello {}!" [msg])))
)
(create-table message)
(set-message "world")
(greet)
Failure: Tx Failed: Keyset failure (keys-any)
The problem is sovled when I changed the module name from hello-world to something else.
Change this:
;; Define the module.
(module hello-world MODULE_ADMIN
"A smart contract to greet the world."
to this:
;; Define the module.
(module hello-world-YOUR-UNIQUE-NAME MODULE_ADMIN
"A smart contract to greet the world."

Batch Processing of Select query in clojure

I have a clojure application. It accepts a POST request with the following payload.
[
{
"key1": value1,
"key2": value2,
"key3": value3,
"key4": value4
},
{
"key1": value1,
"key2": value2,
"key3": value3,
"key4": value4
}
]
For each element of the array we make a SELECT call to DB using jdbc like so
(defn my-definition [db payload]
(->> payload
(map (fn [p]
(if-let [da (my-db-call-method db)]
(assoc da :key1 (:key1 p))
{:key2 (:key2 p)
:key3 (:key3 p)
})))))
DB call
(defn my-db-call-method [db json-params]
(with-sql-handling "my-db-call-method"
(jdbc/query db [my-query
(to long (:key1 json-params))
(to long (:key2 json-params))
)))
my-query.txt would look something like belowSelect * from my_table where col_name1 = ? and col_name2 =? Now, if 1 DB call takes 100 ms then 100 batch size would take 10 sec. This is too long. Is there a way to batch process the request in Clojure? Please note that code is in working condition but taking a long time due to sequential processing. How can we parallel process this? Can we use any third-party library to achieve this?

clj-kafka - consumer empty

I am trying usgin library clj-kafka.
Here my code
(use [clj-kafka.producer]
[clj-kafka.zk]
[clj-kafka.consumer.zk]
[clj-kafka.core]))
(brokers {"zookeeper.connect" "localhost:2181"})
(def p (producer {"metadata.broker.list" "localhost:9092"
"serializer.class" "kafka.serializer.DefaultEncoder"
"partitioner.class" "kafka.producer.DefaultPartitioner"}))
(send-message p (message "test" (.getBytes "this is my message")))
(def config {"zookeeper.connect" "localhost:2181"
"group.id" "clj-kafka.consumer"
"auto.offset.reset" "smallest"
"auto.commit.enable" "false"})
(with-resource [c (consumer config)]
shutdown
(take 2 (messages c "test"))) ;; return ()
I start zookepper-server and kafka itself with
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
The config/zookepper.properties:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
and config/server.properties:
broker.id=0
listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
zocket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
The 'problem' is when I execute:
(with-resource [c (consumer config)]
shutdown
(take 2 (messages c "test"))) ;; return empty ()
Any idea here?
Thanks in advance
See this github issue. Seems the documentation isn't great. You have to force the realization of the sequence (which is lazy) with doall. try this:
(with-resource [c (consumer config)]
shutdown
(doall (take 2 (messages c "test"))))

Calling external services in scala code with dependencies

I am facing a major issue with my design at this juncture. My method is trying to accomplish the follows:
Insert the passed in object into the database.
Get the autoincremented id from the insert and use it to call webservice1 along with the object.
Get the result from webservice1 and call webservice2 with the original object and some response from webservice1.
Combine the results from webservice1 and 2 and write it into the database.
Get the resulting autoincremented id from the last insert and call webservice3 with the original object that would eventually result into the success or failure of the operation.
I want to design this in a flexible manner since the requirements are in a flux and I do not want to keep on modifying my logic based on any changing. I do realize some amount of change is inevitable but I would like to minimize the damage and respect the open-closed principle.
My initial take was as follows:
def complexOperation(someObject:T) =
dbService.insertIntoDb(someObject) match {
case Left(e:Exception) => Left(e)
case Right(id:Int) => webService.callWebService1(id,someObject) match {
case Left(e:Exception) => Left(e)
case Right(r:SomeResponse1) => webService.callWebservice2(r,someObject) match {
case Left(e:Exception) => webService.rollbackService1();Left(e)
case Right(context:ResponseContext) => dbService.insertContextIntoDb(context) match {
case Left(e:Exception) => Left(e)
case Right(id:Int) => webService.callWebservice3(id,someObject) match {
case Left(e:Exception) => webService.rollbackService3();Left(e)
case Right(r:Response) => Right(r)
}
}
}
}
As you can see, this is a tangled mess. I can neither unit test it, nor extend it nor very easily debug it if things spiral out of control. This code serves its purpose but it will be great to get some ideas on how I should refactor it to make the lives of the people who inherit my code a little more easier.
Thanks
Have a look at scala.util.Try. It's available in Scala 2.10, which may or may not be available to you as an option, but the idea of it is perfect for your scenario.
What you have in your code example is what I like calling the "pyramid" of nesting. The best solution to this is to use flat-mapping wherever you can. But obviously that's an issue when you have stuff like Either[Exception, Result] at every step. That's where Try comes in. Try[T] is essentially a replacement for Either[Exception, T], and it comes with all of the flatMap-ing goodness that you need.
Assuming you can either change the return type of those webService calls, or provide some implicit conversion from Either[Exception, Result] to Try[Result], your code block would become something more like...
for {
id <- dbService.insertIntoDb(someObject)
r <- webService.callWebService1(id,someObject)
context <- webService.callWebservice2(r,someObject)
id2 <- dbService.insertContextIntoDb(context)
response <- webService.callWebservice3(id,someObject).recoverWith {
case e: Exception => webService.rollbackService3(); Failure(e)
}
} yield response
Lift has a similar mechanism in net.liftweb.common.Box. It's like Option, but with a container for Exceptions too.
edit: It looks like you can use the left or right method of an Either, and it will let you use flatMap-ing almost exactly the way I described with Try. The only difference is that the end result is an Either[Exception, Result] instead of a Try[Result]. Check out LeftProjection for details/examples.
You can use for comprehension to reduce the noise in the code.
#Dylan had the right idea above. Let me see if I can help translate what you want to do into idiomatic Scala 2.9.1 code.
This version doesn't attempt any rollbacks:
// 1: No rollbacks, just returns the first exception in Left
def complexOperation1(someObject:T): Either[Exception, Response] = {
for {
id <- dbService.insertIntoDb(someObject).right
r <- webService.callWebService1(id, someObject).right
context <- webService.callWebservice2(idResp, someObject).right
id2 <- dbService.insertContextIntoDb(context).right
response <- webService.callWebservice3(id,someObject).right
} yield response
}
Now, let's try to do the rollbacks exactly as you had them above:
// 2: Rolls back all web services and returns first exception in Left
def complexOperation1(someObject:T): Either[Exception, Response] = {
for {
id <- dbService.insertIntoDb(someObject).right
r <- webService.callWebService1(id, someObject).right
context <- webService.callWebservice2(idResp, someObject).left.map { e =>
webService.rollbackService1()
e
}.right
id2 <- dbService.insertContextIntoDb(context).right
response <- webService.callWebservice3(id,someObject).left.map { e =>
webService.rollbackService3()
e
}.right
} yield response
}
If you define a function which does the effect (the rollback) on the left, it get's a little cleaner and easier to test, for example:
// 3: Factor out the side-effect of doing the follbacks on Left
def rollbackIfLeft[T](f: => Either[Exception, T], r: => Unit): Either[Exception, T] = {
val result = f
result.left.foreach(_ => r) // do the rollback if any exception occured
result
}
def complexOperation1(someObject:T): Either[Exception, Response] = {
for {
id <- dbService.insertIntoDb(someObject).right
r <- webService.callWebService1(id, someObject).right
context <- rollbackIfLeft(webService.callWebservice2(idResp, someObject),
webService.rollbackService1()).right
id2 <- dbService.insertContextIntoDb(context).right
response <- rollbackIfLeft(webService.callWebservice3(id,someObject),
webService.rollbackService3()).right
} yield response
}
You can try out rollbackIfLeft in the scala REPL to get a sense of it:
scala> rollbackIfLeft(Right(42), println("hey"))
res28: Either[Exception,Int] = Right(42)
scala> rollbackIfLeft(Left(new RuntimeException), println("ERROR!"))
ERROR!
res29: Either[Exception,Nothing] = Left(java.lang.RuntimeException)
Hope this helps!