Clojure REST API: Logging issue - clojure

I have figured out one issue with logging in clojure api.
I try to implement logging mechanism for API Requests and SQL statements.
Project.clj
(defproject clojure-dauble-business-api "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:ring {:handler clojure-dauble-business-api.routes/app}
:dependencies [[org.clojure/clojure "1.8.0"]
[metosin/compojure-api "1.1.10"]
[ring/ring-core "1.4.0"]
[ring/ring-jetty-adapter "1.4.0"]
[ring/ring-defaults "0.2.0"]
[org.clojure/java.jdbc "0.6.1"]
[yesql/yesql "0.5.3"]
[org.postgresql/postgresql "9.4.1212"]
[org.clojure/data.json "0.2.6"]
[cheshire "5.7.1"]
[com.googlecode.log4jdbc/log4jdbc "1.2"]
[org.slf4j/slf4j-simple "1.7.12"]
[org.clojure/tools.logging "0.3.1"]
[log4j/log4j "1.2.17" :exclusions [javax.mail/mail
javax.jms/jms
com.sun.jmdk/jmxtools
com.sun.jmx/jmxri]]]
:profiles {:dev {:resource-paths ["resources"]}}
:plugins [[lein-ring "0.12.0"]]
:main clojure-dauble-business-api.routes)
and
log4j.properties
# Application Logging
log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=./log/app.log
log4j.appender.file.MaxFileSize=1MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=[%d][%p][%c] %m%n
# # the appender used for the JDBC API layer call logging above, sql only
log4j.appender.sql=org.apache.log4j.RollingFileAppender
log4j.appender.sql.layout=org.apache.log4j.PatternLayout
log4j.appender.sql.layout.ConversionPattern= \u001b[0;31m (SQL)\u001b[m %d{yyyy-MM-dd HH:mm:ss.SSS} \u001b[0;32m %m \u001b[m %n
#
# # ==============================================================================
# # JDBC API layer call logging :
# # INFO shows logging, DEBUG also shows where in code the jdbc calls were made,
# # setting DEBUG to true might cause minor slow-down in some environments.
# # If you experience too much slowness, use INFO instead.
#
log4jdbc.drivers=org.postgresql.Driver
#
# # Log all JDBC calls except for ResultSet calls
log4j.logger.jdbc.audit=INFO,sql
log4j.additivity.jdbc.audit=false
#
# # Log only JDBC calls to ResultSet objects
log4j.logger.jdbc.resultset=INFO,sql
log4j.additivity.jdbc.resultset=false
#
# # Log only the SQL that is executed.
log4j.logger.jdbc.sqlonly=INFO,sql
log4j.additivity.jdbc.sqlonly=false
#
# # Log timing information about the SQL that is executed.
log4j.logger.jdbc.sqltiming=INFO,sql
log4j.additivity.jdbc.sqltiming=false
#
# # Log connection open/close events and connection number dump
log4j.logger.jdbc.connection=INFO,sql
log4j.additivity.jdbc.connection=false
here
[com.googlecode.log4jdbc/log4jdbc "1.2"]
[org.slf4j/slf4j-simple "1.7.12"]
overrides logging mechanism. Means, Logs are getting displayed on console instead creating app.log file(as per log4j.properties configurations).
Only API requests logs are getting displayed on console and It does not show SQL Logs.
Why logs are getting displayed on console? and How can I logs SQL logs in file?

your project should depend on slf4j-log4j12 instead of slf4j-simple? Since
Binding for Simple implementation, which outputs all events to System.err. Only messages of level INFO and higher are printed. This binding may be useful in the context of small applications.
https://www.slf4j.org/manual.html
It should be a java question.
edit:
This is a second question, actually.
The log4jdbc library using a proxy datasource to log sql statement, so you should use the datasource class provided by log4jdbc net.sf.log4jdbc.Log4jdbcProxyDataSource
you can get more info by just search google by log4jdbc
http://blog.jhades.org/logging-the-actualreal-sql-queries-of-a-springhibernate-application/

I have figured answer to my issue.
Modified project.clj
(defproject clojure-dauble-business-api "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:ring {:handler clojure-dauble-business-api.routes/app}
:dependencies [[org.clojure/clojure "1.8.0"]
[metosin/compojure-api "1.1.10"]
[ring/ring-core "1.4.0"]
[ring/ring-jetty-adapter "1.4.0"]
[ring/ring-defaults "0.2.0"]
[org.clojure/java.jdbc "0.6.1"]
[yesql/yesql "0.5.3"]
[org.postgresql/postgresql "9.4.1212"]
[org.clojure/data.json "0.2.6"]
[cheshire "5.7.1"]
[com.googlecode.log4jdbc/log4jdbc "1.2"]
[org.slf4j/slf4j-log4j12 "1.7.21"]
[org.clojure/tools.logging "0.3.1"]
[log4j/log4j "1.2.17" :exclusions [javax.mail/mail
javax.jms/jms
com.sun.jmdk/jmxtools
com.sun.jmx/jmxri]]]
:profiles {:dev {:resource-paths ["resources"]}}
:plugins [[lein-ring "0.12.0"]]
:main clojure-dauble-business-api.routes)
Modified log4j.properties file is here
# Application Logging
log4j.rootLogger=info, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=./log/app.log
log4j.appender.file.MaxFileSize=1MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=[%d][%p][%c] %m%n
#
# # ==============================================================================
# # JDBC API layer call logging :
# # INFO shows logging, DEBUG also shows where in code the jdbc calls were made,
# # setting DEBUG to true might cause minor slow-down in some environments.
# # If you experience too much slowness, use INFO instead.
#
log4jdbc.drivers=org.postgresql.Driver
#
# # Log all JDBC calls except for ResultSet calls
log4j.logger.jdbc.audit=INFO,sql
log4j.additivity.jdbc.audit=false
#
# # Log only JDBC calls to ResultSet objects
log4j.logger.jdbc.resultset=INFO,sql
log4j.additivity.jdbc.resultset=false
#
# # Log only the SQL that is executed.
log4j.logger.jdbc.sqlonly=INFO,sql
log4j.additivity.jdbc.sqlonly=false
#
# # Log timing information about the SQL that is executed.
log4j.logger.jdbc.sqltiming=INFO,sql
log4j.additivity.jdbc.sqltiming=false
#
# # Log connection open/close events and connection number dump
log4j.logger.jdbc.connection=INFO,sql
log4j.additivity.jdbc.connection=false
! the appender used for the JDBC API layer call logging above, sql only
log4j.appender.sql=org.apache.log4j.FileAppender
log4j.appender.sql.File=./log/sql.log
log4j.appender.sql.Append=false
log4j.appender.sql.layout=org.apache.log4j.PatternLayout
log4j.appender.sql.layout.ConversionPattern=[%d][%p][%c] %m%n

Related

Problems creating an executeable for a luminus clojure project

I am building a server program to provide an API to a mongo database containing details of photos. All works well running it from the REPL, or if I start it using lein run.
The problems happen when I try and run the uberjar. Building the uberjar works ok:
(master) photo-api: lein uberjar
Compiling photo-api.env
Compiling photo-api.config
Compiling photo-api.core
WARNING: find-keyword already refers to: #'clojure.core/find-keyword in namespace: image-lib.core, being replaced by: #'image-lib.core/find-keyword
Compiling photo-api.db.core
Compiling photo-api.handler
Compiling photo-api.layout
Compiling photo-api.middleware
Compiling photo-api.routes.home
Compiling photo-api.routes.services
Created /Users/iain/Code/Clojure/Luminus/photo-api/target/uberjar/photo-api-0.1.0-SNAPSHOT.jar
Created /Users/iain/Code/Clojure/Luminus/photo-api/target/uberjar/photo-api.jar
(master) photo-api:
but running the jar gives an error:
(master) photo-api: java -jar target/uberjar/photo-api.jar
WARNING: find-keyword already refers to: #'clojure.core/find-keyword in namespace: image-lib.core, being replaced by: #'image-lib.core/find-keyword
read config from resource: "config.edn"
Exception in thread "main" java.lang.RuntimeException: could not start [#'photo-api.db.core/db*] due to
at mount.core$up$fn__420.invoke(core.cljc:92)
at mount.core$up.invokeStatic(core.cljc:92)
at mount.core$up.invoke(core.cljc:90)
at mount.core$bring.invokeStatic(core.cljc:210)
at mount.core$bring.invoke(core.cljc:202)
at mount.core$start.invokeStatic(core.cljc:252)
at mount.core$start.doInvoke(core.cljc:244)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at mount.core$start_with_args.invokeStatic(core.cljc:350)
at mount.core$start_with_args.doInvoke(core.cljc:346)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at photo_api.core$start_app.invokeStatic(core.clj:42)
at photo_api.core$start_app.invoke(core.clj:41)
at photo_api.core$_main.invokeStatic(core.clj:50)
at photo_api.core$_main.doInvoke(core.clj:49)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at photo_api.core.main(Unknown Source)
Caused by: java.lang.NullPointerException
at com.mongodb.ConnectionString.<init>(ConnectionString.java:222)
at com.mongodb.MongoClientURI.<init>(MongoClientURI.java:189)
at com.mongodb.MongoClientURI.<init>(MongoClientURI.java:171)
at monger.core$connect_via_uri.invokeStatic(core.clj:230)
at monger.core$connect_via_uri.invoke(core.clj:230)
at photo_api.db.core$fn__12620.invokeStatic(core.clj:29)
at photo_api.db.core$fn__12620.invoke(core.clj:28)
at mount.core$record_BANG_.invokeStatic(core.cljc:86)
at mount.core$record_BANG_.invoke(core.cljc:85)
at mount.core$up$fn__420.invoke(core.cljc:93)
... 18 more
Or the SNAPSHOT jar.
(master) photo-api: java -jar target/uberjar/photo-api-0.1.0-SNAPSHOT.jar
Exception in thread "main" java.lang.NoClassDefFoundError: clojure/lang/Var
at photo_api.core.<clinit>(Unknown Source)
Caused by: java.lang.ClassNotFoundException: clojure.lang.Var
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
(master) photo-api:
What am I missing here? I'm guessing it's something to do with dev build and production build differences, but I can't spot anything obvious and the error messages are not a lot of help to me.
This is the project file which as far as I remember is mostly straight from the original luminus template:
(defproject photo-api "0.1.0-SNAPSHOT"
:description "API for geting data from photos db"
:url "http://soulflyer.com"
:dependencies [[cider/cider-nrepl "0.14.0"]
[clj-time "0.13.0"]
[com.google.guava/guava "20.0"]
[com.novemberain/monger "3.1.0" :exclusions [com.google.guava/guava]]
[compojure "1.6.0"]
[cprop "0.1.10"]
[funcool/struct "1.0.0"]
[luminus-immutant "0.2.3"]
[luminus-nrepl "0.1.4"]
[luminus/ring-ttl-session "0.3.2"]
[markdown-clj "0.9.99"]
[metosin/compojure-api "1.1.10"]
[metosin/muuntaja "0.3.1"]
[metosin/ring-http-response "0.9.0"]
[mount "0.1.11"]
[org.clojure/clojure "1.8.0"]
[org.clojure/tools.cli "0.3.5"]
[org.clojure/tools.logging "0.4.0"]
[org.clojure/data.json "0.2.6"]
[org.webjars.bower/tether "1.4.0"]
[org.webjars/bootstrap "4.0.0-alpha.5"]
[org.webjars/font-awesome "4.7.0"]
[org.webjars/jquery "3.1.1"]
[ring-webjars "0.2.0"]
[ring/ring-core "1.6.1"]
[ring/ring-defaults "0.3.0"]
[selmer "1.10.7"]
[image-lib "0.2.1-SNAPSHOT"]]
:min-lein-version "2.0.0"
:jvm-opts ["-server" "-Dconf=.lein-env"]
:source-paths ["src/clj"]
:test-paths ["test/clj"]
:resource-paths ["resources"]
:target-path "target/%s/"
:main ^:skip-aot photo-api.core
:plugins [[lein-cprop "1.0.3"]
[org.clojars.punkisdead/lein-cucumber "1.0.5"]
[lein-immutant "2.1.0"]]
:cucumber-feature-paths ["test/clj/features"]
:profiles
{:uberjar {:omit-source true
:aot :all
:uberjar-name "photo-api.jar"
:source-paths ["env/prod/clj"]
:resource-paths ["env/prod/resources"]}
:dev [:project/dev :profiles/dev]
:test [:project/dev :project/test :profiles/test]
:project/dev {:dependencies [[prone "1.1.4"]
[ring/ring-mock "0.3.0"]
[ring/ring-devel "1.6.1"]
[pjstadig/humane-test-output "0.8.2"]
[clj-webdriver/clj-webdriver "0.7.2"]
[org.apache.httpcomponents/httpcore "4.4"]
[org.clojure/core.cache "0.6.3"]
[org.seleniumhq.selenium/selenium-server "2.48.2"]]
:plugins [[com.jakemccrary/lein-test-refresh "0.19.0"]
[refactor-nrepl "2.3.0-SNAPSHOT"]]
:source-paths ["env/dev/clj"]
:resource-paths ["env/dev/resources"]
:repl-options {:init-ns user}
:injections [(require 'pjstadig.humane-test-output)
(pjstadig.humane-test-output/activate!)]}
:project/test {:resource-paths ["env/test/resources"]}
:profiles/dev {}
:profiles/test {}})
I think you're just missing required configuration - at least Mongo connection string.
When running in dev mode, the luminus (and cprops configuration library) uses the config from your profiles.clj.
However, this is the leiningen thing and it's not available when running from uberjar.
You can specify all your "production" configuration in "config.edn" file and run uberjar like this:
java -Dconf=config.edn -jar target/uberjar/photo-api.jar
You can also use java system properties to set configuration values one by one.
See http://www.luminusweb.net/docs/environment.md for more details.

How to create an uberjar correctly with "lein uberjar"?

if I use lein run the project is no problem.
But when I use jave -jar blog.jar after lein uberjar It happen exceptions.
16-Jul-20 11:28:05 DESKTOP-C3SC9AR INFO [slf4j-timbre.adapter] - >> starting.. *db*
Exception in thread "main" java.lang.RuntimeException: could not start [*db*] due to
.....
Caused by: java.lang.Exception: :jdbc-url, :datasource, or :datasource-
AND my project.clj file
(defproject blog "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:dependencies ....
:min-lein-version "2.0.0"
:uberjar-name "blog.jar"
:jvm-opts ["-server"]
:main blog.core
:migratus {:store :database}
:plugins [[lein-environ "1.0.1"]
[migratus-lein "0.2.0"]]
:profiles
{:uberjar {:omit-source true
:env {:production true}
:aot :all
:source-paths ["env/prod/clj"]}
:dev [:project/dev :profiles/dev]
:test [:project/test :profiles/test]
:project/dev ...
:project/test ...
:profiles/dev {:env {:database-url "jdbc:postgresql://localhost/blog?user=postgres&password=root"}}
:profiles/test {}})
Maybe it can't find the key of "database-url"?
Data provided via environment settings in project.clj are not provided when you use java directly. It's your responsibility to make sure the environment settings visible to the app provide any credentials or connection info needed.
It could look something like:
DATABASE_URL="jdbc...." java -jar my-project-standalone.jar

Ragtime Migrate with Environment Variables throwing Error (Heroku Deployment)

I'm trying to run lein ragtime migrate on a heroku dyno. Normally, I would set the database path in my project.clj like so:
(defproject my-project "0.1.0-SNAPSHOT"
:min-lein-version "2.0.0"
:dependencies [[org.clojure/clojure "1.6.0"]
[org.clojure/java.jdbc "0.3.7"]
[postgresql "9.3-1102.jdbc41"]
[ragtime "0.3.9"]
[ring "1.4.0-RC1"]
[ring/ring-defaults "0.1.2"]]
:plugins [[lein-ring "0.8.13"]
[ragtime/ragtime.lein "0.3.9"]]
...
:ragtime {:migrations ragtime.sql.files/migrations
:database (System/getenv "DATABASE_URL")}
...
:profiles
{:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
[ring-mock "0.1.5"]]
:test {:ragtime {:database (System/getenv "DATABASE_URL")}}})
When I run the command, I get the following error both locally and depolying over Heroku
java.lang.IllegalArgumentException: No method in multimethod 'connection' for dispatch value: postgres
Any pointers in the right direction would be very appreciated.
Ragtime 0.3.9 uses the scheme from the connection url as the dispatch value for the connection multimethod. The code is here and here. But the DATABASE_ENV from heroku doesn't have a "jdbc" but a "postgres" scheme (which makes sense, it has to be generic).
A workaround could be to add the "jdbc://" prefix:
:ragtime {:migrations ragtime.sql.files/migrations
:database ~(str "jdbc://" (System/getenv "DATABASE_URL"))}
You can also upgrade to [ragtime "0.4.0"] which doesn't use the scheme to find out how to create the connection. See the wiki for info about the upgrade path from 0.3.x

Resource not found : Exception Directory does not exist: resources ring.middleware.file/ensure-dir (file.clj:12)

I am deploying a Clojure project built on luminus on Ubuntu server. When I try to start my server I get the following exception:
Exception Directory does not exist: resources ring.middleware.file/ensure-dir (file.clj:12)
This is my project.clj file
(defproject big5 "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
[lib-noir "0.5.0"]
[compojure "1.1.5" :exclusions [ring/ring-core]]
[com.cemerick/friend "0.1.5"]
[ring-server "0.2.7"]
[clabango "0.5"]
[korma "0.3.0-RC6"]
[mysql/mysql-connector-java "5.1.6"]
[com.taoensso/timbre "2.7.1"]
[com.taoensso/tower "2.0.1"]
[com.postspectacular/rotor "0.1.0"]
[markdown-clj "0.9.19"]
[clj-json "0.5.3"]
[clj-time "0.6.0"]
[log4j "1.2.17"
:exclusions
[javax.mail/mail
javax.jms/jms
com.sun.jdmk/jmxtools
com.sun.jmx/jmxri]]
[org.slf4j/slf4j-log4j12 "1.7.5"]]
:plugins [[lein2-eclipse "2.0.0"]
[lein-ring "0.8.7"]
[lein-midje "3.0.0"]]
:ring {:handler big5.handler/war-handler
:init big5.handler/init
:destroy big5.handler/destroy}
:profiles
{:production {:ring {:open-browser? false
:stacktraces? false
:auto-reload? false}}
:dev {:dependencies [[ring-mock "0.1.3"]
[ring/ring-devel "1.1.8"]
[midje "1.5.1"]]}}
:min-lein-version "2.0.0")
I believe Leiningen uses the resources directory as the classpath root. Typically resources/public is where you store your static web assets like images, javascript, and stylesheets. If you create those directories you should be good.
Here are some references for you:
Resources in Clojure applications
Serving static files with ring/compojure - from a war
Finally, from the luminus documentation itself:
http://www.luminusweb.net/docs#anatomy_of_a_luminus_application

Auto building Clojurescript files for Compojure app

I have a web application where i'm using Compojure on the server and Clojurescript on the client. I'm using the leing-cljsbuild plugin to automatically compile cljs files to js.
I'm able to generate the required client side files and load them in the browser when I set the optimizations to :whitespace or :simple, but when I set optimizations to none, the js files reference their dependencies using the local file-system path, which leads to the files not loading at all in the browser.
So, my question is how do I make the generated files use server urls instead of local file paths when they are generated by the clojurescript compiler.
Here's my project.clj file
(defproject my-proj-clj "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:dependencies [[org.clojure/clojure "1.5.1"]
[compojure "1.1.6"]
[org.clojure/tools.nrepl "0.2.3"]
[hiccup "1.0.3"]
[com.novemberain/monger "1.5.0"]
[org.clojure/clojurescript "0.0-2127"]
[jayq "2.5.0"]
]
:plugins [[lein-ring "0.8.8"]
[lein-cljsbuild "1.0.1"]
]
:ring {:handler my-proj-clj.handler/app
}
:cljsbuild { :builds
[{
:source-paths ["src/my-proj-clj"]
:compiler {
:output-dir "./resources/public/js"
:output-to "./resources/public/js/cljs-file.js"
:pretty-print true
:source-map "./resources/public/js/cljs-file.js.map"
:optimizations :none
}}]}
:profiles {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]]}}
I believe the only valid optimization values are :whitespace, :simple, or :advanced. See line 96 at https://github.com/emezeske/lein-cljsbuild/blob/1.0.1/sample.project.clj.
Thus I would use :whitespace as the optimization level (at least to get something working).
Per your post, an optimization level of :whitespace works? Thus, perhaps you can elaborate.
What results are you expecting from an optimization level of ":none". How does your expected result differ from what an optimization level of :whitespace produces?
An optimization level of :none means cljsbuild is not generating js from your cljs source (it will generate a few goog.include statements but nothing else). Try using an interactive repl to help you prototype. Try running the following : lein trampoline cljsbuild repl-rhino
Hope that helps.
I have roughly same setup, optimizations set to :none, generated files use local paths. However, browser does load the scripts.
What I have is this in index.html:
<script src="js/development/goog/base.js" type="text/javascript"></script>
<script src="js/development/main.js" type="text/javascript" ></script>
<script type="text/javascript">goog.require("ixtlan.core");</script>
this in project.clj:
:cljsbuild {
:builds [{:id "dev"
:source-paths ["src/cljs"]
:compiler {
:output-to "resources/public/js/development/main.js"
:output-dir "resources/public/js/development"
:optimizations :none
:source-map true}}
...
and routes contain:
(defroutes routes
(GET "/" [] (index))
(route/files "/" {:root "resources/public"}))
Hope, this helps.