I deployed my Clojure app to AWS and I also have Datomic Transactor(EC2) with DynamoDB when my users started to use my app something happened after a while and Datomic Transactor rebooted(for some reason), my app became unresponsive so I re-started my application server(to re-establish transactor connection) then there were no users data, no data at all!?
I was using same URI("datomic:ddb://us-east-1/my-table/my-db"), also I don't call delete-db or some function like that.
I only call d/create-database and d/connect for connecting to Datomic.
It happened me on development but I did not care that time because I thought I did something wrong or something like that.
How can I restore that old data?
P.S: Also I can see that DynamoDB table has some storage size like 900KB I assume some how my data in there? There were just couple of users tho.
My Datomic Version: "0.9.5561"
Here is my code:
(defn establish-conn
[]
(try
;conf/get returns DB URI like: "datomic:ddb://us-east-1/my-table/my-db"
(d/create-database (conf/get :db-uri))
(reset! conn (d/connect (conf/get :db-uri)))
(catch Throwable t
(log/error "Could not establish db conn." t))))
Also I have this code for recovering connection from Transactor failure/unavailable etc.
(defn fix-if-conn-ex
;;getting exception's error message
[err-msg]
(when (any? (str/includes? err-msg %) [":db.error/connection-released"
":db.error/transactor-unavailable"])
(db/establish-conn)))
I could not reproduce the issue with the latest Datomic version "0.9.5561.56" on my staging environment, I assume it's resolved with that one.
Related
I have deployed a datomic transactor to an AWS cloud formation, connected to a DyanmoDB as storage. I'm able to connect to the transactor if I go to my local datomic directory (version 0.9.5.44) and use the repl:
=> (def uri "datomic:ddb://us-east-1/datemo/test-db?aws_access_key_id={xxx}&aws_secret_key={yyy}")
=> (def conn (d/connect uri))
#object[datomic.peer.Connection 0x5e5ddfbc "{:unsent-updates-queue 0, :pending-txes 0,:next-t 1000, :basis-t 63, :index-rev 0, :db-id \"test-db-ab4b8028-e25b-4431-b2a2-c7173b4c9918\"}"]
If, however, I do the same in my project's repl, it does not work, and instead I get the following error:
CompilerException java.lang.NoClassDefFoundError: Could not initialize
class datomic.ddb_cluster__init,
compiling:(form-init1336530578825920965.clj:1:11)
Incidentally, if I try to connect to my usual datomic dev transactor it works just fine. Does anyone know what I might be doing wrong here? The error seems to suggest that the class is not even present...
You need to include both the Datomic Peer library and the AWS SDK as dependencies in your project to connect to a DDB-backed instance.
This section of the documentation: http://docs.datomic.com/storage.html#sec-6-1 contains details of including the AWS SDK.
-Marshall
While running my tests with lein doo phantom, I receive a -1 status response and an empty string as the body. However, when I run the test in the repl, I am able to retrieve the request data with a 200 status response and the appropriate data in the body. Is this because a manytomany channel is being returned first as mentioned below, thus giving me the inappropriate response? If so, how could I account for this?
https://github.com/r0man/cljs-http#async-response-handling
I also thought maybe I need to use a timeout to wait for the request to complete. If so, how would I apply that appropriately with my existing code? It looks like cljs-http has :timeout as a parameter but I haven't been ably to get it to work appropriately (assuming this is the cause of the issue).
(deftest test-async
(async done
(go (let [response (<! (http/get "http://localhost:3000/api/user/1"
{:with-credentials? false
:query-params {"id" 1}}))]
(is (= {:status 200}
(select-keys response [:status]))))
(done))))
Since you are running your test under phantomjs. Phantomjs default disable cross domain XHR access and your tests js are running on localhost,all external ajax calls are denied.
you can set the --web-security=false to allow your test to do cross domain ajax.
In your project.clj add this
:doo {:paths {:phantom "phantomjs --web-security=false"}}
more info about phantomjs
http://phantomjs.org/api/command-line.html
We have a Clojure web application that is used by multiple projects (>20) that have multiple users logging in simultaneously. All projects have their own MySQL database. We have tried to figure out a way to use one application instance to serve requests from users that are delivered from their project's database.
The following script shows the principles of our multiple connections and should be executable in REPL (with correct database setup).
(ns testmultiple.core
(:require
[clojure.java.jdbc :as jdbc]
[compojure.core :refer [defroutes GET ANY routes context]]
[conman.core :as conman]
[mount.core :refer [defstate]]))
(def database-urls {:DB1 "jdbc:mysql://localhost:3306/DB1?user=DB1_user&password=DB1_password"
:DB2 "jdbc:mysql://localhost:3306/DB2?user=DB2_user&password=DB2_password"})
;; Connects to all databases in pool-specs
(defn connect!
[pool-specs]
(reduce merge (map (fn [pool-spec]
{(keyword (key pool-spec)) (conman/connect! {:jdbc-url (val pool-spec)})}) pool-specs)))
;; Disconnect from all databases in db-connections
(defn disconnect!
[db-connections]
(map (fn [db] (conman/disconnect! (val db))) db-connections))
;; Establish connections to all databases
;; and store connections in *dbs*
(defstate ^:dynamic *dbs*
:start (connect!
database-urls)
:stop (disconnect! *dbs*))
;; Bind queries to *db* dynamic variable which is bound
;; to each clients database before executing queries
;; The queries file defines the query get-user which
;; returns user by user id
(def ^:dynamic *db* nil)
(conman/bind-connection *db* "sql/queries.sql")
(mount.core/start)
; Define function that executes in current *db* binding
(defn getuser [id] (get-user {:id id}))
; Works, the user with Id 670 is returned from DB1
(with-bindings {#'*db* (:DB1 *dbs*)} (getuser 670))
; Works, the user with Id 670 is returned from DB2
(with-bindings {#'*db* (:DB2 *dbs*)} (getuser 670))
More specifically, the project is inferred from the URL request in the router. The following code shows the principle for the router. Accessing www.example.com/DB1/page1 and www.example.com/DB2/page2 will show page1 with data from DB1 and page2 with data from DB2, respectively.
(defn serve-page1 [] (str "page1" (getuser 670)))
(defn serve-page2 [] (str "page2" (getuser 670)))
(def home-routes
(context "/:project" [project]
(if (contains? *dbs* (keyword project))
(routes
(GET "/page1" []
(with-bindings {#'*db* ((keyword project) *dbs*)}
(serve-page1)))
(GET "/page2" []
(with-bindings {#'*db* ((keyword project) *dbs*)}
(serve-page2))))
(ANY "*" [] (str "Project not found")))))
This will be an application with considerable traffic. Notably, we are still in development phase and have thus not been able to test this solution with more than a couple of databases running on localhost. Our questions are
Is establishing multiple connections like this reasonable, stable and scalable?
Are there other better methods for the routing and dynamic binding of the project's database?
Is establishing multiple connections like this reasonable, stable and scalable?
Yes, this is a very reasonable approach. Very few database systems are limited by the number of outgoing connections. Both JDBC and Korma will handle this just fine in clojure. You do need to be aware of which requests are dependent on which DB when building the monitoring and ops related components of course. So you can tell which DB is causing problems.
Are there other better methods for the routing and dynamic binding of the project's database?
My only suggestion would be to explicitly pass the DB to each function rather than using a binding, though this is a personal style opinion and your approach will clearly work.
I want to start the elasticsearch service on a machine, originally I was going to do this using pallet.actions.service but the documentation says this is now deprecated in favour of pallet.crate.service/service.
However I cannot find any guide to use it to simply start an initd service that is already installed on the node.
Below is my first attempt at getting an es node running using the old service function. Unfortunately I couldn't get this to start the service either, it installed elasticsearch but did not run the service. When I was playing around with this I noticed that it was deprecated:
(def elasticsearch-server
(server-spec
:extends [(javacrate/server-spec {})]
:phases
{:install (plan-fn
(add-es-source)
(package-manager :update)
(package "elasticsearch")
(service "elasticsearch" :action :start))}))
I don't recall if service does something extra other than give you an idiomatic interface to service, so what you can do is use exec-script*, which is what I do in my own code without any trouble. In fact, I even do this with Elasticsearch (among other things) and with my services using runit for supervision.
(exec-script* "service elasticsearch start")
Perhaps service does a better job of recovering, though, when the service fails to start? If service elasticsearch start doesn't return 0 then your whole plan will fail.
I am working on an application that has to download some external resources and make them accessible through a public/static directory in Ring.
But.. I have a problem saving the resources into a static directory in my application, when developing I use the ring-jetty-adapter; the test & production servers are running Tomcat.
I added :web-content "public" to my leiningen project and added the public directory in the root of the project, then I have a download function using http-agent and duck-streams:
(defn download
[file-name url]
(h/http-agent url
:handler (fn [agnt]
(let [fname file-name]
(with-open [w (d/writer fname)]
(d/copy (h/stream agnt) w))))))
If I am booting Jetty from the REPL and use savepath: "public/my.file", the downloaded file is placed correctly in the public directory.
But when I deploy it using a .war file to Tomcat, it looks for a public directory in the Tomcat root directory, and not under the application context path.
I tried to add a middleware wrapper to determine the context path and from there build the correct save path, but I cannot find any way to access the HttpServlet or a way to determine if the application is running in the adapter or if it is deployed under a specific context.
Here the wrapper:
(defn wrap-context-info [handler]
(fn [req]
(let [resp (handler req)]
(assoc resp :servlet (:servlet req) :req (:servlet-request req)))))
both :servlet and :req are nil.
Looking at the ring-servlet source, it appears that the ring servlet adapter associates the HttpServlet, HttpServletRequest, and HttpServletResponse objects with the ring request map under :servlet, :servlet-request, and :servlet-response keys, accordingly.
It also adds a :servlet-context entry to the request map with the value of (.getServletContext servlet) for convenience.
Inside your handler, you might want to check for the presence of these keys in the request map, and then pull further information that you need from the associated objects.