External jars with Dropwizard - classloader

I am trying to write a Dropwizard application and its doc tells me that I need to ship everything as an uber jar.
However, in my application I need to support multiple databases and this requires multiple database JDBC driver jars in my classpath, all of which are not expected to be shipped together with my application. Users are expected to place the corresponding JDBC jar like mysql-connector-java-5.1.39.jar in a particular folder by their own.
After reading Dropwizard's documentation I am not sure if this kind of usage is supported. Does anyone have experience making it to work this way?

Since java 6, you can wildcard classpaths.
Using the application plugin, the generated bin folder will have a start script that contains the classpath. What we want to do, is to instead of listing every possible jar in the bin folder, we simply include all of them.
Note: You can also do the same thing with different folders if you want the classpath in a different location.
This can be achieved (in a workaround manner since there are problems with this plugin in my version) in the easiest way as follows. In build.gradle you do:
startScripts {
doLast {
def windowsScriptFile = file getWindowsScript()
def unixScriptFile = file getUnixScript()
windowsScriptFile.text = windowsScriptFile.text.replaceAll('CLASSPATH=.*', 'CLASSPATH=\\$APP_HOME/lib/*')
unixScriptFile.text = unixScriptFile.text.replaceAll('CLASSPATH=.*', 'CLASSPATH=\\$APP_HOME/lib/*')
}
}
This will wildcard your lib folder in the start scripts. When starting up, your classpath will simply be
lib/*
When you drop jars into that folder, they will automatically be picked up (on startup, not on runtime).
I hope this helps,
Artur

Related

Gradle native c++ non source files resources or assets folder location?

Context: I am primarily a Java developer and I use gradle for all my builds. As a hobby I've been playing around with C/C++ and found that gradle has the ability to build these as well. So instead of learning cmake/make just for some small hobby projects I thought I'd use gradle since I am familiar with it.
Question: How do I define locations for non-source files?
With Java there is a resources folder that you can put things like images, text files, etc... in and gradle will put these in the jar so in your code you can load them with relative paths pretty easily.
src > main > java for source files.
src > main > resources for assets.
Is there an equivalent way to do this for c++ projects?
Also if there is a default folder that would be good to know, but also how to define it in the build.gradle file to a different location would also be appreciated.
For reference here is my simple build file right now:
apply plugin: "cpp"
model {
components {
main(NativeExecutableSpec) {
sources {
cpp {
source {
srcDir "src"
}
}
}
}
}
}
In my code I'd like to be able to load an image, for example, with something like:
HoweverYouLoadAnImageInCpp("imageName.png");
While having a simple structure like:
root
--src
--images
If there is no way to currently do this, is there a workaround or a more standard way people do this in C/C++?
As mentioned in the comments, unlike jars in java, c/c++ does not seem to have a standard way of including assets in the executable. There do seem to be platform specific ways and gradle does seem to have support for Windows resource files, so if that is what you are looking for see the gradle docs.
I prefer not to do any platform specific things though so I thought I'd answer this with what I decided to do in case someone else finds this question with a similar need. If another, better, answer pops up and I notice I will change the selection.
In the end the executable will look for paths relative to where it is executed (from what I can tell at least.) So I just made a copy task to put the assets in a parallel folder.
task copyAssets {
copy {
from "."
into "build/exe/main"
include "images/**" // This will take the whole images folder from project root
}
}
build.dependsOn copyAssets
So when I gradle build now it will copy my images folder to the same folder that it builds my 'main' cpp source executable. And in my code I can access those images with:
HoweverYouGetImages("images/imagename.png");
You could of course get more fancy with your task and zip it up or compress your images and decompress on loading in your code.

How publish a directory with ring middleware?

I am trying to use Clojure + Compojure + Ring in combination with the qooxdoo JS library. This is actually going well, but qooxdoo runs in two modes "build" (that works for me) and "source" (not so good). In the latter case, the JS generated by qooxdoo actually hardcodes references (well, using relative addresses ../../..) back to the qooxdoo installation and at run time it asks for sth like:
http://localhost:3000/opt/qooxdoo-5.0.1-sdk/framework/source/class/qx/bom/client/OperatingSystem.js
...since I have the library installed under /opt/qooxdoo-5.0.1-sdk.
Serious sanity check: if I directly open the index.html in the browser, it works fine. So it seems I just have to figure out how to serve the static files under the /opt library install.
I have tried wrap-file from the ring.middleware.file because that sounds like what I want but no matter what path I give it I get hundreds of 404s as it tries to pick up each framework file individually from the installed library.
I can work OK under "build" (qooxdoo cobbles together a single minified .js I serve with wrap-resource) but on occasion I need to run in source mode to find JS bugs.
Am I missing something simple?
The correct way to handle this is to configure Qooxdoo to tell it what URIs you would like to use - by default the source build does just use relative paths, but you can easily override this by editing the config.json.
In your config.json you will have a "jobs" section containing a "libraries" section, containing a "library" array - your application is a library, as is Qooxdoo, as is any contribs so it will look something like this:
"jobs" : {
"libraries" : {
"=library" : [ {
"manifest" : "${QOOXDOO_PATH}/framework/Manifest.json"
}, {
"manifest" : "Manifest.json"
}
},
Each "library" object can have a "uri" property, so for your example you probably want something like this:
"jobs" : {
"libraries" : {
"=library" : [ {
"manifest" : "${QOOXDOO_PATH}/framework/Manifest.json",
"uri" : "/opt/qooxdoo-5.0.1-sdk"
}, {
"manifest" : "Manifest.json"
}
},
Simple indeed: (wrap-file "/")
In development/source mode qooxdoo works off the installation directory instead of code pulled into my local tree, and does so by hardcoding relative paths that resolve to an absolute /opt/qooxdoo-etc path.
This looks to the server like a relative "opt/qooxdoo..." file request, so I had to offer root ("/") as a search directory.
(wrap-file "/") is a serious security issue since you're offering the root directory for all.
What you can potentially do is to make a dedicate directory to server your static file, and serve your content from there.
su
mkdir -p /var/clojure/static/opt
cd /var/clojure/static/opt
ln -s /opt/qooxdoo-5.0.1-sdk qooxdoo-5.0.1-sdk
chown -r YOUR-USER-ID /var/clojure/static/opt
And use the middleware: (wrap-file "/var/clojure/static") to serve your file.
There is a section in the manual that deals with the issue of serving a source version through a web server.
I guess you found the basic insight yourself, that the web server must export a root where the relative paths used in the generated loader match URL paths under that web server. (The rational behind this is that the source version uses all the JS files from all involved libraries directly from disk.)
In the worst case that might mean you need to export the file system root ("/") from your web server. As you are doing this on a local development machine this shouldn't be much of a problem. If security is a concern you might want to collect your qooxdoo environment under some innocent path like /home/kenny/devel/qooxdoo, containing the qooxdoo SDK, your app, and all libs/contribs you might be using.
If you follow the above link you will also find some help from the qooxdoo tool chain. E.g. if you run generate.py source-httpd-config[*] it will tell you which path on your local disk is the closest parent directory that will encompass all necessary libraries, i.e. needs to be exported in your web server for the source version to work.
Alternatively, as John wrote you can export each qooxdoo lib through an individual path under your web server, and then tell your main application where to find it. You might actually need to give a proper URL like http://localhost/libs/qooxdoo-5.0.1-sdk/framework, not just a file system path as John suggests. Also remember that you have to go as far as the directory where the Manifest file resides (hence the above path ending in .../framework). See this manual section for a deep dive.

Pentaho DI / Kettle / Spoon Configuration with External Jars

I am deveoloping a custom plugin for Pentaho DI / Kettle 4.4 and use custom Validation and XSLT jars as part of it.
Those jars are referenced in plugin.xml along with the plugin's jar itself and deployed in the same location as the plugin's jar.
When I try to load this plugin in Spoon GUI, I can see and run the plugin fine till I reach the reference to the external jars and after that I get ClassNotFOund Exception.
I have tried following so far but has not worked :
Add the expernal Jars to libext folder and expect those to be picked up automatically. - Failed. Jars not loaded.
Add -cp command line option to Spoon.bat and specify the Jar files - Did not work.
Turned on -verbose option and search everywhere in the output but the Jars are ignored.
Deployed the Jars next to the plugin's Jar file in the plugin's folder. - Jars are not recognized.
From Within the plugin, if I dynamically load (using Reflection and URL Class Loader) I am able to force the loading of the Jars and it works that way but I would like a simpler classpath fix.
Any class that I reference in the Jar through XSL externsion (XALAN-J Java Extension) is still not recognized / loaded and XSL terminates at the beginning of Extension call.
Searched Google and this site for something similar but did not find specific answers on why SPoon behaves so differently and avoids external code and content.
A similar issue is reported here though its too old and has not shown final conclusive fix. I tried all that it said. http://forums.pentaho.com/archive/index.php/t-77190.html
Requesting someone to help me identify whats missing. Thanks in advance,
Thanks to this seemingly obscure website and it fixed all pain.
http://www.arulraj.net/2014/09/how-to-access-external-java-libraries-with-in-pentaho.html
Basically, go to the launcher folder and edit the launcher.properties file to include any path, library etc. and it fixes all the issues I reported above. Wish Kettle documentatation had answers like this.

Using AsConfigured and still be able to get UnitTest results in TFS

So I am running into an issue when I go to build my projects using tfs build controller using the Output location "AsConfigred" it will not detect my unit tests. Let me give a little info on my setup.
TFS 2013 Update 2, Default Process Template
Here is a few screenshots that can hopefully help fill in what I can't in typing. I am copying my build out to a file share on our network so that we can use other utilities use the output. I don't want to use "PerProject" or "SingleFolder" because they mess up the file structure we have configured (These both will run the tests). So i have the files copy to folder names "SingleOutputFolder" which is a child of the DropLocation. I would like to be able to run from the drop folder or run from the bin folder for each of my tests (I don't care which). However it doesn't seem to detect/run ANY of the tests. Any help would be greatly appreciated. Please let me know if you need any additional information.
I have tried using ***test*.dll, Install\SingleFolderOutput**.test.dll, and $(TF_BUILD_DROPLOCATION)\Install\SingleFolderOutput*test*.dll
But I am not sure what variables are available and understand where the scope of its execution is.
Given that you're using Build Output location set to AsConfigured you have to change the default values of the Test sources spec setting to allow build to find the test libraries in the bin folders. Here's an example.
If the full path to the unit test libraries is:
E:\Builds\7\<TFS Team Project>\<Build Definition>\src\<Unit Test Project>\bin\Release\*test*.dll
use
..\src\*UnitTest*\bin\*\*test*.dll;
This question was asked on MSDN forums here.
MSDN Forums Suggested Workaround
The suggested workaround in the accepted answer (as of 8 a.m. on June 20) is to specify the full path to the test projects' binary folders: For example:
C:\Builds\{agentId}\{teamProjectName}\{buildDefinitionName}\src\{solutionName}\{testProjectName}\bin*\Debug\*test*.dll*
which really should have been shown as
{agentWorkingFolder}\src\{relativePathToTestProjectBinariesFolder}\*test*.dll
However this approach is very brittle, for the following reasons:
Any new test projects you add to the solution will not be executed until you add them to the build definition's list of test sources:
It will break under any of the following circumstances:
the build definition is renamed
the working folder in build agent properties is modified
you have multiple build agents, and a different agent than the one you specified in {id} runs the build
Improved Workaround
My workaround mitigates the issues listed in #2 (can't do anything about #1).
In the path specified above, replace the initial part:
{agentWorkingFolder}
with
..
so you have
..\src\{relativePathToTestProjectBinariesFolder}\*test*.dll
This works because the internal working directory is apparently the \binaries\ folder that is a sibling of the \src\ folder. Navigating up to the parent folder (whatever it is named, we don't care) and back in to \src\ before specifying the path to the test projects binaries does the trick.
Note: If you have multiple test projects, you add additional entries, separated with semicolons:
..\src\{relativePathToTestProjectONEBinariesFolder}\*test*.dll;..\src\{relativePathToTestProjectTWOBinariesFolder}\*test*.dll;..\src\{relativePathToTestProjectTHREEBinariesFolder}\*test*.dll;
What I ended up doing was adding a post build event to copy all of the test.dll into the staging location folder in the specific build that is basically equivalent to where it would go on a SingleFolder build and do that on each test project.
if "$(TeamBuildOutDir)" == "" (
echo "Building Interactively not in TFS"
) else (
echo "Building in TFS"
xcopy "$(TargetDir)*.*" "$(TeamBuildBinaries)\" /Y /E /S
)
MSBUILD parameter in the build def that told it to basically drop in the folder that TFS looks for them.
/p:TeamBuildBinaries="$(TF_BUILD_BINARIESDIRECTORY)"
Kept the default Test assembly file specification:
**\*test*.dll
View this link for the information on the variable that I used and what relative path it exists at.
Another solution is to do the reverse.
Leave all of the files in the root so that all of the built in functionality works. There is more than just test execution in there. What about static code analysis, impact analysis..among others. You would have to do something custom for them all.
Instead use a pre-drop powershell script to create your Install arrangement from the root files.
If it is an application then you can use the _ApplicationFolder Nuget package to create an _PublishApplications folder same as you get for web applications.

Gradle Jetty plugin locking files

Is there a way to fix the file locking issue caused by jetty entirely from gradle?
Some clarification:
When using the Gradle Jetty plugin by running gradle jettyRun, jetty causes the static resource files (html, css, js, etc.) to be locked when using Windows.
You can see a description of the problem in Files locked on Windows.
The same article also describes how you can fix that. Basically you have to either:
Disable the use of file mapped buffer
Not use NIO at all.
Both things require to add some jetty specific configuration files to the project, which I do not want to do - the jetty plugin is used only for convenience, and maintaining configuration for it does not feel right.
I do not need NIO for testing on the local machine, so any solution works.
Edit:
For now, I picked the option at which you set useFileMappedBuffer to false. This is how to do it:
Specify a path to your webdefault.xml like
[jettyRun, jettyRunWar,jettyStop]*.with {
//other configs
webDefaultXml = file("${project.webAppDir}/WEB-INF/jetty-webdefault.xml")
}
Get file from the latest 6.1.x distribution of jetty. The plugin seems to support only jetty 6. You can localte it at jetty-6.1.26\etc\webdefault.xml. Obviously, you have to place it at the path specified at the previous step.
Change the default servlet init parameter useFileMappedBuffer to false
I will research the option of using embeded jetty insted of the plugin.
I found a plugin that seem to be a better alternative:
https://github.com/akhikhl/gretty
Positives
Does not lock your files and support hot deployment (even something Gretty call "fast reload")
Gretty 1.2.0 uses Jetty 9.2.9.v20150224. Jetty plugin provided by Gradle 2.2.1 uses Jetty 6.1.25.
same task is used jettyRun (or more simply run).
"Press any key to stop the server". Jetty plugin required CTRL+C then Y.
From what I can tell, the documentation seem to be awesome (Gradle's not so much)
Negatives
A bit more bloated code to setup the buildscript's classpath dependency or apply plugin directly from URL (see doc)
Gretty crash unless you explicitly apply plugin: 'war' (Jetty plugin extends the War plugin)
Kiril answered his own question, many thanks. You should follow Kiril's instructions and this will help you find the appropriate webdefault.xml.
To find out what version of Jetty is started by Gradle, execute
gradle jettyRun -i
And you'll see something like this:
...
Tmp directory = determined at runtime
Web defaults = org/mortbay/jetty/webapp/webdefault.xml
Web overrides = none
Webapp directory = C:\dev\my-project\src\main\webapp
Starting jetty 6.1.25 ...
jetty-6.1.25
...
It took me a while to find a copy of Jetty 6.1.25 as it is no longer listed on the Jetty download page (not even in the archive section!).
You can then grab the appropriate copy of webdefault.xml from here, adjusting the version number as appropriate for your needs:
http://grepcode.com/file/repo1.maven.org/maven2/org.mortbay.jetty/jetty/6.1.25/org/mortbay/jetty/webapp/webdefault.xml