Description
I am developing AWS Java Lambdas, with Gradle as my build tool.
AWS requires a "self-contained" Java archive (.jar, .zip, ...) to be uploaded, which has to include everything, my source code, the dependencies etc.
There is the Gradle plugin shadow for this purpose, it can be included like this:
import com.github.jengelman.gradle.plugins.shadow.transformers.Log4j2PluginsCacheFileTransformer
...
shadowJar {
archiveName = "${project.name}.jar"
mergeServiceFiles()
transform(Log4j2PluginsCacheFileTransformer)
}
build.dependsOn shadowJar
gradle build produces a file somefunction.jar, in my case it is 9.5MB in size.
AWS documentation suggests to
putting your dependency .jar files in a separate /lib directory
There are specific instructions how to do this on Creating a ZIP Deployment Package for a Java Function.
task buildZip(type: Zip) {
archiveName = "${project.name}.zip"
from compileJava
from processResources
into('lib') {
from configurations.runtimeClasspath
}
}
build.dependsOn buildZip
gradle build produces a file build/distributions/somefunction.zip, in my case it is 8.5MB in size.
Both archives, zip and jar, can be upload to AWS and run fine. Performance seems to be the same.
Question
Which archive to favor, Zip or (shdow)Jar?
More specific questions, which come to my mind:
AWS documetation says "This [putting your dependency .jar files in a separate /lib directory] is faster than putting all your function’s code in a single jar with a large number of .class files." Does anyone know, what exactly is faster? Build-time? Cold/warm start? Execution time?
When build the Zip, I am not using the shadowJar features mergeServiceFiles() and Log4j2PluginsCacheFileTransformer. Not using mergeServiceFiles should in worst case decrease the execution time. As long as I omit Log4j2 plugins, I can omit Log4j2PluginsCacheFileTransformer. Right?
Are there any performance considerations using the one or the other?
Related
In an iOS c++/Qt application, I need to ship a few files and to keep them in their directory structure.
For the Android version, we bundle a zip which we unzip on the target before creating the QApplication.
On iOS, it seems that CMake is not capable of bundling files in a tree:
https://cmake.org/cmake/help/latest/prop_tgt/RESOURCE.html#prop_tgt:RESOURCE
https://cmake.org/cmake/help/latest/prop_sf/MACOSX_PACKAGE_LOCATION.html
I am not sure if this is a limitation of cmake or if this is a global limitation on iOS.
From the docs about iOS bundles:
It uses a relatively flat structure with few extraneous directories in an effort to save disk space and simplify access to the files.
What would be the preferred approach?
Is there a solution to ship the files from CMake directly?
If not, how can I achieve this so that they are available before the QApplication is created?
The xcode command
Thanks to #Cy-4AH, I added the folder in Xcode and could get the command to do this:
CpResource _PATH_TO_DIRECTORY_ _APP_BUNDLE_DIRECTORY_/_RESOURCE_DIR_NAME_
cd /Users/denis/opt/qfield/ios/QField
export PATH="....."
builtin-copy -exclude .DS_Store -exclude CVS -exclude .svn -exclude .git -exclude .hg -strip-debug-symbols -strip-tool /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip -resolve-src-symlinks _PATH_TO_DIRECTORY_ _APP_BUNDLE_DIRECTORY_
But how can I create this from cmake? builtin-copyis an xcode command.
Simple system copy command
From an old (2008) discussion, we could use simple cp commands.
This works up to signing, but then I get an error unsealed contents present in the bundle root.
From this answer, it seems related that I cannot simply add folders in the resource directory. From the docs anatomy of framework bundles: Nonlocalized resources reside at the top level of the Resources directory
(Disclaimer: I'm not a CMake user, and there may be a more CMake-ey way to do this)
If you can set up post-build action, the following terminal script can efficiently sync files into your bundle from another location. I use it in my game engine because it only copies updated or new files upon subsequent builds, and preserves directory structure:
mkdir -p PATHTO/ORIGINFOLDERNAME
mkdir -p PATHTOBUILDFOLDER/PROJECTNAME.app/Contents/Resources/DESTINATIONFOLDERNAME
rsync -avu --delete --exclude=".*" PATHTO/ORIGINFOLDERNAME/ PATHTOBUILDFOLDER/PROJECTNAME.app/Contents/Resources/DESTINATIONFOLDERNAME
The mkdir commands are only to ensure that the folders are generated, if they were deleted.
So apparently the CMake method also works for directories.
target_sources(${QT_IOS_TARGET} PRIVATE ${_resource})
set_source_files_properties(${_resource} PROPERTIES MACOSX_PACKAGE_LOCATION Resources)
It will just be added at the root directory of the bundle and not within the Resources.
If the embedded file is not too big, you might consider :
in your source tree, generating a C++ file embedding that file as a constant array. For example, if your file contains just hello, world with a new line, you could have something like
/// file contents.cc
const char file_contents[] = "hello, world\n";
and at the beginning of your program (perhaps in your main function, before your QApplication) call a C++ function which writes such a file (perhaps in /tmp/).
in your build automation (e.g your Makefile or your qmake things), have something which generates the C++ contents.cc file from the genuine source
This is with a POSIX/Linux point of view, adapt my answer to your iOS.
I am a noob. What is the 'artifacts' in the buildsepc yaml file?
I read on https://docs.aws.amazon.com/codebuild/latest/userguide/getting-started-create-build-spec-console.html,
"Artifacts represents the set of build output artifacts that CodeBuild uploads to the output bucket. files represents the files to include in the build output."
Maybe, I am not understanding it correctly. Given the settings in the screenshot above, I expect two zipped files(template.yml and outputtemplate.yml) to be uploaded to the output bucket, say BUCKET=MYBUCKET.
But, when I check my S3 bucket after I build and deploy it, I have 2 files named like c7e84f72729709f7a0.
Also, just to understand what's going on, I tried removing the lines: 'type: zip' and '-template.yml', and built and deployed again. I expected only 1 file since I removed line 8 and 10. But, the result was still two files sitting in my bucket. What exactly are the artifacts? And What is the type?(this I can't even find any documentations). Why is the type in most cases, if not always, zip? when in fact the uploaded file is not actually not a zip file?
Thank you.
The file c7e84f72729709f7a0 is your zip file. It will contain both yml files. Just unpack it as any other zip file. May need to add extension .zip if required by your unpacking software.
I don't know where the type: zip comes from. The reference docs for the buldspec.yml do not document such a field.
And artifacts are outcomes files of your build. For example, when you are building a C++ project, it would be executable or library files resulting from compilation of your C++ source code.
The artifices are also carried over to a next stage of your CI/CD pipeline, such as integration testing or deployment with CodeDeploy.
Just to add to Marcin's answer, the type property of artifacts was deprecated in version 0.2 (the one that's being used here). You can see the changes across buildspec versions at the bottom of this page.
Context: I am primarily a Java developer and I use gradle for all my builds. As a hobby I've been playing around with C/C++ and found that gradle has the ability to build these as well. So instead of learning cmake/make just for some small hobby projects I thought I'd use gradle since I am familiar with it.
Question: How do I define locations for non-source files?
With Java there is a resources folder that you can put things like images, text files, etc... in and gradle will put these in the jar so in your code you can load them with relative paths pretty easily.
src > main > java for source files.
src > main > resources for assets.
Is there an equivalent way to do this for c++ projects?
Also if there is a default folder that would be good to know, but also how to define it in the build.gradle file to a different location would also be appreciated.
For reference here is my simple build file right now:
apply plugin: "cpp"
model {
components {
main(NativeExecutableSpec) {
sources {
cpp {
source {
srcDir "src"
}
}
}
}
}
}
In my code I'd like to be able to load an image, for example, with something like:
HoweverYouLoadAnImageInCpp("imageName.png");
While having a simple structure like:
root
--src
--images
If there is no way to currently do this, is there a workaround or a more standard way people do this in C/C++?
As mentioned in the comments, unlike jars in java, c/c++ does not seem to have a standard way of including assets in the executable. There do seem to be platform specific ways and gradle does seem to have support for Windows resource files, so if that is what you are looking for see the gradle docs.
I prefer not to do any platform specific things though so I thought I'd answer this with what I decided to do in case someone else finds this question with a similar need. If another, better, answer pops up and I notice I will change the selection.
In the end the executable will look for paths relative to where it is executed (from what I can tell at least.) So I just made a copy task to put the assets in a parallel folder.
task copyAssets {
copy {
from "."
into "build/exe/main"
include "images/**" // This will take the whole images folder from project root
}
}
build.dependsOn copyAssets
So when I gradle build now it will copy my images folder to the same folder that it builds my 'main' cpp source executable. And in my code I can access those images with:
HoweverYouGetImages("images/imagename.png");
You could of course get more fancy with your task and zip it up or compress your images and decompress on loading in your code.
How I can build local sources and dependancies with flatpak-builder?
I can build local sources
flatpak build ../dictionary ./configure --prefix=/app
I can extract and build application with dependancies with a .json
flatpak-builder --repo=repo dictionary2 org.gnome.Dictionary.json
But no way to build dependancies and local sources? I don't find sources type
like dir or other, only archive, git (no hg?) ...
flatpak-builder is meant to automate the whole build process, with a single entry-point: the JSON manifest.
Everything else it obtains from Git, Bazaar or tarballs. Note that for these the "url" property may be a local URL starting with file://.
(There is indeed no support for Hg. If that's important for you, feel free to request it.)
In addition to that, there are a few more source types (see the flatpak-manifest(5) manpage), which can be used to modify the extracted sources:
file which point to a local file to copy somewhere in the extracted sources;
patch which point to a local patch file to apply to the extracted sources;
script which creates a script in the extracted sources, from an array of commands;
shell which modifies the extracted sources by running an array of commands;
Adding a dir source type might be useful.
However (and I only flatpaked a few apps, and contributed 2 or 3 patches to the code, so I might be completely wrong) care must be taken as this would easily make builds completely unreproducible, which is one thing flatpak-builder tries very hard to enable.
For example, when using a local file source, flatpak-builder will base64-econde the content of that file and use it as a data:text/plain;charset=utf8;base64,<content> URL for the file which it stores in the manifest included inside the final build.
Something similar might be needed for a dir source (tar the folder then base64-encode the content of the tar?), otherwise it would be impossible to reproduce the build. I've just been told (after submitting this answer) that this changed in Git master, in favour of a new flatpak-builder --bundle-sources option. This would probably make it easier to support reproducible builds with a dir source type.
In any case, feel free to start the conversation around a new dir source type in the upstream bug tracker. :)
There's a expermental cli tool if you want to use it https://gitlab.com/csoriano/flatpak-dev-cli
You can read the docs
http://docs.flatpak.org/en/latest/building-simple-apps.html
http://docs.flatpak.org/en/latest/flatpak-builder.html
In a nutshell this is what you need to use flatpak as develop workbench
https://github.com/albfan/gnome-builder/wiki/flatpak
I am trying to write a Dropwizard application and its doc tells me that I need to ship everything as an uber jar.
However, in my application I need to support multiple databases and this requires multiple database JDBC driver jars in my classpath, all of which are not expected to be shipped together with my application. Users are expected to place the corresponding JDBC jar like mysql-connector-java-5.1.39.jar in a particular folder by their own.
After reading Dropwizard's documentation I am not sure if this kind of usage is supported. Does anyone have experience making it to work this way?
Since java 6, you can wildcard classpaths.
Using the application plugin, the generated bin folder will have a start script that contains the classpath. What we want to do, is to instead of listing every possible jar in the bin folder, we simply include all of them.
Note: You can also do the same thing with different folders if you want the classpath in a different location.
This can be achieved (in a workaround manner since there are problems with this plugin in my version) in the easiest way as follows. In build.gradle you do:
startScripts {
doLast {
def windowsScriptFile = file getWindowsScript()
def unixScriptFile = file getUnixScript()
windowsScriptFile.text = windowsScriptFile.text.replaceAll('CLASSPATH=.*', 'CLASSPATH=\\$APP_HOME/lib/*')
unixScriptFile.text = unixScriptFile.text.replaceAll('CLASSPATH=.*', 'CLASSPATH=\\$APP_HOME/lib/*')
}
}
This will wildcard your lib folder in the start scripts. When starting up, your classpath will simply be
lib/*
When you drop jars into that folder, they will automatically be picked up (on startup, not on runtime).
I hope this helps,
Artur