I have a maven project that converts text files of a specific format to another format.
For testing I have put in src/test/resources a large amount of test files.
I also have another project that uses the first one to do the conversion and then do some extra stuff on the output format. I also want to test this project against the same test dataset, but I dont want to have duplicate data sets and I want to be able to test the first project alone since it is also a standalone converter project.
Is there any common solution for doing that? I dont mind not having the test dataset inside the projects source tree as long as each project can access the data set independently of the other. I dont want to setup a database for that also. I am thinking something like a repository of test data simpler than an RDBMS. Is there any application for this kind of need that I can use with a specific maven plugin? Ease of setup and simplicity is my priority. Also I m thinking something like packaging the test data and putting it in a internal maven repo and then downloading it and unzip it in the junit code. Or better, is there a maven plugin that can do this for me?
Any ideas?
It is possible to share resources with Maven, for example with the help of the Assembly and Dependency plugins. Basically:
Create a module with a packaging of type pom to hold the shared resources
Use the assembly plugin to package the module as a zip
Declare a dependency on this zip in "consumer" modules
And use dependency:unpack-dependencies (plus some exclusion rules) to unpack the zip
This approach is detailed in How to share resources across projects in Maven. It requires a bit of setup (which is "provided") and is not too complicated.
Just place them in a tree in the src/main/resources directory of a separate module specially to share the test data. They will be added to the jar file and me nicely compressed and versioned in your nexus repository, file-share, ~/.m2/repository or whatever you use to store/distribute maven artifacts.
Then just add a dependency in the projects you need the data in the test scope and use resource loading to get them from the jars.
You do not need any special plugins or other infrastructure. This just works.
Related
I‘m looking for a way to add unit tests to my documentation. I generate my docs website from asciidoc using antora. Works perfectly fine. Now I am looking for a way to unit test my docs (asciidoc files as well). But this is not my question right now.
What I want right now is a way to test the filesystem structure of my docs. I have several repos with similar base setup and basic docs. The details differ but e.g. I want to ensure every docs have their main docs in PROJECT/docs/modules/ROOT and the navigation file is named nav.adoc in every module etc.
Basically I want the same docs structure for every project.
So I wonder if there is a way to test the filesystem (foldernames, Filenames). Ultimately these tests should be part of my gitlab ci pipeline.
Can archunit help with that? I’d consider it if there is no better way, but since archunit targets java projects it seems like overkill for my uaecase.
It is important to remember that Antora can gather content from multiple repos. When Antora gathers content, that content is held in a virtual filesystem until the transformation to HTML is complete. That means that wherever you run Antora, the filesystem structure of the repos cannot be directly interrogated unless it happens to be local (you're using "author" mode, or you have an up-to-date checkout of the repo(s) locally).
It is possible to write an extension for Antora (in JavaScript, since Antora is a Node.js application) to investigate the virtual filesystem and complain about any structural problems found.
Outside of Antora, you'd basically be writing a repo-specific test. That test could be invoked any way you like, say by running a test script when make is run. Or, you might add a git commit hook to your repo that prevents commits when the appropriate file structure does not exist.
The TL;DR answer is "yes, this can be done". However, it requires some implementation effort to achieve.
Ember don't seems to play well with requirejs or building with r.js without some hackery.
Bit hard to manage a large application, I like to break down my file structure as HMVC, which manages modules bit easier :
app
- blah
- modules
-module1
- controller
- model
- view.
Has anyone come up a build stack to automate the compilation into single file with dependency management?
Update: You should now be starting new projects using ember-cli, which provides all the build/dev tools plus many other useful features.
Original answer:
An example Ember project using grunt.js as a build system:
https://github.com/trek/ember-todos-with-build-tools-tests-and-other-modern-conveniences
I've used this as a starting point for some Ember apps and it works well.
Ember don't seems to play well with requirejs or building with r.js without some hackery.
Why do you want to use require.js in the first place?
As for making this all work you'll need a build tool. I recommend brunch (Node) or iridium (Ruby). Brunch is more simple and supports many different module formats. Iridium is much more powerful. It uses Minispade for modules. Require.js/AMD is not needed because your js is always delivered as a single concatenated file.
For stand-alone ember dev I use ember-skeleton as a starting point, which is itself based primarily on rake-pipeline.
https://github.com/interline/ember-skeleton
This provides the whole kit-and-kaboodle for beginning an app, but the meat of it is the rake-p Assetfile, which describes how rake-pipeline will digest all the files, filter their contents, and ultimately combine them into the final handful of files. Additionally, the combination of loader.js and the minispade filter allows the use of 'require()' statements for managing dependencies amongst your files.
We are moving our build and testing system to Jenkins, and are looking for an easy way (where we don't have to code all the logic ourselves) to manage the build artifacts.
Basically we need an organised way to store them in order by the type of the build, the user who built it and such for example:
johnd/nightly/r543241/win32/program.zip
johnd/nightly/trunk/lin64/program.tar.gz
master/release/2.1/win32/program.zip
This way we can upload when a build is complete, and retrieve the required artifact in the testing stage with ease.
Until now we simply stored the files in directories on NFS, but recently started considering an artifact manager. I've looked at Artifcatory, Archiva and Nexus. But all seem very Java centered or at least required maven to work with. Since I don't want to introduce more complexity (We mainly work with python, scons is our build tool) and I don't want to introduce maven to the mix, I'm looking for something that has an easy command line (or better REST/Python interface) to upload, download, manage artifacts.
If you don't use an artifact manager, but use some other clever method to manage your C++ artifacts for release/test needs I'd be happy to hear that also.
I have exactly the same issue. I didn't succeed to integrate artifactory easilly, so I forgot it.
For the moment I use the copy artefact plugin to copy by scp the file to an NFS share. But this is not a good solution since shares may be mounted differently on two machines, and windows cannot access them directly.
But a real artifact manager would be so such a good thing for our project.
In my previous job, I was using a documentation CMS (LiveLink) which provided good services:
- self organising folder.
- persistant url (we can move, rename folders and files, a public URL to a given files never changes, that was SOOO great : "http://server/go/123456789123456789" always pointed to the same file. So you just had to give this url to wget and you get the file
- file versionning.
- meta data, advanced search
I'm still searching such a solution in opensource software and don't find it.
In you just need to share artefact between two jobs, simple use the "copy artifact for another build" plugin. I use it to split my job into a a build job and a test job (actually there are 3 test jobs, one very fast done after each commit, one quite slow done every day, and one very slow done each week end).
Im new to GIT and dont know yet how much it will fit my needs, but it looks impressive.
I have a single webapp that i use for differents customers (django+javascript)
I plan to use GIT to handle these differents customers version as branches. Each customer can have custom files, folders and settings, improved versions... but the should share the same 'core'. We are a small team and suscribed a github account.
Is the branch the good way to handle this case ?
About the settings file, how would you proceed ? Would you .gitignore the customer specific settings file and add a settings.xml.sample file for example is the repo ?
Also, is there any way to prevent some files to be merged into master ? (but commited to the customer branch). For example, id like to save some customer data to the customer branch but prevent from to being commited to master.
Is the .gitignore file branch-specific ? YES
EDIT
After reading all your answers (thanks!) i decided to first refactor my django project structure to isolate the core and my differents apps in an apps subfolder. Doing this makes a cleaner project, and tweaking the .gitignore file makes it easy to use git branches to manage the differents customers and settings !
Ju.
In addition to cpharmston's answer, it sounds like you need to do some refactoring to separate out what is truly custom for each client and what isn't. Then you may consider adding additional repositories to track the customizations for each client (entirely new repos, not branches). Then your deployment can pull your "core" from your main repo, and the client-specific stuff from that repo.
I would not use branches to accomplish what you are trying to do.
In source control, branches are intended to be used for things that are meant to be merged back into trunk. For example, Alex Gaynor spent his summer of code working on a branch of Django that allows for support for multiple databases, with the goal of eventually merging it back into the Django trunk.
Checkouts (or clones, in Git's case) might better suit what you are trying to do. You would create a repo containing all of the project's base files (and .sample files, if you will), and clone the repo to all the various locations you wish to deploy the code. Then manually create the configuration and customization files at each deployment (take care not to add them to the repo). Whenever you update the code in the repo, run a pull on each deployment to update the code. Viola!
Other answers are correct that you'll be in the best shape for maintenance to the extent that you separate out your core code from the custom per-client code. However, I'll break from the crowd and say that if you're unable to do that (say because you need to add extra functionality to core code for a certain client), DVCS branches would work just fine for what you want to do. Though I'd probably recommend per-directory branches rather than in-repo branches for this purpose (git can do per-directory branches as well, it's nothing but a cloned repo that diverges).
I use hg, not git, but all of my Django projects are cloned from the same base "project template" repo that has utility scripts, a basic common set of INSTALLED_APPS, etc. This means when I make changes to that project template, I can easily merge those common updates into existing projects. This isn't exactly the same as what you're planning, but is similar. You will occasionally have to deal with merge conflicts, if you modify the same area of code in the core that you've already customized for a specific client.
Matthew Talbert is correct, you really need to separate the custom stuff from non-custom stuff. If you can refactor all the core code to be contained in one directory, your clients can use it as a read-only git submodule. The added benefit is that you lock them into an explicit version of the core code. This means that they would have to consciously update to a newer revision, which is what you want for production code.
After reading all your answers (thanks!) i decided to first refactor my django project structure to isolate the core and my differents apps in an apps subfolder. Doing this makes a cleaner project, and tweaking the .gitignore in the differents branches file makes it easy to use git branches to manage the differents customers and settings !
I'm just getting started with a project that combines GWT, Google App Engine and the Google Eclipse plugin. Where is the best place to store my tests? I normally keep my code organized Maven-style, with src/main/java, and tests in src/test/java. The default setup I get from the plugin dumped my source directly into src, which I'm not too fond of, but I'd prefer not to fight against the tools. What's the "standard" place to put unit tests in such a project?
Solution:
create src/main/java, move the existing code under there
create src/test/java, add your tests here
go to Project -> Properties -> Java Build Path, add the new locations as Source Folders.
I've faced a kind of problem woth GAE testing: Some tests require an appengine-testing.jar wich conflicts with the main appengine-api-xxx.jar of the poject. That way, I was able to run tests for GAE but it conflicted with a normal run/debug launch. To be able to run the app in my local machine, I had to remove the appengine-testing.jar and then, a lot of compilation errors appeared in my test/ clases.
If you want an advice, set your test clases in another project (where you can use the jars without conflict)
Otherwise, if you got make it work, please, tell me how did you do.
Thanks a lot.
Put it where it pains you least.
GWT on Google App Engine is pretty new at this point; you are
optimistic to expect there is a "standard" place, especially since
you've already found an inconsistency in what the tools do.
Since you've already accepted the source starting at "src/", why not
put the test source in "test/"? This is certainly standard in many
contexts.