I‘m looking for a way to add unit tests to my documentation. I generate my docs website from asciidoc using antora. Works perfectly fine. Now I am looking for a way to unit test my docs (asciidoc files as well). But this is not my question right now.
What I want right now is a way to test the filesystem structure of my docs. I have several repos with similar base setup and basic docs. The details differ but e.g. I want to ensure every docs have their main docs in PROJECT/docs/modules/ROOT and the navigation file is named nav.adoc in every module etc.
Basically I want the same docs structure for every project.
So I wonder if there is a way to test the filesystem (foldernames, Filenames). Ultimately these tests should be part of my gitlab ci pipeline.
Can archunit help with that? I’d consider it if there is no better way, but since archunit targets java projects it seems like overkill for my uaecase.
It is important to remember that Antora can gather content from multiple repos. When Antora gathers content, that content is held in a virtual filesystem until the transformation to HTML is complete. That means that wherever you run Antora, the filesystem structure of the repos cannot be directly interrogated unless it happens to be local (you're using "author" mode, or you have an up-to-date checkout of the repo(s) locally).
It is possible to write an extension for Antora (in JavaScript, since Antora is a Node.js application) to investigate the virtual filesystem and complain about any structural problems found.
Outside of Antora, you'd basically be writing a repo-specific test. That test could be invoked any way you like, say by running a test script when make is run. Or, you might add a git commit hook to your repo that prevents commits when the appropriate file structure does not exist.
The TL;DR answer is "yes, this can be done". However, it requires some implementation effort to achieve.
Related
I want to intercept ClassLoader's loadClass method. In order to show the process of loading class by ClassLoader. But I don't know the working principle of aspectj. It can modify the source code of the JDK?
You could just use a debugger and step through the process in order to understand it.
AspectJ can weave aspects into existing class files during compilation (CTW, compile-time weaving) or during class-loading (LTW, load-time weaving).
LTW does not work for JDK classes because those classes are always loaded before the weaving agent. So you have a bootstrapping problem there.
What you can do, however, is to apply CTW to the class files in e.g. rt.jar from your JRE/JDK distribution, zip the woven class files into another JAR and replace the original one with it. Then if you start the JVM with aspectjrt.jar on the boot classpath, you see the aspects in action. I have done that before and it works. But really, this is not what AspectJ was designed for. But you asked, so I told you. It is a very advanced topic though, and I doubt that it is the road you should take as a complete AspectJ greenhorn - no offense meant.
Update: Because I have been asked this question so often, I created a little demo project showing how to weave aspects into the JRE/JDK. I still do not think it makes sense to use it under normal circumstances, but what the heck: Why do people climb the Mount Everest? Because it exists. ;-)
We are moving our build and testing system to Jenkins, and are looking for an easy way (where we don't have to code all the logic ourselves) to manage the build artifacts.
Basically we need an organised way to store them in order by the type of the build, the user who built it and such for example:
johnd/nightly/r543241/win32/program.zip
johnd/nightly/trunk/lin64/program.tar.gz
master/release/2.1/win32/program.zip
This way we can upload when a build is complete, and retrieve the required artifact in the testing stage with ease.
Until now we simply stored the files in directories on NFS, but recently started considering an artifact manager. I've looked at Artifcatory, Archiva and Nexus. But all seem very Java centered or at least required maven to work with. Since I don't want to introduce more complexity (We mainly work with python, scons is our build tool) and I don't want to introduce maven to the mix, I'm looking for something that has an easy command line (or better REST/Python interface) to upload, download, manage artifacts.
If you don't use an artifact manager, but use some other clever method to manage your C++ artifacts for release/test needs I'd be happy to hear that also.
I have exactly the same issue. I didn't succeed to integrate artifactory easilly, so I forgot it.
For the moment I use the copy artefact plugin to copy by scp the file to an NFS share. But this is not a good solution since shares may be mounted differently on two machines, and windows cannot access them directly.
But a real artifact manager would be so such a good thing for our project.
In my previous job, I was using a documentation CMS (LiveLink) which provided good services:
- self organising folder.
- persistant url (we can move, rename folders and files, a public URL to a given files never changes, that was SOOO great : "http://server/go/123456789123456789" always pointed to the same file. So you just had to give this url to wget and you get the file
- file versionning.
- meta data, advanced search
I'm still searching such a solution in opensource software and don't find it.
In you just need to share artefact between two jobs, simple use the "copy artifact for another build" plugin. I use it to split my job into a a build job and a test job (actually there are 3 test jobs, one very fast done after each commit, one quite slow done every day, and one very slow done each week end).
I would love to hear ideas on how to best move code from development server to production server.
A list of gotcha's, don't do this list would be helpful.
Any tools to help automate the steps of.
Make backups of existing code, given these list of files
Record the Deployment of these files from dev to production
Allow easier rollback if deployment or app fails in any way...
I have never worked at a company that had a deployment process, other than a very manual, ftp files from dev to production.
What have you done in your companies, departments, etc?
Thank you...
Yes, I am a coldfusion programmer, but files are files, and this should be language agnostic question.
OK, I'll bite. There's the technology aspect of this problem, which other answers have already covered. But the real issue is a process problem. Where the real focus should be ensuring a meaningful software development life cycle (SDLC) - planning, development, validation, and deployment. I'll cover each in turn. What you want is a repeatable activity at each phase.
Planning
Articulating and recording what's to be delivered. Often tickets or user stories are enough. Sometimes you do more, like a written requirements document, that a customer signs off on, that's translated into various artifacts such as written use cases - ultimately what you want though is something recorded in an electronic system where you can associate changes to code with it. Which leads me to...
Development
Remember that electronic system? Good. Now when you make changes to code (you're committing to source control right?) you associate those change with something in this electronic system - typically tickets. I like Trac, but have also heard good things about Atlassian's suite. This gives you traceability. So you can assert what's been done and how. Then you can use this system and source control to create a build - all the bits needed for whatever's changed - and tag that build in source control - that's your list of what's changed. Even better, have a build contain everything, so that it's standalone entity that can easily be deployed on it's own. The build is then delivered for...
Validation
Perhaps the most important step that many shops ignore - at their own peril. Defects found in production are exponentially more expensive to fix then when they're discovered earlier in the process. And validation is often the only step where this occurs in many shops - so make sure yours does it.
This should not be done by the programmer! That's like the fox watching the hen house. And whoever is doing is should be following some sort of plan. We use Test Link. This means each build is validated the same way, so you can identify regression bugs. And, this build should be deployed in the same way as you would into production.
If all goes well (we usually need a minimum of 3 builds) the build is validated. And this goes to...
Deployment
This should be a non-event, because you're taking a validated build following the same steps as you did in testing. Could be first it hits a staging server, where there's an automated copying process, but the point being is that is shouldn't be an issue at this point, because you validated with the same process.
Conclusion
In terms of knowing what's where, what you really want is a logical way to group changes together. This is where the idea of a build comes in. It's really the unit that should segue between steps in the SDLC. If you already have that, then the ability to understand the state of a given system becomes trivial.
Check out Ant or Maven - these are build and deployment tools used in the Java world which can help you copy / ftp files, backup and even check out code from SVN.
You can automate your deployment steps using these tools, for example Ant will allow you declare a set of tasks as part of your deployment. So you could, for example:
Check out a revision using SVNAnt or similar to a directory
Copy (and perhaps zip first) these files to a backup directory
FTP all the files to your web server(s)
Create a report to email to the team illustrating the deployment
Really you can do almost anything you wish to put time into using Ant. Maven is a little more strucutred (and newer) and you can see a discussion of the differences here.
Hope that helps!
In a nutshell...
You should start with some source control solution - probably Subversion or Git. Once that's in place you can create a script that generates a clean build of your source code and deploys it to your production server(s).
You could do this with a simple batch script or use something like Ant for more control. Here is a simple example of a batch file using Subversion:
svn copy svn://path/to/your/project/trunk -r HEAD svn://path/to/your/project/tags/%version%
svn checkout svn://path/to/your/project/trunk -r HEAD //path/to/target/directory
Ant makes it easy to do things like automatically run unit tests and sync directories. For example:
<sync todir="//path/to/target/directory" includeEmptyDirs="true" overwrite="true">
<fileset dir="${basedir}">
<exclude name="**/*.svn"/>
<exclude name="**/test/"/>
</fileset>
</sync>
This is really just a starting point. A next step might be a continuous integration solution like Hudson. I would also recommend reading "Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Applications".
One ColdFusion specific gotcha is to make sure you clear the Application scope when required (to update any singleton components). A common approach here is to use a URL parameter that causes onRequestStart() to call onApplicationStart(). You may also have to clear the trusted cache.
We use a system called AnthillPro: http://www.anthillpro.com
It's commercial software, but it allows us to completely automate our deployment process across multiple servers and operating systems (We currently use it for both ColdFusion and Java, but it can be used for most languages. It has a ton of 3rd party integrations:
http://www.anthillpro.com/html/products/anthillpro/tool-integrations.html
I have a maven project that converts text files of a specific format to another format.
For testing I have put in src/test/resources a large amount of test files.
I also have another project that uses the first one to do the conversion and then do some extra stuff on the output format. I also want to test this project against the same test dataset, but I dont want to have duplicate data sets and I want to be able to test the first project alone since it is also a standalone converter project.
Is there any common solution for doing that? I dont mind not having the test dataset inside the projects source tree as long as each project can access the data set independently of the other. I dont want to setup a database for that also. I am thinking something like a repository of test data simpler than an RDBMS. Is there any application for this kind of need that I can use with a specific maven plugin? Ease of setup and simplicity is my priority. Also I m thinking something like packaging the test data and putting it in a internal maven repo and then downloading it and unzip it in the junit code. Or better, is there a maven plugin that can do this for me?
Any ideas?
It is possible to share resources with Maven, for example with the help of the Assembly and Dependency plugins. Basically:
Create a module with a packaging of type pom to hold the shared resources
Use the assembly plugin to package the module as a zip
Declare a dependency on this zip in "consumer" modules
And use dependency:unpack-dependencies (plus some exclusion rules) to unpack the zip
This approach is detailed in How to share resources across projects in Maven. It requires a bit of setup (which is "provided") and is not too complicated.
Just place them in a tree in the src/main/resources directory of a separate module specially to share the test data. They will be added to the jar file and me nicely compressed and versioned in your nexus repository, file-share, ~/.m2/repository or whatever you use to store/distribute maven artifacts.
Then just add a dependency in the projects you need the data in the test scope and use resource loading to get them from the jars.
You do not need any special plugins or other infrastructure. This just works.
Im new to GIT and dont know yet how much it will fit my needs, but it looks impressive.
I have a single webapp that i use for differents customers (django+javascript)
I plan to use GIT to handle these differents customers version as branches. Each customer can have custom files, folders and settings, improved versions... but the should share the same 'core'. We are a small team and suscribed a github account.
Is the branch the good way to handle this case ?
About the settings file, how would you proceed ? Would you .gitignore the customer specific settings file and add a settings.xml.sample file for example is the repo ?
Also, is there any way to prevent some files to be merged into master ? (but commited to the customer branch). For example, id like to save some customer data to the customer branch but prevent from to being commited to master.
Is the .gitignore file branch-specific ? YES
EDIT
After reading all your answers (thanks!) i decided to first refactor my django project structure to isolate the core and my differents apps in an apps subfolder. Doing this makes a cleaner project, and tweaking the .gitignore file makes it easy to use git branches to manage the differents customers and settings !
Ju.
In addition to cpharmston's answer, it sounds like you need to do some refactoring to separate out what is truly custom for each client and what isn't. Then you may consider adding additional repositories to track the customizations for each client (entirely new repos, not branches). Then your deployment can pull your "core" from your main repo, and the client-specific stuff from that repo.
I would not use branches to accomplish what you are trying to do.
In source control, branches are intended to be used for things that are meant to be merged back into trunk. For example, Alex Gaynor spent his summer of code working on a branch of Django that allows for support for multiple databases, with the goal of eventually merging it back into the Django trunk.
Checkouts (or clones, in Git's case) might better suit what you are trying to do. You would create a repo containing all of the project's base files (and .sample files, if you will), and clone the repo to all the various locations you wish to deploy the code. Then manually create the configuration and customization files at each deployment (take care not to add them to the repo). Whenever you update the code in the repo, run a pull on each deployment to update the code. Viola!
Other answers are correct that you'll be in the best shape for maintenance to the extent that you separate out your core code from the custom per-client code. However, I'll break from the crowd and say that if you're unable to do that (say because you need to add extra functionality to core code for a certain client), DVCS branches would work just fine for what you want to do. Though I'd probably recommend per-directory branches rather than in-repo branches for this purpose (git can do per-directory branches as well, it's nothing but a cloned repo that diverges).
I use hg, not git, but all of my Django projects are cloned from the same base "project template" repo that has utility scripts, a basic common set of INSTALLED_APPS, etc. This means when I make changes to that project template, I can easily merge those common updates into existing projects. This isn't exactly the same as what you're planning, but is similar. You will occasionally have to deal with merge conflicts, if you modify the same area of code in the core that you've already customized for a specific client.
Matthew Talbert is correct, you really need to separate the custom stuff from non-custom stuff. If you can refactor all the core code to be contained in one directory, your clients can use it as a read-only git submodule. The added benefit is that you lock them into an explicit version of the core code. This means that they would have to consciously update to a newer revision, which is what you want for production code.
After reading all your answers (thanks!) i decided to first refactor my django project structure to isolate the core and my differents apps in an apps subfolder. Doing this makes a cleaner project, and tweaking the .gitignore in the differents branches file makes it easy to use git branches to manage the differents customers and settings !