I'm using TeamCity 7 as CI Server, and I'd have to test several Web Application plugins, mainly written with PHP. I'm familiar with ANT and *Unit, but I have an issue to solve: to properly test a plugin, my idea would be the following:
Cleanup the testing environment.
Install a clean copy of the Web Application which will host the plugin.
Install the plugin.
Enable the plugin.
Run the tests.
Obviously, running the tests on an installed environment is the easy part. Most tests are fired by directly calling plugin's classes' methods, yet the framework must be configured, even with minimal settings, to allow calling its bootstrap file and perform due initialization. I tried running the tests in an environment I prepared manually, and they run as expected.
The issue is now automating the installation of the standard Web Application, and, most importantly, its configuration. The basic steps are the following:
Unzip framework somewhere (done).
Create a Database (done).
Create a Database User ans assign it propert privileges (done).
Run Web Application's setup.
The tricky part is that not all Web Application implement a command line interface, such as drush for Drupal, hence I came out with two possible ways to complete the installation:
Simulate manual installation via CURL
Take note of the installation steps and the forms that need to be filled.
POST data to each of the forms using CURL.
I tried this method, still manually, with acceptable results. The Web Application gets installed as expected, and it can be used.
However, this requires a Web Server where the application can run. As far as I know, TeamCity Agents work in their own, random named directories and anything "installed" in them cannot be accessed via HTTP requests.
Backup/Restore
Manually pre-install a Web Application and configure its basic settings.
Zip the Application's directory and a backup of its database.
Before running the tests, unzip the directory in Agent's working directory.
Restore the backed-up database. The application will now be "configured".
This method is a bit "rough", but it doesn't require a Web Server to be running. Although the Web Application won't be able to server HTTP requests, that doesn't necessarily matter, as the tests will be run against plugin's classes.
This method has two major drawbacks, though:
Tests involving interaction with the Web Application (e.g. hooks, event handlers, and so on) can't be run.
Since the Web Application and its database are pre-configured, their parameters will be the same at every run. Therefore, it wouldn't be possible to run two Agents at the same time, for example to test two different plugins.
I'm now wondering if there's a better solution, as both the above look less than optimal to me.
Please note that, although I'm using TeamCity, the CI Server itself should not be a big deal, as I'm running everything with ANT. Therefore, any suggestion, even related to another CI Serverm, will be welcome (I know Hudson, CruiseControl and BuildMaster, I can adapt a concept easily). Thanks.
Related
I have been rushing to get my app, a mixture between nodejs server driving an sqlite database and a litElement based client side providing the ui, into a useable state as a beta release. I achieved that a couple of days ago and now I am (belatedly I know) thinking how to put together a test framework. However I am really struggling to understand how to best test the client side. I think its because I am having difficulty understanding conceptually what the two main choices of framework are. Before I go into more detail, let me explain the structure of the app in top level terms.
At the project root level there are three main directories node_modules which comprises all the modules I've pulled in (including lit-element and web-components-loader which are client side elements - but see below) server which contains all the code for the server side of my application and client which consist of all the code for the client side of my app. I run rollup ONLY at module install time to "package" lit-element and the directives I use and the web-component-loader and effectively treeshake and copy them to the client/libs. As a result of this my client is coded to assume the modules are in the libs directory AND I DO NOT NEED OR HAVE any build stage. I guess the root of the client is index.html which pulls in a service-worker.js and main-app.js. main-app is the root of a tree in lit-element based components that make up the entire client app. Nginx is the web server for all the static files in the client, but also acts as a proxy to pass any urls that start /api to a standard node http web server (not even express, although I do use the router, body-parser and final-handler modules) and these get passed to various api handlers each of which is a separate javascript file - although these can "require" a few common modules that I have written and those in the node_modules directory.
I plan on using jest as a test environment. For my server I think it is easy. For each api handler I want to test I can build a test script that "requires" the javascript file I want to test. I am in two minds about whether to use a sqlite database for testing or mock something - I am leaning towards the former as I am using better-sqlite3 and it is totally synchronous and very fast. I already have scripts to create empty databases, so I have no worry about test isolation.
Client testing is where I get confused. I "think" that in essence jest can run tests the same way as for the server, one element at a time. BUT, these elements and my test scripts are going to need a "web-platform" set of APIs - not least of which is the entire shadow dom and custom components stuff that lit-element uses. This is where, I think, puppeteer or electron in with there associated jest plugins which can put these platform apis into the test environment. But, and this is the essence of my confusion, puppeteer instructions all start with a something like
const browser = await puppeteer.launch({headless: true});
const page = await browser.newPage()
await page.goto(SOME URL);
What is this URL? - do I have to also run a server? I cannot relate this snippet to running a test controlled by jest. All the examples seem to use webpack and typescript, neither of which I know anything about.
The other module I have seen mentioned is electron, in particular this article, which everything else seems to point to (or the same text).
https://www.ninkovic.dev/blog/2020/testing-web-components-with-jest-and-lit-element
From the code snippets in this article it "seems" like it might be what I want, BUT ...
I cannot find very many references to electron other than on its own web site. Here it is telling you to use electron as a tool to build a cross platform application, but nowhere can I find what it is - it assumes you already know. I don't want a UI for my unit testing I want it to be headless like in puppeteer.
Hence my confusion and why I am unsure how to achieve what I want. Can someone give me some pointers as to
How can I set up puppeteer to run headless tests without needing a server OR
What exactly is electron (and can I use it to
a) run my tests
b) provide me with tools to examine the dom elements I have created to see I have created the right ones) and how is it different from puppeteer and can I use it to conduct headless tests of my client.
UPDATE
I've done some more digging and am beginning to understand the differences. Let me summarise what I think I have found.
Puppeteer is great for end to end testing of your site. You run the tests by launching the included puppeteer at the home page of your site (or the more likely scenario of a development test site) and programatically pretend to be a user who can click on buttons etc. You can use various methods, including functions such as document.querySelector() to check your UI has behaved how you think, or you can take screen shots and compare with standardised version. I could possible use it for unit tests, but I would have to run a server, create a test fixture html page for every test and navigate to it. jest-puppeteer is a package with some of that built in.
Electron is a platform for building apps. What the url I was referencing was using a test runner app jest-electron built using electron. So worrying about electron is a red herring, I should be worrying about jest-electron.
My main concern right now, I think, is that I need different jest configurations for my three scenarios
unit tests on the server
unit tests on the client
end to end testing of the complete app.
Given I have only one package.json file and one set of node_modules I need to figure out a way to have three different jest-config.js files.
I want to setup an cloud development environment for my personal use.
Requirements:
1. Have a cloud web server (basically any linux system) serving my Elixir (backend language) app.
2. Connect Sublime Text / Atom to this server (via sftp maybe), and make code changes and save. Automatic compilation and other stuff will be taken care by mix or task runner.
3. Multiple device connectivity to this setup.
Reasons for this setup:
I want to be able to develop from anywhere (office, home, etc), just configure the IDE and continue to work from where I left off last from any device.
Better productivity and less setup required.
Secure as well
Current solution I have:
Had setup a linux instance with sftp server enabled.
Created projects under the root of the sftp directory.
Run task runners in those projects to auto compile and server with other stuff.
Connected sublime text to this sftp server and start working. On save it uploads the file to the server.
I connect another laptop to this server and can start working on last saved state.
This setup works fine till now, but if there is a better way for this, I would love to know.
Since you are using git, you don't need a separate cloud server for syncing your dev environments. The easiest way to meet your needs would be to create a branch in git called workinprogress (for example), and then push and pull to it from your various locations. When you have something you want to publish to the main branch you can do an interactive rebase before merging, which enables you to rewrite the history of your workinprogress branch, squashing and rewording the commit messages as much as you like. Then once you have everything you want on the main branch, you either delete workinprogress and start a new one, or just git checkout workinprogress && git reset --hard master.
If you still want to have your Elixir app on a live server somewhere, then you can just pull from Github on that server and get latest updates for the app.
I work from various places too, and use this work flow. No problems so far.
I have an automation framework that uses static html pages within its project directory to perform certain aws operations such as dynamoDB scan and Aws Lambda executions. Due to some performance bottleneck in a dependent api component for the test we are trying to move the framework to an ec2 instance with Amazon linux and run the tests from there.
Since we have methods in the TestNG class that actually uses selenium web driver to spin up a browser and open up the static page in order to perform the required Aws operations I am pretty sure this test is going to run into issues.
There are two potential approach I see for solving this issue:
Implement AWSUtil classes and use necessary aws clients to replace the web dependent logics (Will require some effort and re-engineering)
Use a headless chrome browser (or any compatible one) in order to run the web dependent steps.
I am pretty sure that number 1 can be easily achieved, just a matter of time and effort. However, would love to know if there is an easy way of accomplishing #2 since this would not require any code rewrite.
We got the same issue and been successful with puppeteer,
https://github.com/GoogleChrome/puppeteer
If you don't want to install the latest version of node, you can dockerize your tests.
puppeteer can run headless or with browser.
Hope it helps.
There is no need to change anything in your tests, just the setup and the execution. The tests can run headlessly on a Continuous Integration (CI) server. There is no out-of-the-box setup since there is no display output for the browser to launch in. However with Xvfb you can launch the browser virtually. Straight from the docs:
Xvfb (short for X virtual framebuffer) is an in-memory display server for UNIX-like operating system (e.g., Linux). It enables you to run graphical applications without a display
Depending from do you want to keep Xvfb running in the background until the process is killed, there are two options:
Xvfb :99 &
export DISPLAY=:99
run-your-tests-here
or
xvfb-run run-your-tests-here
Here is a Linux tutorial. I am using this for my Docker based Jenkins setup and works like a charm, every time.
I'm developing an tool using Django for internal use at my organization. It's used to search and tag documents (using Haystack and Solr), and will be employed on different projects. My team currently has a working prototype and we want to deploy it 'in the wild.'
Our security environment is strict. Project documents are located on subfolders on a network drive, and access to these folders is restricted based on users' Windows credentials (we also have an MS SQL server that uses the same credentials). A user can only access the projects they are involved in. Since we're an exclusively Microsoft shop, if we want to deploy our app on the company intranet, we'll need to use an IIS server to deal with these permissions. No one on the team has the requisite knowledge to work with IIS, Active Directory, and our IT department is already over-extended. In short, we're not web developers and we don't have immediate access to anybody experienced.
My hacky solution is to forgo IIS entirely and have each end user run a lightweight server locally (namely, CherryPy) while each retaining access to a common project-specific database (e.g. a SQLite DB living on the network drive or a DB on the MS SQL server). In order to use the tool, they would just launch an all-in-one batch script and point their browser to 127.0.0.1:8000. I recognize how ugly this is, but I feel like it leverages the security measures already in place (note that never expect more than 10 simultaneous users on a given project). Is this a terrible idea, and if so, what's a better solution?
I've dealt with a similar situation (primary development was geared toward a normal deployment situation, but some users have a requirement to use the application on a standalone workstation). Rather than deploy web and db servers on a standalone workstation, I just run the app with the Django internal development server and a SQLite DB. I didn't use CherryPy, but hopefully this is somewhat useful to you.
My current solution makes a nice executable for users not familiar with the command line (who also have trouble remembering the URL to put in their browser) but is also relatively easy development:
Use PyInstaller to package up the Django app into single executable. Once you figure this out, don't continue to do it by hand, add it to your continuous integration system (or at least write a script).
Modify the manage.py to:
Detect if the app is frozen by PyInstaller and there are no arguments (i.e.: user executed it by double clicking it) and if so, then run execute_from_command_line(..) with arguments to start the Django development server.
Right before running the execute_from_command_line(..), pop off a thread that does a time.sleep(2) (to let the development server come up fully) and then webbrowser.open_new("http://127.0.0.1:8000").
Modify the app's settings.py to detect if frozen and change things around such as the path to the DB server, enabling the development server, etc.
A couple additional notes.
If you go with SQLite, Windows file locking on network shares may not be adequate if you have concurrent writing to the DB; concurrent readers should be fine. Additionally, since you'll have different DB files for different projects you'll have to figure out a way for the user to indicate which file to use. Maybe prompt in app, or build the same app multiple times with different settings.py files. Variety of a ways to hit this nail...
If you go with MSSQL (or any client/server DB), the app will have to know the DB credentials (which means they could be extracted by a knowledgable user). This presents a security risk that may not be acceptable. Basically, don't try to have the only layer of security within the app that the user is executing. The DB credentials used by the app that a user is executing should only have the access that the user is allowed.
I am trying to work out a good way to run a staging server and a production server for hosting multiple Coldfusion sites. Each site is essentially a fork of a repo, with site specific changes made to each. I am looking for a good way to have this staging server move code (upon QA approval) to the production server.
One fanciful idea involved compiling the sites each into EAR files to be run on the production server, but I cannot seem to wrap my head around Coldfusion archives, plus I cannot see any good way of automating this, especially the deployment part.
What I have done successfully before is use subversion as a go between for a site, where once a site is QA'd the code is committed and then the production server's working directory would have an SVN update run, which would then trigger a code copy from the working directory to the actual live code. This worked fine, but has many moving parts, and still required some form of server access to each server to run the commits and updates. Plus this worked for an individual site, I think it may be a nightmare to setup and maintain this architecture for multiple sites.
Ideally I would want a group of developers to have FTP access with the ability to log into some control panel to mark a site for QA, and then have a QA person check the site and mark it as stable/production worthy, and then have someone see that a site is pending and click a button to deploy the updated site. (Any of those roles could be filled by the same person mind you)
Sorry if that last part wasn't so much the question, just a framework to understand my current thought process.
Agree with #Nathan Strutz that Ant is a good tool for this purpose. Some more thoughts.
You want a repeatable build process that minimizes opportunities for deltas. With that in mind:
SVN export a build.
Tag the build in SVN.
Turn that export into a .zip, something with an installer, etc... idea being one unit to validate with a set of repeatable deployment steps.
Send the build to QA.
If QA approves deploy that build into production
Move whole code bases over as a build, rather than just changed files. This way you know what's put into place in production is the same thing that was validated. Refactor code so that configuration data is not overwritten by a new build.
As for actual production deployment, I have not come across a tool to solve the multiple servers, different code bases challenge. So I think you're best served rolling your own.
As an aside, in your situation I would think through an approach that allows for a standardized codebase, with a mechanism (i.e. an API) that allows for the customization you're describing. Otherwise managing each site as a "custom" project is very painful.
Update
Learning Ant: Ant in Action [book].
On Source Control: for the situation you describe, I would maintain a core code base and overlays per site. Export core, then site specific over it. This ensures any core updates that site specific changes don't override make it in.
Call this combination a "build". Do builds with Ant. Maintain an Ant script - or perhaps more flexibly an ant configuration file - per core & site combination. Track version number of core and site as part of a given build.
If your software is stuffed inside an installer (Nullsoft Install Shield for instance) that should be part of the build. Otherwise you should generate a .zip file (.ear is a possibility as well, but haven't seen anyone actually do this with CF). Point being one file that encompasses the whole build.
This build file is what QA should validate. So validation includes deployment, configuration and functionality testing. See my answer for deployment on how this can flow.
Deployment:
If you want to automate deployment QA should be involved as well to validate it. Meaning QA would deploy / install builds using the same process on their servers before doing a staing to production deployment.
To do this I would create something that tracks what server receives what build file and whatever credentials and connection information is necessary to make that happen. Most likely via FTP. Once transferred, the tool would then extract the build file / run the installer. This last piece is an area I would have to research as to how it's possible to let one server run commands such as extraction or installation remotely.
You should look into Ant as a migration tool. It allows you to package your build process with a simple XML file that you can run from the command line or from within Eclipse. Creating an automated build process is great because it documents the process as well as executes it the same way, every time.
Ant can handle zipping and unzipping, copying around, making backups if needed, working with your subversion repository, transferring via FTP, compressing javascript and even calling a web address if you need to do something like flush the application memory or server cache once it's installed. You may be surprised with the things you can do with Ant.
To get started, I would recommend the Ant manual as your main resource, but look into existing Ant builds as a good starting point to get you going. I have one on RIAForge for example that does some interesting stuff and calls a groovy script to do some more processing on my files during the build. If you search riaforge for build.xml files, you will come up with a great variety of them, many of which are directly for ColdFusion projects.