I'm working on a project where we create a web-app and right now we run our unit-tests on our desktop browsers. However someone on the team expressed their concern with us not doing unit-testing on the mobile browsers the app is supposed to run on.
Is there a way to run jasmine tests using karma on mobile devices, and if so how?
There are a few companies that have infrastructure for this. One I saw a presentation from is http://appurify.com/. Googling around there are a few others.
I am sure each is different, but generally there is a VM that runs on the device, and you can tunnel to it and run unit, load, network, etc tests.
No relationship with appurify. Just happen to hear of them.
cheers-
Dustin
Related
I have been rushing to get my app, a mixture between nodejs server driving an sqlite database and a litElement based client side providing the ui, into a useable state as a beta release. I achieved that a couple of days ago and now I am (belatedly I know) thinking how to put together a test framework. However I am really struggling to understand how to best test the client side. I think its because I am having difficulty understanding conceptually what the two main choices of framework are. Before I go into more detail, let me explain the structure of the app in top level terms.
At the project root level there are three main directories node_modules which comprises all the modules I've pulled in (including lit-element and web-components-loader which are client side elements - but see below) server which contains all the code for the server side of my application and client which consist of all the code for the client side of my app. I run rollup ONLY at module install time to "package" lit-element and the directives I use and the web-component-loader and effectively treeshake and copy them to the client/libs. As a result of this my client is coded to assume the modules are in the libs directory AND I DO NOT NEED OR HAVE any build stage. I guess the root of the client is index.html which pulls in a service-worker.js and main-app.js. main-app is the root of a tree in lit-element based components that make up the entire client app. Nginx is the web server for all the static files in the client, but also acts as a proxy to pass any urls that start /api to a standard node http web server (not even express, although I do use the router, body-parser and final-handler modules) and these get passed to various api handlers each of which is a separate javascript file - although these can "require" a few common modules that I have written and those in the node_modules directory.
I plan on using jest as a test environment. For my server I think it is easy. For each api handler I want to test I can build a test script that "requires" the javascript file I want to test. I am in two minds about whether to use a sqlite database for testing or mock something - I am leaning towards the former as I am using better-sqlite3 and it is totally synchronous and very fast. I already have scripts to create empty databases, so I have no worry about test isolation.
Client testing is where I get confused. I "think" that in essence jest can run tests the same way as for the server, one element at a time. BUT, these elements and my test scripts are going to need a "web-platform" set of APIs - not least of which is the entire shadow dom and custom components stuff that lit-element uses. This is where, I think, puppeteer or electron in with there associated jest plugins which can put these platform apis into the test environment. But, and this is the essence of my confusion, puppeteer instructions all start with a something like
const browser = await puppeteer.launch({headless: true});
const page = await browser.newPage()
await page.goto(SOME URL);
What is this URL? - do I have to also run a server? I cannot relate this snippet to running a test controlled by jest. All the examples seem to use webpack and typescript, neither of which I know anything about.
The other module I have seen mentioned is electron, in particular this article, which everything else seems to point to (or the same text).
https://www.ninkovic.dev/blog/2020/testing-web-components-with-jest-and-lit-element
From the code snippets in this article it "seems" like it might be what I want, BUT ...
I cannot find very many references to electron other than on its own web site. Here it is telling you to use electron as a tool to build a cross platform application, but nowhere can I find what it is - it assumes you already know. I don't want a UI for my unit testing I want it to be headless like in puppeteer.
Hence my confusion and why I am unsure how to achieve what I want. Can someone give me some pointers as to
How can I set up puppeteer to run headless tests without needing a server OR
What exactly is electron (and can I use it to
a) run my tests
b) provide me with tools to examine the dom elements I have created to see I have created the right ones) and how is it different from puppeteer and can I use it to conduct headless tests of my client.
UPDATE
I've done some more digging and am beginning to understand the differences. Let me summarise what I think I have found.
Puppeteer is great for end to end testing of your site. You run the tests by launching the included puppeteer at the home page of your site (or the more likely scenario of a development test site) and programatically pretend to be a user who can click on buttons etc. You can use various methods, including functions such as document.querySelector() to check your UI has behaved how you think, or you can take screen shots and compare with standardised version. I could possible use it for unit tests, but I would have to run a server, create a test fixture html page for every test and navigate to it. jest-puppeteer is a package with some of that built in.
Electron is a platform for building apps. What the url I was referencing was using a test runner app jest-electron built using electron. So worrying about electron is a red herring, I should be worrying about jest-electron.
My main concern right now, I think, is that I need different jest configurations for my three scenarios
unit tests on the server
unit tests on the client
end to end testing of the complete app.
Given I have only one package.json file and one set of node_modules I need to figure out a way to have three different jest-config.js files.
Can I run my e2e test developed using Protractor on the AWS device farm?
Because I want to complete mobile testing of my project using the AWS device farm, and do not really understand can I do that or not. I found 3 types about that on the AWS forum, but it is too old from 2018.
First forum discussion
Second forum discussion
Third forum discussion
Maybe something changed?
I have protractor e2e tests written for the desktop browser and want to use those ones for the mobile browser too.
I will answer this for both mobile browsers and desktop testing.
Mobile Browsers
AWS Device Farm has 2 execution modes: Standard Mode and Custom Mode.
Standard mode gives you granular reporting if you don't generate a report for your tests locally. This splits up the artifacts for each test.
Custom mode gives you as close as possible execution state and results as you would get locally. It does not give you the granular reporting which is fine for most as you already get reports locally which will be available on Device Farm as well. It is recommended for customers to use custom mode as that is the one that is most up to date and adds supports for latest frameworks unless of course they absolutely need granular reporting.
Protractor on Device Farm
It is not officially support today.
However, Device Farm supports Appium Nodejs in custom mode. You get a yaml file where you can run shell commands on the host machine where the tests will be executed. So in case of protractor you could select this test type (Appium Nodejs), install the missing dependencies needed for the tests, start your server, and run your tests.
The points to evaluate: Since Device Farm takes your tests as inputs, you will have to upload the zip file of your tests. I would highly recommend checking the instructions for nodejs tests and using the same. Alternatively, you can also download your tests on the fly using the yaml file.
Desktop Browsers
Device Farm has a selenium grid that you can connect to from your local machine and run your tests. The browsers Chrome and Firefox run on Windows platform and Safari is not supported today. If you use a selenium grid on your local machine for your tests, then you most likely should be able to run the same tests using the Selenium grid on Device Farm. Of course, pending validation.
If you need more help on any of these items feel free to reach out to aws-devicefarm-support#amazon.com and I can help you further.
You can test in chrome with an emulated mobile mode:
You can add "mobileEmulation" in a new protractor.conf-mobile.js
chromeOptions: {
args: ['--disable-infobars', '--headless', '--disable-gpu', '--window-size=1920,1080'],
'mobileEmulation' : { 'deviceName': 'Galaxy S5' },
I have an app with ParseServer back-end and Ionic2 front-end. I need to simulate multiple users to stress test the back-end.
What load testing tools would you recommend to use for such setup?
Thanks.
You need to split your process into 2 phases:
Server-side testing. You need to load test your backend to ensure that it is in position to simulate anticipated number of users. In fact any tool capable of sending HTTP Requests will fit, the most popular free and open source load testing solutions are JMeter, Grinder, Gatling, and Tsung. All of them come with record-and-replay functionality so you will be able to build your test by just interacting with your mobile application and using the load testing tool as a proxy. See Open Source Load Testing Tools: Which One Should You Use? article for main features highlighted and compared.
Client-side testing. Even if your server responds very fast, handles enormous loads, able to scale, etc. your application user experience may be not that good as client side performance also matters. You can go for Chrome Dev Tools Remote Debugging and/or Intel XDK which is capable of profiling existing applications.
You can try using ZebraTester and record the script for this test. The trial version allows you to have up to 20 Virtual Users and these can run multiple loops depending on the length of your test. The same tool can record the script as well as run the load test from your local machine.
I use to test parse server
in http://jmeter.apache.org/
is a free tool you can install your computer then start testing
I wonder if you have any mobile test automation tool that you can recommend? After trying several tools I simply cannot find one that meets every criteria. Ranorex can be used on both devices but after facing too many issues I have chosen not to use it. The best tool I have encountered is robotium, although it only works on Android and therefore is not interesting.
Criteria:
Possible to CI integrate.
Can be used with Nunit or a similar structure.
Can be used on both iOS and Android.
Is stable.
Please do not recommend one of these recording tools, since they (in my opinion) do not test anything.
Have a nice day an thank you for all the help.
Automation of applications on real devices for Hybrid/Native/Mobile Web can be achieved by using Appium which is an open-source framework.
Appium allows you to create once and use everywhere capability for mobile web apps, e.g., same script would be running on Android as well as iOS.
Its rich function library support enables you to automate even complex mobile application gestures e.g., tap, pinch, precise tap, swipe etc.
For more details on appium, visit: http://appium.io/
Maybe Tosca Mobile+ would be of interest for you:
http://www.tricentis.com/tricentis-tosca-testsuite/tosca-mobile-plus/
Tosca follows a model based approach - it's not script based, and it's also not a Capture & Replay tool. The identification of the controls is done via technology-related properties and IMHO the most reliable way for a stable automation.
Beside of Android & iOS, Tosca does support many more technologies (UI and Non-UI).
Tosca can also be completely integrated into your CI (using e.g. Jenkins, TeamCity, QuickBuild or any other build tool).
For more details you can also look at the Tosca Mobile+ Documentation:
https://documentation.tricentis.com/en/840/index.htm#tbox/mobile.htm%3FTocPath%3DTBox%7CTosca%2520Mobile%252B%7C_____0
May be it is bit late to answer this question - you can try M-eux from Jamo Solutions. It works well for android and iOS. Supports native, hybrid and web apps on both android and iOS. Continuous integration is possible. By default there is no out of the box integration but you can write unit test cases in junit. Compile them and put them in CI using jenkins.
Other option could be seetest from Experitest. Same features are available with this tool as well. Both of these tools are not open source or freeware.
You could try out Calabash. It can be used with both IOS and android and is easily integrated into a CI pipeline.
Have you tried SIGOS App Experience (formerly Keynote Mobile Testing/DeviceAnywhere)?
It integrates with many of the leading CI/agile tools including Appium, HP UFT, Selenium, and SAP. They offer testing on real devices and you can choose what type of deployment you want-- public cloud, private cloud, on-premise.
SIGOS App Experience also offers app performance monitoring, which not all test automation platforms do. https://appexperience.sigos.com/
I'm familiar with TDD and use it in both my workplace and my home-brewed web applications. However, every time I have used TDD in a web application, I have had the luxury of having full access to the web server. That means that I can update the server then run my unit tests directly from the server. My question is, if you are using a third party web host, how do you run your unit tests on them?
You could argue that if your app is designed well and your build process is sound and automated, that running unit tests on your production server isn't necessary, but personally I like the peace of mind in knowing that everything is still "green" after a major update.
For everyone who has responded with "just test before you deploy" and "don't you have a staging server?", I understand where you're coming from. I do have a staging server and a CI process set up. My unit tests do run and I make sure they all pass before an an update to production.
I realize that in a perfect world I wouldn't be concerned with this. But I've seen it happen before. If a file is left out of the update or a SQL script isn't run, the effects are immediately apparent when running your unit tests but can go unnoticed for quite some time without them.
What I'm asking here is if there is any way, if only to satisfy my own compulsive desires, to run a unit test on a server that I cannot install applications on or remote into (e.g. one which I will only have FTP access to in order to update files)?
I think I probably would have to argue that running unit tests on your production server isn't really part of TDD because by the time you deploy to your production environment technically speaking, you're past "development".
I'm quite a stickler for TDD, and when I'm preaching the benefits to clients I often find myself saying "you can't half adopt TDD, it's all or nothing"
What you probably should have is some form of automated testing that you perform "after" deployment but these are not part of TDD.
Maybe you should look at your process again.
You could write functional tests in something like WATIR, WATIN or Selenium that test what is returned in the reponse page after posting certain form data or requesting specific URLs.
For clarification: what sort of access do you have to your web server? FTP or WebDAV only? From your question, I'm guessing ssh access isn't available - you're dropping files in a directory to deploy. Is that correct?
If so, the answer for unit testing is likely 'do it before you deploy'. You can set up functional testing driven by an automated tool like Selenium to test your app remotely via the web interface, but that's not really unit testing the sense that you're restricted to testing the system as a whole.
Have you considered setting up a staging server, perhaps as a VMWare instance, that mirrors or at least mimics your deployment environment?
What's preventing you from running unit tests on the server? If you can upload your production code and let it run there, why can't you upload this other code and run it as well?
I've written test tools for sites using python and httplib/urllib2 generally it would have been overkill but it was suitable in these cases. Not sure it's going to be of general use though.