We are developing a PC application that runs on Windows.
What would be a good approach for the developers to get some kind of load testing done before it goes to Test? -
Using an already existing tool for GUI testing like Selenium? Any other easy to use tool? Our code is in C++ (Open GL) on Windows.
Add a test stub within our code to simulate a load of the events that would get generated every time a button is clicked.
Use of a run time memory leak tool like Purify Plus or drmemory
We are developers and are primarily looking at some bare minimum load tests before the test team runs extensive tests.
Is a combination of #2 and #3 or just #3 helpful for this?
Please let me know if you have any other comments..
So you typically don't load test desktop applications unless that application is making calls to your server (in that case you would be load testing the sever).
I've never seen a desktop application which isn't able to handle user load on a single desktop. Unless you are querying a db on the user's desktop and need to worry about loading a massive amount of data and pegging the CPU, you should be fine.
When programming in C/C++, I've found GNU gprof invaluable in building a program which successfully utilizes the CPU's cache. GNU gprof is really invaluable beacuse it will provide you with the flat profile tool, which will give you the total execution time spent in each function and its percentage of the total running time, along with function call counts.
For UI desktop application testing, these two tools are really great and have been recommended by Joel (from Joel on Software):
TestComplete
Sikuli
You can learn more about about GNU gprof here.
Both Silk Performer and Load Runner offer GUI load testing support. I've used both and have ran up to 250 Virtual User loads which have simulated a few thousand users per/day. I've sometimes have run into applications where all protocol level load testing tools don't support the applications and the GUI approach works great. I've also been learning to Selenium with JMeter to run the GUI load tests but getting the selenium script to be reliable has been very difficult. If you run into application that don't need huge amounts of users you'll find the GUI approach to be much faster and sometime more accurate than the protocol based approach.
Related
I am writing an application using Qt and want to try and deploy it as a web-application. I want user's to be able to use my application by accessing it through a web browser. I'm guessing that's what a web-application is? What kind of options do I have? I've never looked into doing anything like this but I'd like to learn something new.
EDIT: What if I deployed my application on a Linux server and had users access/run it through a terminal? I think writing web application is going to be more complicated than I had originally thought.
If all you have is a Qt application, then the best you can do is use Qt 5 and run it using a remote visualization package:
Use WebGL streaming, introduced in Qt 5.10. Qt exposes a browser-connectible interface directly, without need for third-party code.
For Qt 5.0-5.9, you can use the vnc platform plugin. Then connect using a web-browser based vnc client.
For many uses it might be sufficient, and certainly it's much less effort than coding up a web app.
You're looking for Wt which provides a different set of drawing routines for many Qt GUI elements, turning them from lines on screen to HTML controls.
http://www.webtoolkit.eu/wt
It also handles websocket calls to provide interactivity. It seems a great idea, let us know how it works in practice.
For the case of QML there is QmlWeb which is a JavaScript library that is able to parse QML-code and create a website out of it using normal HTML/DOM elements and absolute positions within CSS, translating the QML properties into CSS properties.
QmlWeb is a small project by Lauri Paimen that he’s already developing for a few years now. QmlWeb of course doesn’t yet support everything Qt’s implementation of QML does, but it already supports a quite usable subset of it. It supports nearly all of the most basic QML syntax. Moreover it has support for HTML input elements (Button, TextInput, TextArea are currently supported, more to come).
Well, QmlWeb is not finished. I hope Digia help with this project to make it ready with mature features.
Interestingly, it is possible to compile Qt applications to javascript using emscripten-qt. These run fairly fast with Firefox's asm.js interpreter:
http://vps2.etotheipiplusone.com:30176/redmine/projects/emscripten-qt/wiki
Try "Qt for Webassembly".
Webassembly allows the C/C++ code to be compiled and run natively inside majority of the browsers:
WebAssembly (Wasm, WA) is a web standard that defines a binary format and a corresponding assembly-like text format for executable code in Web pages. ... It is executed in a sandbox in the web browser after a verification step. Programs can be compiled from high-level languages into Wasm modules and loaded as libraries from within JavaScript applets ... Its initial aim is to support compilation from C and C++, though support for other source languages such as Rust and .NET languages is also emerging.
To run a Qt application unchanged over the web so users can operate it in a browser, you can compile it for Android using the x86 Android ABI, run it inside an Android emulator on a server and supply the Android Cast videostream to users' browsers. You'll also need to have JavaScript in place that records the keyboard and mouse events on the web clients and relays them back to the server.
I had previously tried Qt WebGL streaming and found it to be good over the local network but too slow over the Internet. A 10 s application startup time is acceptable, but 3 s to show a new screen is rather not. I had the exact same experience with the Qt VNC platform plugin. Compared with that, the Android Cast streaming based appetize.io solution (see below) was much faster, providing a well usable user experience even over my 8 Mbit/s connection.
Existing solutions
Here is an overview of commercial products and open source software components that I found that can help you with this approach:
appetize.io. This is a commercial product to run Android applications over the web for demo and testing purposes. I have just done this with a Qt QML based application and liked the outcome. When choosing an Android 9 / 10 device you can see that the "Screencast" setting is on; which is why I believe that this solution uses the Android Cast technology.
runthatapp.com. This is another commercial offer. Not as sophisticated (yet) as appetize.io, but providing a nice pay-as-you-go scheme.
ScreenStream. An open source Android app that provides a web server to view the screen of one Android device in a web browser, also relying on the Android Cast technology. That Android device could be an emulator running on a web server. And to make this multi-user capable you can employ a small load balancer similar to a technique that I developed for Qt WebGL streaming. The ScreenStream README shows that the application might consume up to 20 Mbit/s per client in short bursts.
Ideas for future improvements
Serving your Qt app as an interactive live video stream seems a promising idea to me, given that I found it already less sluggish than VNC and similar solutions. There are ways to make this even faster, such as using a hardware H.265 video encoder to create a video stream with very little delay. By operating multiple such encoders on a single server, the server could serve multiple clients and still keep its CPU load low. Maybe there are even better video formats for such a purpose, given that user interfaces of programs lend themselves well to lossless compression.
Some hints for appetize.io
Finally: since I used the appetize.io product for a Qt application over the last few days, here are some tips from that experience:
It is necessary to compile your Qt application for the x86 Android ABI. The default armeabi-v7a ABI will not work because most appetize.io devices are actually server-based Android emulators and the only ARM based device ("Nexus 5 Physical") failed to start any Qt application I tried to use with it.
The x86_64 ABI may also work, but you might then have to also compile Qt yourself for it, as not all versions of Qt come pre-compiled for that architecture.
All appetize.io links (both for standalone pages and embeddable iframes) support GET parameters to configure the app presentation format. Especially relevant here is screenOnly=true to show the app without a picture of a phone or tablet around it.
Features that rely on phone hardware (camera, position etc.) will not work or only show dummy data. But if you really wanted, you could create a hybrid application combined with client-side JavaScript. It would run device-dependent code in the user's browser, for example to take a photo with the webcam, and then provide the results to the Qt application via the appetize.io cross-document messaging protocol. The following message types seem suitable to build a simple communication protocol: pasteText(value), keypress(key, shiftKey) and openUrl(value).
In the default appetize.io standalone app demo pages, only the key events of ordinary letter keys are sent to the app, not keyboard shortcuts or function keys like F2 and Esc. This might be possible to fix with JavaScript on an own page embedding the appetize.io iframe, as their cross-document messaging protocol provides the keypress(key, shiftKey) message type.
Qt does not support writing browser based web applications. Unfortunately.
You need to use common web programming technologies for this. There are a lot of ways, but Qt is not one of them.
I am having a product which runs on Vxworks on end product hardware. But development is done in Visual studio using cross compiler and downloaded to hardware for testing. I am planning to write unit test cases for product. My question is because my development is done on windows and how can I run unit test cases as it is not resemebling real scenario?
Any inputs are welcome
I suspect that you have code that interacts a lot with VxWorks through system calls. Putting a layer of abstraction in there will be hard.
Are you using c or c++?
If you are using c++ and you can identify parts of the system that:
are subject to frequent change; and
are mostly handling internal data; or
just relating to a predefined/formalized subset of the surrounding system (e.g. protocol handling or individual PLC control logic modules).
Then you should first inject c++ interface(s) between the module and the rest of the system. This module should only relate to the interface(s)/adapter(s). Then you have an isolated piece that can be strapped into a visual studio test harness.
Then you should try to identify areas in your system that are error prone to bugs, subject to (frequent) change or audit. You will probably never achieve even 50% coverage on the target system, but you can achieve a system where 90% of the daily coding happens within the covered 40% of the code base.
Not possible without additional effort (new project; compile twice; for your host and let it run on your host) In this case search for an development environment that support your target for unit tests, for example http://www.parasoft.com/jsp/products/embedded_cpptest.jsp
Go on reading at parasoft C++ unit test question .
You do it just like any other system:
Write the unit tests
compile and load into target system
run unit tests
verify results
Where it runs is immaterial. The biggest difficulty is on an embedded system with limited output capability. But even if there is only a single LED, it should still be possible to signal success and failure. Only it is a little more abstract than showing "passed".
Coming from C++ & MFC background, is there any better (maintainability/customization) platform in developing application GUI ?
We are developing industrial applications (machine vision), where :
-Performance-critical (mostly image processing in CPU atm, but GPU is up next)
-Low level hardware interfacing (inhouse PCI device, frame grabber, motion card)
-Real-time data visualization (images / statistical graph)
-Future roadmap includes networkability for distributed processing and remote access.
Cross-platform will not be important for us since the system runs in controlled-environment (customer only cares whether the system runs and they got their output).
There are also concerns on migration cost (3rd party dependencies, training cost for developers and service personnel)
Edit
Clarification on the "image processing" mentioned above:
I'm referring to "picture" (2D information in matrix format) rather than graphic (commonly 3D vectorized). Currently we uses 3rd party imaging library (for spatial domain processing like segmentation, OCR/OCV, morphology, pattern match) and incorporate our result logic.
If you need performance-critical graphics processing, then C++/DirectX or C++/OpenGL are your best bets, hands down. C++/DirectX is arguably the more maintainable of the two.
That said... depending on the actual processing you're doing, you might consider moving portions of your UI to a more maintainable platform. The .NET framework / WPF can do some pretty amazing things, and with good implementation of patterns like MVVM and can be amazingly maintainable. Ditto the networking side; WCF abstracts a lot of common protocols away from the code, making for cleaner, more maintainable networking code. You can even write your translation layer between your unmanaged processing and your managed layer in C++/CLI.
That said, it's all very subjective. I can't tell enough from your bullet points to make a good judgement on whether or not you can offload some or even all of your processing to .NET/C#. It's worth considering, but my gut tells me that it's probably not your best bet.
As a fan of Qt I would be remiss not to mention it.
Although cross-platform is not one of your criteria, it is a nice bonus.
Qt also has good video hardware support through OpenGL (I'm not sure that it will help with capture hardware though).
It is open-source so you can get your hands as dirty as you like.
It is highly customizable.
It is actively developed and has a large community.
MFC programmers should not have much trouble coming up to speed.
You should also read through some of these questions and answers:
Good C++ GUI library for Windows
https://stackoverflow.com/questions/610/gui-programming-apis
What I had did before when developing a C++ scientific application is that, it will develop it completely under console based application. The console based application will able accept various type of command from user keyboard, and perform action accordingly. For example :
image_processor > load input.png
image_processor > save out.png
The good thing on this is, I can 100% in concentrating my algorithm design, without worry much on how to fit into the GUI framework. Either they are MFC or QT.
At the end of day, instead of taking input from keyboard input stream, I will just simply hook my console based application's STDIN, to the GUI application communication channel. My GUI application will then string based command, to talk and receive feedback from the console application.
Guess what I use to develop the GUI? Java Swing :)
I guess I am taking Unix people approach. See what Joel says :
Suppose you take a Unix programmer and a Windows programmer and give them each the task of creating the same end-user application. The Unix programmer will create a command-line or text-driven core and occasionally, as an afterthought, build a GUI which drives that core. This way the main operations of the application will be available to other programmers who can invoke the program on the command line and read the results as text. The Windows programmer will tend to start with a GUI, and occasionally, as an afterthought, add a scripting language which can automate the operation of the GUI interface.
I realize by taking Windows approach, you will end up with a more user friendly application. However, if your main concern is to get the sophisticated algorithm written well and GUI is the secondary, I would suggest that you go for Unix approach.
I'm looking for a free code profiler (performance profiler, not a memory profiler) which works for VSTO applications (specifically Excel in my case, but I gather that if it works for one, it should work for all).
I have tried Eqatec Profiler, but it can't seem to launch the application from the profiler (The Run app button is greyed out, which I'm gathering is because the VSTO application is compiled into a dll which runs in-process with Excel.exe).
If anyone has successfully used Eqatec or another free code profiler to profile a VSTO application, I'd be interested to hear how you did it.
What you're doing should be possible with EQATEC Profiler.
When you're simply instrumenting one or more DLLs then there is no distinct "application to run": you simply have some instrumented DLLs, but the profiler doesn't know in what way you intend to use them.
However, the instrumented assemblies are (along with the runtime-module, EQATEC.Profiler.RuntimeFullNet.dll) always completely "self-contained", meaning that they perform the timing themselves as soon as the methods within them are used. So you simply need to have your VSTO app load the DLL as usual and voila, the DLL will itself start accumulating timing information and start listening for a connection from the profiler with which you can then take snapshots etc.
So to summarize:
Build you DLL and keep the profiler running
Have your VSTO app load the DLL (however you do that)
The profiler and instrumented DLL should now be connected (see the Run-tab)
From within the profiler you can now take snapshots
I hope this helps. If not then let me know.
Have you tried Microsoft's CLR Profiler?
Check out How To: Use CLR Profiler at http://msdn.microsoft.com/en-us/library/ms979205.aspx.
Download CLR Profiler for the .NET Framework 2.0 is available at https://github.com/MicrosoftArchive/clrprofiler.
I have used it with great success many times.
Any good place to learn about POST and how to design and code one? I am a C++ programmer and quite baffeled with the term.
Thanks
You might want to take a look at the code for coreboot, a free software (open source) BIOS project that runs on many different kinds of hardware.
You can checkout the OpenBIOS project.
They have information on numberous opensource bios/firmware implementations.
Being open source you can grab the code from svn or read it online for all of them.
BIOS? That's not too common in the embedded world, the one place where people still write POSTs. Typically, they happen before the OS itself starts, or alternatively, as the OS starts.
The goal is to figure out whether the device can run, run in degraded mode, or should signal malfunction. A typical sequence is test CPU and XIP flash, then memory, fixed hardware, and then optional hardware. You define a series of tests. A test has a start function and a check function. The start functions kicks off the test; the check polls to see if a result is already available. Tests have dependencies, and the test controller starts those tests for which the dependencies have passed (CPU and RAM being the special cases, if they're broken it's not feasible to have a nice test controller).
As you can infer from the CPU and RAM tests, you don't have the luxury of C++. You can't even assume you can use all of C. During the first part of the POST, you might not even have a stack (!)
Open source EFI BIOS, with documentation and specs (good way to learn):
https://www.tianocore.org/
Background In June of 2004 Intel
announced that it would release the
"Foundation Code" of its next
generation firmware technology - a
successor to the PC BIOS - under an
open source license later in the year.
The Foundation Code, developed by
Intel as part of a project code named
Tiano, is Intel's "preferred
implementation" of the Extensible
Firmware Interface (EFI)
Specification. The code to be released
includes the core of the Foundation
Code as well as a driver development
kit. To follow through with its
intentions to release the code as open
source, Intel partnered with
Collabnet, an industry leader in
providing tools and services to
support an open source initiative, to
create a community for this effort.
The result of this partnership is this
Open Source Website.
Since there are more projects that are
EFI-based working in parallel with the
Foundation Code, it was decided to
release the EFI Shell Application and
the EFI Self Certification Test (SCT)
project to the open source community.
POST (Power On Self Test) is part of the Bios, and writing a POST, but not other parts of the BIOS, seems like an odd task indeed.
The documentation section of the processor manufacturer's web site would be a good start for BIOS programming. I remember writing an 80186 BIOS and POST a long time ago, and I worked exclusively with the Intel specs.
And btw, you will be doing this in Assembler, not C++.