I have an application that issues LOTS of command-line operations (e.g., at the "console"), from different threads. For this I'm using QProcess (Qt C++):
QProcess* p = new QProcess();
// ...maybe set QProcessEnvironment, set up stdout/stderr, etc....
p->start("cmd.exe");
p->write("dir\n");
p->closeWriteChannel();
p->waitForBytesWritten(-1/*forever*/);
p->waitForReadyRead(-1/*forever*/);
p->waitForFinished(-1/*forever*/);
// ...read all text from process
The above works fine. I could not get it to work properly under any permutation without QProcess::closeWriteChannel(), but I'd be interested if anyone is aware of an option (see below).
My understanding is that since I called QProcess::closeWriteChannel(), I can never re-open that channel. However, since I went through the work of instantiating the QProcess and setting the QProcessEnvironment, I could merely start the process again (which re-opens the write-channel).
// ...after previous run:
p->start("cmd.exe");
// ...use it again...
This works fine too.
So, I'm comparing TWO design options:
Instantiate QProcess each time. No re-use beyond a single command-line invocation. Simple.
Re-use QProcess. Do the work of "remembering" which QProcess instance has which QProcessEnvironment already set-up, so I can run commands in the appropriate environment. Faster (no re-instantiation of QProcess, no re-set-up of QProcessEnvironment()), but requires more "book-keeping".
QUESTION: How significant is the runtime overhead for instantiating QProcess and setting up the QProcessEnvironment?
Significant? I need to execute many thousands of command-line operations across many threads. If it is "big", that steers me towards (2) (do-able, I've figured out how). Otherwise, (1) is really simple, and I don't want to complicate things for no reason.
Yes, "Premature optimization is the root of all evil." I'm not trying to do that. This system needs to massively scale.
Thoughts?
QProcess* p = new QProcess();
// ...maybe set QProcessEnvironment, set up stdout/stderr, etc....
p->start("cmd.exe");
p->write("dir\n");
p->closeWriteChannel();
p->waitForBytesWritten(-1/*forever*/);
p->waitForReadyRead(-1/*forever*/);
p->waitForFinished(-1/*forever*/);
// ...read all text from process
The above works fine. I could not get it to work properly under any
permutation without QProcess::closeWriteChannel(), but I'd be
interested if anyone is aware of an option (see below).
Currently you are writing "\n" as to characters.
You need to write them as a single character.
p->write(QString("dir") + '\n');
instead of
p->write("dir\n");
p->closeWriteChannel();
With this method the write channel stays open and you can react should the process query for input.
Related
A post in this (Are system() calls evil?) thread says:
Your program's privileges are inherited by its spawned programs. If your application ever runs as a privileged user, all someone has to do is put their own program with the name of the thing you shell out too, and then can execute arbitrary code (this implies you should never run a program that uses system as root or setuid root).
But system("PAUSE") and system("CLS") shell to the OS, so how could a hacker possibly intervene if it ONLY shells to a specific secure location on the hard-drive?
Does explicitly flush—by using fflush or _flushall—or closing any stream before calling system eliminate all risk?
The system function passes command to the command interpreter, which executes the string as an operating-system command. system uses the COMSPEC and PATH environment variables to locate the command-interpreter file CMD.exe. If command is NULL, the function just checks whether the command interpreter exists.
You must explicitly flush—by using fflush or _flushall—or close any stream before you call system.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/system-wsystem
In case, there are any doubts here's the actual snippet from the MS' implementation (very simple and straightforward):
// omitted for brevity
argv[1] = _T("/c");
argv[2] = (_TSCHAR *) command;
argv[3] = NULL;
/* If there is a COMSPEC defined, try spawning the shell */
/* Do not try to spawn the null string */
if (argv[0])
{
// calls spawnve on value of COMSPEC vairable, if present
// omitted for brevity
}
/* No COMSPEC so set argv[0] to what COMSPEC should be. */
argv[0] = _T("cmd.exe");
/* Let the _spawnvpe routine do the path search and spawn. */
retval = (int)_tspawnvpe(_P_WAIT,argv[0],argv,NULL);
// clean-up part omitted
As to concerns of what _tspawnvpe may actually be doing, the answer is: nothing magical. The exact invocation sequence for spawnvpe and friends goes as following (as anybody with licensed version of MSVC can easily learn by inspecting the spanwnvpe.c source file):
Do some sanity checks on parameters
Try to invoke _tspawnve on the passed file name. spawnve will succeed if the file name represents an absolute path to an executable or a valid path relative to the current working directory. No further checks are done - so yes, if a file named cmd.exe exists in current directory it will be invoked first in the context of system() call discussed.
In a loop: obtain the next path element using `_getpath()
Append the file name to the path element
Pass the resulted path to spwanvpe, check if it was successful
That's it. No special tricks/checks involved.
The original question references POSIX not windows. Here there is no COMSPEC (there is SHELL but system() deliberately does not use it); however /bin/sh is completely, utterly vulnerable.
Suppose /opt/vuln/program does system("/bin/ls"); Looks completely harmless, right? Nope!
$ PATH=. IFS='/ ' /opt/vuln/program
This runs the program called bin in the current directory. Oops. Defending against this kind of thing is so difficult it should be left to the extreme experts, like the guys who wrote sudo. Sanitizing environment is extremely hard.
So you might be thinking what is that system() api for. I don't actually know why it was created, but if you wanted to do a feature like ftp has where !command is executed locally in the shell you could do ... else if (terminalline[0] == '!') system(terminalline+1); else ... Since it's going to be completely insecure anyway there's no point in making it secure. Of course a truly modern use case wouldn't do it that way because system() doesn't look at $SHELL but oh well.
The Qt documentation gives this explanation:
QProcess::start:
Starts the given program in a new process, if none is already running,
passing the command line arguments in arguments.
QProcess::startDetached:
Starts the program program with the arguments arguments in a new
process, and detaches from it.
What is the difference between the two? Is the difference only that you can start just one instance of a program using QProcess::start and many instances using QProcess::startDetached?
If you use start, termination of the caller process will cause the termination of the called process as well. If you use startDetached, after the caller is terminated, the child will continue to live. For example:
QProcess * p = new QProcess();
p->start("some-app");
delete p;// <---some-app will be terminated
QProcess * p = new QProcess();
p->startDetached("some-app");
delete p;// <---some-app will continue to live
The start() function is a member function, while startDetached is a static class function.
If you look at the documentation of QProcess, you'll see that there are functions to allow you to do things with the process such as: -
Receive the output or error streams from the running process (readAllStandardOutput / readAllStandardError)
Redirect the output to a file (setStandardOutputFile)
Use a file for standard input into the process (setStandardInputFile)
Communicate via channels
Get notified when the process has finished
These are just some of the things that you can only do with an instance of a QProcess. If, however, you want a simple and quick way of starting a process without having to create an instance and you don't need the extra functionality, you can simply call QProcess::startDetached.
Also, as the docs state for startDetached: -
If the calling process exits, the detached process will continue to live.
At the moment I'm using an eclipse-like IDE and the corresponding debug perspective, that most of you are probably familiar with. While debugging code I quite often find myself stepping through many lines of code and observing variables and double checking if everything is as it is supposed to be.
But suppose there is something like this:
1. important line, e.g. generating a new object;
2. another important line, e.g. some tricky class method;
3. for (int i = 0; i < some_limit; ++i)
4. some_array[i]++;
5. more important stuff;
Obviously I'm interested in what happens in lines 1,2 and 5 (I know this is a poor example, but please bear with me for a little while longer) but I don't want to step through all hundreds (or even thousands) of iterations of lines 3/4.
So, finally, my question: Is there some way to step directly over the for-cycle? What I do right now is set a new breakpoint at line 5 and let the program run as soon as I hit line 3 and I believe this is not an optimal solution.
edit: The eclipse implementation of what ks1322 proposed is called "Run to line" and is mapped to ctrl-r
Use until command instead of next.
From gdb documentation:
Continue running until a source line past the current line, in the
current stack frame, is reached. This command is used to avoid single
stepping through a loop more than once.
If you will use until instead of next, gdb will step over loops only once, which almost exactly what you want.
I've got a series of cpp source file and I want to write another program to JUDGE if they can run correctly (give input and compare their output with standart output) . so how to:
call/spawn another program, and give a file to be its standard input
limit the time and memory of the child process (maybe setrlimit thing? is there any examples?)
donot let the process to read/write any file
use a file to be its standard output
compare the output with the standard output.
I think the 2nd and 3rd are the core part of this prob. Is there any way to do this?
ps. system is Linux
To do this right, you probably want to spawn the child program with fork, not system.
This allows you to do a few things. First of all, you can set up some pipes to the parent process so the parent can supply the input to the child, and capture the output from the child to compare to the expected result.
Second, it will let you call seteuid (or one of its close relatives like setreuid) to set the child process to run under a (very) limited user account, to prevent it from writing to files. When fork returns in the parent, you'll want to call setrlimit to limit the child's CPU usage.
Just to be clear: rather than directing the child's output to a file, then comparing that to the expected output, I'd capture the child's output directly via a pipe to the parent. From there the parent can write the data to a file if desired, but can also compare the output directly to what's expected, without going through a file.
std::string command = "/bin/local/app < my_input.txt > my_output_file.txt 2> my_error_file.txt";
int rv = std::system( command.c_str() );
1) The system function from the STL allows you to execute a program (basically as if invoked from a shell). Note that this approach is inherenly insecure, so only use it in a trusted environment.
2) You will need to use threads to be able to achieve this. There are a number of thread libraries available for C++, but I cannot give you recommendation.
[After edit in OP's post]
3) This one is harder. You either have to write a wrapper that monitors read/write access to files or do some Linux/Unix privilege magic to prevent it from accessing files.
4) You can redirect the output of a program (that it thinks goes to the standard output) by adding > outFile.txt after the way you would normally invoke the program (see 1)) -- e.g. otherapp > out.txt
5) You could run diff on the saved file (from 3)) to the "golden standard"/expected output captured in another file. Or use some other method that better fits your need (for example you don't care about certain formatting as long as the "content" is there). -- This part is really dependent on your needs. diff does a basic comparing job well.
I am using Qsettings for non gui products to store its settings into xml files. This is written as a library which gets used in C, C++ programs. There will be 1 xml file file for each product. Each product might have more than one sub products and they are written into xml by subproduct grouping as follows -
File: "product1.xml"
<product1>
<subproduct1>
<settings1>..</settings1>
....
<settingsn>..</settingsn>
</subproduct1>
...
<subproductn>
<settings1>..</settings1>
....
<settingsn>..</settingsn>
</subproductn>
</product1>
File: productn.xml
<productn>
<subproduct1>
<settings1>..</settings1>
....
<settingsn>..</settingsn>
</subproduct1>
...
<subproductn>
<settings1>..</settings1>
....
<settingsn>..</settingsn>
</subproductn>
</productn>
The code in one process does the following -
settings = new QSettings("product1.xml", XmlFormat);
settings.setValue("settings1",<value>)
sleep(20);
settings.setValue("settings2", <value2>)
settings.sync();
When the first process goes to sleep, I start another process which does the following -
settings = new QSettings("product1.xml", XmlFormat);
settings.remove("settings1")
settings.setValue("settings3", <value3>)
settings.sync();
I would expect the settings1 to go away from product1.xml file but it still persist in the file - product1.xml at the end of above two process. I am not using QCoreApplication(..) in my settings library. Please point issues if there is anything wrong in the above design.
This is kind of an odd thing that you're doing, but one thing to note is that the sync() call is what actually writes the file to disk. In this case if you want your second process to actually see the changes you've made, then you'll need to call sync() before your second process accesses the file in order to guarantee that it will actually see your modifications. Thus I would try putting a settings.sync() call right before your sleep(20)
Maybe you have to do delete settings; after the sync() to make sure it is not open, then do the writing in the other process?
Does this compile? What implementation of XmlFormat are you using and which OS? There must be some special code in your project for storing / reading to and from Xml - there must be something in this code which works differently from what you expect.