i have an application that queries a specific folder for its contents with a quick interval. Up to the moment i was using FindFirstFile but even by applying search pattern i feel there will be performance problems in the future since the folder can get pretty big; in fact it's not in my hand to restrict it at all.
Then i decided to give FindFirstFileEx a chance, in combination with some tips from this question.
My exact call is the following:
const char* search_path = "somepath/*.*";
WIN32_FIND_DATA fd;
HANDLE hFind = ::FindFirstFileEx(search_path, FindExInfoBasic, &fd, FindExSearchNameMatch, NULL, FIND_FIRST_EX_LARGE_FETCH);
Now i get pretty good performance but what about compatibility? My application requires Windows Vista+ but given the following, regarding FIND_FIRST_EX_LARGE_FETCH:
This value is not supported until Windows Server 2008 R2 and Windows 7.
I can pretty much compile it on my Windows 7 but what will happens if someone runs this on a Vista machine? Does the function downgrade to a 0
(default) in this case? It's safe to not test against operating system?
UPDATE:
I said above about feeling the performance is not good. In fact my numbers are following on a fixed set of files (about 100 of them):
FindFirstFile -> 22 ms
FindFirstFile -> 4 ms (using specific pattern; however all files may wanted)
FindFirstFileEx -> 1 ms (no matter patterns or full list)
What i feel about is what will happen if folder grows say 50k files? that's about x500 bigger and still not big enough. This is about 11 seconds for an application querying on 25 fps (it's graphical)
Just tested under WinXP (compiled under win7). You'll get 0x57 (The parameter is incorrect) when it calls ::FindFirstFileEx() with FIND_FIRST_EX_LARGE_FETCH. You should check windows version and dynamically choose the value of additional parameter.
Also FindExInfoBasic is not supported before Windows Server 2008 R2 and Windows 7. You'll also get run-time 0x57 error due to this value. It must be changed to another alternative if binary is run under old windows version.
at first periodic queries a specific folder for its contents with a quick interval - not the best solution I think.
you need call ReadDirectoryChangesW - as result you will be not need do periodic queries but got notifies when files changed in directory. the best way bind directory handle with BindIoCompletionCallback or CreateThreadpoolIo and call first time direct call ReadDirectoryChangesW. then when will be changes - you callback will be automatic called and after you process data - call ReadDirectoryChangesW again from callback - until you got STATUS_NOTIFY_CLEANUP (in case BindIoCompletionCallback) or ERROR_NOTIFY_CLEANUP (in case CreateThreadpoolIo) in callback (this mean you close directory handle for stop notify) or some error.
after this (first call to ReadDirectoryChangesW ) you need call FindFirstFileEx/FindNextFile loop but only once - and handle all returned files as FILE_ACTION_ADDED notify
and about performance and compatibility.
all this is only as information. not recommended to use or not use
if you need this look to - ZwQueryDirectoryFile - this give you very big win performance
you only once need open File handle, but not every time like with
FindFirstFileEx
but main - look to ReturnSingleEntry parameter. this is key
point - you need set it to FALSE and pass large enough buffer to
FileInformation. if set ReturnSingleEntry to TRUE function
and return only one file per call. so if folder containing N files -
you will be need call ZwQueryDirectoryFile N times. but with
ReturnSingleEntry == FALSE you can got all files in single call, if buffer will be large enough. in all case you serious reduce
the number of round trips to the kernel, which is very costly
operation . 1 query with N files returned much more faster than N
queries. the FIND_FIRST_EX_LARGE_FETCH and do this - set
ReturnSingleEntry to TRUE - but in current implementation (i check this on latest win 10) system do this only in
FindNextFile calls, but in first call to
FindFirstFileEx it (by unknown reason) still use
ReturnSingleEntry == TRUE - so will be how minimum 2 calls to the ZwQueryDirectoryFile, when possible have single call
(if buffer will be large enough of course) and if direct use
ZwQueryDirectoryFile - you control buffer size. you can
allocate once say 1MB for buffer, and then use it in periodic
queries. (without reallocation). how large buffer use
FindFirstFileEx with FIND_FIRST_EX_LARGE_FETCH - you can
not control (in current implementation this is 64kb - quite
reasonable value)
you have much more reach choice for FileInformationClass - less
informative info class - less data size, faster function worked.
about compatibility? this exist and worked from how minimum win2000 to latest win10 with all functional. (in documentation - "Available starting with Windows XP", however in ntifs.h it declared as #if (NTDDI_VERSION >= NTDDI_WIN2K) and it really was already in win2000 - but no matter- XP support more than enough now)
but this is undocumented, unsupported, only for kernel mode, no lib file.. ?
documented - as you can see, this is both for user and kernel mode - how you think - how is FindFirstFile[Ex] / FindNextFile - working ? it call ZwQueryDirectoryFile - no another way. all calls to kernel only through ntdll.dll - this is fundamental. ( yes still possible that ntdll.dll stop export by name and begin export by ordinal only for show what is unsupported really was). lib file exist, even two ntdll.lib and ntdllp.lib (here more api compare first) in any WDK. headers, where declared ? #include <ntifs.h>. but it conflict with #include <windows.h> - yes conflict, but if include ntifs.h in namespace with some tricks - possible avoid conflicts
Related
This might be a general question, but this problem is really getting me confused.
I have two different C++ applications, compiled with Visual Studio 2012, needing an instance of the same object. I have put a breakpoint before the creation of each object to measure the RAM usage by stepping my programs. The first one takes approximately 2.5 MiB more RAM after creating the object, while the second app is taking 30 MiB!
Both objects are created using a simple call to new with the same parameters. The code behind the constructors is the same.
As a detail: the first project contains much fewer .cpp files than the second one. So I thought it might be a problem of internal fragmentation of the exe. Plus, I've also tried to break BEFORE any code was executed inside the main function, and the memory usage was already much different (6 MiB for the first app, 35 MiB for the second one).
Does anyone have any idea what might be going on?
EDIT : The said object is a DirectX context, with a constructor creating a Direct3D instance and a device. Both the instance AND device are created the same way, but both have different RAM usage between the two apps.
Here is the code for the D3D device creation :
d3d_presentParams = new D3DPRESENT_PARAMETERS; ZeroMemory( d3d_presentParams, sizeof(D3DPRESENT_PARAMETERS) );
d3d_presentParams->Windowed = !window->isFullscreen();
d3d_presentParams->SwapEffect = D3DSWAPEFFECT_DISCARD;
d3d_presentParams->hDeviceWindow = window->getHwnd();
d3d_presentParams->MultiSampleType = antiAlias;
d3d_presentParams->EnableAutoDepthStencil = true;
d3d_presentParams->AutoDepthStencilFormat = D3DFMT_D32F_LOCKABLE;
d3d_presentParams->PresentationInterval = (info.m_vsync ? D3DPRESENT_INTERVAL_ONE : D3DPRESENT_INTERVAL_IMMEDIATE);
{
d3d_presentParams->BackBufferCount = 1;
d3d_presentParams->BackBufferFormat = D3DFMT_A8R8G8B8;
d3d_presentParams->BackBufferWidth = m_viewportSize.x;
d3d_presentParams->BackBufferHeight = m_viewportSize.y;
}
d3d->CreateDevice(
D3DADAPTER_DEFAULT,
D3DDEVTYPE_HAL,
window->getHwnd(),
D3DCREATE_HARDWARE_VERTEXPROCESSING | D3DCREATE_MULTITHREADED,
d3d_presentParams,
&d3d_device);
EDIT 2 : Problem has been "solved" for now. See the answer below.
Alright, I "solved" it, kind of. You're not going to believe what the problem was.
TL;DR : Renaming the executable lowers the RAM usage from 80 MiB to 28 MiB. It seems that Windows is suspicious of non-verified applications, as I discovered a directory full of logs inside C:/Users/<Me>/AppVerifierLogs/.
It seemed that RAM usage was linked to how many sources files it contained. Thus, I tried copying one of my projects entirely. I had to rename it, because a solution cannot have two projects with the same name... And guess what, memory usage was down to 7.3 MiB at the beginning of main(), instead of 30 MiB previously.
Later, I found out that it was a specific executable name that caused it to use more memory. I simply renamed the .exe file, and the problem was gone. I even tried to rename Firefox's exe by my app's name, and it made it crash.
And finally, I discovered in C:/Users/<me>/AppVerifierLogs/ at least 3000 files named after my application. This immediately made me think about the way Windows tries to verify applications (e.g. in order to see if they're not viruses).
I have no idea where this is coming from, nor how to turn off this verfier, but it's really annoying as a dev to have Windows panicking about an application you're developping, and injecting (possibly useless) stuff inside it.
The fix for now is just to rename the executable. If anyone knows about which application might have access to this "AppVerifierLogs" directory, please mention it, because even Google searches dont't seem to be able to help at this point...
I have a code written in Atmel Studio to read/write data from a SD card. I am using FatFs here. My problem is the code doesn't compile when I use some of the functions (f_chdir, f_getcwd...) in FatFs. Some functions works fine (f_puts, f_open, f_mount, f_mkdir...). All these functions are located in same header file (ff.h, ff.c)
The error says "undefined reference to -function-, ld returned 1 exit status". When I go to the error it shows the end of the code while it is suppose to show where the error is.
I cannot understand the problem with my code. Any help is appreciated.
Just ran into this using the SD card for a project using SAM4S Xplained Pro (Atmel 7, ASF 3.20).
Make sure you have all the asf projects (fatfs, sd_mmc, memory access control, and then the other basics e.g. pmc, gpio, and maybe a few more). My asf did NOT include sd_mmc_mem.c and sd_mmc_mem.h for some reason, so I had to include those myself. Also remember to do the sd_mmc_init() at the top of your main loop. As for the configuration...
If you look closely at ffconf.h, the first thing it does is include conf_fatfs.h, which (wait for it!) is EXACTLY the same file line by line as ffconf.h. All the variables are defined there first and foremost (and guarded by an #ifndef FFCONF and NOT an CONF_FATFS) aka that's where it counts..
Go into conf_fatfs.h and change _USE_STRFUNC to 1 or 2 et voila.
Also note that in the places where you use it, you'll have to #include the ff.h preceded by either ffconf.h or conf_access.h
ASF is a real snake pit if you don't know what you're looking for.
Enjoy.
By default, the memory control access interface is disabled in the ASF wizard. In order to enable the memory control access, please follow the steps below.
Open the ASF wizard (Alt + W).
Enable the Memory Control Access as follows.
ASF SD sd_mmc_mem.h memory access enable
Finally, click the “Apply” option to make the changes.
This adds sd_mmc_mem.h /.c files
Open the ffconf.h in your favorite editor and set _FS_RPATH to 2. From ffconf.h:
#define _FS_RPATH 0
/* This option configures relative path feature.
/
/ 0: Disable relative path feature and remove related functions.
/ 1: Enable relative path feature. f_chdir() and f_chdrive() are available.
/ 2: f_getcwd() function is available in addition to 1.
/
/ Note that directory items read via f_readdir() are affected by this option. */
Which features of the fatfs library are included in your build is configurable, so that you don't have to lose valuable ROM space (as well as a few bytes of RAM) for functions you're not using.
For versions of the FatFS library prior to 0.8a, _FS_RPATH supports only values 0 and 1; f_getcwd is not available in these versions.
Additionally, in versions prior to 0.8, it is necessary for C++ code to explicitly include its headers as C headers to avoid name mangling:
extern "C" {
#include "ff.h"
}
From version 0.8 onwards, it does this internally. You can find the new versions here if you're still working with an old one -- the comment you left leads me to believe that this might be the case.
Check if _FS_MINIMZE in ffconf.h is 0 to have all functions available.
In my version that I downloaded from elm-chan it was by default set to 3 and lead to the compiler error: undefined reference.
In file ffconf.h, set #define _USE_FIND to 1.
/* This option switches filtered directory read functions, f_findfirst() and
/ f_findnext(). (0:Disable, 1:Enable 2:Enable with matching altname[] too) */
I needed to use f_findfirst() and f_findnext() functions and i was getting undefined reference errors.
Now this solved my problem.
Drive/directory handling functions are under #if _FS_RPATH >= 1 (or similar preprocessors) .
I started to mess with Ypsilon, which is a C++ implementation of Scheme.
It conforms R6RS, features fast garbage collector, supports multi-core CPUs and Unicode but has a LACK of documentation, C++ code examples and comments in the code!
Authors provide it as a standalone console application.
My goal is to use it as a scripting engine in an image processing application.
The source code is well structured, but the structure is unfamiliar.
I spent two weeks penetrating it, and here's what I've found out:
All communication with outer world is done via C++ structures called
ports, they correspond to Scheme ports.
Virtual machine has 3 ports: IN, OUT and ERROR.
Ports can be std-ports (via console), socket-ports,
bytevector-ports, named-file-ports and custom-ports.
Each custom port must provide a filled structure called handlers.
Handlers is a vector containing 6 elements: 1st one is a boolean
(whether
port is textual), and other five are function pointers (onRead, onWrite, onSetPos, onGetPos, onClose).
As far as I understand, I need to implement 3 custom ports (IN, OUT and ERROR).
But for now I can't figure out, what are the input parameters of each function (onRead, onWrite, onSetPos, onGetPos, onClose) in handlers.
Unfortunately, there is neither example of implementing a custom port no example of following stuff:
C++ to Scheme function bindings (provided examples are a bunch of
.scm-files, still unclear what to do on the C++ side).
Compiling and
running bytecode (via bytevector-ports? But how to compile text to
bytecode?).
Summarizing, if anyone provides a C++ example of any scenario mentioned above, it would significantly save my time.
Thanks in advance!
Okay, from what I can read of the source code, here's how the various handlers get called (this is all unofficial, based purely on source code inspection):
Read handler: (lambda (bv off len)): takes a bytevector (which your handler will put the read data into), an offset (fixnum), and a length (fixnum). You should read in up to len bytes, placing those bytes into bv starting at off. Return the number of bytes actually read in (as a fixnum).
Write handler: (lambda (bv off len)): takes a bytevector (which contains the data to write), an offset (fixnum), and a length (fixnum). Grab up to len bytes from bv, starting at off, and write them out. Return the number of bytes actually written (as a fixnum).
Get position handler: (lambda (pos)) (called in text mode only): Allows you to store some data for pos so that a future call to the set position handler with the same pos value will reset the position back to the current position. Return value ignored.
Set position handler: (lambda (pos)): Move the current position to the value of pos. Return value ignored.
Close handler: (lambda ()): Close the port. Return value ignored.
To answer another question you had, about compiling and running "bytecode":
To compile an expression, use compile. This returns a code object.
There is no publicly-exported approach to run this code object. Internally, the code uses run-vmi, but you can't access this from outside code.
Internally, the only place where compiled code is loaded and used is in its auto-compile-cache system.
Have a look at heap/boot/eval.scm for details. (Again, this is not an official response, but based purely on personal experimentation and source code inspection.)
I've got a series of cpp source file and I want to write another program to JUDGE if they can run correctly (give input and compare their output with standart output) . so how to:
call/spawn another program, and give a file to be its standard input
limit the time and memory of the child process (maybe setrlimit thing? is there any examples?)
donot let the process to read/write any file
use a file to be its standard output
compare the output with the standard output.
I think the 2nd and 3rd are the core part of this prob. Is there any way to do this?
ps. system is Linux
To do this right, you probably want to spawn the child program with fork, not system.
This allows you to do a few things. First of all, you can set up some pipes to the parent process so the parent can supply the input to the child, and capture the output from the child to compare to the expected result.
Second, it will let you call seteuid (or one of its close relatives like setreuid) to set the child process to run under a (very) limited user account, to prevent it from writing to files. When fork returns in the parent, you'll want to call setrlimit to limit the child's CPU usage.
Just to be clear: rather than directing the child's output to a file, then comparing that to the expected output, I'd capture the child's output directly via a pipe to the parent. From there the parent can write the data to a file if desired, but can also compare the output directly to what's expected, without going through a file.
std::string command = "/bin/local/app < my_input.txt > my_output_file.txt 2> my_error_file.txt";
int rv = std::system( command.c_str() );
1) The system function from the STL allows you to execute a program (basically as if invoked from a shell). Note that this approach is inherenly insecure, so only use it in a trusted environment.
2) You will need to use threads to be able to achieve this. There are a number of thread libraries available for C++, but I cannot give you recommendation.
[After edit in OP's post]
3) This one is harder. You either have to write a wrapper that monitors read/write access to files or do some Linux/Unix privilege magic to prevent it from accessing files.
4) You can redirect the output of a program (that it thinks goes to the standard output) by adding > outFile.txt after the way you would normally invoke the program (see 1)) -- e.g. otherapp > out.txt
5) You could run diff on the saved file (from 3)) to the "golden standard"/expected output captured in another file. Or use some other method that better fits your need (for example you don't care about certain formatting as long as the "content" is there). -- This part is really dependent on your needs. diff does a basic comparing job well.
I'm having serious trouble with the function stat(). I have an application compiled under cygwin ond Windows 7 and the same app compiled with MSVC++ on Windows 7. The app contains the following code:
struct stat stb;
memset( &stb, 0, sizeof( stb ) );
stat( szPath, &stb );
cout << hex << uppercase << setw(8) << stb.st_mtime << endl;
szPath is a path to a file. The file does not get modified in any way by the app. The problem is, that i get different results for some files. For example:
cygwin version: 40216D72
MSVC++ version: 40217B82
The difference is always E10 = 3600 = 1 hour
By using google, i found this, which seems to be exactly the same issue i'm seeing. Is there a portable way how to fix this? I cannot use any WinAPI calls. The most simple and reliable solution is what i'm looking for, but if it needs to be complicated, so be it. Reliability and portability (win + linux) is the most important thing here.
To obtain both reliability and portability here (or in most situations of this sort where two platforms do different things with what should be the "same" code), you will probably need to use some form of target-dependent code, like:
#ifdef _MSC_VER
// do MSVC++-specific code
#else
// do Linux/Cygwin/generic code
#endif
You then ought to be able to use WinAPI calls in the _MSC_VER section, because that will only be compiled when you're using MSVC++
Apparently per http://support.microsoft.com/kb/190315 this is actually a FEATURE although it really seems like a bug to me. They say you can work around it by clearing "Automatically adjust clock for daylight saving changes" in the Date/Time Properties dialog box for the system clock.
If you have the date of the file, you can use the relative state of dst to determine if you need to make a one-hour adjustment yourself in MSVC only, but that's hacky as well.
Not sure about the functions you are using but I do know that Windows and Linux use the system clock differently. Windows stores local time (including DST) on the system clock. Linux (at least traditionally) stores GMT (or UTC to be precise) on the system clock. I don't if this applies to cygwin.
If a linux system shares hardware with windows it needs to be configured to use the system clock like windows or get messed-up every time windows adjusts DST.