I'm writing some rather complex unit tests in Go and I have something like:
err = os.Mkdir(tempDir, 0755)
assert.NoError(t, err, "error creating temporary directory")
defer func() {
log.Info("cleaning up temporary directory '%s'..", tempDir)
err = os.RemoveAll(tempDir)
assert.NoError(t, err, "error removing temporary directory")
}()
The problem is that this won't work and temporary dir won't be deleted if I hit Ctrl+C to interrupt the test for whatever the reason (it generates temporary data and it takes a lot of time, it can be beneficial to just SIGINT the process if I realize something is wrong).
Any clean way to do this?
Related
I am having problems using the sqlite3 C/C++ amalgamation within the context of Unreal Engine 4...
I can open a connection to the database write data to it, verifying this using DB Browser, but running SELECT statements using sqlite3_prepare_v2 and sqlite3_step is giving unexpected behaviours. I'm checking the return values and not seeing any error codes, but I am seeing a SQLITE_DONE when I'd expect to see SQLITE_ROW returned by sqlite3_step. If I log out the result of sqlite3_expanded_sql, I get the expected statement with all parameters properly inserted and if I copy & paste this into DB Browser, I get the expected results.
The data being pulled is specific to each run of the program so I know that I'm not seeing old or irrelevant data in DB Browser, and I know that my "write" functions are also operating as expected.
What's particularly strange is that after commenting out all references to SQLPreparedStatementString (which I was using in an attempt to debug the issue) the code performs differently: I do now get one row back (instead of the expected 4).
This almost makes me suspect that there's a memory issue going on, perhaps sqlite3 and Unreal are both writing to the same sections of memory without communicating that to each other? I tend to rely on UE4's memory management and don't have a lot of C/C++ experience outside of UE4, so my knowledge of things like malloc is limited pretty much just to the general feeling that "here be dragons" etc. I also understand that sqlite3_prepare_v2 and sqlite3_step are functions that tend to do a lot of memory allocation, but if that's what's causing the problem, I'm surprised that it's only occuring so rarely, since I'm successfully calling them several thousand times (to add data) on each game run.
I understand that it's possible to set compile-time options for sqlite that pre-allocate it a memory block so that it doesn't use malloc at all during runtime, but I've not quite worked out how to do that in the context of including the amalgamation in UE4 just yet.
Relevant code below:
TArray<FName> UOzyBrainLogger::GetLoggedCivsForTurn(const int32 TurnNumber)
{
TArray<FName> rVal;
sqlite3_stmt *SQLStatement;
FString SQLCommand = "SELECT DISTINCT CivName FROM Decision WHERE (GameID = ? AND TurnNumber = ?) ;";
int rc;
rc = sqlite3_prepare_v2(DB, TCHAR_TO_UTF8(*SQLCommand), -1, &SQLStatement, NULL);
if (rc == SQLITE_OK)
{
//FString SQLPreparedStatementString;
UE_LOG(OzyBrainLogger, Log, TEXT("Getting Civs logged for turn %d in game %s"), TurnNumber, *GameID);
if(sqlite3_bind_text( SQLStatement, 1, TCHAR_TO_UTF8(*GameID), -1, SQLITE_STATIC) == SQLITE_OK &&
sqlite3_bind_int( SQLStatement, 2, TurnNumber) == SQLITE_OK)
{
//SQLPreparedStatementString = UTF8_TO_TCHAR(sqlite3_expanded_sql(SQLStatement));
//UE_LOG(OzyBrainLogger, Log, TEXT("Successfully created statement: %s"), *SQLPreparedStatementString);
rc = sqlite3_step(SQLStatement);
if (rc == SQLITE_DONE)
{
UE_LOG(OzyBrainLogger, Log, TEXT("No results found?"));
}
else if (rc != SQLITE_ROW)
{
UE_LOG(OzyBrainLogger, Log, TEXT("Return code was: %d"), rc);
}
while (rc == SQLITE_ROW)
{
FName CivName = FName(*FString(UTF8_TO_TCHAR((const char*) sqlite3_column_text(SQLStatement, 0))));
//UE_LOG(OzyBrainLogger, Log, TEXT("Returning %s for turn %d via statement: %s"), *CivName.ToString(), TurnNumber, *SQLPreparedStatementString);
UE_LOG(OzyBrainLogger, Log, TEXT("Returning %s for turn %d"), *CivName.ToString(), TurnNumber);
rVal.Add(CivName);
rc = sqlite3_step(SQLStatement);
}
}
else
{
UE_LOG(OzyBrainLogger, Warning, TEXT("%s"), UTF8_TO_TCHAR(sqlite3_errmsg(DB)));
}
}
else
{
UE_LOG(OzyBrainLogger, Warning, TEXT("%s"), UTF8_TO_TCHAR(sqlite3_errmsg(DB)));
}
sqlite3_finalize(SQLStatement);
return rVal;
}
I might be missing something really obvious, but I'm not sure how to investigate this further to make progress so any advice on avenues to explore would be very helpful.
Am feeling a bit lost!
I think I managed to fix it! The problem seems to have been a combination of:
Calling sqlite3_open in my object's constructor expecting singleton-like behaviour when UE4 calls constructors silently from time to time (e.g. on compile).
Using SQLITE_STATIC in place of SQLITE_TRANSIENT
Adding return code (rc) checking absolutely everywhere
Fixing both of these issues seems to have resolved the main problem.
The final issue (getting only 2 rows back instead of 4) was a more pedestrian rookie error (I'd rearranged the call to sqlite3_step into a do ... while loop and accidentally ended up calling sqlite3_step twice per iteration!
The things that didn't fix the main issue included calling sqlite3_config with SQLITE_CONFIG_MEMSTATUS, SQLITE_CONFIG_LOOKASIDE and SQLITE_CONFIG_HEAP, however: when I started checking the return codes for these calls properly and finding errors, that was a major clue to realising that something had already started the database connection without me realising it (i.e. UE4's silent constructor-calling on compile)
This question already has answers here:
How to create a temporary directory in C++?
(6 answers)
Closed 3 years ago.
I'm trying to create a unique temporary directory in the system temp folder and have been reading about the security and file creation issues of tmpnam().
I've written the below code and was wondering whether it would satisfy these issues, is my use of the tmpnam() function correct and the throwing of the filesystem_error? Should I be adding checks for other things (e.g. temp_directory_path, which also throws an exception)?
// Unique temporary directory path in the system temporary directory path.
std::filesystem::path tmp_dir_path {std::filesystem::temp_directory_path() /= std::tmpnam(nullptr)};
// Attempt to create the directory.
if (std::filesystem::create_directories(tmp_dir_path)) {
// Directory successfully created.
return tmp_dir_path;
} else {
// Directory could not be created.
throw std::filesystem_error("directory could not be created.");
}
From cppreference.com:
Although the names generated by std::tmpnam are difficult to guess, it is possible that a file with that name is created by another process between the moment std::tmpnam returns and the moment this program attempts to use the returned name to create a file.
The problem is not with how you use it, but the fact that you do.
For example, in your code sample, if a malicious user successfully guesses and creates the directory right in between the first and second line, it might deny service (DOS) from your application, which might be critical, or not.
Instead, there is a way to do this without races on POSIX-compliant systems:
For files see mkstemp(3) for more information
For dirs see mkdtemp(3) for more information.
Your code is fine. Because you try to create the directory the OS will arbitrate between your process and another process trying to create the same file so, if you win, you own the file and if you lose you get an error.
I wrote a similar function recently. Whether you throw an exception or not depends on how you want to use this function. You could, for example simply return either an open or closed std::fstream and use std::fstream::is_open as a measure of success or return an empty pathname on failure.
Looking up std::filesystem::create_directories it will throw its own exception if you don't supply a std::error_code parameter so you don't need to throw your own exception:
std::filesystem::path tmp_dir_path {std::filesystem::temp_directory_path() /= std::tmpnam(nullptr)};
// Attempt to create the directory.
std::filesystem::create_directories(tmp_dir_path));
// If that failed an exception will have been thrown
// so no need to check or throw your own
// Directory successfully created.
return tmp_dir_path;
I have some test cases and they will all write some string to 'tmp.txt' and read it back. And the tests will fail. If I only test one test case at a time, or change the temp file name from 'tmp.txt' to some random generated filenames, the tests will pass. What is the possible reason for that? Are test cases of googletest running sequentially?
TEST(xxxx) {
string s = generate_some_string();
string filename = "tmp.txt";
ofstream tmpFile(filename);
tmpFile << s;
tmpFile.close();
cv::VideoCapture cap(filename);
// read the content back using cap, and do some check
remove(filename.c_str());
}
By default googletest should run the tests in sequence; though there is a special testrunner that runs tests in parallel, and the test plugin for visual studio too can be configured to run tests in parallel.
My suspicion would rather lie on the cv::VideoCapture cap(filename); call - can you verify that the file is indeed closed when the call returns, or at the very latest when you call remove(filename.c_str());?
I am just starting learning golang and am not sure if my error is conceptual or with the language.
It is peculiar, because the error only occurs, when unit-testing my code. If I "go run" everything works fine. As a sqlite-driver I use mattn/go-sqlite3.
Here is where the problem occurs:
func dbExec(command *string) {
db, err := sql.Open("sqlite3", dbPath) // Path and driver are set correcrtly
defer db.Close()
if err != nil { //No problem here
panic(err)
}
_, err = db.Exec(*command)
if err != nil { //Here the errorNo14 "Unable to open the database file" occurs
panic(err)
}
}
So it seems to me, that the database file can be found, but it can't be open because of other limits. I have no concurrency in all of my code. I have only one test-method so far, so even if the testing would be done concurrently, it would so far be only one thread. Maybe someone has an idea! The problem seems to basic, there really is not much more to my code then this.
sql.Open usually does not perform any database operations, it merely initializes the driver (see sql.Open docs for details). The driver's Open() method is called in the background.
This means that the first error is usually encountered on trying to perform an operation, in your case: the db.Exec.
As for the actual error: if this is in tests it is likely that your path is not correct (eg: if the test database is in the test's sub-directory, the test runner's current directory will be different, I think). Try to switch to absolute paths (or paths computed at runtime).
And a small nit: move defer db.Close() after the first error check.
I am working with HANDLES, the first one, nextColorFrameEvent is an event handler and the second one is a stream handler. They are being initialized in the following piece of code:
nextColorFrameEvent = CreateEvent( NULL, TRUE, FALSE, NULL );
hr = nui->NuiImageStreamOpen(
NUI_IMAGE_TYPE_COLOR,
NUI_IMAGE_RESOLUTION_640x480,
0,
2,
nextColorFrameEvent,
&videoStreamHandle);
I want to properly deal with them on destruction, while not creating errors at the same time. Sometimes the initializer wont be called, so both HANDLEs are still NULL when the software comes to an end. Thats why I want to check first if the HANDLEs are properly initialized etc. and if they are, I want to close them. I got my hands on the following piece of code for this:
if (nextColorFrameEvent && nextColorFrameEvent != INVALID_HANDLE_VALUE)CloseHandle(nextColorFrameEvent);
#ifdef QT_DEBUG
DWORD error = GetLastError();
qDebug()<< error;
#endif
if (videoStreamHandle && videoStreamHandle != INVALID_HANDLE_VALUE)CloseHandle(videoStreamHandle);
#ifdef QT_DEBUG
error = GetLastError();
qDebug()<< error;
#endif
But this is apperently incorrect: if I do not run the initializer and then close the software this piece of code runs and gives me a 6:
Starting C:\...\Qt\build-simpleKinectController-Desktop_Qt_5_0_2_MSVC2012_64bit-Debug\debug\simpleKinectController...
6
6
C:\...\Qt\build-simpleKinectController-Desktop_Qt_5_0_2_MSVC2012_64bit-Debug\debug\simpleKinectController exited with code 0
which means:
ERROR_INVALID_HANDLE 6 (0x6) The handle is invalid.
Which means that closeHandle ran anyway despite the tests. What tests should I do to prevent closing when the handle is not a valid HANDLE?
Bonus question: if I run the initializer this error will no longer appear when only closing colorFrameEvent, but will still appear when closing videoStreamHandle:
Starting C:\...\Qt\build-simpleKinectController-Desktop_Qt_5_0_2_MSVC2012_64bit-Debug\debug\simpleKinectController...
0
6
C:\...\Qt\build-simpleKinectController-Desktop_Qt_5_0_2_MSVC2012_64bit-Debug\debug\simpleKinectController exited with code 0
Do I need a diffent function to close a stream handler?
nui->NuiImageStreamOpen(...) does not create a valid Windows handle for the stream but instead it creates an internal handle inside the driver side.
So you can not use windows API to release/close stream handle !!!
To do that just call nui->NuiShutdown(). I have not yet used the callback event but I think its a valid windows handle and should be closed normally.
if you need just to change settings you can always call nui->NuiImageStreamOpen(...) with new settings. No need to shutdown ...
I would also welcome function nui->NuiImageStreamClose(...); because current state of API complicates things for long term running aps with changing sensor configurations.
CreateEvent (http://msdn.microsoft.com/en-us/library/windows/desktop/ms682396(v=vs.85).aspx) returns NULL if an event was not created.
You are checking against INVALID_HANDLE_VALID which is not NULL.
You are probably trying to double-close a handle. That is likely to generate ERROR_INVALID_HANDLE 6. You can't detect this with your test, because the first CloseHandle(nextColorFrameEvent); did not change nextColorFrameEvent.
The solution is to use C++ techniques, in particular RAII. There are plenty of examples around how to use shared_ptr with HANDLE. shared_ptr is the standard solution to run cleanup code at most once, after everyone is done, and only if anybody actually allocated a resource.
there is a good way of debugging that I'm particularly fond of, despite being all writen in macros, which are nasty, but in this case they work wonders.
Zed's Awesome Debug Macros
there are a couple of things I like to change though. They make extensive use of goto, which I tend to avoid, specially in c++ projects, because otherwise you wouldn't be able to declare variables mid-code. This is why I use exit(-1) instead, or, in some projects, I mod the code to try, throw, catch c++. Since you are working with Handles, a good thing would be setting a variable and telling the program to close itself.
here is what I mean. Take this piece of code from the macros (i assume you would read the exercise and familiarize with the macros):
#define check(A, M, ...) if(!(A)) { log_err(M, ##__VA_ARGS__); errno=0; goto error; }
i'd change
goto error;
to something like
error = true;
the syntax inside the program would be something like, and I took it from a multithread program I'm writing myself:
pRSemaphore = CreateSemaphore(NULL, 0, MAX_TAM_ARQ, (LPCWSTR) "p_read_semaphore");
check(pRSemaphore, "Impossible to create semaphore: %d\n", GetLastError());
As you can see, GetLastError is only called when pRSemaphore is set to null. There are somewhat fancy mechanisms behind the macro (at least they are fancy for me), but they are hidden inside the "check" mask, so you needn't worry about them.
next step is to treat the error with something like:
inline void ExitWithError(bool &err) {
//close all handles
//tell other related process to do the same if necessary
exit(-1);
}
or you could just call it inside the macro like
#define check(A, M, ...) if(!(A)) { log_err(M, ##__VA_ARGS__); errno=0; ExitWithError(); }
hope I could be of any help