libcurl FTP-Download incomplete in the Unreal Engine - c++

Question

I am currently trying to download a File from a public git-Repository using curl in my Unreal C++ Project. Here is the code I'm trying to execute that I derived from the FTP-Example:
// This is in the .h file
struct FFtpFile {
FILE* File;
const char* Filename;
};
void FtpFetch(const std::string URL, const char* Filename) {
CURL* Curl = curl_easy_init();
const FFtpFile FtpFile {
nullptr,
Filename
};
if (!Curl) {
UE_LOG(LogTemp, Warning, TEXT("Error Initiating cURL"));
return;
}
curl_easy_setopt(Curl, CURLOPT_URL, URL.c_str());
curl_easy_setopt(Curl, CURLOPT_VERBOSE, 1);
curl_easy_setopt(Curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(Curl, CURLOPT_SSL_VERIFYPEER, false);
// Data Callback
const auto WriteCallback = +[](void* Contents, const size_t Size, const size_t NumMem, FFtpFile* FileStruct) -> size_t {
if (!FileStruct->File) {
fopen_s(&FileStruct->File, FileStruct->Filename, "wb");
if (!FileStruct->File) {
return CURLE_WRITE_ERROR;
}
}
return fwrite(Contents, Size, NumMem, FileStruct->File);
};
curl_easy_setopt(Curl, CURLOPT_WRITEFUNCTION, DownloadCallback);
curl_easy_setopt(Curl, CURLOPT_WRITEDATA, FtpFile);
const CURLcode Result = curl_easy_perform(Curl);
if (Result != CURLE_OK) {
const FString Message(curl_easy_strerror(Result));
UE_LOG(LogTemp, Warning, TEXT("Error Getting Content of the Model File: %s"), *Message);
return;
}
curl_easy_cleanup(Curl);
// Close the Stream after Cleanup
UE_LOG(LogTemp, Log, TEXT("Successfully Fetched FTP-File. Closing Write Stream"))
if (FtpFile.File) {
fclose(FtpFile.File);
}
}
Note that this is executed on a separate Thread using the Unreal Async function:
void AsyncFetchModelFile(const std::string URL) {
std::string Path = ...
TFunction<void()> Task = [Path, URL]() {
FtpFetch(URL, Path.c_str());
};
UE_LOG(LogTemp, Log, TEXT("Fetching FTP on Background Thread"))
Async(EAsyncExecution::Thread, Task, [](){UE_LOG(LogTemp, Warning, TEXT("Finishied FTP on Background Thread!"))});
}
I already removed the curl_global calls, as the documentation states those are not thread-safe. I also tried running the code on the main thread, but the same error happens there too.
To the error itself: The download runs almost flawlessly, but the downloaded file (in this case a .fbx file) always misses the last ~800 Bytes and is therefore incomplete. Also, the file keeps being open in Unreal, so I can't delete/move the file unless I close the Editor.
Before writing this Unreal code I tried running the same code in a pure C++ setting and there it worked flawlessly. But for some reason doing the same in Unreal doesn't work.
I also tried using a private method instead of the lambda-Function, but that didn't make any difference.
Any help would be appreciated
~Okaghana

Answer 1

I didn't manage to get it working in the end (I suspect it's an Unreal-Related Bug), but I found another way using the included Unreal HTTP Module:
FString Path = ...
// Create the Callback when the HTTP-Request has finished
auto OnRequestComplete = [Path](FHttpRequestPtr Request, FHttpResponsePtr Response, bool bWasSuccessful) {
if (bWasSuccessful) {
FFileHelper::SaveArrayToFile(Response->GetContent(), *Path);
UE_LOG(LogTemp, Log, TEXT("Successfully downloaded the file to '%s'"), *Path)
} else {
UE_LOG(LogTemp, Warning, TEXT("Error downloading the file (See EHttpResponseCodes): %s"), Response->GetResponseCode())
}
};
// Create a HTTP-Request and Fetch the file
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> Request = FHttpModule::Get().CreateRequest();
Request->SetVerb("GET");
Request->SetURL(URL);
Request->OnProcessRequestComplete().BindLambda(OnRequestComplete);
Request->ProcessRequest();