I am using ecCodes library in my project, and I have encountered an issue that memory is not freed between reading the files.
The minimal example representing the problem is this (and is basically a combination of those two library API usage examples [1](https://confluence.ecmwf.int/display/ECC/grib_get_keys) [2]:
#include <string>
#include <vector>
#include <iostream>
#include "eccodes.h"
int main() {
std::string filenames[] = {"../data/era5_model.grib", "../data/era5_model2.grib", "../data/era5_model3.grib",
"../data/era5_model4.grib"};
std::vector<long> vec = {};
for (auto & filename : filenames) {
FILE* f = fopen(filename.c_str(), "r");
int err = 0;
codes_handle* h;
while ((h = codes_handle_new_from_file(nullptr, f, PRODUCT_GRIB, &err)) != nullptr) {
long k1 = 0;
err = codes_get_long(h, "level", &k1);
vec.push_back(k1);
}
codes_handle_delete(h);
fclose(f);
}
std::cout << vec[52];
return 0;
}
In the example the program reads 4 identical ERA5 files, each of size 1.5GB. Before opening new file previous one is closed with codes_handle_delete() and fclose().
Therefore, the expected behaviour would be for the memory usage to stay at about 1.5GB. However, in reality the memory usage steadily increases to about 6.5GB and is freed when program closes (see screenshot below).
This particular example has been run on CLion with CMake (Release configuration), but the issue occurs with every other configuration and also in my other Rust project which calls ecCodes with FFI.
The library seems well tested and supported so it seems unlikely that it is a library bug. Therefore, is that an expected behaviour or is my code wrong? If the latter, how can I correct it?
I am using Ubuntu 21.04 and ecCodes 2.20.0 installed with apt
So I contacted the library authors and realized that I have not read this example carefully enough.
For the ecCodes to correctly free the memory codes_handle should be deleted every time it is created (analogically to how you should free the memory every time you alloc it). Therefore in my example codes_handle_delete() should be INSIDE the while loop:
while ((h = codes_handle_new_from_file(nullptr, f, PRODUCT_GRIB, &err)) != nullptr) {
long k1 = 0;
err = codes_get_long(h, "level", &k1);
vec.push_back(k1);
codes_handle_delete(h);
}
After that change memory usage is almost unnoticeable.
Related
I want to allocate my buffers according to memory available. Such that, when I do processing and memory usage goes up, but still remains in available memory limits. Is there a way to get available memory (I don't know will virtual or physical memory status will make any difference ?). Method has to be platform Independent as its going to be used on Windows, OS X, Linux and AIX. (And if possible then I would also like to allocate some of available memory for my application, someone it doesn't change during the execution).
Edit: I did it with configurable memory allocation.
I understand it is not good idea, as most OS manage memory for us, but my application was an ETL framework (intended to be used on server, but was also being used on desktop as a plugin for Adobe indesign). So, I was running in to issue of because instead of using swap, windows would return bad alloc and other applications start to fail. And as I was taught to avoid crashes and so, was just trying to degrade gracefully.
On UNIX-like operating systems, there is sysconf.
#include <unistd.h>
unsigned long long getTotalSystemMemory()
{
long pages = sysconf(_SC_PHYS_PAGES);
long page_size = sysconf(_SC_PAGE_SIZE);
return pages * page_size;
}
On Windows, there is GlobalMemoryStatusEx:
#include <windows.h>
unsigned long long getTotalSystemMemory()
{
MEMORYSTATUSEX status;
status.dwLength = sizeof(status);
GlobalMemoryStatusEx(&status);
return status.ullTotalPhys;
}
So just do some fancy #ifdefs and you'll be good to go.
There are reasons to do want to do this in HPC for scientific software. (Not game, web, business or embedded software). Scientific software routinely go through terabytes of data to get through one computation (or run) (and run for hours or weeks) -- all of which cannot be stored in memory (and if one day you tell me a terabyte is standard for any PC or tablet or phone it will be the case that the scientific software will be expected to handle petabytes or more). The amount of memory can also dictate the kind of method/algorithm that makes sense. The user does not always want to decide the memory and method - he/she has other things to worry about. So the programmer should have a good idea of what is available (4Gb or 8Gb or 64Gb or thereabouts these days) to decide whether a method will automatically work or a more laborious method is to be chosen. Disk is used but memory is preferable. And users of such software are not encouraged to be doing too many things on their computer when running such software -- in fact, they often use dedicated machines/servers.
There is no platform independent way to do this, different operating systems use different memory management strategies.
These other stack overflow questions will help:
How to get memory usage at run time in c++?
C/C++ memory usage API in Linux/Windows
You should watch out though: It is notoriously difficult to get a "real" value for available memory in linux. What the operating system displays as used by a process is no guarantee of what is actually allocated for the process.
This is a common issue when developing embedded linux systems such as routers, where you want to buffer as much as the hardware allows. Here is a link to an example showing how to get this information in a linux (in C):
http://www.unix.com/programming/25035-determining-free-available-memory-mv-linux.html
Having read through these answers I'm astonished that so many take the stance that OP's computer memory belongs to others. It's his computer and his memory to do with as he sees fit, even if it breaks other systems taking a claim it. It's an interesting question. On a more primitive system I had memavail() which would tell me this. Why shouldn't the OP take as much memory as he wants without upsetting other systems?
Here's a solution that allocates less than half the memory available, just to be kind. Output was:
Required FFFFFFFF
Required 7FFFFFFF
Required 3FFFFFFF
Memory size allocated = 1FFFFFFF
#include <stdio.h>
#include <stdlib.h>
#define MINREQ 0xFFF // arbitrary minimum
int main(void)
{
unsigned int required = (unsigned int)-1; // adapt to native uint
char *mem = NULL;
while (mem == NULL) {
printf ("Required %X\n", required);
mem = malloc (required);
if ((required >>= 1) < MINREQ) {
if (mem) free (mem);
printf ("Cannot allocate enough memory\n");
return (1);
}
}
free (mem);
mem = malloc (required);
if (mem == NULL) {
printf ("Cannot enough allocate memory\n");
return (1);
}
printf ("Memory size allocated = %X\n", required);
free (mem);
return 0;
}
Mac OS X example using sysctl (man 3 sysctl):
#include <stdio.h>
#include <stdint.h>
#include <sys/types.h>
#include <sys/sysctl.h>
int main(void)
{
int mib[2] = { CTL_HW, HW_MEMSIZE };
u_int namelen = sizeof(mib) / sizeof(mib[0]);
uint64_t size;
size_t len = sizeof(size);
if (sysctl(mib, namelen, &size, &len, NULL, 0) < 0)
{
perror("sysctl");
}
else
{
printf("HW.HW_MEMSIZE = %llu bytes\n", size);
}
return 0;
}
(may also work on other BSD-like operating systems ?)
The code below gives the total and free memory in Megabytes. Works for FreeBSD, but you should be able to use same/similar sysctl tunables on your platform and do to the same thing (Linux & OS X have sysctl at least)
#include <stdio.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/sysctl.h>
#include <sys/vmmeter.h>
int main(){
int rc;
u_int page_size;
struct vmtotal vmt;
size_t vmt_size, uint_size;
vmt_size = sizeof(vmt);
uint_size = sizeof(page_size);
rc = sysctlbyname("vm.vmtotal", &vmt, &vmt_size, NULL, 0);
if (rc < 0){
perror("sysctlbyname");
return 1;
}
rc = sysctlbyname("vm.stats.vm.v_page_size", &page_size, &uint_size, NULL, 0);
if (rc < 0){
perror("sysctlbyname");
return 1;
}
printf("Free memory : %ld\n", vmt.t_free * (u_int64_t)page_size);
printf("Available memory : %ld\n", vmt.t_avm * (u_int64_t)page_size);
return 0;
}
Below is the output of the program, compared with the vmstat(8) output on my system.
~/code/memstats % cc memstats.c
~/code/memstats % ./a.out
Free memory : 5481914368
Available memory : 8473378816
~/code/memstats % vmstat
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id
0 0 0 8093M 5228M 287 0 1 0 304 133 0 0 112 9597 1652 2 1 97
Linux currently free memory: sysconf(_SC_AVPHYS_PAGES) and get_avphys_pages()
The total RAM was covered at https://stackoverflow.com/a/2513561/895245 with sysconf(_SC_PHYS_PAGES);.
Both sysconf(_SC_AVPHYS_PAGES) and get_avphys_pages() are glibc extensions to POSIX that give instead the total currently available RAM pages.
You then just have to multiply them by sysconf(_SC_PAGE_SIZE) to obtain the current free RAM.
Minimal runnable example at: C - Check available free RAM?
The "official" function for this is was std::get_temporary_buffer(). However, you might want to test whether your platform has a decent implemenation. I understand that not all platforms behave as desired.
Instead of trying to guess, have you considered letting the user configure how much memory to use for buffers, as well as assuming somewhat conservative defaults? This way you can still run (possibly slightly slower) with no override, but if the user know there is X memory available for the app they can improve performance by configuring that amount.
Here is a proposal to get available memory on Linux platform:
/// Provides the available RAM memory in kibibytes (1 KiB = 1024 B) on Linux platform (Available memory in /proc/meminfo)
/// For more info about /proc/meminfo : https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-proc-meminfo
long long getAvailableMemory()
{
long long memAvailable = -1;
std::ifstream meminfo("/proc/meminfo");
std::string line;
while (std::getline(meminfo, line))
{
if (line.find("MemAvailable:") != std::string::npos)
{
const std::size_t firstWhiteSpacePos = line.find_first_of(' ');
const std::size_t firstNonWhiteSpaceChar = line.find_first_not_of(' ', firstWhiteSpacePos);
const std::size_t nextWhiteSpace = line.find_first_of(' ', firstNonWhiteSpaceChar);
const std::size_t numChars = nextWhiteSpace - firstNonWhiteSpaceChar;
const std::string memAvailableStr = line.substr(firstNonWhiteSpaceChar, numChars);
memAvailable = std::stoll(memAvailableStr);
break;
}
}
return memAvailable;
}
I am new to c++ programming and StackOverflow, but I have some experience with core Java. I wanted to participate in programming Olympiads and I choose c++ because c++ codes are generally faster than that of an equivalent Java code.
I was solving some problems involving recursion and DP at zonal level and I came across this question called Sequence game
But unfortunately my code doesn't seem to work. It exits with exit code 3221225477, but I can't make anything out of it. I remember Java did a much better job of pointing out my mistakes, but here in c++ I don't have a clue of what's happening. Here's the code btw,
#include <iostream>
#include <fstream>
#include <cstdio>
#include <algorithm>
#include <vector>
#include <set>
using namespace std;
int N, minimum, maximum;
set <unsigned int> result;
vector <unsigned int> integers;
bool status = true;
void score(unsigned int b, unsigned int step)
{
if(step < N)
{
unsigned int subtracted;
unsigned int added = b + integers[step];
bool add_gate = (added <= maximum);
bool subtract_gate = (b <= integers[step]);
if (subtract_gate)
subtracted = b - integers[step];
subtract_gate = subtract_gate && (subtracted >= minimum);
if(add_gate && subtract_gate)
{
result.insert(added);
result.insert(subtracted);
score(added, step++);
score(subtracted, step++);
}
else if(!(add_gate) && !(subtract_gate))
{
status = false;
return;
}
else if(add_gate)
{
result.insert(added);
score(added, step++);
}
else if(subtract_gate)
{
result.insert(subtracted);
score(subtracted, step++);
}
}
else return;
}
int main()
{
ios_base::sync_with_stdio(false);
ifstream input("input.txt"); // attach to input file
streambuf *cinbuf = cin.rdbuf(); // save old cin buffer
cin.rdbuf(input.rdbuf()); // redirect cin to input.txt
ofstream output("output.txt"); // attach to output file
streambuf *coutbuf = cout.rdbuf(); // save old cout buffer
cout.rdbuf(output.rdbuf()); // redirect cout to output.txt
unsigned int b;
cin>>N>>b>>minimum>>maximum;
for(unsigned int i = 0; i < N; ++i)
cin>>integers[i];
score(b, 0);
set<unsigned int>::iterator iter = result.begin();
if(status)
cout<<*iter<<endl;
else
cout<<-1<<endl;
cin.rdbuf(cinbuf);
cout.rdbuf(coutbuf);
return 0;
}
(Note: I intentionally did not use typedef).
I compiled this code with mingw-w64 in a windows machine and here is the Output:
[Finished in 19.8s with exit code 3221225477] ...
Although I have an intel i5-8600, it took so much time to compile, much of the time was taken by the antivirus to scan my exe file, and even sometimes it keeps on compiling for long without any intervention from the anti-virus.
(Note: I did not use command line, instead I used used sublime text to compile it).
I even tried tdm-gcc, and again some other peculiar exit code came up. I even tried to run it on a Ubuntu machine, but unfortunately it couldn't find the output file. When I ran it on a Codechef Online IDE, even though it did not run properly, but the error message was less scarier than that of mingw's.
It said that there was a run-time error and "SIGSEGV" was displayed as an error code. Codechef states that
A SIGSEGV is an error(signal) caused by an invalid memory reference or
a segmentation fault. You are probably trying to access an array
element out of bounds or trying to use too much memory. Some of the
other causes of a segmentation fault are : Using uninitialized
pointers, dereference of NULL pointers, accessing memory that the
program doesn’t own.
It's been a few days that I am trying to solve this, and I am really frustrated by now. First when i started solving this problem I used c arrays, then changed to vectors and finally now to std::set, while hopping that it will solve the problem, but nothing worked. I tried a another dp problem, and again this was the case.
It would be great if someone help me figure out what's wrong in my code.
Thanks in advance.
3221225477 converted to hex is 0xC0000005, which stands for STATUS_ACCESS_VIOLATION, which means you tried to access (read, write or execute) invalid memory.
I remember Java did a much better job of pointing out my mistakes, but here in c++ I don't have a clue of what's happening.
When you run into your program crashing, you should run it under a debugger. Since you're running your code on Windows, I highly recommend Visual Studio 2017 Community Edition. If you ran your code under it, it would point exact line where the crash happens.
As for your crash itself, as PaulMcKenzie points out in the comment, you're indexing an empty vector, which makes std::cin write into out of bounds memory.
integers is a vector which is a dynamic contiguous array whose size is not known at compile time here. So when it is defined initially, it is empty. You need to insert into the vector. Change the following:
for(unsigned int i = 0; i < N; ++i)
cin>>integers[i];
to this:
int j;
for(unsigned int i = 0; i < N; ++i) {
cin>> j;
integers.push_back(j);
}
P.W's answer is correct, but an alternative to using push_back is to pre-allocate the vector after N is known. Then you can read from cin straight into the vector elements as before.
integers = vector<unsigned int>(N);
for (unsigned int i = 0; i < N; i++)
cin >> integers[i];
This method has the added advantage of only allocating memory for the vector once. The push_back method will reallocate if the underlying buffer fills up.
I am writing a program which requires writing a large binary file (about 12 GiB or more) to a disk. I have created a small test program to test this functionality. Although allocating the RAM memory for the buffer is not a problem, my program does not write the data to a file. The file remains empty. Even for 3.72 GiB files.
//size_t bufferSize=1000; //ok
//size_t bufferSize=100000000; //ok
size_t bufferSize=500000000; //fails although it is under 4GiB, which shouldn't cause problem anyways
double mem=double(bufferSize)*double(sizeof(double))/std::pow(1024.,3.);
cout<<"Total memory used: "<<mem<<" GiB"<<endl;
double *buffer=new double[bufferSize];
/* //enable if you want to fill the buffer with random data
printf("\r[%i \%]",0);
for (size_t i=0;i<(size_t)bufferSize;i++)
{
if ((i+1)%100==0) printf("\r[%i %]",(size_t)(100.*double(i+1)/bufferSize));
buffer[i]=rand() % 100;
}
*/
cout<<endl;
std::ofstream outfile ("largeStuff.bin",std::ofstream::binary);
outfile.write ((char*)buffer,((size_t)(bufferSize*double(sizeof(double)))));
outfile.close();
delete[] buffer;
I actually compiled and ran the code exactly as you have pasted there and it works. It creates a 4GB file.
If you are on a FAT32 filesystem the max filesize is 4GB.
Otherwise I suggest you check:
The amount of free disk space you have.
Whether your user account has any disk usage limits in place.
The amount of free RAM you have.
Whether there are any runtime errors.
#enhzflep's suggestion about the number of prints (although that is
commented out)
It seems that you want to have a buffer that contains the whole file's contents prior to writing it.
You're doing it wrong, through: the virtual memory requirements are essentially double of what they need to be. Your process retains the buffer, but when you write that buffer to disk it gets duplicated in operating system's buffers. Now, most OSes will notice that you write sequentially and may discard their buffers quickly, but still: it's rather wasteful.
Instead, you should create an empty file, grow it to its desired size, then map its view into memory, and do the modifications on the file's view in memory. For 32 bit hosts your file size is limited to <1GB. For 64 bit hosts, it's limited by the filesystem only. On modern hardware, creating and filling a 1GB file that way takes on the order of 1 second (!) if you have enough free RAM available.
Thanks to the wonders of RAII, you don't need to do anything special to release the mapped memory, or to close/finalize the file. By leveraging boost you can avoid writing platform-specific code, too.
// https://github.com/KubaO/stackoverflown/tree/master/questions/mmap-boost-40308164
#include <boost/interprocess/file_mapping.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <boost/filesystem.hpp>
#include <cassert>
#include <cstdint>
#include <fstream>
namespace bip = boost::interprocess;
void fill(const char * fileName, size_t size) {
using element_type = uint64_t;
assert(size % sizeof(element_type) == 0);
std::ofstream().open(fileName); // create an empty file
boost::filesystem::resize_file(fileName, size);
auto mapping = bip::file_mapping{fileName, bip::read_write};
auto mapped_rgn = bip::mapped_region{mapping, bip::read_write};
const auto mmaped_data = static_cast<element_type*>(mapped_rgn.get_address());
const auto mmap_bytes = mapped_rgn.get_size();
const auto mmap_size = mmap_bytes / sizeof(*mmaped_data);
assert(mmap_bytes == size);
element_type n = 0;
for (auto p = mmaped_data; p < mmaped_data+mmap_size; ++p)
*p = n++;
}
int main() {
const uint64_t G = 1024ULL*1024ULL*1024ULL;
fill("tmp.bin", 1*G);
}
I have 2 applications.
VB application is written in .NET 3.5. It is pretty big application. I can't rewrite this to C++ for few reasons. Im not sure if that matters, but it is x86 application.
C++ application is written in .NET 4.0. It is x64 build and there will be no x86 support. For now - it is managed code with a bit of assembler code. I will mix managed and unmanaged later when I learn more about C++. It is x64 build and has to stay like this.
It is supposed to extend VB application features - capture frames from camera, do something with them and send processed images to VB application. Images are pretty big (1920x1080x24bpp) and I need to process 30-60 frames per second like that, so it must be efficent way.
My goals:
"Send" bitmap from C++ application to VB application, and VB application should start some method when that bitmap came.
"Send" some information the other way, from VB application to C++ application. It is supposed to change C++ application processing parameters from VB application GUI.
If possible - send just a pointer and size of bitmap instead of copying whole data in RAM.
Lets say, I want something like this:
VB side:
Function receivedBitmapFromCpp(BMP_POINTER?, BMP_SIZE_X?, BMP_SIZE_Y?, BMP_BPP?) As Integer Handles ????
End Function
C++ side:
void sendBitmapToVb(BMP_POINTER?, BMP_SIZE_X?, BMP_SIZE_Y?, BMP_BPP?)
{
int bitmapsendingresult = ?????
}
It may be System.Drawing.Bitmap, or just some array that I will convert to System.Drawing.Bitmap in VB application. It doesn't matter that much.
My question:
Can someone explain, how can I:
send some object data (like System.Drawing.Bitmap for example), or better pointer to that data from VB application to C++ application
receive that data in C++ application
start some C++ function (with some event?) when data is received/ready
Use shared memory circular buffer to exchange data between processes. This could be implemented using boost interprocess as a C++ dll and then that dll could be imported into your .Net applications. Note that you will need to build 32 and 64 bit versions of boost and your shared memory dll. I prepared an example of 32 bit and 64 bit apps which you can run and see how fast that is. I think it should be fast enough but if is not then still multithreading could be used.
64 bit producer:
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <boost/interprocess/sync/interprocess_mutex.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>
#include <cstring>
#include <cstdlib>
#include <string>
#include <iostream>
#include <chrono>
struct shmem_info
{
boost::interprocess::interprocess_mutex mutex;
uint64_t pos;
bool run;
};
int main(int argc, char *argv[])
{
using namespace boost::interprocess;
struct shm_remove
{
shm_remove() { shared_memory_object::remove("MySharedMemory"); shared_memory_object::remove("MySharedMemoryInfo");}
//~shm_remove() { shared_memory_object::remove("MySharedMemory"); shared_memory_object::remove("MySharedMemoryInfo");}
} remover;
const size_t width = 1920;
const size_t height = 1080;
const size_t bytes_per_pixel = 3;
const size_t frame_size = width*height*bytes_per_pixel;
const size_t frames = 60;
const size_t shmem_frames = 3 * frames;
const size_t shmem_size = width * height * bytes_per_pixel * shmem_frames;
std::cout << "Generating data ..." << std::endl;
std::vector<uint8_t> frame(frame_size);
// generate frame data
for (size_t x = 0; x < width*height; ++x)
for (size_t y = 0; y < bytes_per_pixel; ++y)
frame[x*bytes_per_pixel + y] = (x%252) + y;
std::cout << "Creating shared memory files ..." << std::endl;
shared_memory_object shm(create_only, "MySharedMemory", read_write);
shared_memory_object shm_info(create_only, "MySharedMemoryInfo", read_write);
//Set size
shm.truncate(shmem_size);
shm_info.truncate(sizeof(shmem_info));
//Map the whole shared memory in this process
mapped_region region(shm, read_write);
mapped_region region_info(shm_info, read_write);
shmem_info *info = new (region_info.get_address()) shmem_info;
{
scoped_lock<interprocess_mutex> lock(info->mutex);
info->pos = 0;
info->run = true;
}
char c;
std::cout << "Ready. Now start client application and wait for it to be ready." << std::endl;
std::cout << "Then press a key and enter to start" << std::endl;
std::cin >> c;
std::cout << "Running ..." << std::endl;
std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now();
size_t times = 10;
for (size_t t = 0; t < times; ++t)
{
for (size_t f = 0; f < shmem_frames; ++f)
{
// get pointer to the beginning of shared memory
uint8_t *ptr = static_cast<uint8_t*>(region.get_address());
// move pointer to the next frame
ptr += f*frame_size;
// modify first data point for testing purposes
frame[0] = f;
frame[1] = f + 1;
frame[2] = f + 2;
// copy data to shared memory
memcpy(ptr, &frame[0], frame_size);
// update the position each "frames" number, doing that too frequently kills the performance
if (f % frames == 0)
{
// this will lock access to the pos for the time of updating the pos only
scoped_lock<interprocess_mutex> lock(info->mutex);
info->pos += frames;
std::cout << "write pos = " << info->pos << std::endl;
}
}
}
std::chrono::steady_clock::time_point end = std::chrono::steady_clock::now();
size_t ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();
std::cout << (double(times*shmem_frames*1000) / double(ms)) << " fps." << std::endl;
winapi::sleep(2000);
// stop run
{
scoped_lock<interprocess_mutex> lock(info->mutex);
info->run = false;
}
return 0;
}
32 bit consumer:
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <boost/interprocess/sync/interprocess_mutex.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>
#include <cstring>
#include <cstdlib>
#include <string>
#include <iostream>
#include <chrono>
struct shmem_info
{
boost::interprocess::interprocess_mutex mutex;
uint64_t pos;
bool run;
};
int main(int argc, char *argv[])
{
using namespace boost::interprocess;
const size_t width = 1920;
const size_t height = 1080;
const size_t bytes_per_pixel = 3;
const size_t frame_size = width*height*bytes_per_pixel;
const size_t frames = 60;
const size_t shmem_frames = 3 * frames;
const size_t shmem_size = width * height * bytes_per_pixel * shmem_frames;
std::vector<uint8_t> frame(frame_size);
std::cout << "Opening shared memory files ..." << std::endl;
//Open already created shared memory object.
shared_memory_object shm(open_only, "MySharedMemory", read_write);
shared_memory_object shm_info(open_only, "MySharedMemoryInfo", read_write);
//Map the whole shared memory in this process
mapped_region region(shm, read_only);
mapped_region region_info(shm_info, read_write);
shmem_info *info = static_cast<shmem_info*>(region_info.get_address());
std::cout << "Ready." << std::endl;
bool run = true;
// first wait for processing to be started
while (true)
{
{
scoped_lock<interprocess_mutex> lock(info->mutex);
if (info->run)
break;
}
winapi::Sleep(1);
}
std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now();
uint64_t pos = 0;
uint64_t shm_pos = 0;
while(run)
{
// wait for a new data
{
scoped_lock<interprocess_mutex> lock(info->mutex);
run = info->run;
if (info->pos == pos)
{
winapi::Sleep(1);
continue;
}
// we've got the new data
shm_pos = info->pos;
}
while (pos < shm_pos)
{
// get pointer to the beginning of shared memory
uint8_t *ptr = static_cast<uint8_t*>(region.get_address());
// calculate the frame position in circular buffer and move pointer to that frame
ptr += (pos%shmem_frames)*frame_size;
// copy data from shared memory
memcpy(&frame[0], ptr, frame_size);
//winapi::Sleep(1);
++pos;
if (pos % frames == 0)
std::cout << "read pos: " << pos << std::endl;
}
}
std::chrono::steady_clock::time_point end = std::chrono::steady_clock::now();
size_t ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();
ms -= 2000; // producer waits 2 seconds before sets run=false
std::cout << (double(pos*1000) / double(ms)) << " fps." << std::endl;
return 0;
}
I used boost 1.58, first circle is always slow, you may want to run a warm up circle before start using the shared memory. The data needs to be copied into shared memory but for reading the shmem pointer to the frame could be passed to the .Net application. Then you need to ensure your .Net app reads the data on time before it gets overwritten.
Useful links:
boost interpocess
Simple example
EDIT: I've modified the source code to show number of frames per seconds that roughly can be achieved. On my machine that is 190+ fps so I would expect it to be still above the required 60 fps taking into account the overhead of transferring data/pointer between .Net app and c++ dll.
Above code should give you a good start, you need to refactor producer and consumer shared memory code into a common class and make it a dll. There are few ways of importing c++ dll into .Net. How-to-Marshal-a-C-Class explains some of them quite well.
Now to your questions:
Can someone explain, how can I:
send some object data (like System.Drawing.Bitmap for example), or better pointer to that data from VB application to C++ application
You will need to get HBITMAP from Bitmap using GetHbitmap() method and pass it down to c++ dll. Then in c++ dll copy pixel data and other bitmap info if required into shared memory (pointer to data won't work). How to do that between .Net and c++ see c-get-raw-pixel-data-from-hbitmap. Especially useful will be this answer.
receive that data in C++ application
Then to read data from shared memory you will probably need to first create empty Bitmap of same size like in shared memory and pass its HBITMAP down to C++ dll to fill the pixel data.
start some C++ function (with some event?) when data is received/ready
You just need to continuously poll shared memory for new data like in the above code.
you should compile your VB application to x64 architecture. Since one of your binaries is in x64 so you should be ok with it. From your description I understood that youre using managed C++ .NET. Your C++ project should be compiled to a dll, because it doest not do anything else than just extending VB. So you can just import that .NET dll into your VB application. Now you can use managed C++ .NET functionality inside VB.
If you however not using managed C++ .NET. You can choose one of fallowing.
You can make native C wrapper to your C++. It can be pretty basic, you can make a function witch takes some arguments, and a function pointer as a callback (you wanted a callback) to get a bitmap. Use inter operations to probe you C++ dll for those functions (inter operators are found at Runtime.Interop namespace). Make a VB wrapper around those functions inside you VB application. You can convert methods inside VB to delegates representing function pointers, ant pass those delegates as callbacks.
Or you can make managed C++ .NET wrapper around your native C++ methods. Again, it can be basic, you can make a basic class methods inside managed C++ .NET witch just forwards arguments to native code. Compile everything to managed C++ .NET dll, and include that dll to you VB application and use its functionality. Cheers.
I would use a Named Pipe. It will allow you to send and receive data between processes. Once you receive the data, you can do whatever you want with it. That said, you won't be able to share objects between processes, so don't try sending a pointer. The good news is that named piped is supported by .NET 3.5 and .NET 4.0 according to this page. If you want examples, there are plenty online.
You can do the following:
create a folder that the c++ application generate the images in
in you vb application add FileSystemWatcher so when the first
application (c++) add any file to the common folder, automatically
the second application will read it
you can use this link to learn more about the FileSystemWatcher
you might let the first application to name the file in a specific way for exmaple p-dd-mm-yyyy-hh-mm-ss and the second application rename it to c-dd-mm-yyyy-hh-mm-ss wher p: pending, c: completed and the dd-mm-yyyy-hh-mm-ss datetime value
based on your comment, hope you can find a solution mentioned here How to do CreateFileMapping in a C++ DLL and access it in C#, VB and C++
You are using .Net. So you can use the interop lib. It contains a IntPtr class you can use to capsule the access to the c++ pointer. The bitmap class even has a constructor with an IntPtr. Have a look at the MSDN for Bitmap and IntPtr.
I have the following problem.
I use the following function to receive a string from a buffer until a newline occurs.
string get_all_buf(int sock) {
int n = 1, total = 0, found = 0;
char c;
char temp[1024*1024];
string antw = "";
while (!found) {
n = recv(sock, &temp[total], sizeof(temp) - total - 1, 0);
if (n == -1) {
break;
}
total += n;
temp[total] = '\0';
found = (strchr(temp, '\n') != 0);
if (found == 0){
found = (strchr(temp, '\r\n') != 0);
}
}
antw = temp;
size_t foundIndex = antw.find("\r\n");
if (foundIndex != antw.npos)
antw.erase ( antw.find ("\r\n"), 2 );
foundIndex = antw.find("\n");
if (foundIndex != antw.npos)
antw.erase ( antw.find ("\n"), 2 );
return answ;
}
So use it like this:
string an = get_all_buf(sClient);
If I create an exe file everything works perfectly.
But if I create a dll and run it using rundll32 the application closes at "string an = get_all_buf(sClient);" without any error message...
I tried to fix this for hours now, and I am currently a bit desperate...
P.S. sorry for obvious errors or bad coding style, I just started learning C++.
char temp[1024*1024];
declares a 1Mb structure on the stack. This may be too large and overflow available stack memory. You could instead give it static scope
static char temp[1024*1024];
or allocate it dynamically
char* temp = (char*)malloc(1024*1024);
// function body
free(temp);
Alternatively, assuming the mention of run32.dll means you're working on Windows, you could investigate keeping it on the stack by using the /STACK linker option. This probably isn't the best approach - you've already found it causes problems when you change build settings or try to target other platforms.
Instead of creating temp variable on the stack, I'd create it dynamically (on the heap), but not using raw malloc and free as showed in a previous answer, but using modern C++ and std::vector:
#include <vector>
std::vector<char> temp(1024*1024);
This is exception safe, and you don't have to pay attention to release the allocated memory: std::vector's destructor will do that automatically (also in case of exceptions thrown).
Instead of sizeof(temp), in your code you can use temp.size() (which will return the count of elements in the vector, and since this is a vector of chars, it will return just the total vector size in chars i.e. in bytes).
You can still use operator[] for std::vector, as you do for raw C arrays.
Note also that if you are building a DLL and the above function is exposed at the DLL interface, since this function has a C++ interface with a STL class (std::string) at the boundary, you must pay attention that both your DLL and your clients are built with dynamic linking to the same CRT, and with the same compiler and the same compiler settings (e.g. you can't mix a DLL built with VS2008/VC9 with a .EXE built with VS2010/VC10, or a release-build DLL with a debug-build EXE built with the same compiler).