I'm new to libgphoto2 and had some questions about where the actual image memory is located. It seems that photos are accessed via an object called a CameraFile. There's an example included in the source called sample-capture.c that has a function called capture_to_memory.
In that function, a CameraFile object is created and initialized using gp_file_new. The implementation of gp_file_new creates a CameraFile object whose accesstype member is set to GP_FILE_ACCESSTYPE_MEMORY.
That CameraFile object is then passed to gp_camera_file_get, and I'm not really clear what happens in this function.
After the call to gp_camera_file_get, the file is sent to gp_file_get_data_and_size. Looking at the implementation of that function, and given that the files accesstype is GP_FILE_ACCESSTYPE_MEMORY, the gp_file_get_data_and_size function will not allocate any memory and instead merely assign the pointer to the CameraFile's data member (same with size).
So where is the memory this pointer points to? Is it on the actual camera? Or is it somewhere in my ram? I'm just trying to avoid as many memory operations as possible. I've found a nice blog post
http://sepharads.blogspot.com/2011/11/camera-tethered-capturing-using.html?m=1
where the CameraFile object is created via gp_file_new_from_fd. This sets the files accesstype to GP_FILE_ACCESSTYPE_FD. When gp_file_get_data_and_size is called on a file with this accesstype, new memory is allocated and the file's data is copied into it. However, this is read from disk, or at least I assume it is since FD means Unix File Descriptor, and it needs an actual filename string. Does that mean that gp_camera_file_get reads the data into a file on disk if the CameraFile's accestype is GP_FILE_ACCESSTYPE_FD?
There is a function called gp_file_free. If it's an FD file it closes the file handle, but it doesn't call free(file->data) anywhere, so I'm pretty confused...
What does gp_camera_file_get do? The implementation is there, but it kind of threw me. Does it actually bring the file from my camera's memory to my computer's memory and allocate CameraFile->data in ram? Or does it just give me a handle to something stored on my camera?
Sorry if this is the wrong place to ask, I did see some sort of mailing list for libgphoto2 but I couldn't tell how active it was.
Never mind, I think I've found it.
So gp_camera_file_get ends up calling gp_filesystem_get_file, which calls gp_filesystem_get_file_impl. That function looks at the CameraFileType, and if it's GP_FILE_TYPE_NORMAL (which it is, in my case), it calls gp_file_copy with an instance of CameraFileSystemFile as the source.
gp_file_copy ends up calling malloc if GP_FILE_ACCESSTYPE_MEMORY is the destination file's accesstype, and writes to disk if the accesstype is GP_FILE_ACCESSTYPE_FD.
Then, the call to gp_file_free calls gp_file_clean, which calls free if the accesstype is GP_FILE_ACCESSTYPE_MEMORY.
Related
Recently I ran into a problem at work where you have two functions; one opens a file descriptor (which is a local variable in the function), and passes it to another function where it is used for reading or writing. Now, when one of the operations read/write fails the function that was doing the read/write closes this file descriptor, and returns.
The question is, whose responsibility is to close the file descriptor, or let's say do cleanup:
the function which created the fd
the function which experienced the error while read/write
Is there a design rule for these kind of cases; let's say creation and cleanup.
BTW, the problem was that both functions attempted to close the fd, which resulted in a crash on the second call to close.
There are two parts to this answer — the general design issue and the detailed mechanics for your situation.
General Design
Handling resources such as file descriptors correctly, and making sure they are released correctly, is an important design issue. There are multiple ways to manage the problem that work. There are some others that don't.
Your tags use C and C++; be aware that C++ has extra mechanisms available to it.
In C++, the RAII — Resource Acquisition Is Initialization — idiom is a great help. When you acquire a resource, you ensure that whatever acquires the resource initializes a value that will be properly destructed and will release the resource when destructed.
In both languages, it is generally best if the function responsible for allocating the resource also releases it. If a function opens a file, it should close it. If a function is given an open file, it should not close the file.
In the comments, I wrote:
Generally, the function that opened the file should close it; the function that experienced the error should report the error, but not close the file. However, you can work it how you like as long as the contract is documented and enforced — the calling code needs to know when the called code closed the file to avoid double closes.
It would generally be a bad design for the called function to close the file sometimes (on error), but not other times (no error). If you must go that way, then it is crucial that the calling function is informed that the file is closed; the called function must return an error indication that tells the calling code that the file is no longer valid and should neither be used nor closed. As long as the information is relayed and handled, there isn't a problem — but the functions are harder to use.
Note that if a function is designed to return an opened resource (it is a function that's responsible for opening a file and making it available to the function that called it), then the responsibility for closing the file falls on the code that calls the opening function. That is a legitimate design; you just have to make sure that there is a function that knows how to close it, and that the calling code does close it.
Similar comments apply to memory allocation. If a function allocates memory, you must know when the memory will be freed, and ensure that it is freed. If it was allocated for the purposes of the current function and functions it calls, then the memory should be released before return. If it was allocated for use by the calling functions, then the responsibility for release transfers to the calling functions.
Detailed mechanics
Are you sure you're using file descriptors and not FILE * (file streams)? It's unlikely that closing a file descriptor twice would cause a crash (error, yes, but not a crash). OTOH, calling fclose() on an already closed file stream could cause problems.
In general, in C, you pass file descriptors, which a small integers, by value, so there isn't a way to tell the calling function that the file descriptor is no longer valid. In C++, you could pass them by reference, though it is not conventional to do so. Similarly with FILE *; they're most usually passed by value, not by reference, so there isn't a way to tell the calling code that the file is not usable any more by modifying the value passed to the function.
You can invalidate a file descriptor by setting it to -1; that is never a valid file descriptor. Using 0 is a bad idea; it is equivalent to using standard input. You can invalidate a file stream by setting it to 0 (aka NULL). Passing the null pointer to functions that try to use the file stream will tend to cause crashes. Passing an invalid file descriptor typically won't cause crashes — the calls may fail with EBADF set in errno, but that's the limit of the damage, usually.
Using file descriptors, you will seldom get a crash because the file descriptor is no longer valid. Using file streams, all sorts of things can go wrong if you try using an invalid file stream pointer.
I have an DLL that might get called by multiple applications at the same time.
This DLL memory-maps a file.
I have 2 questions:
1) Each application will create its own instance of the DLL, right?
And thus, the file will be memory-mapped multiple times
2) If this is true, I don't understand what is happening here:
a) Application A calls the DLL.
b) Application B calls the DLL.
c) I quit application A, and the DLL will unmap the file.
d) Application B calls the DLL, and the memory-mapped file is not available anymore, and the call fails.
I don't understand this.
Does anybody do?
Thank you.
This happens because your assumption from 1) is false. A dll is by definition shared; both applications are using the same dll instance, so when you release the file in one application, it won't be available to the others.
To get around your issue, you should implement some reference counting mechanism in order to unmap the file only when no process is using it.
Edit: #sumeet is right. Each process has its own address space; when two processes load the same dll, they might share its read-only data for increased efficiency, but their writable data is local to each process. Nevertheless, a memory-mapped file is a kernel object, like semaphores, pipes and shared memory. Thus, if you unmap it in a process, you unmap it for all.
Edit2: From MSDN (Remarks section):
Multiple processes can share a view of the same file by either using a
single shared file mapping object or creating separate file mapping
objects backed by the same file. A single file mapping object can be
shared by multiple processes through inheriting the handle at process
creation, duplicating the handle, or opening the file mapping object
by name. For more information, see the CreateProcess, DuplicateHandle
and OpenFileMapping functions.
[...]
Mapped views of a file mapping object maintain internal references to
the object, and a file mapping object does not close until all
references to it are released. Therefore, to fully close a file
mapping object, an application must unmap all mapped views of the file
mapping object by calling UnmapViewOfFile and close the file mapping
object handle by calling CloseHandle. These functions can be called in
any order.
First of all, from the first paragraph, how is each app initializing the view?
From the second paragraph, I gather that calling UnmapViewofFile and CloseHandle from each app will release all references to the memory file, and then Windows will automatically release the associated resources (i.e. he keeps the reference count, you don't need to do it).
Post your memory mapping initialization and shutdown code for both apps.
I have a block of memory which I have allocated using mmap. I then want to write this block of memory to a file on the file system. I am wondering if there is a way to point the existing block of memory directly to a file descriptor using mmap() or will I have to copy the whole block of memory to a new mmap() area which has been created with the file descriptor?
You can use vmsplice with SPLICE_F_GIFT to place the memory into a pipe, after that splice or copy_file_range may do what is required.
You will have top copy it, unless you can create the mmap with a temporary file in the first place, and then just rename the file when you want to "save" it.
I'm not familiar with all implementations, I have looked quite closely at the Linux implementation, but I believe this applies in all cases: When you do not have a file when you create a mmap region, the OS will use the "swap" as a file for the mmap region. Since there is a mapping between the file-content and the memory content, you can't trivially just change what that mapping is (although it could, perhaps, be technically possible to achieve, it could be rather "expensive" if you have a 4GB mmap region, where only a small part of it is in physical RAM, the rest is "on disk").
Assuming we're talking pure POSIX, there's no way to attach an mmap'd region to a new fd. I'm also not aware of any way to do this in Linux or OS X.
You'll have to either copy, or mmap the file that you want to write to in the first place. If you don't want the file to appear immediately for some reason, there's the following workaround:
open a temporary file in the filesystem where you want the final file (the same directory is a safe bet)
mmap it, modify it, write out changes
rename the file to its final location.
I'm very new to Memory Mapped Files, and I'm a little lost on something.
I know that if I had a file, I could load it and access it from various processes at once using MMaps.
But in my situation, I'm creating a DLL attached to Process A, and that DLL has been given a pointer to a cSurface which Process A has prepared. I need to share that cSurface's data with Process B. I really don't want to have to call up a blank MMap and copy my Process A's surface into it, only to copy it out again in process B.
Is it possible to map my surface as if it were a file so the MMap already points to the surface data when it's created (as it would were I loading SomeTextFile.txt)?
My plan, in theory, would be to receive a pointer to the surface in Proc A, tell windows to share that surface's memory with a given name, and use Mutexes to coordinate access - the idea being that both processes read the same physical copy of the surface with no cumbersome copying.
Is that possible?
Yes, it's all there in the documentation of CreateFileMapping.
You can give your memory mapped file a name. If another process open a mmap with the same name it will point to the same memory. It the handle in CreateFileMapping is put to INVALID_HANDLE_VALUE it keeps it purely in memory. Check the documentation of CreateFileMapping
Thanks for all your comments.
I did some further research and found the answer.
You can use Memory Mapped Files to share either a file, or a blank memory space. If you want to share data already initialised in memory prior to setting up the map, you have to create a blank map and subsequently copy your data into it.
I currently try to implement some interprocess communication using the Windows CreateFileMapping mechanism. I know that I need to create a file mapping object with CreateFileMapping first and then create a pointer to the actual data with MapViewOfFile. The example then puts data into the mapfile by using CopyMemory.
In my application I have an image buffer (1 MB large) which I want to send to another process. So now I inquire a pointer to the image and then copy the whole image buffer into the mapfile. But I wonder if this is really necessary. Isn't it possible to just copy an actual pointer in the shared memory which points to the image buffer data? I tried a bit but didn't succeed.
Different processes have different address spaces. If you pass a valid pointer in one process to another process, it will probably point to random data in the second process. So you will have to copy all the data.
I strongly recommend you use Boost::interprocess. It has lots of goodies to manage this kind of stuff & even includes some special Windows-only functions in case you need to interoperate w/ other processes that use particular Win32 features.
The most important thing is to use offset pointers rather than regular pointers. Offset pointers are basically relative pointers (they store the difference between where the pointer is and where the thing pointed to is). This means that even if the two pointers are mapped to different address spaces, as long as the mappings are identical in structure then you are fine.
I've used all kinds of complicated data structures with offset smart pointers and it worked like a charm.
Shared Memory doesn't mean sending and receiving of Data. Its a memory created for number of processes without violation. For that you have to follow some mechanisms like locks so that the data will not corrupt.
In process 1 :
CreateFileMapping() : It will create the Shared Memory Block, with the name provided in last parameter, if it is not already present and returns back a handle (you may call it a pointer), if successful.
MapViewOfFile() : It maps (includes) this shared block in the process address space and returns a handle (again u can say a pointer).
With this pointer returned by MapViewOfFile() only you can access that shared block.
In process 2 :
OpenFileMapping() : If the shared memory block is successfully created by CreateFileMapping(), you can use it with the same name (name used to create the shared memory block).
UnmapViewOfFile() : It will unmap (you can remove the shared memory block from that process address space). When you are done using the shared memory (i.e. access, modification etc) call this function .
Closehandle() : finally to detach the shared memory block from process , call this with argument,handle returned by OpenFileMapping() or CreateFileMapping().
Though these functions look simple, the behaviour is tricky if the flags are not selected properly.
If you wish to read or write shared memory, specify PAGE_EXECUTE_READWRITE in CreateFileMapping().
Whenever you wish to access shared memory after creating it successfully, use FILE_MAP_ALL_ACCESS in MapViewOfFile().
It is better to specify FALSE (do not inherit handle from parent process) in OpenFileMapping() as it will avoid confusion.
You CAN get shared memory to use the same address over 2 processes for Windows. It's achieveable with several techniques.
Using MapViewOfFileEx, here's the significant experpt from MSDN.
If a suggested mapping address is
supplied, the file is mapped at the
specified address (rounded down to the
nearest 64K-boundary) if there is
enough address space at the specified
address. If there is not enough
address space, the function fails.
Typically, the suggested address is
used to specify that a file should be
mapped at the same address in multiple
processes. This requires the region of
address space to be available in all
involved processes. No other memory
allocation can take place in the
region that is used for mapping,
including the use of the VirtualAlloc
or VirtualAllocEx function to reserve
memory.
If the lpBaseAddress parameter
specifies a base offset, the function
succeeds if the specified memory
region is not already in use by the
calling process. The system does not
ensure that the same memory region is
available for the memory mapped file
in other 32-bit processes.
Another related technique is to use a DLL with a section marked Read + Write + Shared. In this case, the OS will pretty much do the MapViewOfFileEx call for you and for any other process which loads the DLL.
You may have to mark your DLL to a FIXED load address, not relocateable etc.. naturally.
You can use Marshalling of pointers.
If it's possible, it would be best to have the image data loaded/generated directly into the shared memory area. This eliminates the memory copy and puts it directly where it needs to be. When it's ready you can signal the other process, giving it the offset into your shared memory where the data begins.