I want to send data from one file descriptor to another via linux-aio without buffering and without transferring data to and from user space. Is such a sendfile64() funktion possible with linux-aio?
I looked at some linux-aio examples (in C/C++) and simple file-copy programs. All these examples do reading -> buffer -> writing.
regards,
philip
It's possible if you mmap the file, then you can make a aio sendfile. It's faster then sendfile via do_splice and should not sync at i_mutex.
Look at the lighttp linux_aio module.
Related
I am trying to build a new layer using scapy. I want to read a pcap file and store each packet in an array so that I can access all the information inside the packet using the index of the array.
The scapy function that you're looking for is rdpcap
from scapy.all import *
packets = rdpcap('file.pcap')
packets[0].show()
As you can read everywhere Scape is SUPER slow for pcap files >10MB. I made a couple of tools always using dpkt. Just to give you an idea. Having a 670MB file using dpkt splitting the specific bytes I need compared to full fletched scapy (reading raw packages (the fastest solution with scapy)) is:
dpkt needs 150s (6GB ram), scapy >750s (>10GB)
I currently have some code reading files which are not compressed, it uses the following approach to read a file in C++
FILE* id = fopen("myfile.dat", "r");
after obtaining id, different parts of the code access the file using fread, fseek, etc.
I would like to adapt my code so as to open a gzip version of the file, e.g. "myfile.dat.gz" without needing to change too much.
Ideally I would implement a wrapper to fopen, call it fopen2, which can read both myfile.dat and myfile.dat.gz, i.e. it should return a pointer to a FILE object, so that the remaining of the code does not need to be changed.
Any suggestions?
Thank you.
PS: it would be fine to decompress the whole file in memory, if this approach provides a solution
zlib provides analogs of fopen(), fread(), etc. called gzopen(), gzread(), etc. for reading and writing gzip files. If the file is not gzip-compressed, it will be read just as the f functions would. So you would only need to change the function names and link in zlib.
I wanted to read contents off a .qcow2 image using bdrv_pread(...) functions in QEMU.
Say, the full path of my image is /path/to/myimage.qcow2, I want to be able to read 'n' bytes of data off of this image at a particular offset. Now the bdrv_pread functions takes these arguments 'BlockDriverState *bs, int64_t offset, void *buf, int count1', how exactly do I initialize the BlockDriverState (device?) from the path of the image. All other parameters other than BlockDriverState are clear to me.
Thanks.
If your goal is to access a qcow2 file from your own program I would recommend not trying to use the QEMU functions. These are going to have a lot of state associated with QEMU that is not necessary if all you want to do is read the contents of the qcow2 file. Instead you could look at the qcow2 specification or if you want to work at one higher level of abstraction you could look at the libguestfs library, which states that it has an API for accessing the supported VM disk formats (although I have never used it myself). There is some example code here that can help you get started.
Can anyone show me how to directly access metafiles ($MFT, $Volume, $Bitmap...) ?
I need to get info from these files.
You can use the ioctl function, FSCTL_GET_NTFS_FILE_RECORD.
I am trying to load a bitmap from an archive. The bitmap class I have takes a character pointer to a filename and then loads it if it is in the same directory. The bitmap loading class is well tested and I don't want to mess with it too much. Problem is it uses a file pointer to load and do all of its file manipulation. Is there any way to emulate a file pointer and actually have it read from a chunk in memory instead?
Sorry if this is a bizarre question.
Refactor it and create functions that takes the exact same parameters as before : If you used fopen, fread and fseek that read from disk, create mopen, mread and mseek that read file from memory. You'll only have to fix the name of the functions.
It should be easy without risk and code won't look like an dirty hack in the end.
You can also use a pipe. A pipe is a piece of memory where you can read and write using file primitives. Which is basically what you want
(Assuming POSIX Operating system)
create a pipe:
int p[2];
pipe(p);
use fdopen() to turn the pipe file descriptor into a FILE*
FILE *emulated_file = fdopen(p[0], "r");
then write whatever you want to the write end of the pipe :
write(p[1], 17 ,"whatevereyouwant");
Now :
buf[32];
fread(&buf,1,32, emulated_file);
cout<<buf<<endl;
willl output "whateveryouwant".
Check out John Ratcliff's File Interface replacement for standard file I/O. It supports the feature you need.
You'll still need to refactor the bitmap loading code to use the new interface. However, this interface supports loading from file on disk, or memory chunk in memory (as well as writing to file on disk, or to expandable memory chunks).