Can we use snappy file as input file to Map reduce application without using a Custom Input Class?
I was not able to find any information about this.
Regards,
Nish
Yes you can do that but you may have to decompress the file first to use it.
CompressionCodec codec = (CompressionCodec) ReflectionUtils.newInstance(SnappyCodec.class, new Configuration());
After creating an instance of codec, call createInputStream and pass the hadoop path.
codec.createInputStream
Related
I am trying to build a new layer using scapy. I want to read a pcap file and store each packet in an array so that I can access all the information inside the packet using the index of the array.
The scapy function that you're looking for is rdpcap
from scapy.all import *
packets = rdpcap('file.pcap')
packets[0].show()
As you can read everywhere Scape is SUPER slow for pcap files >10MB. I made a couple of tools always using dpkt. Just to give you an idea. Having a 670MB file using dpkt splitting the specific bytes I need compared to full fletched scapy (reading raw packages (the fastest solution with scapy)) is:
dpkt needs 150s (6GB ram), scapy >750s (>10GB)
I currently have some code reading files which are not compressed, it uses the following approach to read a file in C++
FILE* id = fopen("myfile.dat", "r");
after obtaining id, different parts of the code access the file using fread, fseek, etc.
I would like to adapt my code so as to open a gzip version of the file, e.g. "myfile.dat.gz" without needing to change too much.
Ideally I would implement a wrapper to fopen, call it fopen2, which can read both myfile.dat and myfile.dat.gz, i.e. it should return a pointer to a FILE object, so that the remaining of the code does not need to be changed.
Any suggestions?
Thank you.
PS: it would be fine to decompress the whole file in memory, if this approach provides a solution
zlib provides analogs of fopen(), fread(), etc. called gzopen(), gzread(), etc. for reading and writing gzip files. If the file is not gzip-compressed, it will be read just as the f functions would. So you would only need to change the function names and link in zlib.
Can anyone show me how to directly access metafiles ($MFT, $Volume, $Bitmap...) ?
I need to get info from these files.
You can use the ioctl function, FSCTL_GET_NTFS_FILE_RECORD.
Is it possible to change the frame rate of an avi file using the Video for windows library? I tried the following steps but did not succeed.
AviFileInit
AviFileOpen(OF_READWRITE)
pavi1 = AviFileGetStream
avi_info = AviStreamInfo
avi_info.dwrate = 15
EditStreamSetInfo(dwrate) returns -2147467262.
I'm pretty sure the AVIFile* APIs don't support this. (Disclaimer: I was the one who defined those APIs, but it was over 15 years ago...)
You can't just call EditStreamSetInfo on an plain AVIStream, only one returned from CreateEditableStream.
You could use AVISave, then, but that would obviously re-copy the whole file.
So, yes, you would probably want to do this by parsing the AVI file header enough to find the one DWORD you want to change. There are lots of documents on the RIFF and AVI file formats out there, such as http://www.opennet.ru/docs/formats/avi.txt.
I don't know anything about VfW, but you could always try hex-editing the file. The framerate is probably a field somewhere in the header of the AVI file.
Otherwise, you can script some tool like mencoder[1] to copy the stream to a new file under a different framerate.
[1] http://www.mplayerhq.hu/
HRESULT: 0x80004002 (2147500034)
Name: E_NOINTERFACE
Description: The requested COM interface is not available
Severity code: Failed
Facility Code: FACILITY_NULL (0)
Error Code: 0x4002 (16386)
Does it work if you DON'T call EditStreamSetInfo?
Can you post up the code you use to set the stream info?
do you have a working code to share.
I’m trying to figure out how to save to a file an IBitmapImage image.
I need to resize existing .jpg file and it seems like the only API for Windows Mobile. I managed to load it convert it to IImage -> IBitmapImage -> IBasicBitmapOps and resize it finally, but I have no clue how to save it properly to a new file.
Use IBitmapImage::LockBits to get access to the image data via its BitmapData* lockedBitmapData parameter. Use the BitmapData to prepare a bitmap file info header, then write that one and the image data in BitmapData::Scan0 to a file using regular file writing with ::WriteFile or higher level ones if you use such.