Related
I use fuse to build my own file system in MIT 6.824 lab, and the read operation is implemented in this function.
void
fuseserver_read(fuse_req_t req, fuse_ino_t ino, size_t size,
off_t off, struct fuse_file_info *fi)
{
std::string buf;
int r;
if ((r = yfs->read(ino, size, off, buf)) == yfs_client::OK) {
char* retbuf = (char *)malloc(buf.size());
memcpy(retbuf,buf.data(),buf.size());
//Print the information of the result.
printf("debug read in fuse: the content of %lu is %s, size %lu\n",ino,retbuf, buf.size());
fuse_reply_buf(req,retbuf,buf.size());
} else {
fuse_reply_err(req, ENOENT);
}
//global definition
//struct fuse_lowlevel_ops fuseserver_oper;
//In main()
// fuseserver_oper.read = fuseserver_read;
I print the information of the buf before it return.
The write operation is also implemented, of course.
Then I run a simple test to read out some words.
//test.c
int main(){
//./yfs1 is the mount point of my filesystem
int fd = open("./yfs1/test-file",O_RDWR | O_CREAT,0777);
char* buf = "123";
char* readout;
readout = (char *)malloc(3);
int writesize = write(fd,buf,3);
int readsize = read(fd,readout,3);
printf("%s,%d\n",buf,writesize);
printf("%s,%d\n",readout,readsize);
close(fd);
}
I can get nothing by read(fd,readout,3), but the information printed by the fuseserver_read shows that the buffer is read out successfully before fuse_reply_buf
$ ./test
123,3
,0
debug read in fuse: the content of 2 is 123, size 3
So why the read() in test.c can not read anything from my file system??
Firstly, I've made a mistake to write my test file. The file pointer will point to the end of the file after "write" and of course can read nothing later. So simply reopen the file can make the test work.
Secondly, before read() operation of FUSE, the FUSE will getattr() first and truncate the result of the read() operation with the "size" attribute of the file. So it must be very careful to manipulate the attribute of a file.
There is also a need to notify that you have finished reading by sending an empty buffer, as an "EOF". You can do that by using reply_buf_limited.
Take a look at hello_ll example in the fuse source tree:
static void tfs_read(fuse_req_t req, fuse_ino_t ino, size_t size,
off_t off, struct fuse_file_info *fi) {
(void) fi;
assert(ino == FILE_INO);
reply_buf_limited(req, file_contents, file_size, off, size);
}
static int reply_buf_limited(fuse_req_t req, const char *buf, size_t bufsize,
off_t off, size_t maxsize)
{
if (off < bufsize)
return fuse_reply_buf(req, buf + off,
min(bufsize - off, maxsize));
else
return fuse_reply_buf(req, NULL, 0);
}
I'm getting a strange file handle from fopen; the pointer itself isn't NULL, but the file it represents has no size, and feof(file) is already set... what could be causing this?
(I triple checked the file itself, it exists, has data, and the file's permissions are okay... and filename is set to "source/test_file.object.txt")
bool tagFile(const char * filename){
FILE * file = fopen(filename, "r");
if(file == NULL){ // file is not NULL
printf(" Couldn't open the file %s", filename);
return false;
}
int size = fseek(file, 0, SEEK_END); // size is 0
rewind(file);
while(feof(file)){
.... // never executes because feof(file) fails
}
}
0x002cc410 // value of 'file' in the debugger
_ptr=0x00000000 <Bad Ptr>
_cnt=0
_base=0x00000000 <Bad Ptr>
_flag = 1
_file = 3
The most common cause of this is simply opening a file which is empty. There are no bytes to read so EOF is immediate, but the file exists so fopen can't return NULL.
I using libzip to work with zip files and everything goes fine, until i need to read file from zip
I need to read just a whole text files, so it will be great to achieve something like PHP "file_get_contents" function.
To read file from zip there is a function "int
zip_fread(struct zip_file *file, void *buf, zip_uint64_t nbytes)".
Main problem what i don't know what size of buf must be and how many nbytes i must read (well i need to read whole file, but files have different size). I can just do a big buffer to fit them all and read all it's size, or do a while loop until fread return -1 but i don't think it's rational option.
You can try using zip_stat to get file size.
http://linux.die.net/man/3/zip_stat
I haven't used the libzip interface but from what you write it seems to look very similar to a file interface: once you got a handle to the stream you keep calling zip_fread() until this function return an error (ir, possibly, less than requested bytes). The buffer you pass in us just a reasonably size temporary buffer where the data is communicated.
Personally I would probably create a stream buffer for this so once the file in the zip archive is set up it can be read using the conventional I/O stream methods. This would look something like this:
struct zipbuf: std::streambuf {
zipbuf(???): file_(???) {}
private:
zip_file* file_;
enum { s_size = 8196 };
char buffer_[s_size];
int underflow() {
int rc(zip_fread(this->file_, this->buffer_, s_size));
this->setg(this->buffer_, this->buffer_,
this->buffer_ + std::max(0, rc));
return this->gptr() == this->egptr()
? traits_type::eof()
: traits_type::to_int_type(*this->gptr());
}
};
With this stream buffer you should be able to create an std::istream and read the file into whatever structure you need:
zipbuf buf(???);
std::istream in(&buf);
...
Obviously, this code isn't tested or compiled. However, when you replace the ??? with whatever is needed to open the zip file, I'd think this should pretty much work.
Here is a routine I wrote that extracts data from a zip-stream and prints out a line at a time. This uses zlib, not libzip, but if this code is useful to you, feel free to use it:
#
# compile with -lz option in order to link in the zlib library
#
#include <zlib.h>
#define Z_CHUNK 2097152
int unzipFile(const char *fName)
{
z_stream zStream;
char *zRemainderBuf = malloc(1);
unsigned char zInBuf[Z_CHUNK];
unsigned char zOutBuf[Z_CHUNK];
char zLineBuf[Z_CHUNK];
unsigned int zHave, zBufIdx, zBufOffset, zOutBufIdx;
int zError;
FILE *inFp = fopen(fName, "rbR");
if (!inFp) { fprintf(stderr, "could not open file: %s\n", fName); return EXIT_FAILURE; }
zStream.zalloc = Z_NULL;
zStream.zfree = Z_NULL;
zStream.opaque = Z_NULL;
zStream.avail_in = 0;
zStream.next_in = Z_NULL;
zError = inflateInit2(&zStream, (15+32)); /* cf. http://www.zlib.net/manual.html */
if (zError != Z_OK) { fprintf(stderr, "could not initialize z-stream\n"); return EXIT_FAILURE; }
*zRemainderBuf = '\0';
do {
zStream.avail_in = fread(zInBuf, 1, Z_CHUNK, inFp);
if (zStream.avail_in == 0)
break;
zStream.next_in = zInBuf;
do {
zStream.avail_out = Z_CHUNK;
zStream.next_out = zOutBuf;
zError = inflate(&zStream, Z_NO_FLUSH);
switch (zError) {
case Z_NEED_DICT: { fprintf(stderr, "Z-stream needs dictionary!\n"); return EXIT_FAILURE; }
case Z_DATA_ERROR: { fprintf(stderr, "Z-stream suffered data error!\n"); return EXIT_FAILURE; }
case Z_MEM_ERROR: { fprintf(stderr, "Z-stream suffered memory error!\n"); return EXIT_FAILURE; }
}
zHave = Z_CHUNK - zStream.avail_out;
zOutBuf[zHave] = '\0';
/* copy remainder buffer onto line buffer, if not NULL */
if (zRemainderBuf) {
strncpy(zLineBuf, zRemainderBuf, strlen(zRemainderBuf));
zBufOffset = strlen(zRemainderBuf);
}
else
zBufOffset = 0;
/* read through zOutBuf for newlines */
for (zBufIdx = zBufOffset, zOutBufIdx = 0; zOutBufIdx < zHave; zBufIdx++, zOutBufIdx++) {
zLineBuf[zBufIdx] = zOutBuf[zOutBufIdx];
if (zLineBuf[zBufIdx] == '\n') {
zLineBuf[zBufIdx] = '\0';
zBufIdx = -1;
fprintf(stdout, "%s\n", zLineBuf);
}
}
/* copy some of line buffer onto the remainder buffer, if there are remnants from the z-stream */
if (strlen(zLineBuf) > 0) {
if (strlen(zLineBuf) > strlen(zRemainderBuf)) {
/* to minimize the chance of doing another (expensive) malloc, we double the length of zRemainderBuf */
free(zRemainderBuf);
zRemainderBuf = malloc(strlen(zLineBuf) * 2);
}
strncpy(zRemainderBuf, zLineBuf, zBufIdx);
zRemainderBuf[zBufIdx] = '\0';
}
} while (zStream.avail_out == 0);
} while (zError != Z_STREAM_END);
/* close gzip stream */
zError = inflateEnd(&zStream);
if (zError != Z_OK) {
fprintf(stderr, "could not close z-stream!\n");
return EXIT_FAILURE;
}
if (zRemainderBuf)
free(zRemainderBuf);
fclose(inFp);
return EXIT_SUCCESS;
}
With any streaming you should consider the memory requirements of your app.
A good buffer size is large, but you do not want to have too much memory in use depending on your RAM usage requirements. A small buffer size will require you call your read and write operations more times which are expensive in terms of time performance. So, you need to find a buffer in the middle of those two extremes.
Typically I use a size of 4096 (4KB) which is sufficiently large for many purposes. If you want, you can go larger. But at the worst case size of 1 byte, you will be waiting a long time for you read to complete.
So to answer your question, there is no "right" size to pick. It is a choice you should make so that the speed of your app and the memory it requires are what you need.
I have this code that basically reads from file and creates new file and write the content from the source to the destination file. It reads the buffer and creates the file, but fwrite
doesn't write the content to the newly created file, I have no idea why.
here is the code. (I have to use only this with _sopen, its part of legacy code)
#include <stdio.h>
#include <stdlib.h>
#include <io.h>
#include <fcntl.h>
#include <string>
#include <share.h>
#include <sys\stat.h>
int main () {
std::string szSource = "H:\\cpp\\test1.txt";
FILE* pfFile;
int iFileId = _sopen(szSource.c_str(),_O_RDONLY, _SH_DENYNO, _S_IREAD);
if (iFileId >= 0)
pfFile = fdopen(iFileId, "r");
//read file content to buffer
char * buffer;
size_t result;
long lSize;
// obtain file size:
fseek (pfFile , 0 , SEEK_END);
lSize = ftell (pfFile);
fseek(pfFile, 0, SEEK_SET);
// buffer = (char*) malloc (sizeof(char)*lSize);
buffer = (char*) malloc (sizeof(char)*lSize);
if (buffer == NULL)
{
return false;
}
// copy the file into the buffer:
result = fread (buffer,lSize,1,pfFile);
std::string szdes = "H:\\cpp\\test_des.txt";
FILE* pDesfFile;
int iFileId2 = _sopen(szdes.c_str(),_O_CREAT,_SH_DENYNO,_S_IREAD | _S_IWRITE);
if (iFileId2 >= 0)
pDesfFile = fdopen(iFileId2, "w+");
size_t f = fwrite (buffer , 1, sizeof(buffer),pDesfFile );
printf("Error code: %d\n",ferror(pDesfFile));
fclose (pDesfFile);
return 0;
}
You can make main file and try it see if its working for you .
Thanks
Change your code to the following and then report your results:
int main () {
std::string szSource = "H:\\cpp\\test1.txt";
int iFileId = _sopen(szSource.c_str(),_O_RDONLY, _SH_DENYNO, _S_IREAD);
if (iFileId >= 0)
{
FILE* pfFile;
if ((pfFile = fdopen(iFileId, "r")) != (FILE *)NULL)
{
//read file content to buffer
char * buffer;
size_t result;
long lSize;
// obtain file size:
fseek (pfFile , 0 , SEEK_END);
lSize = ftell (pfFile);
fseek(pfFile, 0, SEEK_SET);
if ((buffer = (char*) malloc (lSize)) == NULL)
return false;
// copy the file into the buffer:
result = fread (buffer,(size_t)lSize,1,pfFile);
fclose(pfFile);
std::string szdes = "H:\\cpp\\test_des.txt";
FILE* pDesfFile;
int iFileId2 = _sopen(szdes.c_str(),_O_CREAT,_SH_DENYNO,_S_IREAD | _S_IWRITE);
if (iFileId2 >= 0)
{
if ((pDesfFile = fdopen(iFileId2, "w+")) != (FILE *)NULL)
{
size_t f = fwrite (buffer, (size_t)lSize, 1, pDesfFile);
printf ("elements written <%d>\n", f);
if (f == 0)
printf("Error code: %d\n",ferror(pDesfFile));
fclose (pDesfFile);
}
}
}
}
return 0;
}
[edit]
for other posters, to show the usage/results of fwrite - what is the output of the following?
#include <stdio.h>
int main (int argc, char **argv) {
FILE *fp = fopen ("f.kdt", "w+");
printf ("wrote %d\n", fwrite ("asdf", 4, 1, fp));
fclose (fp);
}
[/edit]
sizeof(buffer) is the size of the pointer, i.e. 4 and not the number of items in the buffer
If buffer is an array then sizeof(buffer) would potentially work as it returns the number of bytes in the array.
The third parameter to fwrite is sizeof(buffer) which is 4 bytes (a pointer). You need to pass in the number of bytes to write instead (lSize).
Update: It also looks like you're missing the flag indicating the file should be Read/Write: _O_RDWR
This is working for me...
std::string szdes = "C:\\temp\\test_des.txt";
FILE* pDesfFile;
int iFileId2;
err = _sopen_s(&iFileId2, szdes.c_str(), _O_CREAT|_O_BINARY|_O_RDWR, _SH_DENYNO, _S_IREAD | _S_IWRITE);
if (iFileId2 >= 0)
pDesfFile = _fdopen(iFileId2, "w+");
size_t f = fwrite (buffer , 1, lSize, pDesfFile );
fclose (pDesfFile);
Since I can't find info about _sopen, I can only look at man open. It reports:
int open(const char *pathname, int flags);
int open(const char *pathname, int flags, mode_t mode);
Your call _sopen(szdes.c_str(),_O_CREAT,_SH_DENYNO,_S_IREAD | _S_IWRITE); doesn't match either one of those, you seem to have flags and 'something' and modes / what is SH_DENY?
What is the result of man _sopen?
Finally, shouldn't you close the file descriptor from _sopen after you fclose the file pointer?
Your final lines should look like this, btw :
if (iFileId2 >= 0)
{
pDesfFile = fdopen(iFileId2, "w+");
size_t f = fwrite (buffer , 1, sizeof(buffer),pDesfFile ); //<-- the f returns me 4
fclose (pDesfFile);
}
Since you currently write the file regardless of whether or not the fdopen after the O_CREAT succeeded. You also do the same thing at the top, you process the read (and the write) regardless of the success of the fdopen of the RDONLY file :(
You are using a mixture of C and C++. That is confusing.
The sizeof operator does not do what you expect it to do.
Looks like #PJL and #jschroedl found the real problem, but also in general:
Documentation for fwrite states:
fwrite returns the number of full items actually written, which may be less than count if an error occurs. Also, if an error occurs, the file-position indicator cannot be determined.
So if the return value is less than the count passed, use ferror to find out what happened.
The ferror routine (implemented both as a function and as a macro) tests for a reading or writing error on the file associated with stream. If an error has occurred, the error indicator for the stream remains set until the stream is closed or rewound, or until clearerr is called against it.
I have embedded a custom stream in a dump (i.e. passed the UserStreamParam argument to MiniDumpWriteDump function). Now, I'm trying to extract the stream from a WinDbg extension. (Note that I have verified that I can retrieve the stream using the MiniDumpReadDumpStream function).
I'm using the IDebugAdvanced2::Request method with DEBUG_REQUEST_READ_USER_MINIDUMP_STREAM request. I am able to retrieve data from standard streams. For example, the following snippet will correctly retrieve the contents of the misc info stream.
DEBUG_READ_USER_MINIDUMP_STREAM rums = {};
rums.StreamType = MiscInfoStream;
rums.Buffer = &buf;
rums.BufferSize = sizeof buf;
hr = p->Request(DEBUG_REQUEST_READ_USER_MINIDUMP_STREAM,
&rums, sizeof rums, 0, 0, 0);
However, trying to retrieve my own stream will result in an error (0x80070570, ERROR_FILE_CORRUPT) and WinDbg outputs
Dir entry 11, ??? stream has unknown stream type 6381921
Note that the same message appears as a part of the .dumpdebug output.
Stream 11: type ??? (6381921), size 00000038, RVA 00033FA9
Dir entry 11, ??? stream has unknown stream type 6381921
What is the problem? How do I retrieve contents of my user stream?
very late answer
StreamType cannot be UserDefined StreamTypes
jen-lung chiu of ms posted so in osronline windbg lists long back
do not know if the latest dbgeng has this limitation eliminated
you either retrieve it with a dbghelp function independently
(using dbghelp functions inside windbg extensions are not recommended )
or parse the stream yourself with fopen() fread() like below
userstream:\>type ..\usrstr.cpp
#include <stdio.h>
#include <engextcpp.hpp>
#include <dbghelp.h>
const ULONG MBUFFSIZE = 0x1000;
PVOID Buff = 0;
int __cdecl ReadUserStream (char *dmpfile)
{
PMINIDUMP_HEADER MiniHeader = 0;
PMINIDUMP_DIRECTORY MiniDir = 0;
PMINIDUMP_USER_STREAM userstream = 0;
size_t result = 0;
ULONG Streams =0;
ULONG i = 0;
FILE * fp = fopen(dmpfile,"rb");
if (fp)
{
result = fread(Buff, 1, sizeof(MINIDUMP_HEADER), fp );
if ( result == sizeof(MINIDUMP_HEADER) )
{
MiniHeader = (PMINIDUMP_HEADER) Buff;
Streams = MiniHeader->NumberOfStreams;
for (i = 0; i < Streams; i++ )
{
result = fread( Buff, 1, sizeof(MINIDUMP_DIRECTORY), fp );
if ( result == sizeof(MINIDUMP_DIRECTORY) )
{
MiniDir = (PMINIDUMP_DIRECTORY) Buff;
if ( MiniDir->StreamType > LastReservedStream )
{
userstream = (PMINIDUMP_USER_STREAM)Buff;
ULONG savedbuffsize = userstream->BufferSize;
ULONG savedtype = userstream->Type;
PCHAR savedbufferptr = (PCHAR)userstream->Buffer;
long pos = ftell(fp);
fseek(fp, (long)savedbufferptr,SEEK_SET);
result = fread( Buff, 1, savedbuffsize, fp );
if ( result == savedbuffsize )
{
printf(
"\n"
"Datastream Type = %.8x\n"
"Buffer Size = %.8x\n"
"Buffer = %p\n"
"Buffer content = %s\n"
"\n",
savedtype,
savedbuffsize,
savedbufferptr,
Buff
);
fseek(fp,pos,SEEK_SET);
continue;
}
else
{
printf(
"failed to read buffer contents at offset %p of
user stream %x\n",
savedbufferptr,
savedtype);
fseek(fp,pos,SEEK_SET);
continue;
}
}
}
else
{
printf("failed to fread Minidump directory exiting \n");
goto getout;
}
}
}
else
{
printf("failed to fread Minidump header exiting \n");
goto getout;
}
}
else
{
printf("failed to open dmp file exiting \n");
goto getout;
}
getout:
if (fp)
fclose(fp);
return 0;
}
int __cdecl main (int argc, char * argv[])
{
if (argc !=2)
{
printf("Usage %s %s\n",argv[0],"somedump.dmp");
return 0;
}
Buff = malloc( MBUFFSIZE );
if (Buff)
{
ReadUserStream(argv[1]);
free(Buff);
return 0;
}
else
{
printf("malloc failed exiting\n");
return 0;
}
}
output from an userdump that has userStreams in it
(oleg staradumov debuginfo.com writeuserstream.cpp )
userstream:\>usrstr.exe
Usage usrstr.exe somedump.dmp
userstream:\>usrstr.exe test.dmp
Datastream Type = 00010000
Buffer Size = 00000021
Buffer = 000010B6
Buffer content = This is the first data stream...
Datastream Type = 00010001
Buffer Size = 00000023
Buffer = 000010D7
Buffer content = and this is the second data stream
I found this topic while looking for a method to read out user stream from the dbg file.
#blabb 's answer is correct in basics and helped me a lot, but it has a two flaws:
You should use the MINIDUMP_HEADER.StreamDirectoryRva to locate the MINIDUMP_DIRECTORY list.
You should not convert the MINIDUMP_DIRECTORY entry to MINIDUMP_USER_STREAM, because that's an error (MINIDUMP_USER_STREAM is a bigger struct than MINIDUMP_DIRECTORY, so you are reading uninitialized memory there) Use the MINIDUMP_DIRECTORY to locate the needed part of the stream.
Even if not tested, it should work if you fill StreamType with a custom value (greater than LastReservedStream = 0xFFFF) instead of MiscInfoStream.