I'm writing a small utility to parse xfs filesystem on windows.
For small size image <5GB my utility is working fine: I'm able to list all files and directories.
But when I try to parse large xfs image > 30GB. It is giving wrong results.
I'm using _fseeki64 and _ftelli64 for seeking and fread for reading particular block.
One of thing I've notice is that _fseeki64 is not wroking properly.
Below is my seek function which seek to particular group number and block number.
int FileSystemReadXFS::SeekToGroupBlock(uint16_t grpNum, uint64_t blockNum)
{
int error = -1;
//Seek to beginning
if(_fseeki64(m_fileSystemInfo.fp, (__int64)0, SEEK_SET) != 0)
{
PRINT_SEEK_ERROR;
goto BAILOUT;
}
__int64 currPtr = 0;
//Seek to destination group
if(grpNum > 0)
{
if(_fseeki64(m_fileSystemInfo.fp, (__int64)(grpNum*m_fileSystemInfo.SizeOfBlockGroup*m_fileSystemInfo.BlockSize), SEEK_SET))
{
PRINT_SEEK_ERROR;
goto BAILOUT;
}
currPtr = _ftelli64(m_fileSystemInfo.fp);
}
//Seek to destination block in group
if(blockNum > 0)
{
if(_fseeki64(m_fileSystemInfo.fp, (__int64)(blockNum*m_fileSystemInfo.BlockSize), SEEK_CUR))
{
PRINT_SEEK_ERROR;
goto BAILOUT;
}
currPtr = _ftelli64(m_fileSystemInfo.fp);
}
error = 0;
BAILOUT:
return error;
}
However the above function is taking me to wrong postion.
For example when I want to seek to group number = 2 with m_fileSystemInfo.SizeOfBlockGroup = 2043982
and m_fileSystemInfo.BlockSize = 4096.
I'm expecting currPrt = 2043982*4096*2 = 16744300544 (0x3E609C000), But _ftelli64 is returning (0xE609C000).
Please suggest what could be going wrong. Also please advice what is the best way to deal with large files on windows in c++.
Update::
I found that the seekOffset is getting limited to 8154365952 (0x1e609c000) instead of actual
value of 16744300544 (0x3e609c000) inspite of me using __int64.
So.
_int64 seekOff = (_int64)(grpNum*m_fileSystemInfo.SizeOfBlockGroup*m_fileSystemInfo.BlockSize)
= 2*2043982*4096
is giving 8154365952 instead of 16744300544.
I'm not sure what could the reason as everything is in __int64.
Apparently the problem was with calculation of seek offset. It was generating integer overflow.
Hence I have to convert everything to __int64 even-though I'm working on 64 bits application. I was thinking compiler might be able to do this for me.
__int64 grpNum = 2;
__int64 sizeOfBlockGroup = 2043982;
__int64 blockSize = 4096;
__int64 seekOffSet = grpNum*sizeOfBlockGroup*blockSize;
which works fine with _fseeki64 and __ftelli64.
Your best bet is to use Win32 API directly and not go through the C RunTime.
Use CreateFile to open the file and SetFilePointerEx to seek
The functions you're calling will end up calling these APIs anyway. In Visual studio you have the CRT code so you can step into _fseeki64 and possibly see where it goes wrong.
Related
Currently, I am working on real time interface with Visual Studio C++.
I faced problem is, when buffer is running for data store, that time .exe is not responding at the point data store in buffer. I collect data as 130Hz from motion sensor. I have tried to increase virtual memory of computer, but problem was not solved.
Code Structure:
int main(){
int no_data = 0;
float x_abs;
float y_abs;
int sensorID = 0;
while (1){
// Define Buffer
char before_trial_output_data[][8 * 4][128] = { { { 0, }, }, };
// Collect Real Time Data
x_abs = abs(inchtocm * record[sensorID].y);
y_abs = abs(inchtocm * record[sensorID].x);
//Save in buffer
sprintf(before_trial_output_data[no_data][sensorID], "%d %8.3f %8.3f\n",no_data,x_abs,y_abs);
//Increment point
no_data++;
// Break While loop, Press ESc key
if (GetAsyncKeyState(VK_ESCAPE)){
break;
}
}
//Data Save in File
printf("\nSaving results to 'RecordData.txt'..\n");
FILE *fp3 = fopen("RecordData.dat", "w");
for (i = 0; i<no_data-1; i++)
fprintf(fp3, output_data[i][sensorID]);
fclose(fp3);
printf("Complete...\n");
}
The code you posted doesn't show how you allocate more memory for your before_trial_output_data buffer when needed. Do you want me to guess? I guess you are using some flavor of realloc(), which needs to allocate ever-increasing amount of memory, fragmenting your heap terribly.
However, in order for you to save that data to a file later on, it doesn't need to be in continuous memory, so some kind of list will work way better than an array.
Also, there is no provision in your "pseudo" code for a 130Hz reading; it processes records as fast as possible, and my guess is - much faster.
Is your prinf() call also a "pseudo code"? Otherwise you are looking for trouble by having mismatch of the % format specifications and number and type of parameters passed in.
I wonder why this code doesn't work:
#include <iostream>
using namespace std;
int main()
{
int *pointer = (int*)0x02F70BCC;
cout<<*pointer;
return 0;
}
In my opinion it should write on the screen value of 0x02F70BCC,
instead of this my programm crashes.
I know that memory with adress 0x02F70BCC stores value of 20.
But like I said no matter what it just doesn't want to show correct number.
Please help me guys, detailed explanation would be very nice of you.
It doesn't work, because you won't get access to every location in memory you want. Not every location in memory is valid, you may want to read about Virtual Address Space.
Some addresses are reserved for device drivers and kernel mode operations. Another range of addresses (for example 0xCCCCCCCC and higher) may be reserved for uninitialized pointers.
Even if some location is valid, operating system may still deny access to write to/read from certain location, if that would cause undefined behaviour or violate system safety.
EDIT
I think you might be interested in creating some kind of "GameHack", that allows you to modify amount of resources, number of units, experience level, attributes or anything.
Memory access is not a simple topic. Different OSes use different strategies to prevent security violations. But many thing can be done here, after all there is a lot software for doing such things.
First of all, do you really need to write your own tool? If you just want some cheating, use ArtMoney - it is a great memory editor, that I have been using for years.
But if you really have to write it manually, you need to do some research first.
On Windows, for example, I would start from these:
ReadProcessMemory
WriteProcessMemory
Also, I am quite certain, that one of possible techniques is to pretend, that you are a debugger:
DebugActiveProcess.
EDIT 2
I have done some research and it looks, that on Windows (I assume this is your platform, since you mentioned gaming; can't imagine playing anything on crappy Linux), steps required to write another process' memory are:
1. Enumerate processes: (EnumProcesses)
const size_t MAX_PROC_NUM = 512;
DWORD procIDs[MAX_PROC_NUM] = { 0 };
DWORD idsNum = 0;
if(!EnumProcesses(procIDs, sizeof(DWORD) * MAX_PROC_NUM, &idsNum))
//handle error here
idsNum /= sizeof(DWORD); //After EnumProcesses(), idsNum contains number of BYTES!
2. Open required process. (OpenProcess,GetModuleFileNameEx)
const char* game_exe_path = "E:\\Games\\Spellforce\\Spellforce.exe"; //Example
HANDLE game_proc_handle = nullptr;
DWORD proc_access = PROCESS_QUERY_INFORMATION | PROCESS_VM_READ | PROCESS_VM_WRITE; //read & write memory, query info needed to get .exe name
const DWORD MAX_EXE_PATH_LEN = 1024;
for(DWORD n = 0 ; n < idsNum ; ++idsNum)
{
DWORD current_id = procIDs[n];
HANDLE current_handle = OpenProcess(proc_access, false, current_id);
if(!current_handle)
{
//handle error here
continue;
}
char current_path[MAX_EXE_PATH_LEN];
DWORD length = GetModuleFileNameEx(current_handle, nullptr, current_path, MAX_EXE_PATH_LEN);
if(length > 0)
{
if(strcmp(current_path, game_exe_path) == 0) //that's our game!
{
game_proc_handle = current_handle;
break;
}
}
CloseHandle(current_handle); //don't forget this!
}
if(!game_proc_handle)
//sorry, game not found
3. Write memory (WriteProcessMemory)
void* pointer = reinterpret_cast<void*>(0x02F70BCC);
int new_value = 5000; //value to be written
BOOL success = WriteProcessMemory(game_proc_handle, pointer, &new_value, sizeof(int), nullptr);
if(success)
//data successfully written!
else
//well, that's... em...
This code is written just 'as is', but I see no errors, so you can use it as your starting point. I also provided links for all functions I used, so with some additional research (if necessary), you can achieve what you are trying to.
Cheers.
When you use,
cout<<*pointer;
the program tries to dereference the value of the pointer and writes the value at the address.
If you want to print just the pointer, use:
cout << pointer;
Example:
int main()
{
int i = 20;
int* p = &i;
std::cout << *p << std::endl; // print the value stored at the address
// pointed to by p. In this case, it will
// print the value of i, which is 20
std::cout << p << std::endl; // print the address that p points to
// It will print the address of i.
}
I have no experience in audio programming and C++ is quite low level language so I have a little problems with it. I work with ASIO SDK 2.3 downloaded from http://www.steinberg.net/en/company/developers.html.
I am writing my own host based on example inside SDK.
For now I've managed to go through the whole sample and it looks like it's working. I have external sound card connected to my PC. I've successfully loaded driver for this device, configured it, handled callbacks, casting data from analog to digital etc. common stuff.
And part where I am stuck now:
When I play some track via my device I can see bars moving in the mixer (device's software). So device is connected in right way. In my code I've picked the inputs and outputs with the names of the bars that are moving in mixer. I've also used ASIOCreateBuffers() to create buffer for each input/output.
Now correct me if I am wrong:
When ASIOStart() is called and driver is in running state, when I input the sound signal to my external device I believe the buffers get filled with data, right?
I am reading the documentation but I am a bit lost - how can I access the data being sent by device to application, stored in INPUT buffers? Or signal? I need it for signal analysis or maybe recording in future.
EDIT: If I had made it to complicated then in a nutshell my question is: how can I access input stream data from code? I don't see any objects/callbacks letting me to do so in documentation.
The hostsample in the ASIO SDK is pretty close to what you need. In the bufferSwitchTimeInfo callback there is some code like this:
for (int i = 0; i < asioDriverInfo.inputBuffers + asioDriverInfo.outputBuffers; i++)
{
int ch = asioDriverInfo.bufferInfos[i].channelNum;
if (asioDriverInfo.bufferInfos[i].isInput == ASIOTrue)
{
char* buf = asioDriver.bufferInfos[i].buffers[index];
....
Inside of that if block asioDriver.bufferInfos[i].buffers[index] is a pointer to the raw audio data (index is a parameter to the method).
The format of the buffer is dependent upon the driver and that can be discovered by testing asioDriverInfo.channelInfos[i].type. The types of formats will be 32bit int LSB first, 32bit int MSB first, and so on. You can find the list of values in the ASIOSampleType enum in asio.h. At this point you'll want to convert the samples to some common format for downstream signal processing code. If you're doing signal processing you'll probably want convert to double. The file host\asioconvertsample.cpp will give you some idea of what's involved in the conversion. The most common format you're going to encounter is probably INT32 MSB. Here is how you'd convert it to double.
for (int i = 0; i < asioDriverInfo.inputBuffers + asioDriverInfo.outputBuffers; i++)
{
int ch = asioDriverInfo.bufferInfos[i].channelNum;
if (asioDriverInfo.bufferInfos[i].isInput == ASIOTrue)
{
switch (asioDriverInfo.channelInfos[i].type)
{
case ASIOInt32LSB:
{
double* pDoubleBuf = new double[_bufferSize];
for (int i = 0 ; i < _bufferSize ; ++i)
{
pDoubleBuf[i] = *(int*)asioDriverInfo.bufferInfos.buffers[index] / (double)0x7fffffff;
}
// now pDoubleBuf contains one channels worth of samples in the range of -1.0 to 1.0.
break;
}
// and so on...
Thank you very much. Your answer helped quite much but as I am inexperienced with C++ a bit :P I find it a bit problematic.
In general I've written my own host based on hostsample. I didn't implement asioDriverInfo structure and use common variables for now.
My first problem was:.
char* buf = asioDriver.bufferInfos[i].buffers[index];
as I got error that I can't cast (void*) to char* but this probably solved the problem:
char* buf = static_cast<char*>(bufferInfos[i].buffers[doubleBufferIndex]);
My second problem is with the data conversion. I've checked the file you've recommended me but I find it a little black magic. For now I am trying to follow your example and:
for (int i = 0; i < inputBuffers + outputBuffers; i++)
{
if (bufferInfos[i].isInput)
{
switch (channelInfos[i].type)
{
case ASIOSTInt32LSB:
{
double* pDoubleBuf = new double[buffSize];
for (int j = 0 ; j < buffSize ; ++j)
{
pDoubleBuf[j] = bufferInfos[i].buffers[doubleBufferIndex] / (double)0x7fffffff;
}
break;
}
}
}
I get error there:
pDoubleBuf[j] = bufferInfos[i].buffers[doubleBufferIndex] / (double)0x7fffffff;
which is:
error C2296: '/' : illegal, left operand has type 'void *'
What I don't get is that in your example there is no table there: asioDriverInfo.bufferInfos.buffers[index] after bufferInfos and even if I fix it... to what kind of type should I cast it to make it work. P
PS. I am sure ASIOSTInt32LSB data type is fine for my PC.
The ASIO input and output buffers are accessible using void pointers, but using memcpy or memmove to access I/O buffer will create a memory copy which is to be avoided if you are doing real-time processing. I would suggest casting the pointer type to int* so you can directly access them.
It's also very slow in real-time processing to cast types 1 by 1 when you have like 100+ audio channels when AVX2 is supported on most CPUs.
_mm256_loadu_si256() and _mm256_cvtepi32_ps() will do the conversion much faster.
I went through this blog and this videocast. In Windows if I want to retrieve information about a range of pages within the virtual address space of a specified process, I can use WinAPI VirtualQueryEx method:
MEMORY_BASIC_INFORMATION meminfo;
unsigned char *addr = 0;
for(;;)
{
if(!VirtualQueryEx(hProc, addr, &meminfo, sizeof(meminfo)))
break;
if(meminfo.State & MEM_COMMIT)
{
//collect some data from meminfo
}
addr = (unsigned char*)meminfo.BaseAddress + meminfo.RegionSize;
}
I wondered how to get similar set of information in Linux using syscalls, but it is not clear for me how using C/C++ can I gather such a data under Linux. I went through this thread when there are suggestions to take a look at /proc/<pid>/mem or /proc/<pid>/maps files. Is it the good direction? How should look the closest implementation to this one provided here, but for Linux?
Yes, the proc filesystem is part of the Linux API, so this is the way to go. A lot of data in that filesystem is usually accessed using a library wrapper, but that's where the data lie.
As far as I know /proc/<pid>/maps is the only reliable and supported way to do it. Even libunwind is using it:
if (maps_init (&mi, getpid()) < 0)
return -1;
unsigned long offset;
while (maps_next (&mi, &low, &hi, &offset)) {
struct dl_phdr_info info;
info.dlpi_name = mi.path;
info.dlpi_addr = low;
I'm currently re-creating a memory modifier application using C++, the original was in C#.
All credit goes to "gimmeamilk" who's tutorials Ive been following on YouTube(video 1 of 8). I would highly recommend these tutorials for anyone attempting to create a similar application.
The problem I have is that my VirtualQueryEx seems to run forever. The process I'm scanning is "notepad.exe" and I am passing to the application via command line parameter.
std::cout<<"Create scan started\n";
#define WRITABLE (PAGE_READWRITE | PAGE_WRITECOPY | PAGE_EXECUTE_READWRITE | PAGE_EXECUTE_WRITECOPY) //These are all the flags that will be used to determin if a memory block is writable.
MEMBLOCK * mb_list = NULL; //pointer to the head of the link list to be returned
MEMORY_BASIC_INFORMATION meminfo; //holder for the VirtualQueryEx return struct
unsigned char *addr = 0; //holds the value to pass to VirtualQueryEx
HANDLE hProc = OpenProcess(PROCESS_ALL_ACCESS,false, pid);
if(hProc)
{
while(1)
{
if(VirtualQueryEx(hProc,addr, &meminfo, sizeof(meminfo)) == 0)
{
break;
}
if((meminfo.State & MEM_COMMIT) && (meminfo.Protect & WRITABLE)) //((binary comparison of meminfos state and MEM_COMMIT, this is basically filtering out memory that the process has reserved but not used)())
{
MEMBLOCK * mb = create_memblock(hProc, &meminfo);
if(mb)
{
mb->next = mb_list;
mb_list = mb;
}
}
addr = (unsigned char *)meminfo.BaseAddress + meminfo.RegionSize;//move the adress along by adding on the length of the current block
}
}
else
{
std::cout<<"Failed to open process\n";
}
std::cout<<"Create scan finished\n";
return mb_list;
The output from this code results in
Create scan started on process:7228
Then it does not return anything else to the console. Unfortunately the example source code linked to via the Youtube video is no longer available.
(7228 will change based on the current pid of notepad.exe)
edit-reply to question #Hans Passant
I still don't understand, what I think Im doing is
Starting a infinite loop
{
Testing using vqx if the address is valid and populating my MEM_BASIC_etc..
{
(has the process commited to using that addr of memory)(is the memory writeable)
{
create memblock etc
}
}
move the address along by the size of the current block
}
My program is x32 and so is notepad (as far as I'm aware).
Is my problem that because I'm using a x64 bit OS that I'm actually inspecting half of a block (a block here meaning the unit assigned by the OS in memory) and its causing it to loop?
Big thanks for your help! I want to understand my problem as well as fix it.
Your problem is you're compiling a 32 bit program and using it to parse the memory of a 64 bit program. You define 'addr' as a unsigned char pointer, which in this case is 32 bits in size. It cannot contain a 64 bit address, which is the cause of your problem.
If your target process is 64 bit, compile your program as 64 bit as well. For 32 bit target processes, compile for 32 bit. This is typically the best technique for dealing with the memory of external processes and is the fastest solution.
Depending on what you're doing, you can also use #ifdef and other conditionals to use 64 bit variables depending on the target, but the original solution is usually easier.