I am trying to make a small kernel for 80386 processor mainly for learning purpose and want to get the full memory map of the available RAM.
I have read that it is possible and better to do so with the help of GRUB than directly querying the BIOS.
Can anybody tell me how do I do it ?
Particularly, for using bios functionality in real mode we use bios interrupts and get the desired values in some registers , what is the actual equivalent way when we want to use GRUB provided functions ?
Here is the process I use in my kernel (note that this is 32bit). In my bootstrap assembly file, I tell GRUB to provide me with a memory map:
.set MEMINFO, 1 << 1 # Get memory map from GRUB
Then, GRUB loads the address of the multiboot info structure into ebx for you (this structure contains the address of the memory map). Then I call into C code to handle the actual iteration and processing of the memory map. I do something like this to iterate over the map:
/* Macro to get next entry in memory map */
#define MMAP_NEXT(m) \
(multiboot_memory_map_t*)((uint32_t)m + m->size + sizeof(uint32_t))
void read_mmap(multiboot_info_t* mbt){
multiboot_memory_map_t* mmap = (multiboot_memory_map_t*) mbt->mmap_addr;
/* Iterate over memory map */
while((uint32_t)mmap < mbt->mmap_addr + mbt->mmap_length) {
// process the current memory map entry
mmap = MMAP_NEXT(mmap);
}
}
where multiboot_info_t and multiboot_memory_map_t are defined as in the Gnu multiboot.h file. As Andrew Medico posted in the comments, here is a great link for getting started with this.
Related
I am fresh in using intel pin tool, and want to track a certain loop in a binary file, but i found in each run the address of the instructions changed in each run, how can i find a specific instruction or a specific loop even it change in each run ? Edit 0: I have the following address, which one of them is the RVA:( the first section of address(small address) are constant for each run, but the last section(big address) changed for each run) Address loop_repeation No._of_Instruction_In_Loop
4195942 1 8
4195972 1 3
....... ... ...
140513052566480 1 2
...... ... ...
the address of the instructions changed in each run, how can i find a specific instruction or a specific loop even it change in each run ?
This is probably because you have ASLR enabled (which is enabled by default on Ubuntu). If you want your analyzed program to load at the same address in each run, you might want to:
1) Disable ASLR:
Disable it system-wide: sysctl -w kernel.randomize_va_space=0 as explained here.
Disable it per process: $> setarch $(uname -m) -R /bin/bash as explained here.
2) Calculate delta (offsets) in your pintool:
For each address that you manipulate, you need to use a RVA (Relative Virtual Address) rather than a full VA (Virtual Address).
Example:
Let's say on your first run your program loads at 0x80000000 (this is the "Base Address"), and a loop starts at 0x80000210
On the second run, the program loads at 0x90000000 ("Base Address") and the loops starts at 0x90000210
Just calculate the offsets of the loops from the Base Address:
Base_Address - Program_Address = offset
0x80000210 - 0x80000000 = 0x210
0x90000210 - 0x90000000 = 0x210
As both resulting offsets are the same, you know you have the exactly the same instruction, independently of the base address of the program.
How to do that in your pintool:
Given an (instruction) address, use IMG_FindByAddress to find the corresponding image (module).
From the image, use IMG_LowAddress to get the base address of the module.
Subtract the module base from the instruction: you have the RVA.
Now you can compare RVA between them and see if they are the same (they also must be in the same module).
Obviously this doesn't work for JITed code as JITed code has no executable module (think mmap() [linux] or VirtualAlloc() [windows])...
Finally there's a good paper (quite old now, but still applicable) on doing a loop detection with pin, if that can help you.
I have a program that is creating a map file, its able to do that call just fine, m_hMap = CreateFileMapping(m_hFile,0,dwProtect,0,m_dwMapSize,NULL);, but when the subsequent function call to MapViewOfFile(m_hMap,dwViewAccess,0,0,0), I get an error code of 8, which is ERROR_NOT_ENOUGH_MEMORY, or error string "error Not enough storage is available to process this command".
So I'm not totally understanding what the MapViewOfFile does for me, and how to fix the situation.
some numbers...
m_dwMapSize = 453427200
dwProtect = PAGE_READWRITE;
dwViewAccess = FILE_MAP_ALL_ACCESS;
I think my page size is 65536
In case of very large file and to read it, it is recommended to read it in small pieces and then process each piece. And MapViewOfFile function is used to map a piece in memory.
Look at http://msdn.microsoft.com/en-us/library/windows/desktop/aa366761(v=vs.85).aspx need offset to do its job properly i.e. in case you want to read a very large file in pieces. Mostly due to fragmentation and related reason very large memory request fails.
If you are working on a 64 bit processor then the system will allocate a total of 4GB memory with bit set LargeaddressAware.
go to Configuration properties->linker->system. in Enable largeaddressware: check
Yes /LARGEADDRESSAWARE and check.
We have an embedded board with ColdFire CPU which runs µC-OS/II. When the embebbed program crashes, the CPU dumps (or copies) the entire RAM in the embedded flash. Then, we have a procedure to retrieve the RAM content (which was dumped into the flash) into a simple .bin file.
When we want to debug, we use GDB (m68k-elf-gdb.exe) combined with the .elf file. For example :
$ gdb our_elf_file
(gdb) print some_var
Cannot access memory at address 0x30617890
(gdb) ptype some_var
type = unsigned int
(gdb)
This allows us to know the address of the variable. Then, we perform a simple offset operation with the previous given address and read the RAM dump at a specific location.
For example, if we want to read some_var located at 0x30617890, we know that the dump represent the RAM content starting from 0x20000000. After that, we read 4 bytes of the .bin file at the offset (0x30617890 - 0x20000000).
(Sometimes we also use objdump (m68k-elf-objdump.exe) for other purposes).
I am completely new to this kind of stuff so maybe my question is stupid, but, is there some way to tell gdb where the RAM content is ?
I need to retrieve the total amount of RAM present in a system and the total RAM currently being used, so I can calculate a percentage. This is similar to: Retrieve system information on MacOS X?
However, in that question the best answer suggests how to get RAM by reading from:
/usr/bin/vm_stat
Due to the nature of my program, I found out that I am not cannot read from that file - I require a method that will provide me RAM info without simply opening a file and reading from it. I am looking for something to do with function calls. Something like this preferably : getTotalRam() and getRamInUse().
I obviously do not expect it to be that simple but I was looking for a solution other than reading from a file.
I am running Mac OS X Snow Leopard, but would preferably get a solution that would work across all current Mac OS X Platforms (i.e. Lion).
Solutions can be in C++, C or Obj-C, however C++ would the best possible solution in my case so if possible please try to provide it in C++.
Getting the machine's physical memory is simple with sysctl:
int mib [] = { CTL_HW, HW_MEMSIZE };
int64_t value = 0;
size_t length = sizeof(value);
if(-1 == sysctl(mib, 2, &value, &length, NULL, 0))
// An error occurred
// Physical memory is now in value
VM stats are only slightly trickier:
mach_msg_type_number_t count = HOST_VM_INFO_COUNT;
vm_statistics_data_t vmstat;
if(KERN_SUCCESS != host_statistics(mach_host_self(), HOST_VM_INFO, (host_info_t)&vmstat, &count))
// An error occurred
You can then use the data in vmstat to get the information you'd like:
double total = vmstat.wire_count + vmstat.active_count + vmstat.inactive_count + vmstat.free_count;
double wired = vmstat.wire_count / total;
double active = vmstat.active_count / total;
double inactive = vmstat.inactive_count / total;
double free = vmstat.free_count / total;
There is also a 64-bit version of the interface.
You're not supposed to read from /usr/bin/vm_stat, rather you're supposed to run it; it is a program. Look at the first four lines of output
Pages free: 1880145.
Pages active: 49962.
Pages inactive: 43609.
Pages wired down: 123353.
Add the numbers in the right column and multiple by the system page size (as returned by getpagesize()) and you get the total amount of physical memory in the system in bytes.
vm_stat isn't setuid on Mac OS, so I assume there is a non-privileged API somewhere to access this information and that vm_stat is using it. But I don't know what that interface is.
You can figure out the answer to this question by looking at the source of the top command. You can download the source from http://opensource.apple.com/. The 10.7.2 source is available as an archive here or in browsable form here. I recommend downloading the archive and opening top.xcodeproj so you can use Xcode to find definitions (command-clicking in Xcode is very useful).
The top command displays physical memory (RAM) numbers after the label "PhysMem". Searching the project for that string, we find it in the function update_physmem in globalstats.c. It computes the used and free memory numbers from the vm_stat member of struct libtop_tsamp_t.
You can command-click on "vm_stat" to find its declaration as a membor of libtop_tsamp_t in libtop.h. It is declared as type vm_statistics_data_t. Command-clicking that jumps to its definition in /usr/include/mach/vm_statistics.h.
Searching the project for "vm_stat", we find that it is filled in by function libtop_tsamp_update_vm_stats in libtop.c:
mach_msg_type_number_t count = sizeof(tsamp->vm_stat) / sizeof(natural_t);
kr = host_statistics(libtop_port, HOST_VM_INFO, (host_info_t)&tsamp->vm_stat, &count);
if (kr != KERN_SUCCESS) {
return kr;
}
You will need to figure out how libtop_port is set if you want to call host_statistics. I'm sure you can figure that out for yourself.
It's been 4 years but I just wanted to add some extra info on calculating total RAM.
To get the total RAM, we should also consider Pages occupied by compressor and Pages speculative in addition to Kyle Jones answer.
You can check out this post for where the problem occurs.
We're using Fogbugz for tracking issues and I am in the middle of writing a C++ wrapper around the XML API for Fogbugz.
The best practice seems to be to use the "scout" field so that similar/same crashes are just counted but not reported again. To do that we need a unique string for a particular cause of a crash.
In Win32 - after getting a dmp file or other crash handler what is a good way to make a unique string for a crash? (we're going to create a dmp file and send it to the fogbugz server)
In previous postings/articles/etc Joel has made various suggestions but much of those counted on a language like C# that use reflection and have a lot of information that is either harder to get or not possible to get.
Have any other people gotten things like stack traces or other things to make scout entries in fogbugz?
EDIT
To clarify - we don;t want a unique id for every incident - there are likely crashes that have the same code path. We want to capture that. I was thinking that we would get the last few stack calls that are in our code (not ones from win32 DLLs) - but not sure how to go about doing this.
Reporting every crash as unique is not right. Reporting all crashes under the same case is not right. Different users repeating a scenario that causes a crash should map to the same incident.
EDIT
What I think we want is a general "signature" of a crash - based on what is on the stack. Similar stacks should have the same signature. For example - take the top 5 methods that are in our app and then the first call (if any) we make into an MS DLL. This would probably be sufficient for a signature and would likely correlate the crashes that are "the same".
So how does one get the list of methods on the stack? And how can you tell if they are from your own app or in another DLL?
EDIT - NOTE
We want to create a "bucket id"/signature while in the exception handler so that we can create the minidump and send it to fogbugz as a scout description. Alternatively we can load up the dump on t he next start of the app and send it then with a signature we generate.
Here in my project I use the Address Memory of the Crash as a "Unique" ID.
IMO the best thing you can use will be bucket id from dump analysis. Use properly configured Debugging Tools for Windows (windbg), one can do !analyze -v and classify your dumps into different buckets based on bucket id. Bucket id guaranteed that if two dumps are the same, their bucket id will be the same. That solves part of the puzzle.
Many times two dumps rooted from same problem will create different bucket id's (maybe version difference, say your 1.0 and 1.1 both crash at same point). You can use faulting module and stack signature to correlate bugs from the same point of fault.
There will be certain things that causes very random dumps (e.g. heap corruption, the faulting module is typically the victim). Therefore dump analysis should be considered best-effort. When you can't, you can't.
I used something like this to generate exceptions in my last app (MSVC), so every error would get logged with the sourcefile and line it occured on:
class Error {
//...
public: Error(string file, string line, string error) ;
};
#define ERROR(err) Error(__FILE__, __LINE__, err)
It's probably a little bit late, but I will add my solution here, too, in case it can help other people.
You can do this using fools from "Debugging Tools for Windows", for example windbg.exe or better kd.exe.
Running the command "kd.exe -z "path_to_dump.dmp" -c "kd;q" >> dumpstack.txt, you might get the following result:
Microsoft (R) Windows Debugger Version 10.0.15063.400 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [d:\work\bugs\14122\myexe.exe.2624.dmp]
User Mini Dump File with Full Memory: Only application data is available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*C:\Symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*C:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 10 Version 15063 MP (4 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
15063.0.x86fre.rs2_release.170317-1834
Machine Name:
Debug session time: Fri Oct 13 00:09:01.000 2017 (UTC + 1:00)
System Uptime: 0 days 0:18:33.797
Process Uptime: 0 days 0:03:40.000
................................................................
.....................................................
Loading unloaded module list
..............................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(a40.2580): Security check failure or stack buffer overrun - code c0000409 (first/second chance not available)
eax=00000001 ebx=00000000 ecx=00000007 edx=77cc4350 esi=00000000 edi=00000000
eip=62ae7666 esp=0b75e17c ebp=0b75e1a8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
msvcr120!abort+0x28:
62ae7666 cd29 int 29h
0:068> kd: Reading initial command 'kb;q'
ChildEBP RetAddr Args to Child
0b75e178 62addc5f 935dda1f 00000000 00000000 msvcr120!abort+0x28
0b75e1a8 0b75e7d4 62a9b436 0b75e1dc 62a52aa5 msvcr120!terminate+0x33
WARNING: Frame IP not in any known module. Following frames may be wrong.
0b75e1ac 62a9b436 0b75e1dc 62a52aa5 00000000 0xb75e7d4
0b75e1b4 62a52aa5 00000000 62a59740 0b75e7d4 msvcr120!__FrameUnwindToState+0x89
0b75e1c8 62a52b33 00000000 00000000 00000000 msvcr120!_EH4_CallFilterFunc+0x12
0b75e1f4 62a5a0f3 62b1f7b8 62a4f7c6 0b75e324 msvcr120!_except_handler4_common+0x8e
0b75e214 77cd6152 0b75e324 0b75e7c4 0b75e344 msvcr120!_except_handler4+0x1e
0b75e238 77cd6124 0b75e324 0b75e7c4 0b75e344 ntdll!ExecuteHandler2+0x26
0b75e30c 77cc4266 0b75e324 0b75e344 0b75e324 ntdll!ExecuteHandler+0x24
0b75e30c 74cf28f2 0b75e324 0b75e344 0b75e324 ntdll!KiUserExceptionDispatcher+0x26
0b75e684 62a59339 e06d7363 00000001 00000003 KERNELBASE!RaiseException+0x62
0b75e6c4 6001821c 0b75e6e4 6004e1bc 946a8f2a msvcr120!_CxxThrowException+0x5b
0b75e6f8 60018042 0b75e720 946a8efa ffffffff mymodule!FunctionC+0x7c
0b75e730 60016544 946a8ece ffffffff 092889d8 mymodule!FunctionB+0x32
0b75e754 600166b8 00842338 6000588d 00000001 myothermodule!FunctionB+0x44
From this stack, you can create a unique bucket if you take for example only your methods from the stack and concatenate them in a string: "mymodule!FunctionC+0x7c;mymodule!FunctionB+0x32;myothermodule!FunctionB+0x44". In order for this to work, you need to have access to you personal symbols server, either using the environment variable _NT_SYMBOL_PATH or with the -y command line switch.
You can alternatively create a string from the return addresses only (second column): "62addc5f,0b75e7d4,62a9b436,62a52aa5,62a52b33,62a5a0f3,77cd6152,77cd6124,77cc4266,74cf28f2,62a59339,6001821c,60018042,60016544,600166b8"
Just use an MD5 string generated from the dump file and you will likely to get a unique string for every crash.
I would start with collecting the data on how often every function in your code has been "flashed" in a crash report stack trace. Every report would have to be added to some kind of database, and every function would have to be indexed so that you could later query, which functions seem to crash more often than others. (And of course, functions like main() will be in every report, but that's understandable).
Or, you think that only crash reports seem to be the problem, you could just remove all those entries from crash stack traces, and then hash the rest (your functions). That way you could see if any particular call chain of your own functions causes a crash repeatedly, no matter what external functions have been called in between.
Then of course, some of the more complicated problems will not be captured this way anyway, as the stack trace will be completely different. To help that, you could record other data from your application along with the stack trace in every report, like sizes of buffers, counters, states of different parts of the application and so on... And then do some statistics on that.