I am trying to perform loop bound analysis for ARMV7m code using Z3 for a big Framework.
I would like to find the memory address that are used by a certain function inside .elf file
for example in a function foo() I have the below basic block
ldr r1, [r3, #0x20]
strb r2, [r3, #6] {__elf_header}
str r2, [r3, #0x24] {__elf_header}
str r2, [r3, #0x20] {__elf_header}
mov r3, r1
cmp r1, #0
bne #0x89f6
How can I get the initial memory location used by this function [r3, #0x20] ? Are there memory segements for every function to access or is it random ?
Given that the above basic block is a loop. Is there a way to know the memory address that will be used during its execution ?
Does the compiler for example save a memory location address from 0x20 to 0x1234 to be only accessed during the execution of such basic block ? In another word, Is there a map between a function and the range of memory address used by it ?
It is confusing as to what you are asking. First off why would any linker put the effort into randomizing things? Perhaps there is one to intentionally make the output not repeatable. But a linker is just a program and normally will do things like process the items on the command line in order, and then process each object from beginning to end...not random.
So far the rest of this seems pretty straight forward just use the tools. Your comment implies gnu tools? Since this is in part tool specific you should have tagged it as such as you cannot really make generalizations across all toolchains ever created.
unsigned int one ( void )
{
return(1);
}
unsigned int two ( void )
{
return(2);
}
unsigned int three ( void )
{
return(3);
}
arm-none-eabi-gcc -O2 -c so.c -o so.o
arm-none-eabi-objdump -d so.o
so.o: file format elf32-littlearm
Disassembly of section .text:
00000000 <one>:
0: e3a00001 mov r0, #1
4: e12fff1e bx lr
00000008 <two>:
8: e3a00002 mov r0, #2
c: e12fff1e bx lr
00000010 <three>:
10: e3a00003 mov r0, #3
14: e12fff1e bx lr
as shown they are all in .text, simple enough.
arm-none-eabi-gcc -O2 -c -ffunction-sections so.c -o so.o
arm-none-eabi-objdump -d so.o
so.o: file format elf32-littlearm
Disassembly of section .text.one:
00000000 <one>:
0: e3a00001 mov r0, #1
4: e12fff1e bx lr
Disassembly of section .text.two:
00000000 <two>:
0: e3a00002 mov r0, #2
4: e12fff1e bx lr
Disassembly of section .text.three:
00000000 <three>:
0: e3a00003 mov r0, #3
4: e12fff1e bx lr
and now each function has its own section name.
So the rest relies heavily on linking and there is no one linker script, you the programmer choose directly or indirectly and how the final binary (elf) is built is a direct result of that choice.
If you have something like this
.text : { *(.text*) } > rom
and nothing else with respect to these functions then all of them will land in this definition, but the linker script or instructions to the linker can indicate something else causing one or more to land in its own space.
arm-none-eabi-ld -Ttext=0x1000 so.o -o so.elf
arm-none-eabi-ld: warning: cannot find entry symbol _start; defaulting to 0000000000001000
arm-none-eabi-objdump -d so.elf
so.elf: file format elf32-littlearm
Disassembly of section .text:
00001000 <one>:
1000: e3a00001 mov r0, #1
1004: e12fff1e bx lr
00001008 <two>:
1008: e3a00002 mov r0, #2
100c: e12fff1e bx lr
00001010 <three>:
1010: e3a00003 mov r0, #3
1014: e12fff1e bx lr
and then of course
arm-none-eabi-nm -a so.elf
00000000 n .ARM.attributes
00011018 T __bss_end__
00011018 T _bss_end__
00011018 T __bss_start
00011018 T __bss_start__
00000000 n .comment
00011018 T __data_start
00011018 T _edata
00011018 T _end
00011018 T __end__
00011018 ? .noinit
00001000 T one <----
00000000 a so.c
00080000 T _stack
U _start
00001000 t .text
00001010 T three <----
00001008 T two <----
which is simply because there is a symbol table in the file
Symbol table '.symtab' contains 22 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 00000000 0 NOTYPE LOCAL DEFAULT UND
1: 00001000 0 SECTION LOCAL DEFAULT 1
2: 00000000 0 SECTION LOCAL DEFAULT 2
3: 00000000 0 SECTION LOCAL DEFAULT 3
4: 00011018 0 SECTION LOCAL DEFAULT 4
5: 00000000 0 FILE LOCAL DEFAULT ABS so.c
6: 00001000 0 NOTYPE LOCAL DEFAULT 1 $a
7: 00001008 0 NOTYPE LOCAL DEFAULT 1 $a
8: 00001010 0 NOTYPE LOCAL DEFAULT 1 $a
9: 00001008 8 FUNC GLOBAL DEFAULT 1 two
10: 00011018 0 NOTYPE GLOBAL DEFAULT 1 _bss_end__
11: 00011018 0 NOTYPE GLOBAL DEFAULT 1 __bss_start__
12: 00011018 0 NOTYPE GLOBAL DEFAULT 1 __bss_end__
13: 00000000 0 NOTYPE GLOBAL DEFAULT UND _start
14: 00011018 0 NOTYPE GLOBAL DEFAULT 1 __bss_start
15: 00011018 0 NOTYPE GLOBAL DEFAULT 1 __end__
16: 00001000 8 FUNC GLOBAL DEFAULT 1 one
17: 00011018 0 NOTYPE GLOBAL DEFAULT 1 _edata
18: 00011018 0 NOTYPE GLOBAL DEFAULT 1 _end
19: 00080000 0 NOTYPE GLOBAL DEFAULT 1 _stack
20: 00001010 8 FUNC GLOBAL DEFAULT 1 three
21: 00011018 0 NOTYPE GLOBAL DEFAULT 1 __data_start
but if
arm-none-eabi-strip so.elf
arm-none-eabi-nm -a so.elf
arm-none-eabi-nm: so.elf: no symbols
arm-none-eabi-objdump -d so.elf
so.elf: file format elf32-littlearm
Disassembly of section .text:
00001000 <.text>:
1000: e3a00001 mov r0, #1
1004: e12fff1e bx lr
1008: e3a00002 mov r0, #2
100c: e12fff1e bx lr
1010: e3a00003 mov r0, #3
1014: e12fff1e bx lr
The elf file format is somewhat trivial you can easily write code to parse it, you do not need a library or anything like that. And with simple experiments like these can easily understand how these tools work.
How can I get the initial memory used by this function ?
Assuming you mean the initial address assuming not relocated. You just read it out of the file. Simple.
Are there memory segments for every function to access or is it random ?
As demonstrated above, the command line option you mention later in a comment (should have been in the question, you should edit the question for completeness) does exactly that makes a custom section name per function. (what happens if you have the same non-global function name in two or more objects? you can easily figure this out on your own)
Nothing is random here, you would need to have a reason to randomize things for security or other, it is more often preferred that a tool outputs the same or at least similar results each time with the same inputs (some tools will embed a build date/time in the file and that may vary from one build to the next).
If you are not using gnu tools then binutils may still be very useful with parsing and displaying elf files anyway.
arm-none-eabi-nm so.elf
00011018 T __bss_end__
00011018 T _bss_end__
00011018 T __bss_start
00011018 T __bss_start__
00011018 T __data_start
00011018 T _edata
00011018 T _end
00011018 T __end__
00001000 T one
00080000 T _stack
U _start
00001010 T three
00001008 T two
nm so.elf (x86 binutils not arm)
00001000 t $a
00001008 t $a
00001010 t $a
00011018 T __bss_end__
00011018 T _bss_end__
00011018 T __bss_start
00011018 T __bss_start__
00011018 T __data_start
00011018 T _edata
00011018 T _end
00011018 T __end__
00001000 T one
00080000 T _stack
U _start
00001010 T three
00001008 T two
Or can build with clang and examine with gnu, etc. Obviously disassembly won't work, but some tools will.
If this is not what you were asking then you need to re-write your question or edit it so we can understand what you are actually asking.
Edit
I would like to know if there is a map between a function and the range of memory address used by it ?
In general no. The term function implies but is not limited to high level languages like C, etc. Where the machine code clearly has no clue nor should it and well optimized code does not necessarily have a single exit point from the function, much less a return marking the end. For architectures like the various arm instruction sets the return instruction is not the end of the "function", there is pool data that may follow.
But let's look at what gcc does.
unsigned int one ( unsigned int x )
{
return(x+1);
}
unsigned int two ( void )
{
return(one(2));
}
unsigned int three ( void )
{
return(3);
}
arm-none-eabi-gcc -O2 -S so.c
cat so.s
.cpu arm7tdmi
.eabi_attribute 20, 1
.eabi_attribute 21, 1
.eabi_attribute 23, 3
.eabi_attribute 24, 1
.eabi_attribute 25, 1
.eabi_attribute 26, 1
.eabi_attribute 30, 2
.eabi_attribute 34, 0
.eabi_attribute 18, 4
.file "so.c"
.text
.align 2
.global one
.arch armv4t
.syntax unified
.arm
.fpu softvfp
.type one, %function
one:
# Function supports interworking.
# args = 0, pretend = 0, frame = 0
# frame_needed = 0, uses_anonymous_args = 0
# link register save eliminated.
add r0, r0, #1
bx lr
.size one, .-one
.align 2
.global two
.syntax unified
.arm
.fpu softvfp
.type two, %function
two:
# Function supports interworking.
# args = 0, pretend = 0, frame = 0
# frame_needed = 0, uses_anonymous_args = 0
# link register save eliminated.
mov r0, #3
bx lr
.size two, .-two
.align 2
.global three
.syntax unified
.arm
.fpu softvfp
.type three, %function
three:
# Function supports interworking.
# args = 0, pretend = 0, frame = 0
# frame_needed = 0, uses_anonymous_args = 0
# link register save eliminated.
mov r0, #3
bx lr
.size three, .-three
.ident "GCC: (GNU) 10.2.0"
we see this is being placed in the file, but what does it do?
.size three, .-three
One reference says this is used so that the linker can remove the function if it is not used. And I have seen that feature in play so good to know (you could have looked this up just as easily as I did)
So in that context the info is there and you can extract it (lesson for the reader).
And then if you use this gcc compiler option that you mentioned
-ffunction-sections
Disassembly of section .text.one:
00000000 <one>:
0: e2800001 add r0, r0, #1
4: e12fff1e bx lr
Disassembly of section .text.two:
00000000 <two>:
0: e3a00003 mov r0, #3
4: e12fff1e bx lr
Disassembly of section .text.three:
00000000 <three>:
0: e3a00003 mov r0, #3
4: e12fff1e bx lr
[ 4] .text.one
PROGBITS 00000000 000034 000008 00 0 0 4
[00000006]: ALLOC, EXEC
[ 5] .rel.text.one
REL 00000000 0001a4 000008 08 12 4 4
[00000040]: INFO LINK
[ 6] .text.two
PROGBITS 00000000 00003c 000008 00 0 0 4
[00000006]: ALLOC, EXEC
[ 7] .rel.text.two
REL 00000000 0001ac 000008 08 12 6 4
[00000040]: INFO LINK
[ 8] .text.three
PROGBITS 00000000 000044 000008 00 0 0 4
[00000006]: ALLOC, EXEC
[ 9] .rel.text.three
REL 00000000 0001b4 000008 08 12 8 4
[00000040]: INFO LINK
That is giving us a size of the sections.
In general with respect to software compiled or in particular assembled, assume that a function doesn't have boundaries. As you can see above the one function is inlined into the two function, invisibly, so how big is an inlined function within another function? How many instances of a function are there in a binary? Which one do you want to monitor and know the size of, performance of, etc? Gnu has this feature with gcc, you can see if it is there with other languages or tools. Assume the answer is no, and then if you happen to find a way, then that is good.
Does the compiler saves a memory segment to be only accessed by a certain function ?
I have no idea what this means. The compiler doesn't make memory segments the linker does. How the binary is put into a memory image is a linker thing not a compiler thing for starters. Segments are just a way to communicate between tools that these bytes are for starters code (read only ideally), initialized data, or uninitialized data. Perhaps extending to read only data and then make up your own types.
If your ultimate goal is to find the bytes that represent the high level concept of "function" in memory (assuming no relocation, etc) by looking at the elf binary using the gnu toolchain. That is possible in theory.
The first thing we appear to know is that the OBJECT contains this information so that a linker feature can remove unused functions for size. But that does not automatically mean that the output binary from the linker also includes this information. You need to find where this .size lands in the object and then look for that in the final binary.
The compiler turns one language into another, often from a higher level to a lower level but not always depends on the compiler and input/output languages. C to assembly or C to machine code or what about Verilog to C++ for a simulation is that higher or lower? The terms .text, .data, .bss are not part of the language but more of a habit based on learned experience and helps as mentioned communicate with the linker so that the output binaries can be more controlled for various targets. Normally as shown above the compiler, gcc in this case, since no generalities can be made in this area across all tools and languages or even all C or C++ tools all the code for all the functions in the source file land in one .text segment by default. You have to do extra work to get something different. So the compiler in general does not make a "segment" or "memory segment" for each...In general. You already solved your problem it seems by using a command line option that turns every function into its own segment and now you have a lot more control over size and location, etc.
Just use the file format and/or the tools. This question or series of questions boils down into just go look at the elf file format. This is not a Stack Overflow question as questions seeking recommendations for external information is not for this site.
Does the compiler for example save a memory location address from 0x20 to 0x1234 to be only accessed during the execution of such basic block ? In another word, Is there a map between a function and the range of memory address used by it ?
"save"? the compiler does not link the linker links. Is that memory "only" accessed during the execution of that block? Well in a pure textbook theory yes, but in reality branch prediction and prefetch or cache line fills can also access that "memory".
Unless doing self-modifying code or using the mmu in interesting ways you do not re-use an address space for more than one function within an application. In general. So function foo() is implemented somewhere and bar() somewhere else. Hand written asm from the good old days you might have foo() branch right into the middle of bar() to save space, get better performance or to make the code harder to reverse engineer or whatever. But compilers are not that efficient, they do their best to turn concepts like functions into first off functional(ly equivalent to the high level code) and then second, if desired smaller or faster or both relative to a straight brute force conversion between languages. So barring inlining and tail (leaf?, I call it tail) optimizations and such, one could say there are some number of bytes at some address that define a compiled function. But due to the nature of processor technology you cannot assume those bytes are only accessed by the processor/chip/system busses only when executing that function.
I have trained a model using Google AutoML Tables. I would like to inspect the general structure of the trained model (which algos were used, what preprocessing if any, etc)
In the "Viewing model architecture with Cloud Logging" section of the docs here, I see:
If more than one model was used to create the final model, the
hyperparameters for each model are returned as an entry in the
modelParameters array, indexed by position (0, 1, 2, and so on)
My modelParameters array is as shown below (with the first and last elements expanded). Im only doing the AutoML Tables quickstart, which uses the Bank marketing open-source dataset, so I'm suprised that it would return such a complex model (25 stacked/ensembled models?). I would think that just model 0 (a single Gradient Boosted Decision Tree with 300 trees and a max depth of 15) would be sufficient.
Also '25' is a suspiciously round number. Are we sure that the docs are correct and this list isn't actually the best 25, stackranked by accuracy score? Is there a better way to understand the end to end model (including preprocessing) that Google AutoML Tables is producing?
modelParameters: [
0: {
hyperparameters: {
Center Bias: "False"
Max tree depth: 15
Model type: "GBDT"
Number of trees: 300
Tree L1 regularization: 0
Tree L2 regularization: 0.10000000149011612
Tree complexity: 3
}
}
1: {…}
2: {…}
3: {…}
4: {…}
5: {…}
6: {…}
7: {…}
8: {…}
9: {…}
10: {…}
11: {…}
12: {…}
13: {…}
14: {…}
15: {…}
16: {…}
17: {…}
18: {…}
19: {…}
20: {…}
21: {…}
22: {…}
23: {…}
24: {
hyperparameters: {
Center Bias: "False"
Max tree depth: 9
Model type: "GBDT"
Number of trees: 500
Tree L1 regularization: 0
Tree L2 regularization: 0
Tree complexity: 0.10000000149011612
}
}
]
}
The docs and your original interpretation of the results are correct. In this case, AutoML Tables created an ensemble of 25 models.
It also provides the full list of individual models tried during the search process if you click on the "Trials" link. That should be much larger list than 25.
I want to get the interpretation of Map-Reduce logs after running a query on Hive-Tez ? What the lines after INFO: conveys ?
Here I have attached a sample
INFO : Session is already open
INFO : Dag name: SELECT a.Model...)
INFO : Tez session was closed. Reopening...
INFO : Session re-established.
INFO :
INFO : Status: Running (Executing on YARN cluster with App id application_14708112341234_1234)
INFO : Map 1: -/- Map 3: -/- Map 4: -/- Map 7: -/- Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
INFO : Map 1: -/- Map 3: 0/118 Map 4: 0/118 Map 7: 0/1 Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
INFO : Map 1: 0/118 Map 3: 0/118 Map 4: 0/118 Map 7: 0/1 Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
INFO : Map 1: 0/118 Map 3: 0/118 Map 4: 0(+5)/118 Map 7: 0/1 Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
INFO : Map 1: 0/118 Map 3: 0(+5)/118 Map 4: 0(+7)/118 Map 7: 0(+1)/1 Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
INFO : Map 1: 0/118 Map 3: 0(+15)/118 Map 4: 0(+18)/118 Map 7: 0(+1)/1 Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13
The log you posted is DAG execution log. The DAG consists of
Map 1,Map 3,Map 4,Map 7 mappers vertices and reducers: Reducer 2,Reducer 5,Reducer 6
Map 1: -/- - this means that the vertex is not initialized, the number of mappers are not calculated yet.
Map 4: 0(+7)/118 - this means that totally there are 118 mappers, 7 of them are running in parallel, 0 completed yet, 118-7=111 are pending.
Reducer 2: 0/15 - this means that totally there are 15 reducers, 0 of them are running, 0 of them are completed (15 reducers pending).
Negative figures (there are no such in your example) = number of failed or killed mappers or reducers
Qubole has explanation about Tez log: https://docs.qubole.com/en/latest/user-guide/hive/using-hive-on-tez/hive-tez-tuning.html#understanding-log-pane
I've a string with some Spanish characters and when I use the U2's OCONV() function with the code 'MCT', it changes the Spanish Character to something else. Does anyone know?
STRING: T r L=16 `CITáN, MOR 32000'
TEST.MCT: 5: STR2 = OCONV(STR,'MCT')
:: S
TEST.MCT: 6: CRT STR2
:: S
Cit?9: Mor 32000
Note that I created the following program, and I do not see the issue.
CT BP SO
SO
0001 STR = "CIT":CHAR(225):"N, MOR 3200"
0002 STR2 = OCONV(STR, "MCT")
0003 PRINT STR
0004 PRINT STR2
0005 PRINT SEQ(STR2[4,1])
When I compile and run it, I get the following:
CITáN, MOR 3200
Citán, Mor 3200
225
>
Note that I tested UniVerse11.2.2, on Windows. Can you try the sample code I provided from the HS.SALES account, and let me know what it does?
If it still has an issue, please let us know the full UniVerse version, and the OS you are running this on.
added info:
note tested on UniVerse 11.1.1 on AIX 6.1 and it worked for me. If you are still having issues, I suggest you contact you UniVerse Maintenance provider.
It is hard to read your output since it combines lines.
My run through RAID shows the correct information.
RAID BP SO
SO: 1: STR = "CIT":CHAR(225):"N, MOR 3200"
:: S
SO: 2: STR2 = OCONV(STR, "MCT")
:: S
SO: 3: PRINT STR
:: S
CITáN, MOR 3200
SO: 4: PRINT STR2
:: S
Citán, Mor 3200
SO: 5: PRINT SEQ(STR2[4,1])
:: S
225
Yet, I have LANG=en_US in my UNIX environment variables.
So, there may be an issue with your environment based on what LANG is set, I suggest you contact you U2 Maintenance provider.
I would like to match the following text:
PokerStars Hand #95528134282: Tournament #2013004001, $0.10+$0.01 USD Hold'em No Limit - Level VI (100/200) - 2013/03/14 15:35:36 WET [2013/03/14 11:35:36 ET]
Table '2013004001 5898' 9-max Seat #1 is the button
Seat 1: Pucharrin (7250 in chips)
Seat 2: pahol (24180 in chips)
Seat 3: dno16 (2000 in chips)
Seat 4: sogd20i07 (150 in chips) is sitting out
Seat 5: koaollie (13680 in chips)
Seat 6: vovik770 (6307 in chips)
Seat 7: gab341978 (6920 in chips)
Seat 8: 19gow63 (1000 in chips)
Seat 9: pokerplayer (9048 in chips)
pahol: posts small blind 100
dno16: posts big blind 200
*** HOLE CARDS ***
Dealt to pokerplayer [3s 9d]
sogd20i07: folds
koaollie: folds
vovik770: folds
gab341978: folds
19gow63: raises 800 to 1000 and is all-in
pokerplayer: folds
Pucharrin: folds
pahol: raises 1000 to 2000
dno16: calls 1800 and is all-in
*** FLOP *** [4s 7c Ah]
*** TURN *** [4s 7c Ah] [Qs]
*** RIVER *** [4s 7c Ah Qs] [Ks]
*** SHOW DOWN ***
pahol: shows [Qc Qd] (three of a kind, Queens)
dno16: shows [6h 2h] (high card Ace)
pahol collected 2000 from side pot
19gow63: shows [Kd 2s] (a pair of Kings)
pahol collected 3000 from main pot
19gow63 re-buys and receives 1000 chips for $0.10
dno16 re-buys and receives 2000 chips for $0.20
*** SUMMARY ***
Total pot 5000 Main pot 3000. Side pot 2000. | Rake 0
Board [4s 7c Ah Qs Ks]
Seat 1: Pucharrin (button) folded before Flop (didn't bet)
Seat 2: pahol (small blind) showed [Qc Qd] and won (5000) with three of a kind, Queens
Seat 3: dno16 (big blind) showed [6h 2h] and lost with high card Ace
Seat 4: sogd20i07 folded before Flop (didn't bet)
Seat 5: koaollie folded before Flop (didn't bet)
Seat 6: vovik770 folded before Flop (didn't bet)
Seat 7: gab341978 folded before Flop (didn't bet)
Seat 8: 19gow63 showed [Kd 2s] and lost with a pair of Kings
Seat 9: pokerplayer folded before Flop (didn't bet)
And I would like to capture the lines starting with "Seat somenumber: someplayername (somenumber in chips) someoptionaltext" as a group.
I tried the following regex:
PokerStars.*?Level .+? \(\d+\/(\d+)\) (?:.|\s)*?((?:Seat \d+: .*? \(\d+ in chips\)(?:.|\s)*?)+)(?:.|\s)*?(.*?) collected (\d+)
but it only captures the first occurence "Seat 1: Pucharrin (7250 in chips) ".
How can I change it to capture the group?
Thanks.
Here is a regexp that captures the data you need. When you specify the syntax/ordering of those elements, I could revamp it for you
(Seat\s(\d)+:\s+([\w\d\-\_\.]+)\s+\((\d+)\s+in\s+chips\)(.*))
I used your service and input to test it, seems to work fine.
-- EDIT --
Unless you want the entire block starting with PokerStars and stopping at the last )
Then maybe:
if (preg_match('/((?:PokerStars.*\s*Table.*[\r\n]*)(?:Seat.*\s+)+)/i', $subject, $regs)) {
$result = $regs[0];
} else {
$result = "";
}