Removing OPENSSL_cleanse from OpenSSL-1.0.1r - c++

I found out that OPENSSL_cleanse wastes a lot of time in my project. For example, if it runs for 25 seconds, 3 seconds are wasted by OPENSSL_cleanse. I checked the code of this function and decided that it isn't doing anything very useful for me. I know it fills memory with garbage data for security reasons but I don't really care about it. So I decided to place return; just before the start of any operations in this function.
void OPENSSL_cleanse(void *ptr, size_t len)
{
return;
// original OpenSSL code goes here
}
I'm using Mac OS and Xcode. I've compiled the lib and installed it in /Users/ForceBru/Desktop/openssl via the --openssldir option of the Configure script. I've added it to my project in Build Settings->Link Binary With Libraries and added include dirs in Build Settings->Search Paths->Header Search Paths and Build Settings->Search Paths->Library Search Paths.
The project compiled fine, but the time profiler still shows pretty expensive calls to OPENSSL_cleanse.
Edit: the C tag is because OpenSSL is written in C, and the C++ tag is because my code is in C++. Maybe this information will be helpful.
The question is, what am I doing wrong? How do I remove the calls to OPENSSL_cleanse? I think this has to do with linking, because the command line includes -lcrypto, which means this library can actually be taken from anywhere (right?), not necessarily from /Users/ForceBru/Desktop/openssl.
Edit #2: I've edited the linker options to use the .a file in /Users/ForceBru/Desktop/openssl and removed it from Build Settings->Link Binary With Libraries. Still no effect.

It turns out that OpenSSL has lots of assembly code generated by some Perl scripts that are located in the crypto directory (*cpuid.pl). These scripts generate assembly code for the following architectures: alpha, armv4, ia64, ppc, s390x, sparc, x86 and x86_64.
When make runs, the appropriate script fires generating a *cpuid.S (where * is one of the architectures mentioned earlier). These files are compiled into the library and seem to override the OPENSSL_cleanse implemented in crypto/mem_clr.c.
What I had to do is to simply change the body of OPENSSL_cleanse to ret in x86_64cpuid.pl:
.globl OPENSSL_cleanse
.type OPENSSL_cleanse,\#abi-omnipotent
.align 16
OPENSSL_cleanse:
ret
# loads of OPENSSL assembly
.size OPENSSL_cleanse,.-OPENSSL_cleanse

This isn't quite the answer that you were looking for, but it may help you along...
Removing OPENSSL_cleanse from OpenSSL-1.0.1r...
I checked the code of this function and decided that it isn't doing anything very useful for me...
That's probably a bad idea, but we would need to know more about your threat model. Zeroization allows you to deterministically remove sensitive material from memory.
Its also a Certification and Accreditation (C&A) item. For example, FIPS 140-2 requires zeroization even at Level 1.
Also, you can't remove OPENSSL_cleanse per se because OPENSSL_clear_realloc, OPENSSL_clear_free and friends call it. Also see the OPENSSL_cleanse man page.
For example, if it runs for 25 seconds, 3 seconds are wasted by OPENSSL_cleanse
OK, so this is a different problem. OPENSSL_cleanse is kind of convoluted, and it does waste some cycles in an effort to survive the optimization pass.
If you check Commit 380f18ed5f140e0a, then you will see it has been changed in OpenSSL 1.1.0 to the following. Maybe you could use it instead?
diff --git a/crypto/mem_clr.c b/crypto/mem_clr.c
index e6450a1..3389919 100644 (file)
--- a/crypto/mem_clr.c
+++ b/crypto/mem_clr.c
## -59,23 +59,16 ##
#include <string.h>
#include <openssl/crypto.h>
-extern unsigned char cleanse_ctr;
-unsigned char cleanse_ctr = 0;
+/*
+ * Pointer to memset is volatile so that compiler must de-reference
+ * the pointer and can't assume that it points to any function in
+ * particular (such as memset, which it then might further "optimize")
+ */
+typedef void *(*memset_t)(void *,int,size_t);
+
+static volatile memset_t memset_func = memset;
void OPENSSL_cleanse(void *ptr, size_t len)
{
- unsigned char *p = ptr;
- size_t loop = len, ctr = cleanse_ctr;
-
- if (ptr == NULL)
- return;
-
- while (loop--) {
- *(p++) = (unsigned char)ctr;
- ctr += (17 + ((size_t)p & 0xF));
- }
- p = memchr(ptr, (unsigned char)ctr, len);
- if (p)
- ctr += (63 + (size_t)p);
- cleanse_ctr = (unsigned char)ctr;
+ memset_func(ptr, 0, len);
}
Also see Issue 455: Reimplement non-asm OPENSSL_cleanse() on OpenSSL's GitHub.
How do I remove the calls to OPENSSL_cleanse?
OK, so this is a different problem. You have to locate all callers and do something with each. It looks like there's about 185 places you will need to modify things:
$ cd openssl
$ grep -IR _cleanse * | wc -l
185
Instead of this:
void OPENSSL_cleanse(void *ptr, size_t len)
{
return;
// original OpenSSL code goes here
}
Maybe you can delete the function, and then:
#define OPENSSL_cleanse(x, y)
Then the function calls becomes a macro that simply disappears during optimization. Be sure to perform a make clean after changing from a function to a macro.
But I would not advise doing so.
The project compiled fine, but the time profiler still shows pretty expensive calls to OPENSSL_cleanse.
My guess here is either (1) you did not perform a make clean after the changes to the OpenSSL library, or (2) you compiled and linked to the wrong version of the OpenSSL library. But I could be wrong on both.
You can see what your executable's runtime dependencies are with otool -L. Make sure its the expected one. Also keep in mind OpenSSL does not use -install_name.
Before you run your executable, you can set DYLD_LIBRARY_PATH to ensure the dylib you are modifying is loaded. Also see the dyld(1) man pages.

Related

clang AddressSanitizer instructs code improperly, false-positive result

FOREWORD
The current question is pretty damn huge and related to my master thesis, so I am humbly asking for your patience. I encountered a problem that is going to be explained further about half a year ago and the problem was needed an exterior look because at that point I was really stuck and I had nobody to help me. In the end I waved a hand at the problem, but now I am back in business (the second wind, let us put it that way).
INTRODUCTION
Crucial technologies used in the project:
C++, llvm/clang 13.0.1, ASAN, libFuzzer
The underlying idea behind the project I was writting is:
Write a parser of C-code projects to find functions that are presumed to be vulnerable (in the frames of the current question it does not matter how I decide that they are vulnerable)
When I find the vulnerable function, I start to write fuzzer code with libFuzzer for the function.
At this point I have an IR file with my vulnerable function, an IR file with my fuzzer code so it is time
to perform a separate compilation of two files. During the compilation process I instruct them with ASAN and libFuzzer by the clang compiler.
So the two files are coalesced together and I have an executable called, for example, 'fuzzer'. Theoretically, I can execute this executable and libFuzzer is going to fuzz my vulnerable function.
ACTUAL PROBLEM (PART 1)
ASAN intructs my code somehow bad. It gives me the wrong result.
How do I know that?
I found and took a vulnerable function. This function is from the old version of libcurl and is called sanitize_cookie_path. I reproduced the bug with AFL++ and it gave me what I wanted. If you pass a single quote to the function, it is going to 'blow'. Something similar I wanted to do with libFuzzer and ASAN, but as I mentioned earlier these two did not give me the expected result. Having spent some time on the problem, I can say that there is something with ASAN.
PROBLEM REPRODUCTION
I have the code (see below) in the file sanitize_cookie_path.c:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>
#include <stddef.h>
static char* sanitize_cookie_path(const char* cookie_path) {
size_t len;
char* new_path = strdup(cookie_path);
if (!new_path) {
return NULL;
}
if (new_path[0] == '\"') {
memmove((void *)new_path, (const void*)(new_path + 1), strlen(new_path));
}
if (new_path[strlen(new_path) - 1] == '\"') {
new_path[strlen(new_path) - 1] = 0x0;
}
if (new_path[0] !='/') {
free(new_path);
new_path = strdup("/");
return new_path;
}
len = strlen(new_path);
if (1 < len && new_path[len - 1] == '/') {
new_path[len - 1] = 0x0;
}
return new_path;
}
int main(int argc, char** argv) {
if (argc != 2) {
exit(1);
}
sanitize_cookie_path('\"');
return 0;
}
My C++ code compiles it with the command:
clang -O0 -emit-llvm path/to/sanitize_cookie_path.c -S -o path/to/sanitize_cookie_path.ll > /dev/null 2>&1
On the IR level of the above code I get rid of the 'main' so only the 'sanitize_cookie_path' function presents.
I generate the simple fuzzer code (see below) for this function:
#include <cstdio>
#include <cstdint>
static char* sanitize_cookie_path(const char* cookie_path) ;
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
(void) sanitize_cookie_path((char*) data);
return 0;
}
Then I compile it with the command:
clang -O0 -emit-llvm path/to/fuzz_sanitize_cookie_path.cc -S -o path/to/fuzz_sanitize_cookie_path.ll > /dev/null 2>&1
Two IR files are being compiled with the separate compilation. NOTE that before the separate compilation I perform some business to get them fit each other. For instance, I ditch the 'static' keyword and resolve name mangling from C++ to C code.
I compile them both together with the command:
clang++ -O0 -g -fno-omit-frame-pointer -fsanitize=address,fuzzer -fsanitize-coverage=trace-cmp,trace-gep,trace-div path/to/sanitize_cookie_path.ll path/to/fuzz_sanitize_cookie_path.ll -o path-to/fuzzer > /dev/null 2>&1
The final 'fuzzer' executable is ready.
ACTUAL PROBLEM (PART 2)
If you execute the fuzzer program, it is not going to give you the same results as AFL++ gives you. My fuzzer tumbles down on the '__interceptor_strdup' function from some standard library (see error snippet below). The crash report done by libFuzzer is literally empty (0 bytes), but ideally it had to find that the error is with a quote ("). Having done my own research I found out that ASAN did instruct the code bad and it gives me a false-position result. Frankly speaking I can fuzz the 'printf' function from stdio.h and find the same error.
[sanitize_cookie_path]$ ./fuzzer
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1016408680
INFO: Loaded 1 modules (11 inline 8-bit counters): 11 [0x5626d4c64c40, 0x5626d4c64c4b),
INFO: Loaded 1 PC tables (11 PCs): 11 [0x5626d4c64c50,0x5626d4c64d00),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
=================================================================
==2804==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000011 at pc 0x5626d4ba7671 bp 0x7ffe43152df0 sp 0x7ffe431525a0
READ of size 2 at 0x602000000011 thread T0
#0 0x5626d4ba7670 in __interceptor_strdup (/path/to/fuzzer+0xdd670)
#1 0x5626d4c20127 in sanitize_cookie_path (/path/to/fuzzer+0x156127)
#2 0x5626d4c20490 in LLVMFuzzerTestOneInput (/path/to/fuzzer+0x156490)
#3 0x5626d4b18940 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/path/to/fuzzer+0x4e940)
#4 0x5626d4b1bae6 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) (/path/to/fuzzer+0x51ae6)
#5 0x5626d4b1c052 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) (/path/to/fuzzer+0x52052)
#6 0x5626d4b0100b in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/path/to/fuzzer+0x3700b)
#7 0x5626d4af0297 in main (/path/to/fuzzer+0x26297)
#8 0x7f8e6442928f (/usr/lib/libc.so.6+0x2928f)
#9 0x7f8e64429349 in __libc_start_main (/usr/lib/libc.so.6+0x29349)
#10 0x5626d4af02e4 in _start /build/glibc/src/glibc/csu/../sysdeps/x86_64/start.S:115
I used gdb to enter into the strdup(cookie_path). gdb shows me that the fuzzer tumbles down on the address 0x0000555555631687.
0x0000555555631684 <+452>: mov %rbp,%rsi
0x0000555555631687 <+455>: addr32 call 0x555555674100 <_ZN6__asan18ReportGenericErrorEmmmmbmjb>
0x000055555563168d <+461>: pop %rax
WHAT I TRIED TO DO
I tried to instuct my sanitize_cookie_path.c and fuzz_sanitize_cookie_path.cc with ASAN right at the beginning, not at the IR level but whatever I did nothing worked.
I passed to the 'fuzzer' the so called corpus directory with pre-cooked data to be passed to the fuzzer. I even passed the quote explicitly to the 'fuzzer', but nothing. Example (with the same directory as the fuzzer):
$ mkdir corpus/; echo "\"" > corpus/input; hexdump corpus/input
0000000 0a22
0000002
$ ./fuzzer corpus/
I also googled everything I could about libFuzzer and ASAN, but nothing gave me the results.
Changed compilation command. I got rid of the '-fno-omit-frame-pointer' and '-fsanitize-coverage=trace-cmp,trace-gep,trace-div'.
If there are some uncertainties in the details I have provided, do not hesitate to ask about them and I will iron them out to be more clear for you.
What are some other sites/forums where I can possibly get heard? I would ideally want to contact the developers of ASAN.
I will be more than happy for any help.
UPDATE 04/10/2022
llvm/clang have been upgraded from 13.0.1 to the latest available version in the Arch repository - 14.0.6. The problem still persists.
Opened an issue in the google/sanitizers repository.
Once more I have reread my question and comments, looked again at the code and additionally ran into this thought:
AddressSanitizer is not expected to produce false positives. If you
see one, look again; most likely it is a true positive!
As #Richard Critten and #chi have correctly pointed out in the comments section strdup function needs a NULL terminated string, so I changed my solution
from
(void) sanitize_cookie_path((char*) data);
to
char* string_ = new char[size + 1];
memcpy(string_, data, size);
string_[size] = 0x0;
(void) sanitize_cookie_path(string_);
delete[] string_;
The about solution converts the raw array of bytes data to a NULL terminated string string_ and passes it to the function. This solution works as it is expected.
It was just a stupid mistake that I had overlooked. Thanks again to #Richard Critten and #chi and everyone that tried to help.
Since there is no bug, I am going to retract my false accusations in google/sanitizers.

How to correctly determine fastest CDN, mirror, download server in C++

The question that I'm struggling with is how to determine in c++ that which is the server with fastest connection for the client do make git clone from or download tarball. So basically I want to choose from collection of known mirrors which one will be used for downloading content from.
Following code I wrote demonstrates that what I am trying to achieve more clearly perhaps, but I believe that's not something one should use in production :).
So lets say I have two known source mirrors git-1.exmple.com and git-2.example.com and I want to download tag-x.tar.gz from one which client has best connectivity to.
CDN.h
#include <iostream>
#include <cstdio>
#include <cstring>
#include <cstdlib>
#include <netdb.h>
#include <arpa/inet.h>
#include <sys/time.h>
using namespace std;
class CDN {
public:
long int dl_time;
string host;
string proto;
string path;
string dl_speed;
double kbs;
double mbs;
double sec;
long int ms;
CDN(string, string, string);
void get_download_speed();
bool operator < (const CDN&);
};
#endif
CDN.cpp
#include "CND.h"
CDN::CDN(string protocol, string hostname, string downloadpath)
{
proto = protocol;
host = hostname;
path = downloadpath;
dl_time = ms = sec = mbs = kbs = 0;
get_download_speed();
}
void CDN::get_download_speed()
{
struct timeval dl_started;
gettimeofday(&dl_started, NULL);
long int download_start = ((unsigned long long) dl_started.tv_sec * 1000000) + dl_started.tv_usec;
char buffer[256];
char cmd_output[32];
sprintf(buffer,"wget -O /dev/null --tries=1 --timeout=2 --no-dns-cache --no-cache %s://%s/%s 2>&1 | grep -o --color=never \"[0-9.]\\+ [KM]*B/s\"",proto.c_str(),host.c_str(),path.c_str());
fflush(stdout);
FILE *p = popen(buffer,"r");
fgets(cmd_output, sizeof(buffer), p);
cmd_output[strcspn(cmd_output, "\n")] = 0;
pclose(p);
dl_speed = string(cmd_output);
struct timeval download_ended;
gettimeofday(&download_ended, NULL);
long int download_end = ((unsigned long long)download_ended.tv_sec * 1000000) + download_ended.tv_usec;
size_t output_type_k = dl_speed.find("KB/s");
size_t output_type_m = dl_speed.find("MB/s");
if(output_type_k!=string::npos) {
string dl_bytes = dl_speed.substr(0,output_type_k-1);
double dl_mb = atof(dl_bytes.c_str()) / 1000;
kbs = atof(dl_bytes.c_str());
mbs = dl_mb;
} else if(output_type_m!=string::npos) {
string dl_bytes = dl_speed.substr(0,output_type_m-1);
double dl_kb = atof(dl_bytes.c_str()) * 1000;
kbs = dl_kb;
mbs = atof(dl_bytes.c_str());
} else {
cout << "Should catch the errors..." << endl;
}
ms = download_end-download_start;
sec = ((float)ms)/CLOCKS_PER_SEC;
}
bool CDN::operator < (const CDN& other)
{
if (dl_time < other.dl_time)
return true;
else
return false;
}
main.cpp
#include "CDN.h"
int main()
{
cout << "Checking CDN's" << endl;
char msg[128];
CDN cdn_1 = CDN("http","git-1.example.com","test.txt");
CDN cdn_2 = CDN("http","git-2.example.com","test.txt");
if(cdn_2 > cdn_1)
{
sprintf(msg,"Downloading tag-x.tar.gz from %s %s since it's faster than %s %s",
cdn_1.host.c_str(),cdn_1.dl_speed.c_str(),cdn_2.host.c_str(),cdn_2.dl_speed.c_str());
cout << msg << endl;
}
else
{
sprintf(msg,"Downloading tag-x.tar.gz from %s %s since it's faster than %s %s",
cdn_2.host.c_str(),cdn_2.dl_speed.c_str(),cdn_1.host.c_str(),cdn_1.dl_speed.c_str());
cout << msg << endl;
}
return 0;
}
So what are your thoughts and how would you approach this. What are the alternatives to replace this wget and achieve same clean way in c++
EDIT:
As #molbdnilo pointed correctly
ping measures latency, but you're interested in throughput.
So therefore I edited the demonstrating code to reflect that, however question remains same
For starters, trying to determine "fastest CDN mirror" is an inexact science. There is no universally accepted definition of what "fastest" means. The most one can hope for, here, is to choose a reasonable heuristic for what "fastest" means, and then measure this heuristic as precisely as can be under the circumstances.
In the code example here, the chosen heuristic seems to be how long it takes to download a sample file from each mirror via HTTP.
That's not such a bad choice to make, actually. You could reasonably make an argument that some other heuristic might be slightly better, but the basic test of how long it takes to transfer a sample file, from each candidate mirror, I would think is a very reasonable heuristic.
The big, big problem here I see here is the actual implementation of this heuristic. The way that this attempt -- to time the sample download -- is made, here, does not appear to be very reliable, and it will end up measuring a whole bunch of unrelated factors that have nothing do with network bandwidth.
I see at least several opportunities here where external factors completely unrelated to network throughput will muck up the measured timings, and make them less reliable than they should be.
So, let's take a look at the code, and see how it attempts to measure network latency. Here's the meat of it:
sprintf(buffer,"wget -O /dev/null --tries=1 --timeout=2 --no-dns-cache --no-cache %s://%s/%s 2>&1 | grep -o --color=never \"[0-9.]\\+ [KM]*B/s\"",proto.c_str(),host.c_str(),path.c_str());
fflush(stdout);
FILE *p = popen(buffer,"r");
fgets(cmd_output, sizeof(buffer), p);
cmd_output[strcspn(cmd_output, "\n")] = 0;
pclose(p);
... and gettimeofday() gets used to sample the system clock before and after, to figure out how long this took. Ok, that's great. But what would this actually measure?
It helps a lot here, to take a blank piece of paper, and just write down everything that happens here as part of the popen() call, step by step:
1) A new child process is fork()ed. The operating system kernel creates a new child process.
2) The new child process exec()s /bin/bash, or your default system shell, passing in a long string that starts with "wget", followed by a bunch of other parameters that you see above.
3) The operating system kernel loads "/bin/bash" as the new child process. The kernel loads and opens any and all shared libraries that the system shell normally needs to run.
4) The system shell process initializes. It reads the $HOME/.bashrc file and executes it, most likely, together with any standard shell initialization files and scripts that your system shell normally does. That itself can create a bunch of new processes, that have to be initialized and executed, before the new system shell process actually gets around to...
5) ...parsing the "wget" command it originally received as an argument, and exec()uting it.
6) The operating system kernel now loads "wget" as the new child process. The kernel loads and open any and all shared libraries that the wget process needs. Looking at my Linux box, "wget" loads no less than 25 separate shared libraries, including kerberos, and ssl libraries. Each one of those shared libraries get initialized.
7) The wget command performs a DNS lookup on the host, to obtain the IP address of the web server to connect to. If the local DNS server doesn't have the CDN mirror's hostname's IP address cached, it often takes several seconds to look up the CDN mirrors's DNS zone's authoritative DNS servers, then query them for the IP address, hopping this way and that way, across the intertubes.
Now, one moment... I seem have forgotten what we were trying to do here... Oh, I remember: which CDN mirror is "fastest", by downloading a sample file from each mirror, right? Yeah, that must be it!
Now, what does all of work done above, all of that work, have to do with determining which content mirror is the fastest???
Err... Not much, from the way it looks to me. Now, none of the above should really be such shocking news. After all, all of that is described in popen()'s manual page. If you read popen's manual page, it tells you that's ...what it does. Starts a new child process. Then executes the system shell, in order to execute the requested command. Etc, etc, etc...
Now, we're not talking about measuring time intervals that last many seconds, or minutes. If we're trying to measure something that takes a long time to execute, the relative overhead of popen()'s approach would be negligible, and not much to worry about. But the expected time to download the sample file, for the purpose of figuring out how fast each content mirror is -- I would expect that the actual download time would be relatively short. But it seems to me that the overhead to doing it this way, of forking an entirely new process, and executing first the system shell, then the wget command, with its massive list of dependencies, is going to be statistically significant.
And as I mentioned in the beginning, given that this is trying to determine the vaguely nebulous concept of "fastest mirror", which is already an inexact science -- it seems to me that you'd really want to get rid of as much utterly irrelevant overhead here -- as much as possible, in order to get as accurate of a result.
So, it seems to me that you don't really want to measure here anything other than what you're trying to measure: network bandwidth. And you certainly don't want to measure any of what transpires before any network activity takes place.
I still think that trying to time a sample download is a reasonable proposition. What's not reasonable here is all the popen and wget bloat. So, forget all of that. Throw it out the window. You want to measure how long it takes to download a sample file over HTTP, from each candidate mirror? Well, why don't you do just that?
1) Create a new socket().
2) Use getaddrinfo() to perform a DNS lookup, and obtain the candidate mirror's IP address.
3) connect() to the mirror's HTTP port.
4) Format the appropriate HTTP GET request, and send it to the server.
The above does pretty much what the popen/wget does, up to this point.
And only now I would start the clock running by grabbing the current gettimeofday(), then wait until I read the entire sample file from the socket, then grab the current gettimeofday() once more, to get the ending time of the transmission, and then calculate the actual time it took to receive the file from the mirror.
Only then, will I have some reasonable confidence that I'll be actually measuring the time it takes to receive a sample file from a CDN mirror, and completely ignoring the time it takes to execute a bunch of completely unrelated processes; and then by taking the same sample from multiple CDN mirrors, have any hope of picking one, using as much of a sensible heuristic, as possible.

Eclipse Invalid arguments error when using gstreamer

Ok, so I want to use gstreamer library.
1. Situation
I have some code:
#include <gst/gstpipeline.h>
#include <gst/gst.h>
...
GstElement* pipe = gst_pipeline_new("PipeName");
Where gst_pipeline_new is declared in gstpipeline.h:
GstElement* gst_pipeline_new (const gchar* name) G_GNUC_MALLOC;
where non obvious "things" :) are defined somewhere in the system:
typedef struct _GstElement GstElement; // gstelement.h
typedef char gchar; // gtypes.h
#define G_GNUC_MALLOC __attribute__((__malloc__)) // gmacros.h
2. Problem
Since I use make for building I have no errors during compilation and linking. Program itself runs OK as well. However...
In Eclipse IDE I have the following error:
Description Resource Path Location Type
Invalid arguments '
Candidates are:
_GstElement * gst_pipeline_new(const ? *)
' file.cc /path/to/file line 106 Semantic Error
I added all include directories which are specified in Makefile to eclipse project configuration (Project->Properties->C/C++ General->Paths and Symbols->Includes->C++). Of course it's a C++ project.
3. Question
How to get rid of that Eclipse error? I have no clue how to do this... And it drives me mad since now I use some legacy code and I have around 100 errors like this one.
So far I've tried:
casting either by reinterpret_cast<>() or C-like casting to const gchar*
adding typedef char gchar at the beginning of the file - before any other include!
including gtypes.h (gchar is defined there) - also before any other include
redeclaring `_GstElement gst_pipeline_new(const gchar* name)'
Nither of those helped...
To me it looks like Eclipse does not see the gchar type since it says that the candidate is _GstElement * gst_pipeline_new(const ? *) Where ? substitutes the real type. But I have no idea how to make (or event force :)) Eclipse to see it...
Most probably eclipse just doesn't know about your include paths (for this specific library) and complains about the unindexed types and declarations.
You can add them under 'Project->Properties->C++ General->Paths and Symbols'
If this doesn't help, you can also switch off semantic error checking (see Code Analysis), either in whole or for particular error types.
As g-maulik suggested, It seems that it was really an indexer problem. After increasing the indexer cache limits everything works fine.
Go to Window->Preferences->C/C++->Indexer tab cache limits and increase (might be machine dependent):
Index Database cache:
Limit relative to the maximum heap size: 15%
Absolute limit: 128 MB
Header file cache:
Absolute Limit: 128 MB

can GCC print out intermediate results?

Check the code below:
#include <avr/io.h>
const uint16_t baudrate = 9600;
void setupUART( void ) {
uint16_t ubrr = ( ( F_CPU / ( 16 * (float) baudrate ) ) - 1 + .5 );
UBRRH = ubrr >> 8;
UBRRL = ubrr & 0xff;
}
int main( void ) {
setupUART();
}
This is the command used to compile the code:
avr-gcc -g -DF_CPU=4000000 -Wall -Os -Werror -Wextra -mmcu=attiny2313 -Wa,-ahlmns=project.lst -c -o project.o project.cpp
ubrr is calculated by the compiler as 25, so far so good. However, to check what the compiler calculated, I have peek into the disassembly listing.
000000ae <setupUART()>:
ae: 12 b8 out UBRRH, r1 ; 0x02
b0: 89 e1 ldi r24, 0x19 ; 25
b2: 89 b9 out UBRRL, r24 ; 0x09
b4: 08 95 ret
Is it possible to make avr-gcc print out the intermediate result at compile time (or pull the info from the .o file), so when I compile the code it prints a line like (uint16_t) ubbr = 25 or similar? That way I can do a quick sanity check on the calculation and settings.
GCC has command line options to request that it dump out its intermediate representation after any stage of compilation. The "tree" dumps are in pseudo-C syntax and contain the information you want. For what you're trying to do, the -fdump-tree-original and -fdump-tree-optimized dumps happen at useful points in the optimization pipeline. I don't have an AVR compiler to hand, so I modified your test case to be self-contained and compilable with the compiler I do have:
typedef unsigned short uint16_t;
const int F_CPU = 4000000;
const uint16_t baudrate = 9600;
extern uint16_t UBRRH, UBRRL;
void
setupUART(void)
{
uint16_t ubrr = ((F_CPU / (16 * (float) baudrate)) - 1 + .5);
UBRRH = ubrr >> 8;
UBRRL = ubrr & 0xff;
}
and then
$ gcc -O2 -S -fdump-tree-original -fdump-tree-optimized test.c
$ cat test.c.003t.original
;; Function setupUART (null)
;; enabled by -tree-original
{
uint16_t ubrr = 25;
uint16_t ubrr = 25;
UBRRH = (uint16_t) ((short unsigned int) ubrr >> 8);
UBRRL = ubrr & 255;
}
$ cat test.c.149t.optimized
;; Function setupUART (setupUART, funcdef_no=0, decl_uid=1728, cgraph_uid=0)
setupUART ()
{
<bb 2>:
UBRRH = 0;
UBRRL = 25;
return;
}
You can see that constant-expression folding is done so early that it's already happened in the "original" dump (which is the earliest comprehensible dump you can have), and that optimization has further folded the shift and mask operations into the statements writing to UBRRH and UBRRL.
The numbers in the filenames (003t and 149t) will probably be different for you. If you want to see all the "tree" dumps, use -fdump-tree-all. There are also "RTL" dumps, which don't look anything like C and are probably not useful to you. If you're curious, though, -fdump-rtl-all will turn 'em on. In total there are about 100 tree and 60 RTL dumps, so it's a good idea to do this in a scratch directory.
(Psssst: Every time you put spaces on the inside of your parentheses, God kills a kitten.)
There might be a solution for printing intermediate results, but it will take you some time to be implemented. So it is worthwhile only for a quite large source code base.
You could customize your GCC compiler; either thru a plugin (painfully coded in C or C++) or thru a MELT extension. MELT is a high-level, Lisp-like, domain specific language to extend GCC. (It is implemented as a [meta-]plugin for GCC and is translated to C++ code suitable for GCC).
However such an approach requires you to understand GCC internals, then to add your own "optimization" pass to do the aspect oriented programming (e.g. using MELT) to print the relevant intermediate results.
You could also look not only the generated assembly (and use -fverbose-asm -S as options to GCC) but also perhaps in the generated Gimple representations (perhaps with -fdump-tree-gimple). For some interactive tool, consider the graphical MELT probe.
Perhaps adding your own builtin (with a MELT extension) like __builtin_display_compile_time_constant might be relevant.
I doubt there is an easy way to determine what the compiler does. There may be some tools in gcc specifically to dump the intermediate form of the language, but it will definitely not be easy to read, and unless you REALLY suspect that the compiler is doing something wrong (and have a VERY small example to show it), it's unlikely you can use it for anything meaningful - simply because it is too much work to follow what is going on.
A better approach is to add temporary variables (and perhaps prints) to your code, if you worry about it being correct:
uint16_t ubrr = ( ( F_CPU / ( 16 * (float) baudrate ) ) - 1 + .5 );
uint8_t ubrr_high = ubrr >> 8
uint8_t ubrr_low = ubrr & 0xff;
UBRRH = ubrr_high;
UBRRL = ubrr_low;
Now, if you have a non-optimized build and step through it in GDB, you should be able to see what it does. Otherwise, adding printouts of some sort to the code to show what the values are...
If you can't print it on the target system because you are in the process of setting up the uart that you will be using to print with, then replicate the code on your local host system and debug it there. Unless the compiler is very buggy, you should get the same values from the same compilation.
Here's a hack: simply automate what you are doing by hand now.
In your makefile, ensure that avr-gcc produces a disassembly (-ahlms=output.lst). Alternatively, use your own dissassembly method as a post-compile step in your makefile.
As a post-compilation step, process your listing file using your favorite scripting language to look for out UBRRH and out UBRRL lines. These are going to be loaded from registers, so your script can pull out the immediately preceeding assignments to registers that will be loaded into UBRRH and UBRRL. The script can then reassemble the UBRR value from the value loaded into the general-purpose registers which whhich are used to set UBRRH and UBRRL.
This sounds to be easier than Basile Starynkevich's very useful suggestion of MELT extension. Now, granted that this solution seems fragile, at first blush, so let's consider this issue:
We know that (at least on your processor) the lines out UBRR_, r__ will appear in the disassembly listing: there is simply no other way to set the registers/write data to port. One thing that might change is the spacing in/around these lines but this can be easily handled by your script
We also know that out instructions can only take place from general-purpose registers, so we know there will be a general-purpose register as the second argument to the out instruction line, so that should not be a problem.
Finally, we also know that this register will be set prior to out instruction. Here we must allow for some variability: instead of LDI (load immediate), avr-gcc might produce some other set of instructions to set the register value. I think as a first pass the script should be able to parse immediate loading, and otherwise dump whatever last instruction it finds involving the register that will be written to UBRR_ ports.
The script may have to change if you change platforms (some processors have UBRRH1/2 registers instea of UBRRH, however in that case you baud code will have to change. If the script complains that it can't parse the disassembly then you'll at least know that your check has not been performed.

Tracking code versions in an executable

I have a reasonable sized ( around 40k lines) machine learning system written in C++. This is still in active development and I need to run experiments regularly even as I make changes to my code.
The output of my experiments is captured in simple text files. What I would like to do when looking at these results is have some way of figuring out the exact version of the code that produced it. I usually have around 5 to 6 experiments running simultaneously, each on a slightly different version of the code.
I would like to know for instance that a set of results was obtained by compiling version 1 of file A, version 2 of file B etc (I just need some identifier and the output of "git describe" will do fine here ).
My idea is to somehow include this info when compiling the binary. This way, this can be printed out along with the results.
Any suggestions how this can be done in a nice way. In particular, any nice way of doing this with git?
I generate a single source file as part of my build process that looks like this:
static const char version_cstr[] = "93f794f674 (" __DATE__ ")";
const char * version()
{
return version_cstr;
}
Then its easy to log the version out on startup.
I originally used a DEFINE on the command line, but that meant every version change everything got recompiled by the build system - not nice for a big project.
Here's the fragment of scons I use for generating it, maybe you can adapt it to your needs.
# Lets get the version from git
# first get the base version
git_sha = subprocess.Popen(["git","rev-parse","--short=10","HEAD"], stdout=subprocess.PIPE ).communicate()[0].strip()
p1 = subprocess.Popen(["git", "status"], stdout=subprocess.PIPE )
p2 = subprocess.Popen(["grep", "Changed but not updated\\|Changes to be committed"], stdin=p1.stdout,stdout=subprocess.PIPE)
result = p2.communicate()[0].strip()
if result!="":
git_sha += "[MOD]"
print "Building version %s"%git_sha
def version_action(target,source,env):
"""
Generate file with current version info
"""
fd=open(target[0].path,'w')
fd.write( "static const char version_cstr[] = \"%s (\" __DATE__ \")\";\nconst char * version()\n{\n return version_cstr;\n}\n" % git_sha )
fd.close()
return 0
build_version = env.Command( 'src/autogen/version.cpp', [], Action(version_action) )
env.AlwaysBuild(build_version)
You can use $Id:$ in your source file, and Git will substitute that with the sha1 hash, if you add the file containing this phrase in .gitattributes with the option "ident" (see gitattributes).