I made a custom source block that is reading switch values on a zedboard. It is accessing them via a proc driver that I wrote. The /var/log/kern.log is reporting proper output. The debug printf in the source block is reporting proper output.
However pushing the data to a filesink as well as a GUI number sink is only reading zeros. Did I not set up the block properly?
#ifdef HAVE_CONFIG_H
#include "config.h"
#endif
#include <gnuradio/io_signature.h>
#include "switches_impl.h"
#include <stdio.h>
#include <stdlib.h>
#include <uinstd.h>
namespace gr {
namespace zedboard {
switches::sptr
switches::make()
{
return gnuradio::get_initial_sptr
(new switches_impl());
}
/*
* The private constructor
*/
switches_impl::switches_impl()
: gr::block("switches",
gr::io_signature::make(0,0,0),
gr::io_signature::make(1, 1, sizeof(unsigned int *)))
{}
/*
* Our virtual destructor.
*/
switches_impl::~switches_impl()
{
}
void
switches_impl::forecast (int noutput_items, gr_vector_int &ninput_items_required)
{
/* <+forecast+> e.g. ninput_items_required[0] = noutput_items */
}
int
switches_impl::general_work (int noutput_items,
gr_vector_int &ninput_items,
gr_vector_const_void_star &input_items,
gr_vector_void_star &output_items)
{
//const <+ITYPE+> *in = (const <+ITYPE+> *) input_items[0];
unsigned int *out = (unsigned int *) output_items[0];
// Do <+signal processing+>
// Tell runtime system how many input items we consumed on
// each input stream.
char buffer[5];
size_t size = 1;
size_t nitems = 5;
FILE* fp;
fp = fopen("/proc/zedSwitches","r");
if (fp == NULL)
{
printf("Cannot open for read\n");
return -1;
}
/*
Expect return format:
0x00
*/
fread(buffer, size, nitems, fp);
fclose(fp);
out=(unsigned int *)strtoul(buffer,NULL,0);
printf("read: 0x%02x",out);
consume_each (noutput_items);
// Tell runtime system how many output items we produced.
return noutput_items;
}
} /* namespace zedboard */
} /* namespace gr */
A pointer is a pointer to data, usually:
unsigned int *out = (unsigned int *) output_items[0];
out refers to the buffer for your output.
But you overwrite that pointer with another pointer:
out=(unsigned int *)strtoul(buffer,NULL,0);
which just bends around your copy of that pointer, and doesn't affect the content of that buffer at all. Basic C!
You probably meant to say something like:
out[0]= strtoul(buffer,NULL,0);
That will put your value into the first element of the buffer.
However, you tell GNU Radio that you not only produced a single item (the line above), but noutput_items:
return noutput_items;
That must read
return 1;
when you're only producing a single item, or you must actually produce as many items as you return.
Your consume_each call is nonsensical – GNU Radio Sources are typically instances of gr::sync_block, which means that you'd write a work() instead of a general_work() method as you did.
From the fact alone that this is a general_work and not a work I'd say you haven't used gr_modtool (with block type set to source!) to generate the stub for this block – you really should. Again, I'd like to point you to the Guided Tutorials which should really quickly explain usage of gr_modtool as well as the underlying GNU Radio concepts.
Related
I need to make Viterbi decoding of some convolutional-encoded signal. My application shall work with large files, therefore I cannot insert all the signal into a heap, so I need to process a data by a sequence of separate buffers. I have found a good library for Viterbi decoding - Encoder and a Viterbi decoder in C++ on the dr. Dobbs. I have applied the decoder from the libarary, it works correct, but doesn't provide a function for continuous use (call a function many times for each signal buffer with considering of previous calculations). Then I have found the GNU Radio C++ library which provide the necessary functions. But I don't understand how to use its functions, because it doesn't provide a documentation. It contains the example of Viterbi decoding with is present below:
extern "C" {
#include <gnuradio/fec/viterbi.h>
}
#include <cstdio>
#include <cmath>
#define MAXCHUNKSIZE 4096
#define MAXENCSIZE MAXCHUNKSIZE*16
int main()
{
unsigned char data[MAXCHUNKSIZE];
signed char syms[MAXENCSIZE];
int count = 0;
// Initialize metric table
int mettab[2][256];
int amp = 100; // What is it? ***
float RATE=0.5;
float ebn0 = 12.0;
float esn0 = RATE*pow(10.0, ebn0/10);
gen_met(mettab, amp, esn0, 0.0, 4);
// Initialize decoder state
struct viterbi_state state0[64];
struct viterbi_state state1[64];
unsigned char viterbi_in[16];
viterbi_chunks_init(state0);
while (!feof(stdin)) {
unsigned int n = fread(syms, 1, MAXENCSIZE, stdin);
unsigned char *out = data;
for (unsigned int i = 0; i < n; i++) {
// FIXME: This implements hard decoding by slicing the input stream
unsigned char sym = syms[i] > 0 ? -amp : amp; // What is it? ***
// Write the symbol to the decoder input
viterbi_in[count % 4] = sym;
// Every four symbols, perform the butterfly2 operation
if ((count % 4) == 3) {
viterbi_butterfly2(viterbi_in, mettab, state0, state1);
// Every sixteen symbols, perform the readback operation
if ((count > 64) && (count % 16) == 11) {
viterbi_get_output(state0, out);
fwrite(out++, 1, 1, stdout);
}
}
count++;
}
}
return 0;
}
File viterbi.c from it also contains the next function viterbi(), without a declaration:
/* Viterbi decoder */
int viterbi(unsigned long *metric, /* Final path metric (returned value) */
unsigned char *data, /* Decoded output data */
unsigned char *symbols, /* Raw deinterleaved input symbols */
unsigned int nbits, /* Number of output bits */
int mettab[2][256] /* Metric table, [sent sym][rx symbol] */
) { ...
Also I found one more implementation for Viterbi decoding - The Spiral project. But it also doesn't contain a normal description and doesn't want to compile. And two more implementation on the ExpertCore and Forward Error Correction DSP library.
My question: Can anyone understand how to use the above GNU Radio's implementation of the Viterbi algorithm for continuous use for binary interleaved digital signal (Encoder parameters of my signal: K=7 rate=1/2, every bit in my file is a demodulated sample of a signal)?
I've tried following another answer but just can't seem to get this right. I have about 8MiB of RBGX bitmap to convert to a jpeg in memory using libjpeg-turbo. If I use jpeg_stdio_dest I can write the whole thing to a file, then read the file back in, and it's all right. However, trying to use jpeg_mem_dest has been a puzzle. I have all the same set up as jpeg_stdio_dest, but using mem seems to only do one allocation of 4KiB and then never allocate any more space.
I can't find docs with further instruction on how to use jpeg_mem_dest, and could really use some direction.
void compress(std::vector<unsigned char>& input) {
jpeg_compress_struct cinfo{};
jpeg_error_mgr err{};
cinfo.err = jpeg_std_error(&err);
jpeg_create_compress(&cinfo);
#if 0 // using this with an open FILE* out works
jpeg_stdio_dest(&cinfo, out);
#endif
cinfo.image_width = kWidth; // constants defined somewhere
cinfo.image_height = kHeight;
cinfo.input_components = 4;
cinfo.in_color_space = JCS_EXT_RGBX;
// what's wrong with this?
unsigned char* buf{};
unsigned long buf_sz{};
jpeg_mem_dest(&cinfo, &buf, &buf_sz);
jpeg_set_defaults(&cinfo);
jpeg_set_quality(&cinfo, 70, true);
jpeg_start_compress(&cinfo, true);
while (cinfo.next_scanline < cinfo.image_height) {
auto row = static_cast<JSAMPROW>(&input[cinfo.next_scanline * 4 * kWidth]);
jpeg_write_scanlines(&cinfo, &row, 1);
// Always prints 4096, and buf never changes
std::cout << "buf_sz: " << buf_sz
<< " buf: " << static_cast<void*>(buf) << '\n';
}
jpeg_finish_compress(&cinfo);
// ...
// in reality, return the compressed data
}
Yeah, that is not intuitive at all. The programmer that proposed the jpeg_mem_dest() tweak did not have much choice, extending an existing api is not that easy when it wasn't designed to support a feature in the first place. What is completely unobvious is that your variables don't get updated until after the jpeg_finish_compress() call. Relevant code in the library is:
METHODDEF(void)
term_mem_destination (j_compress_ptr cinfo)
{
my_mem_dest_ptr dest = (my_mem_dest_ptr) cinfo->dest;
*dest->outbuffer = dest->buffer;
*dest->outsize = (unsigned long)(dest->bufsize - dest->pub.free_in_buffer);
}
Note the word "term". This function is called indirectly through a function pointer:
GLOBAL(void)
jpeg_finish_compress (j_compress_ptr cinfo)
{
//...
/* Write EOI, do final cleanup */
(*cinfo->marker->write_file_trailer) (cinfo);
(*cinfo->dest->term_destination) (cinfo);
//...
}
Nothing much you can do about it. Just tweak your std::cout code, move it after the loop to accommodate the way the library works.
Beware the other gritty detail of this function, also not obvious. You have to free() the buffer that it created. Visible in the provided cjpeg.c sample program, end of main().
I'm trying to write a image via UART on Beaglebone Black. But when I use the write() function in library .
int write(int handle, void *buffer, int nbyte);
Regardless of the agurment nbyte has int type, I can not transfer 70kB at once. I displayed the number of byte which is transfered, and the result is the number of byte = 4111.
length = write(fd,body.c_str(),strlen(body.c_str())); //
cout<<length<<endl; // result length = 4111;
cout<<strlen(body.c_str())<<endl; // result strlen(body.c_str()) = 72255;
I hope to hear from you!
The write call does not assure that you can write the amount of data supplied, that's why it as an integer as its return, not a Boolean. The behavior you see is actually common among different operating systems, it may be due to the underline device might does not have sufficient buffer or storage for you to write 70kb. What you need is to write in a loop, each write will write the amount that is left unwritten:
int total = body.length(); // or strlen(body.c_str())
char *buffer = body.c_str();
int written = 0;
int ret;
while (written < total) {
ret = write(fd, buffer + written, total - written);
if (ret < 0) {
// error
break;
}
written += ret;
}
I am loading a 10GB file into memory and I find that even if I strip away any extra overhead and store the data in nothing but an array it still takes up 53 GB of ram. This seems crazy to me since I am converting some of the text data to longs which take up less room and convert the rest to char * which should take up the same amount of room as a text file. I have about 150M rows of data in the file I am trying to load. Is there any reason why this should take up so much ram when I load it the way I do below?
There are three files here a fileLoader class and its header file and a main that simply runs them.
To answer some questions:
OS is UBUNTU 12.04 64bit
This is on a machien with 64GB of RAM and an SSD hd that I have providing 64GB of swap space for RAM
I am loading all of the data at once becuase of the need for speed. It is critical for the application. All sorting, indexing, and lots of the data intensive work runs on the GPU.
The other reason is that loading all of the data at once made it much simpler for me to write the code. I dont have to worry about indexed files, and mappings to locations in another file for example.
Here is the header file:
#ifndef FILELOADER_H_
#define FILELOADER_H_
#include <iostream>
#include <fstream>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <string>
class fileLoader {
public:
fileLoader();
virtual ~fileLoader();
void loadFile();
private:
long long ** longs;
char *** chars;
long count;
long countLines(std::string inFile);
};
#endif /* FILELOADER_H_ */
Here is the CPP file
#include "fileLoader.h"
fileLoader::fileLoader() {
// TODO Auto-generated constructor stub
this->longs = NULL;
this->chars = NULL;
}
char ** split(char * line,const char * delim,int size){
char ** val = new char * [size];
int i = 0;
bool parse = true;
char * curVal = strsep(&line,delim);
while(parse){
if(curVal != NULL){
val[i] = curVal;
i++;
curVal = strsep(&line,delim);
}else{
parse = false;
}
}
return val;
}
void fileLoader::loadFile(){
const char * fileName = "/blazing/final/tasteslikevictory";
std::string fileString(fileName);
//-1 since theres a header row and we are skipinig it
this->count = countLines(fileString) -1;
this->longs = new long long*[this->count];
this->chars = new char **[this->count];
std::ifstream inFile;
inFile.open(fileName);
if(inFile.is_open()){
std::string line;
int i =0;
getline(inFile,line);
while(getline(inFile,line)){
this->longs[i] = new long long[6];
this->chars[i] = new char *[7];
char * copy = strdup(line.c_str());
char ** splitValues = split(copy,"|",13);
this->longs[i][0] = atoll(splitValues[4]);
this->longs[i][1] = atoll(splitValues[5]);
this->longs[i][2] = atoll(splitValues[6]);
this->longs[i][3] = atoll(splitValues[7]);
this->longs[i][4] = atoll(splitValues[11]);
this->longs[i][5] = atoll(splitValues[12]);
this->chars[i][0] = strdup(splitValues[0]);
this->chars[i][1] = strdup(splitValues[1]);
this->chars[i][2] = strdup(splitValues[2]);
this->chars[i][3] = strdup(splitValues[3]);
this->chars[i][4] = strdup(splitValues[8]);
this->chars[i][5] = strdup(splitValues[9]);
this->chars[i][6] = strdup(splitValues[10]);
i++;
delete[] splitValues;
free(copy);
}
}
}
fileLoader::~fileLoader() {
// TODO Auto-generated destructor stub
if(this->longs != NULL){
delete[] this->longs;
}
if(this->chars != NULL){
for(int i =0; i <this->count;i++ ){
free(this->chars[i]);
}
delete[] this->chars;
}
}
long fileLoader::countLines(std::string inFile){
int BUFFER_SIZE = 16*1024;
int fd = open(inFile.c_str(), O_RDONLY);
if(fd == -1)
return 0;
/* Advise the kernel of our access pattern. */
posix_fadvise(fd, 0, 0, 1); // FDADVICE_SEQUENTIAL
char buf[BUFFER_SIZE + 1];
long lines = 0;
while(size_t bytes_read = read(fd, buf, BUFFER_SIZE))
{
if(bytes_read == (size_t)-1)
return 0;
if (!bytes_read)
break;
for(char *p = buf; (p = (char*) memchr(p, '\n', (buf + bytes_read) - p)); ++p)
++lines;
}
return lines;
}
Here is the file with my main function:
#include "fileLoader.h"
int main()
{
fileLoader loader;
loader.loadFile();
return 0;
}
Here is an example of the data that I am loading:
13|0|1|1997|113|1|4|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
14|0|1|1997|113|1|5|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
15|0|1|1997|113|1|6|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
16|0|1|1997|113|1|7|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
17|0|1|1997|113|1|8|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
18|0|1|1997|113|1|9|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
19|0|1|1997|113|1|10|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
20|0|1|1997|113|1|11|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
21|0|1|1997|113|1|12|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
9|0|1|1997|113|1|13|12408012|C9FF921CA04ADA3D606BF6DAC4A0B092|SEMANAL|66C5E828DC69F857ADE060B8062C923E|113|1
27|0|1|1992|125|1|1|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
28|0|1|1992|125|1|2|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
29|0|1|1992|125|1|3|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
30|0|1|1992|125|1|4|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
31|0|1|1992|125|1|5|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
32|0|1|1992|125|1|6|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
33|0|1|1992|125|1|7|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
34|0|1|1992|125|1|8|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
35|0|1|1992|125|1|9|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
36|0|1|1992|125|1|10|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
37|0|1|1992|125|1|11|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
38|0|1|1992|125|1|12|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
39|0|1|1992|125|1|13|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
40|0|1|1992|125|1|14|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
41|0|1|1992|125|1|15|10183|9EF534D2CF74B24AC28CBD9BE937A412|SEMANAL|375CCE505F5353CCDE85D4E84A9888D8|125|1
10|0|1|1996|126|1|1||||||
You are allocating nine chunks of memory for each line, so you are allocating a total of 1350 million pieces of memory. These allocations have a certain overhead, usually at least twice the size of a pointer, possibly even more. On a 64 bit machine, that is already 16 bytes, so you get 21.6 GB of overhead.
In addition to that, you get the overhead of heap fragmentation and alignment: Even if you only ever store a string in it, the allocator has to align the memory allocations so that you can store the largest possible values in it without triggering misalignment. Alignment may depend on the vector unit of your CPU, which can require very significant alignments, 16 byte alignment not being uncommon.
Doing the calculation with 16 bytes allocation overhead and 16 bytes alignment, we get allocations of 43.2 GB without the original data. With the original data this calculation is already very close to your measurement.
Each of those objects and strings you create has individual memory management overhead. So you load the string "0" from column 2, depending on your memory manager, it probably takes between two and four full words (could be more). Call it 16 to 32 bytes of storage to hold a one byte string. Then you load the "1" from column 3. And so on.
Is it possible, for a c++ program, to track how much memory the program is using at one time?
For example, a function with a prototype:
int getEstimatedTotalMemoryUsage();
I suppose if it's not possible, then one will have to get out of the program, do a system call and check the results from there. If so, what tools are available for such purposes? Assuming such a thing is possible, that is.
edit: I'm using linux, any tools that can do this for you?
Yes - use POSIX getrusage. From the Linux man page:
Synopsis
#include <sys/time.h>
#include <sys/resource.h>
int getrusage(int who, struct rusage *usage);
Description
getrusage() returns current resource usages, for a who of either RUSAGE_SELF or RUSAGE_CHILDREN. The former asks for resources used by the current process, the latter for resources used by those of its children that have terminated and have been waited for.
struct rusage {
struct timeval ru_utime; /* user time used */
struct timeval ru_stime; /* system time used */
long ru_maxrss; /* maximum resident set size */
long ru_ixrss; /* integral shared memory size */
long ru_idrss; /* integral unshared data size */
long ru_isrss; /* integral unshared stack size */
long ru_minflt; /* page reclaims */
long ru_majflt; /* page faults */
long ru_nswap; /* swaps */
long ru_inblock; /* block input operations */
long ru_oublock; /* block output operations */
long ru_msgsnd; /* messages sent */
long ru_msgrcv; /* messages received */
long ru_nsignals; /* signals received */
long ru_nvcsw; /* voluntary context switches */
long ru_nivcsw; /* involuntary context switches */
};
Here is an example of measuring memory used by process on Windows.
#include <windows.h>
#include <Psapi.h>
// [...]
PROCESS_MEMORY_COUNTERS memCounter;
BOOL result = K32GetProcessMemoryInfo(GetCurrentProcess(), &memCounter, sizeof(memCounter));
std::cout << "WorkingSetSize " << memCounter.WorkingSetSize << std::endl;
And explanations of returned values https://learn.microsoft.com/en-gb/windows/win32/api/psapi/ns-psapi-process_memory_counters
I wanted this today, myself, so sharing the tested results here. I believe a call to getmem() will do what the OP asked, on any unix box. Written in very generic C, it will work in C or C++.
// Calling function must free the returned result.
char* exec(const char* command) {
FILE* fp;
char* line = NULL;
// Following initialization is equivalent to char* result = ""; and just
// initializes result to an empty string, only it works with
// -Werror=write-strings and is so much less clear.
char* result = (char*) calloc(1, 1);
size_t len = 0;
fflush(NULL);
fp = popen(command, "r");
if (fp == NULL) {
printf("Cannot execute command:\n%s\n", command);
return NULL;
}
while(getline(&line, &len, fp) != -1) {
// +1 below to allow room for null terminator.
result = (char*) realloc(result, strlen(result) + strlen(line) + 1);
// +1 below so we copy the final null terminator.
strncpy(result + strlen(result), line, strlen(line) + 1);
free(line);
line = NULL;
}
fflush(fp);
if (pclose(fp) != 0) {
perror("Cannot close stream.\n");
}
return result;
}
int getmem() {
pid_t pid = getpid();
char cmd[64];
snprintf(cmd, 64, "/bin/ps -p %d -o size", pid);
char* result = exec(cmd);
if (!result) {
return 0;
}
// Find first newline.
int pos = 0;
while (result[pos] != '\n') {
pos++;
}
// Remove the final newline.
result[strlen(result) - 1] = '\0';
// Convert to integer.
int size = atoi(result + pos + 1);
free(result);
return size;
}
Technically, I suppose the printf(...) line should be fprintf(stderr, ...), but I tend to have stderr redirected for certain environment-specific logging reasons, and this is how I compiled and tested the code, so I'm copying verbatim to avoid breakage.
Get your PID: pid_t getpid(void); // unistd.h
Parse /proc/<id>/smaps
If you don't care about shared libraries in mem total it may be simpler
make a system call to ps -p <id> -o %mem