I'm trying to store double precision data from different blocks into a direct access file, i.e. the data is g(m,n) for one block and they all have the same size. Here's the code I wrote:
OPEN(3,FILE='a.TMP',ACCESS='DIRECT',RECL=8*m*n)
WRITE(3,REC=I) ((g(K,L),K=1,m),L=1,n) ! here "I" is the block number
I have 200 this kind of blocks. However, I got the following error after writing the 157th block data into the file:
severe (66): output statement overflows record, unit 3
I believe that means the record size is too large.
Is there any way to handle this? I wonder if there is the record number has a maximum value.
Related
I'm realizing UART-DMA with STM_HAL library and I want to know if message size is counted by hardware (counting clock ticks till line is idle for example) or by some program method(something like strlen). So if Size in
HAL_UARTEx_RxEventCallback(UART_HandleTypeDef *huart, uint16_t Size)
is counted by hardware, I can send data in pure HEX format, but if it is calculated by something like strline, I may recieve problems if data is 0x00 and have to send data in ASCII.
I've tried to make some research in generated code in Keil but failed (maybe I didn't try hard enough) so maybe somebody can help me.
If you are using UART DMA, it is calculated by hardware.
If you check the call hierarchy of HAL_UARTEx_RxEventCallback using your ide, you can see how the Size variable is calculated.
The function is executed in the following flow.(Depending on the version of HAL Driver, it may be slightly different)
UART Idle Interrupt occur
Call HAL_UART_IRQHandler()
If DMA mod is enabled, Call HAL_UARTEx_RxEventCallback(huart, (huart->RxXferSize - huart->RxXferCount))
Therefore, Size variable is calculated as (huart->RxXferSize - huart->RxXferCount)
huart->RxXferSize is a set value when initializing RX DMA.
huart->RxXferCount is (huart->hdmarx)->Instance->NDTR
NDTR is a value calculated by hardware as the size of the buffer remaining after DMA transfer data to memory!!
I'm trying to communicate with a device over a serial communication protocol and having some trouble finding out about which checksum/crc algorithm that is used for the last 2 bytes of the messages. I've tried several CRC16 algorithms in various online crc utilities, like:
http://www.sunshine2k.de/coding/javascript/crc/crc_js.html
http://www.zorc.breitbandkatze.de/crc.html
I've also tried reverse engineering, with the help of REVENG software, but it only gives some occasional random hits (depending on which of the examples from the captured messages I try together), that does not seem to be a correct algorithm that matches all examples.
I have not found any documentation of the device, which can indicate the CRC16 algorithm used or if its some other variant like the lowest bytes of a CRC32.
Below are 2 types of messages each with some different examples and variations. The first 4 bytes of the message tells the renaming number of bytes of the message. Most probably these 4 first bytes should not be included in the CRC calculation, but that is just a guess. What I believe is a 16 wide CRC is the last 2 bytes of each message.
Message type 1 (examples):
0000000908100300180a4621a8
0000000901100300180a463a11
0000000909100300180a461f26
0000000902100300180a4649f9
000000090a100300180a466cce
0000000903100300180a46fb58
000000090b100300180a46de6f
Message type 2 (examples):
0000001f09131900180a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a7be0
0000001f0913190018141414141414141414141414141414141414141414141414f3a5
0000001f09131900181e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e1e3d38
0000001f0913190018282828282828282828282828282828282828282828282828e82f
0000001f09131900183232323232323232323232323232323232323232323232321762
0000001f001319001814ffffffffffffffffffffffffffffffffffffffffffffff3d16
0000001f00131900181effffffffffffffffffffffffffffffffffffffffffffff2e93
0000001f001319001828ffffffffffffffffffffffffffffffffffffffffffffff3438
0000001f00131900185fffffffffffffffffffffffffffffffffffffffffffffffac2b
Anyone out there with some knowledge about CRC that can point me in the right direction to figure this out?
It's not a CRC-16. I ran a simple brute-force search.
Some more messages where only one or two of the last "data" bytes are varying that show some sort of more logical behaviour on the last 2 checksum bytes, but still no clear view of how the algorithm is constructed for the calculation.
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF065B7C
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF075A7C
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF08557C
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF09547C
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0A577C
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0B567C
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0C517C
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF06FFA285
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF07FFA284
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF08FFA28B
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF09FFA28A
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0AFFA289
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0BFFA288
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0CFFA28F
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0DFFA28E
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF06065B85
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF07075A84
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0808558B
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0909548A
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0A0A5789
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0B0B5688
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0C0C518F
0113190018FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0D0D508E
Full set of 1200 messages at: http://dpaste.com/0DSMZJV
I am having trouble getting a custom block to operate at high frequency.
The block I would like to use is going to take in data from an external radio.
I am using an Ettus USRP block to stream data in from this radio, and I can display this on the QT Scope. I can set this block's sample rate to 15 MHz, and with the scope this seems to work ok.
Problem:
I have tried making a simple block with the gnuradio gr_modtool which takes in 2 floats as input and has 0 outputs. The block has private members "timer", a time_t, and "counter", an int. In the "work" function, my code simply does this at the moment:
const float *in_i = (const float *) input_items[0];
const float *in_q = (const float *) input_items[1];
if (count == 0){
if (*in_i > 0.5){
timer = clock();
count = 30000;
}
}else{
count --;
if(count == 0){
timer = clock()-timer;
printf("Count took %d clicks, or %f seconds\n",timer,(float)timer/CLOCKS_PER_SEC);
}
}
// Tell runtime system how many output items we produced.
return 0;
However, when I run this code, it takes longer than the expected time.
For 30000 cycles, it takes 0.872970 to complete, instead of the desired 0.002 seconds. Since the standard gnuradio block generated with gr_modtool is a sync block, and the input stream to the block is coming from the 15 MHz USRP, I would have expected this block to run at that same frequency. This is not currently the case.
Eventually my goal is to be able to store data streaming in over a period of time, and write it to file with certain formatting(A block already exists to do this, but there is some sort of bug that is preventing that block and the USRP block from working at the same time, so I am attempting to write my own.). However, unless I can keep up with the sample rate of 15 MHz, I will lose data. Since this block is fairly simple, I would have hoped it would be able to run quickly enough to keep up. However, the input stream block is able to pull data from the radio and output at 15 MHz, so I know my computer is capable of it.
How can I make this custom block operate more quickly, and keep up with the 15 MHz frequency?(Or, how can I make this sync block operate at the input stream frequency, since it currently does not)
Your block is not consuming any samples. I presume you're writing a sync_block (work function, not general_work), so your number of produced items is identical to the number of consumed items. But as your source code says:
// Tell runtime system how many output items we produced.
return 0;
In other words, your block tells GNU Radio that it didn't use any of the input GNU Radio offered, and produced no output. That means GNU Radio can't do nothing. You must return the number of items you've produced, and for sync blocks, that's the number of items you consumed – even if you're a sink, with zero output streams!
I've run into some trouble when reading chunks of data at random locations all over a big file (>4GB).
The task is to save a 3D datacube to a file and transpose the axes while not loading the whole dataset into RAM.
The storage format is as follows:
I've got 3 Integer at the beginning of the File, storing the dimensions (nX, nY, nZ ).
After that follows the data as lines with length nX.
These Lines are repeated nY times which results in a page and the pages are repeated nZ times.
Meaning:
A line has nX bytes
A page has nX * nY bytes
The file has nX * nY * nZ + 12 bytes
To transpose the dataset i execute the following loop:
for( int i=0;i<nY;i++ )
{
for( int j=0;j<nZ;j++ )
{
read( pBuf, i*nX+j*nY*nX );//read nX bytes from offset i*nX+j*nX*nY
writeNext(pBuf);
}
}
When using fopen, _fseeki64 and fread it happens that after approx. 30% of the overall reads every 6th read or so takes up to 7 s, since there are multiple millions of those reads i can't accept these delays.
Thus i implemented the same algorithm with memory mapped files (CreateFile, CreateFileMapping and MapViewOfFile), but now every 6th read takes about 2 s.
Is there a method/chance of increasing the readout speed?
EDIT1:
I've added some code at http://pastebin.com/MejiTKj0
EDIT2:
Some may notice an inconsistency regarding the offset in the read function. To simplify matters i didn't tell about all variables saved in the file header thus the offset of 15 bytes is okay
If you have a HDD disk on which the files are stored, you should know that seek times dominate heavily when trying to perform random access. You may find you're better off reading the entire file sequentially into memory (a relatively quick operation compared to seek) and then performing your processing on the memory data instead. You may find this is quicker even if you need only a relatively small percentage of the overall file data.
In your loop Z / nZ should be outer most loop and Y should be inner loop. That would save seek times, if the storage memory layout has stored a nZ pages one by one .
In the current code displayed it shows nZ in inner loop, which is no good. The current arrangement of loops is analogous to book reading, with reading first line for each page of the book, then reading second line and so on;
Thank you all very much for your input.
Actually the first thing i should have checked was at fault, being the HDD, which wasn't able to provide the needed datarate.
I'm now thinking about switching to a SSD - Device.
Can any one tell me what does portaudio callback function variable framesperbuffer is?
If i want to play audio stream through PA_WriteStream() by 64 bytes data every iteration then what value i should put in the framesperbuffer?
Also in lsbsndfilelibrary the function for reading wave file expects variable with name frame to be provided.
i.e.
samples=sf_readf_float(file,fptr,frames);
if i put frames=256 then always 64 samples are returned in fptr and rest are garbage whereas returned values from read function is 256.
I have checked through following code
memcpy(array,fptr,samples); //samples returned are 256 always but first 64 contain data
now array[0] to array[63] contain values and array[64] to array[255] contain null value in every iteration of file read.
Now i have to write data read to portaudio audio playing function then what framesperbuffer should be filled in with.
Also in some cases i need to process data and samples reduce to 32 (when i consume two samples to form one output sample)then what value should i put in the framesperbuffer variable?
framesPerBuffer The number of frames passed to the stream callback function, or the preferred block granularity for a blocking read/write stream. The special value paFramesPerBufferUnspecified (0) may be used to request that the stream callback will receive an optimal (and possibly varying) number of frames based on host requirements and the requested latency settings. Note: With some host APIs, the use of non-zero framesPerBuffer for a callback stream may introduce an additional layer of buffering which could introduce additional latency. PortAudio guarantees that the additional latency will be kept to the theoretical minimum however, it is strongly recommended that a non-zero framesPerBuffer value only be used when your algorithm requires a fixed number of frames per stream callback.