I'm converting some OpenCL code to DirectCompute and need to process 8-bit character strings in a compute shader but don't find an HLSL data type for "byte" or "char". OpenCL supports a "char" type, so I was expecting an equivalent. What is the best way to define and access the data?
It seems that the data can be passed by treating it as a series of "uint" types and unpacking it with bit-shifting, AND-ing, etc. but this seems like it will cause unnecessary overhead. What is the correct way?
I've found two ways to do this, although they both require working with int/uint values in the HLSL since I haven't found an 8-bit data type:
Option 1 is to let the "view" handle the translation:
Pass the original data as a byte/char buffer.
Set the Shader Resource View format (D3D11_SHADER_RESOURCE_VIEW_DESC.Format) to DXGI_FORMAT_R8_UINT
Define the HLSL data type as Buffer<uint>
Reference each byte using its byte offset (i.e., treat it as a buffer of bytes not a buffer of uints). Each character is
automatically promoted to a uint value.
Option 2 is to treat each 4-byte sequence as a uint, using the format DXGI_FORMAT_R32_UINT, and manually extract each character using something like this:
Buffer<uint> buffer;
uint offset = ...;
uint ch1, ch2, ch3, ch4;
ch1 = buffer[offset] >> 24;
ch2 = (buffer[offset] & 0x00ff0000) >> 16;
ch3 = (buffer[offset] & 0x0000ff00) >> 8;
ch4 = (buffer[offset] & 0x000000ff);
Either way you end up working with 32-bit values but at least they correspond to individual characters.
Related
my task is to read metadata values from a unsigned char array, which contains the bytes of a binary .shp file (Shapefile)
unsigned char* bytes;
The header of the file which is stored in the array and the order of the information stored in it looks like this:
int32_t filecode // BigEndian
int32_t skip[5] // Uninteresting stuff
int32_t filelength // BigEndian
int32_t version // LitteEndian
int32_t shapetype // LitteEndian
// Rest of the header and of the filecontent which I don't need
So my question would be how can I extract this information (except the skip part of course) under consideration of the endianness and read it into the according variables.
I thought about using ifstream, but I couldnt figure out how to do it properly.
Example:
Read the first four bytes of the binary, ensure big endian byte order, store it in a int32_t. Then skip 5* 4 Bytes (5 * int32). Then read four bytes, ensure big endian byte order, and store it in a int32_t. Then read four bytes, ensure little endian byte order, and again store it in a int32_t and so on.
Thanks for your help guys!
So 'reading' a byte array just means extracting the bytes from the positions in the byte array where you know your data is stored. Then you just need to do the appropriate bit manipulations to handle the endianess. So for example, filecode would be this
filecode = (bytes[0] << 24) | (bytes[1] << 16) | (bytes[2] << 8) | bytes[3];
and version would be this
version = bytes[13] | (bytes[14] << 8) | (bytes[15] << 16) | (bytes[16] << 24);
(An offset of 13 for the version seems a bit odd, I'm just going on what you stated above).
I'm implementing sha512 in OpenCL technology. I have simple definition of kernel function
__kernel void _sha512(__global char *message, const uint length, __global char *hash);
On host I have implemented and successfully tested implementation of sha512 algorithm.
I have a problem with copy data from message array to temporary variable called character.
char character = message[i];
Where i is a loop variable - in range from 0 to message's size.
When I tried to run my program there I got this errors
0x00007FFD9FA03D54 (0x0000000010CD0F88 0x0000000010CD0F88 0x0000000010BAEE88 0x000000001A2942A0), nvvmCompilerProperty() + 0x26174 bytes(s)
...
0x00007FFDDFA70D51 (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), RtlUserThreadStart() + 0x21 bytes(s)
0x00007FFDDFA70D51 (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), RtlUserThreadStart() + 0x21 bytes(s)
I readed about async_work_group_copy() but I can't understand how to use it - in docs I can't found any example code.
I have tried with char character = (__private char) message[i]; but it's not working too.
I don't understand how to pass last parameter into async_work_group_copy() and how to use it to copy data from __global memory into __private memory.
OpenCL by default does not allow single-byte access in kernels: memory access needs to be in multiples of 4 bytes, aligned to 4-byte boundaries. If your implementation supports it, you can enable byte-wise memory accesses. This involves the cl_khr_byte_addressable_store extension, which you need to check for and explicitly enable in your kernel source. Give that a try and see if it solves your problem.
To use async_work_group_copy, try something like this:
#define LOCAL_MESSAGE_SIZE 64 // or some other suitable size for your workgroup
__local char local_message[LOCAL_MESSAGE_SIZE];
event_t local_message_ready = async_work_group_copy(local_message, message, LOCAL_MESSAGE_SIZE, 0);
// ...
// Just before you need to use local_message's content:
wait_group_events(1, &local_message_ready);
// Use local_message from here onwards
Note that async_work_group_copy is not required; you can access global memory directly. Which will be faster depends on your kernel, OpenCL implementation, and hardware.
Another option (the only option if your implementation/hardware do not support cl_khr_byte_addressable_store) is to fetch your data in chunks of at least 4 bytes. Declare your message as a __global uint* and unpack the bytes by shifting and masking:
uint word = message[i];
char byte0 = (word & 0xff);
char byte1 = ((word >> 8) & 0xff);
char byte2 = ((word >> 16) & 0xff);
char byte3 = ((word >> 24) & 0xff);
// use byte0..byte3 in your algorithm
Depending on implementation, hardware, etc. you may find this to be faster than bytewise access. (You may want to check if you need to reverse the unpacking by reading the CL_DEVICE_ENDIAN_LITTLE property using clGetDeviceInfo if you're not sure if all your deployment platforms will be little-endian.)
I have a process that listens to an UDP multi-cast broadcast and reads in the data as a unsigned char*.
I have a specification that indicates fields within this unsigned char*.
Fields are defined in the specification with a type and size.
Types are: uInt32, uInt64, unsigned int, and single byte string.
For the single byte string I can merely access the offset of the field in the unsigned char* and cast to a char, such as:
char character = (char)(data[1]);
Single byte uint32 i've been doing the following, which also seems to work:
uint32_t integer = (uint32_t)(data[20]);
However, for multiple byte conversions I seem to be stuck.
How would I convert several bytes in a row (substring of data) to its corresponding datatype?
Also, is it safe to wrap data in a string (for use of substring functionality)? I am worried about losing information, since I'd have to cast unsigned char* to char*, like:
std::string wrapper((char*)(data),length); //Is this safe?
I tried something like this:
std::string wrapper((char*)(data),length); //Is this safe?
uint32_t integer = (uint32_t)(wrapper.substr(20,4).c_str()); //4 byte int
But it doesn't work.
Thoughts?
Update
I've tried the suggest bit shift:
void function(const unsigned char* data, size_t data_len)
{
//From specifiction: Field type: uInt32 Byte Length: 4
//All integer fields are big endian.
uint32_t integer = (data[0] << 24) | (data[1] << 16) | (data[2] << 8) | (data[3]);
}
This sadly gives me garbage (same number for every call --from a callback).
I think you should be very explicit, and not just do "clever" tricks with casts and pointers. Instead, write a function like this:
uint32_t read_uint32_t(unsigned char **data)
{
const unsigned char *get = *data;
*data += 4;
return (get[0] << 24) | (get[1] << 16) | (get[2] << 8) | get[3];
}
This extracts a single uint32_t value from a buffer of unsigned char, and increases the buffer pointer to point at the next byte of data in the buffer.
This assumes big-endian data, you need to have a well-defined idea of the buffer's endian-mode in order to interpret it.
Depends on the byte ordering of the protocol, for big-endian or so called network byte order do:
uint32_t i = data[0] << 24 | data[1] << 16 | data[2] << 8 | data[3];
Without commenting on whether it's a good idea or not, the reason why it doesn't work for you is that the result of wrapper.substring(20,4).c_str() is (uint32_t *), not (uint32_t). So if you do:
uint32_t * integer = (uint32_t *)(wrapper.substr(20,4).c_str(); it should work.
uint32_t integer = ntohl(*reinterpret_cast<const uint32_t*>(data + 20));
or (handles alignment issues):
uint32_t integer;
memcpy(&integer, data+20, sizeof integer);
integer = ntohl(integer);
The pointer way:
uint32_t n = *(uint32_t*)&data[20];
You will run into problems on different endian architectures though. The solution with bit shifts is better and consistent.
std::string wrapper((char*)(data),length); //Is this safe?
This should be safe since you specified the length of the data.
On the other hand if you did this:
std::string wrapper((char*)data);
The string length would be determined wherever the first 0 byte occurs, and you will more than likely chop off some data.
What's the best way to send float, double, and int16 over serial on Arduino?
The Serial.print() only sends values as ASCII encoded. But I want to send the values as bytes. Serial.write() accepts byte and bytearrays, but what's the best way to convert the values to bytes?
I tried to cast an int16 to an byte*, without luck. I also used memcpy, but that uses to many CPU cycles. Arduino uses plain C/C++. It's an ATmega328 microcontroller.
hm. How about this:
void send_float (float arg)
{
// get access to the float as a byte-array:
byte * data = (byte *) &arg;
// write the data to the serial
Serial.write (data, sizeof (arg));
}
Yes, to send these numbers you have to first convert them to ASCII strings. If you are working with C, sprintf() is, IMO, the handiest way to do this conversion:
[Added later: AAAGHH! I forgot that for ints/longs, the function's input argument wants to be unsigned. Likewise for the format string handed to sprintf(). So I changed it below. Sorry about my terrible oversight, which would have been a hard-to-find bug. Also, ulong makes it a little more general.]
char *
int2str( unsigned long num ) {
static char retnum[21]; // Enough for 20 digits plus NUL from a 64-bit uint.
sprintf( retnum, "%ul", num );
return retnum;
}
And similar for floats and doubles. The code doing the conversion has be known in advance. It has to be told - what kind of an entity it's converting, so you might end up with functions char *float2str( float float_num) and char *dbl2str( double dblnum).
You'll get a NUL-terminated left-adjusted (no leading blanks or zeroes) character string out of the conversion.
You can do the conversion anywhere/anyhow you like; these functions are just illustrations.
Use the Firmata protocol. Quote:
Firmata is a generic protocol for communicating with microcontrollers
from software on a host computer. It is intended to work with any host
computer software package. Right now there is a matching object in a
number of languages. It is easy to add objects for other software to
use this protocol. Basically, this firmware establishes a protocol for
talking to the Arduino from the host software. The aim is to allow
people to completely control the Arduino from software on the host
computer.
The jargon word you need to look up is "serialization".
It is an interesting problem over a serial connection which might have restrictions on what characters can go end to end, and might not be able to pass eight bits per character either.
Restrictions on certain character codes are fairly common. Here's a few off the cuff:
If software flow control is in use, then conventionally the control characters DC1 and DC3 (Ctrl-Q and Ctrl-S, also sometimes called XON and XOFF) cannot be transmitted as data because they are sent to start and stop the sender at the other end of the cable.
On some devices, NUL and/or DEL characters (0x00 and 0x7F) may simply vanish from the receiver's FIFO.
If the receiver is a Unix tty, and the termio modes are not set correctly, then the character Ctrl-D (EOT or 0x04) can cause the tty driver to signal an end-of-file to the process that has the tty open.
A serial connection is usually configurable for byte width and possible inclusion of a parity bit. Some connections will require that a 7-bit byte with a parity are used, rather than an 8-bit byte. It is even possible for connection to (seriously old) legacy hardware to configure many serial ports for 5-bit and 6-bit bytes. If less than 8-bits are available per byte, then a more complicated protocol is required to handle binary data.
ASCII85 is a popular technique for working around both 7-bit data and restrictions on control characters. It is a convention for re-writing binary data using only 85 carefully chosen ASCII character codes.
In addition, you certainly have to worry about byte order between sender and receiver. You might also have to worry about floating point format, since not every system uses IEEE-754 floating point.
The bottom line is that often enough choosing a pure ASCII protocol is the better answer. It has the advantage that it can be understood by a human, and is much more resistant to issues with the serial connection. Unless you are sending gobs of floating point data, then inefficiency of representation may be outweighed by ease of implementation.
Just be liberal in what you accept, and conservative about what you emit.
Does size matter? If it does, you can encode each 32 bit group into 5 ASCII characters using ASCII85, see http://en.wikipedia.org/wiki/Ascii85.
This simply works. Use Serial.println() function
void setup() {
Serial.begin(9600);
}
void loop() {
float x = 23.45585888;
Serial.println(x, 10);
delay(1000);
}
And this is the output:
Perhaps that is best Way to convert Float to Byte and Byte to Float,-Hamid Reza.
int breakDown(int index, unsigned char outbox[], float member)
{
unsigned long d = *(unsigned long *)&member;
outbox[index] = d & 0x00FF;
index++;
outbox[index] = (d & 0xFF00) >> 8;
index++;
outbox[index] = (d & 0xFF0000) >> 16;
index++;
outbox[index] = (d & 0xFF000000) >> 24;
index++;
return index;
}
float buildUp(int index, unsigned char outbox[])
{
unsigned long d;
d = (outbox[index+3] << 24) | (outbox[index+2] << 16)
| (outbox[index+1] << 8) | (outbox[index]);
float member = *(float *)&d;
return member;
}
regards.
`
Structures and unions solve that issue. Use a packed structure with a byte sized union matching the structure. Overlap the pointers to the structure and union (or add the union in the structure). Use Serial.write to send the stream. Have a matching structure/union on receiving end. As long as byte order matches no issue otherwise you can unpack using the "C" hto(s..l) functions. Add "header" info to decode different structures/unions.
For Arduino IDE:
float buildUp(int index, unsigned char outbox[])
{
unsigned long d;
d = (long(outbox[index +3]) << 24) | \
(long(outbox[index +2]) << 16) | \
(long(outbox[index +1]) << 8) | \
(long(outbox[index]));
float member = *(float *)&d;
return member;
}
otherwise not working.
Can't exactly find a way on how to do the following in C/C++.
Input : hexdecimal values, for example: ffffffffff...
I've tried the following code in order to read the input :
uint16_t twoBytes;
scanf("%x",&twoBytes);
Thats works fine and all, but how do I split the 2bytes in 1bytes uint8_t values (or maybe even read the first byte only). Would like to read the first byte from the input, and store it in a byte matrix in a position of choosing.
uint8_t matrix[50][50]
Since I'm not very skilled in formating / reading from input in C/C++ (and have only used scanf so far) any other ideas on how to do this easily (and fast if it goes) is greatly appreciated .
Edit: Found even a better method by using the fread function as it lets one specify how many bytes it should read from the stream (stdin in this case) and save to a variable/array.
size_t fread ( void * ptr, size_t size, size_t count, FILE * stream );
Parameters
ptr - Pointer to a block of memory with a minimum size of (size*count) bytes.
size - Size in bytes of each element to be read.
count - Number of elements, each one with a size of size bytes.
stream - Pointer to a FILE object that specifies an input stream.
cplusplus ref
%x reads an unsigned int, not a uint16_t (thought they may be the same on your particular platform).
To read only one byte, try this:
uint32_t byteTmp;
scanf("%2x", &byteTmp);
uint8_t byte = byteTmp;
This reads an unsigned int, but stops after reading two characters (two hex characters equals eight bits, or one byte).
You should be able to split the variable like this:
uint8_t LowerByte=twoBytes & 256;
uint8_t HigherByte=twoBytes >> 8;
A couple of thoughts:
1) read it as characters and convert it manually - painful
2) If you know that there are a multiple of 4 hexits, you can just read in twobytes and then convert to one-byte values with high = twobytes << 8; low = twobyets & FF;
3) %2x