I'm trying to use System.IO.Pipelines to parse large text files.
But I can't find no conversion function from ReadOnlySequence to ReadOnlySequence. For example like MemoryMarshal.Cast<byte,char>.
IMHO it is pretty useless having a generic ReadOnlySequence<T> if there is only one particular type (byte) applicable.
static async Task ReadPipeAsync(PipeReader reader, IStringValueFactory factory)
{
while (true)
{
ReadResult result = await reader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;
//ReadOnlySequence<char> chars = buffer.CastTo<char>(); ???
}
}
You would have to write a conversion operator to achieve this cast. You cannot cast it explicitly. Be aware that a char[] is two bytes, so you need to choose your encoding algorithm.
IMHO it is pretty useless having a generic ReadOnlySequence<T> if
there is only one particular type (byte) applicable.
While it's true that System.IO.Pipelines will only give you a ReadOnlySequence<byte> because of the fact that a PipeReader is attached to a Stream which is just a stream of bytes, there are other use cases for a ReadOnlySequence<T> eg,
ReadOnlySequence<char> roChars = new ReadOnlySequence<char>("some chars".ToCharArray());
ReadOnlySequence<string> roStrings = new ReadOnlySequence<string>(new string[] { "string1", "string2", "Another String" });
Your conversion operator would have similar logic to the below, but you would set your encoding appropriately.
static void Main(string[] args)
{
// create a 64k Readonly sequence of random bytes
var ros = new ReadOnlySequence<byte>(GenerateRandomBytes(64000));
//Optionally extract the section of the ReadOnlySequence we are interested in
var mySlice = ros.Slice(22222, 55555);
char[] charArray;
// Check if the slice is a single segment - not really necessary
// included for explanation only
if(mySlice.IsSingleSegment)
{
charArray = Encoding.ASCII.GetString(mySlice.FirstSpan).ToCharArray();
}
else
// Could only do this and always assume multiple spans
// which is highly likley for a PipeReader stream
{
Span<byte> theSpan = new byte[ros.Length];
mySlice.CopyTo(theSpan);
// ASCII Encoding - one byte of span = 2 bytes of char
charArray = Encoding.ASCII.GetString(theSpan).ToCharArray();
}
// Convert the char array back to a ReadOnlySegment<char>
var rosChar = new ReadOnlySequence<char>(charArray);
}
public static byte[] GenerateRandomBytes(int length)
{
// Create a buffer
byte[] randBytes;
if (length >= 1)
randBytes = new byte[length];
else
randBytes = new byte[1];
// Create a new RNGCryptoServiceProvider.
System.Security.Cryptography.RNGCryptoServiceProvider rand =
new System.Security.Cryptography.RNGCryptoServiceProvider();
// Fill the buffer with random bytes.
rand.GetBytes(randBytes);
// return the bytes.
return randBytes;
}
Related
I am trying to send CAN Frames with xlCanTransmit method. The problem is that it returns 0 (XL_SUCCESS), but the frame is empty. there are only zeroes in the Frame.
IVxlApi.s_xl_event eventMsg = new IVxlApi.s_xl_event();
...
...
xlStatus = dll.xlCanTransmit(GlobalConfig.g_xlPortHandle, GlobalConfig.g_xlChannelMask[channelIndex],
pEventCount, eventMsg.getPointer());
After Calling the function xlCanTransmit() the frame is sent, but there is no data in it. The function is called from dll file and you can't debug it.
My mapped Structure
public static class s_xl_event extends Structure {
public byte tag;
public byte chanIndex;
public byte[] transId = new byte[SHORT];
public byte[] portHandle = new byte[SHORT];
public byte flags;
public byte reserved;
public byte[] timeStamp = new byte[LONG];
public s_xl_tag_data tagData;
#Override
public void read() {
// read from native memory, populate tag
super.read();
// set union type based on tag
switch (tag) {
case XL_RECEIVE_MSG:
case XL_TRANSMIT_MSG:
tagData.setType(s_xl_can_msg.class);
break;
case XL_CHIP_STATE:
tagData.setType(s_xl_chip_state.class);
break;
case XL_LIN_MSG:
tagData.setType(s_xl_lin_msg_api.class);
break;
case XL_SYNC_PULSE:
tagData.setType(s_xl_sync_pulse.class);
break;
case XL_RECEIVE_DAIO_DATA:
tagData.setType(s_xl_daio_data.class);
break;
case XL_TRANSCEIVER:
tagData.setType(s_xl_transceiver.class);
break;
case XL_RECEIVE_DAIO_PIGGY:
tagData.setType(s_xl_daio_piggy_data.class);
break;
case XL_KLINE_MSG:
tagData.setType(s_xl_kline_data.class);
break;
default:
// add default type or throw exception etc.
tagData.setType(s_xl_can_msg.class);
break;
}
// now read tagData from native memory
tagData.read();
}
#Override
protected List<String> getFieldOrder() {
return Arrays.asList("tag", "chanIndex", "transId", "portHandle", "flags", "reserved", "timeStamp", "tagData");
}
}
CAN Frame in CANoe Trace
enter image description here
The function on the native side looks like this
DECL_STDXL_FUNC(xlCanTransmit, XLCANTRANSMIT, (
XLportHandle portHandle,
XLaccess accessMask,
unsigned int* pEventCount,
void* pEvents)
);
My mapping in java
short xlCanTransmit(long portHandle, long accessMask, IntByReference pEventCount, Pointer pEvents);
I wrote some random data manually and called the xlCanTransmit() function.
But in CANoe trace I see only empty frame with no data and id.
My send function in java
public short send(String txID, String dlc, String[] data, int channelIndex) {
short xlStatus = IVxlApi.XL_ERROR;
IntByReference pEventCount = new IntByReference(1);
IVxlApi.s_xl_event eventMsg = new IVxlApi.s_xl_event();
eventMsg.tag = IVxlApi.XL_TRANSMIT_MSG;
eventMsg.tagData.msg.id[0] = 2;// = Long.parseLong(txID);
eventMsg.tagData.msg.dlc[0] = 8;// = Short.parseShort(dlc);
eventMsg.tagData.msg.flags[0] = 0;
eventMsg.tagData.msg.data[0] = Byte.parseByte(data[0]);
eventMsg.tagData.msg.data[1] = Byte.parseByte(data[1]);
eventMsg.tagData.msg.data[2] = Byte.parseByte(data[2]);
eventMsg.tagData.msg.data[3] = Byte.parseByte(data[3]);
eventMsg.tagData.msg.data[4] = Byte.parseByte(data[4]);
eventMsg.tagData.msg.data[5] = Byte.parseByte(data[5]);
eventMsg.tagData.msg.data[6] = Byte.parseByte(data[6]);
eventMsg.tagData.msg.data[7] = Byte.parseByte(data[7]);
/*if(true){
eventMsg.tagData.msg.id |= IVxlApi.XL_CAN_EXT_MSG_ID;
}*/
eventMsg.write();
xlStatus = dll.xlCanTransmit(GlobalConfig.g_xlPortHandle, GlobalConfig.g_xlChannelMask[channelIndex],
pEventCount, eventMsg.getPointer());
eventMsg.read();
return xlStatus;
}
CAN Frame in CANoe Trace
enter image description here
After calling tagData.read() (line 475) the data initialized with zeroes
enter image description here
The return value of 0 indicates the function preformed properly on the native side, but you haven't copied the native memory back to Java.
This is normally done automatically when you pass a Structure in a function, but you have passed the Pointer to the structure instead. Your last argument is eventMsg.getPointer(). JNA doesn't know how the Pointer maps to your structure until you tell it to do so, and in the current way you're calling it requires using read() to copy the value back to JNA.
You could probably get the results you want by just executing eventMsg.read() after calling the function, but the better solition is to pass the Structure to the function in the first place and let JNA do the conversion to Pointer and auto-read() for you. Change your function call to pass eventMsg as the argument.
So I have a stream, what I want to be able is to transfer it into unsigned char * bytes and back to usable stream.
This stream is image (it is binary if it is important?)
Basically what I am trying now is as follows:
IRandomAccessStream^ inputStream;
DataWriter^ dataWriter = ref new DataWriter(inputStream);
IBuffer^ buffer1 = dataWriter->DetachBuffer();
unsigned char * bytes = GetPointerToPixelData(buffer1);
DataWriter ^writer = ref new DataWriter();
writer->WriteBytes(Platform::ArrayReference<BYTE>(bytes, sizeof(bytes)));
task<DataWriterStoreOperation^>(writer->StoreAsync()).get();
task<Windows::Foundation::IAsyncOperation<bool>>(writer->FlushAsync()).get();
IBuffer ^buffer2 = writer->DetachBuffer();
IRandomAccessStream^ newStream;
task<Windows::Foundation::IAsyncOperationWithProgress<unsigned int, unsigned int>>(newStream->WriteAsync(buffer2)).get();
task<Windows::Foundation::IAsyncOperation<bool>>(newStream->FlushAsync()).get();
UseNewStream(newStream)
I have added all of these task<...> because it is not working without them, and I am not sure how to make it work?
Function GetPointerToPixelData I found online and is following:
byte* GetPointerToPixelData(IBuffer^ buffer)
{
// Cast to Object^, then to its underlying IInspectable interface.
Object^ obj = buffer;
ComPtr<IInspectable> insp(reinterpret_cast<IInspectable*>(obj));
// Query the IBufferByteAccess interface.
ComPtr<IBufferByteAccess> bufferByteAccess;
insp.As(&bufferByteAccess);
// Retrieve the buffer data.
byte* pixels = nullptr;
bufferByteAccess->Buffer(&pixels);
return pixels;
}
Thanks! :)
Firstly, if you want to transfer the image stream to bytes, you need to read the stream by DataReader, not by DataWriter which is for write data. Secondly, the DetachBuffer() method is for "Detaches the buffer that is associated with the data reader", not read the buffer. Lastly, DataReader can directly read bytes by ReadBytes(Byte[]) method. For example:
uint64 length = 0;
BYTE *extracted;
void CleanImagetobyte::MainPage::btnconverttobyte_Click(Platform::Object^ sender, Windows::UI::Xaml::RoutedEventArgs^ e)
{
create_task(KnownFolders::GetFolderForUserAsync(nullptr /* current user */, KnownFolderId::PicturesLibrary))
.then([this](StorageFolder^ picturesFolder)
{
return picturesFolder->GetFileAsync("B.jpg");
}).then([this](task<StorageFile^> task)
{
try
{
StorageFile^ file = task.get();
auto name = file->Name;
return file->OpenAsync(FileAccessMode::Read);
}
catch (Exception^ e)
{
// I/O errors are reported as exceptions.
}
}).then([this](task<IRandomAccessStream^> task)
{
IRandomAccessStream^ inputStream = task.get();
length = inputStream->Size;
IBuffer^ buffer = ref new Buffer(inputStream->Size);
inputStream->ReadAsync(buffer, inputStream->Size, InputStreamOptions::None);
DataReader^ reader = DataReader::FromBuffer(buffer);
extracted = new BYTE[buffer->Length];
reader->ReadBytes(Platform::ArrayReference<BYTE>(extracted, buffer->Length));
});
}
void CleanImagetobyte::MainPage::btnconvertback_Click(Platform::Object^ sender, Windows::UI::Xaml::RoutedEventArgs^ e)
{
create_task(KnownFolders::GetFolderForUserAsync(nullptr /* current user */, KnownFolderId::PicturesLibrary))
.then([this](StorageFolder^ picturesFolder)
{
return picturesFolder->CreateFileAsync("newB.jpg", CreationCollisionOption::ReplaceExisting);
}).then([this](task<StorageFile^> task)
{
StorageFile^ file = task.get();
Array<byte>^ arr = ref new Array<byte>(extracted, length);
FileIO::WriteBytesAsync(file, arr);
});
}
More details for read and write a file please reference this document.
I am porting the openvr sample to jogl, after we created the binding with jna.
Almost at the end (before rendering the controllers and the tracking base stations), I got stuck trying to translate a char pointer in C to a String in Java.
C++ code here:
//-----------------------------------------------------------------------------
// Purpose: Helper to get a string from a tracked device property and turn it
// into a std::string
//-----------------------------------------------------------------------------
std::string GetTrackedDeviceString( vr::IVRSystem *pHmd, vr::TrackedDeviceIndex_t unDevice, vr::TrackedDeviceProperty prop, vr::TrackedPropertyError *peError = NULL )
{
uint32_t unRequiredBufferLen = pHmd->GetStringTrackedDeviceProperty( unDevice, prop, NULL, 0, peError );
if( unRequiredBufferLen == 0 )
return "";
char *pchBuffer = new char[ unRequiredBufferLen ];
unRequiredBufferLen = pHmd->GetStringTrackedDeviceProperty( unDevice, prop, pchBuffer, unRequiredBufferLen, peError );
std::string sResult = pchBuffer;
delete [] pchBuffer;
return sResult;
}
GetStringTrackedDeviceProperty here:
/** Returns a string property. If the device index is not valid or the property is not a string type this function will
* return 0. Otherwise it returns the length of the number of bytes necessary to hold this string including the trailing
* null. Strings will generally fit in buffers of k_unTrackingStringSize characters. */
virtual uint32_t GetStringTrackedDeviceProperty( vr::TrackedDeviceIndex_t unDeviceIndex, ETrackedDeviceProperty prop, VR_OUT_STRING() char *pchValue, uint32_t unBufferSize, ETrackedPropertyError *pError = 0L ) = 0;
Where VR_OUT_STRING() is defined here as:
# define VR_CLANG_ATTR(ATTR)
#define VR_OUT_STRING() VR_CLANG_ATTR( "out_string: ;" )
I have already done something similar where I had to call a function that expect the pointer to an array of TrackedDevicePose_t structures:
private TrackedDevicePose_t.ByReference trackedDevicePosesReference = new TrackedDevicePose_t.ByReference();
public TrackedDevicePose_t[] trackedDevicePose
= (TrackedDevicePose_t[]) trackedDevicePosesReference.toArray(VR.k_unMaxTrackedDeviceCount);
I created first the reference and then from it the actual array.
But here I can't have a class extending the char array..
private String getTrackedDeviceString(IVRSystem hmd, int device, int prop, IntBuffer propError) {
int requiredBufferLen = hmd.GetStringTrackedDeviceProperty.apply(device, prop, Pointer.NULL, 0, propError);
if(requiredBufferLen == 0) {
return "";
}
CharArray.ByReference charArrayReference = new CharArray.ByReference();
char[] cs = charArrayReference.toArray(requiredBufferLen);
return null;
}
Where apply (here) is:
public interface GetStringTrackedDeviceProperty_callback extends Callback {
int apply(int unDeviceIndex, int prop, Pointer pchValue, int unBufferSize, IntBuffer pError);
};
CharArray class, crap attempt here
Any ideas?
I've done some porting of C and C++ code to Java, and while it's probably horribly hacky, the best I've come up with to solve cases where a pointer to an int primitive or a char*/String is needed for a function call, is to create a small wrapper class with a single property, pass that object into the function, change the property as needed, and retrieve the new value after the function call. So something like:
public class StringPointer {
public String value = "";
}
StringPointer pchBuffer = new StringPointer();
unRequiredBufferLen = pHmd.GetStringTrackedDeviceProperty( unDevice, prop, pchBuffer, unRequiredBufferLen, peError );
String sResult = pchBuffer.value;
and inside GetStringTrackedDeviceProperty()
...
pchValue.value = "some string";
...
In this case, you can use a String, since that's what your code is doing with the char* after the function call, but if it actually really needs to be a char[], you can just create char[] pchBuffer = new char[unRequiredBufferLen]; and pass that into the function. It will be just like you were using a char* in C++, and any changes you make inside the array will be visible after the function ends, and you can even do String sResult = new String(pchBuffer);.
I am attempting to convert an array of bytes to a base64 encoded String using the EncodeBytesToString method of the TBase64Encoding class. The documentation for EncodeBytesToString states:
"Returns a string with the input array of bytes encoded up to the specified number of bytes."
Therefore, I attempted to encode my byte array like so:
TFile * File = new TFile();
TBytes Bytes = File->ReadAllBytes("D:\\Sample.pdf");
TBase64Encoding * Encoder = new TBase64Encoding();
String EncodedBytes = Encoder->EncodeBytesToString(Bytes, Bytes.Length);
However, I get the following error:
E2285 Could not find a match for 'TNetEncoding::EncodeBytesToString(TByteDynArray,int)'
I am confused, as the documentation seems to say that I should pass a TBytes object and an int into this function. What am I missing here?
Try this:
//------------------------------------------------------------------------------
String __fastcall BytesToBase64( TByteDynArray _ArrayIn )
{
TBase64Encoding * Encoding = new TBase64Encoding( 64, '\n' );
String Result = Encoding->EncodeBytesToString( &_ArrayIn[0], _ArrayIn.High );
delete Encoding;
return Result;
}
//------------------------------------------------------------------------------
TByteDynArray __fastcall Base64ToBytes( String _64String )
{
TByteDynArray My64Bytes = _64String.BytesOf();
return TNetEncoding::Base64->Decode(&My64Bytes[0], My64Bytes.High);
}
//------------------------------------------------------------------------------
System.NetEncoding.TNetEncoding provides the static property Base64 to retrieve an instance of TNetEncoding for base64 encoding.
So this will also work:
String __fastcall BytesToBase64(TByteDynArray _ArrayIn)
{
return TNetEncoding::Base64->EncodeBytesToString(&_ArrayIn[0], _ArrayIn.High);
}
I have been trying to write encrypt and decrypt functions whose signatures require the input and the output strings to be void* type only. The code works fine if the inputs can be specified as IBuffer^ but in the other case the source string and the encrypted->decrypted string do not match.
CodeIBuffer^ byteArrayToIBufferPtr(byte *source, int size)
{
Platform::ArrayReference<uint8> blobArray(source, size);
IBuffer ^buffer = CryptographicBuffer::CreateFromByteArray(blobArray);
return buffer;
}
byte* IBufferPtrToByteArray(IBuffer ^buffer)
{
Array<unsigned char,1U> ^platArray = ref new Array<unsigned char,1U>(256);
CryptographicBuffer::CopyToByteArray(buffer,&platArray);
byte *dest = platArray->Data;
return dest;
}
int DataEncryption::encryptData(EncryptionAlgorithm algo, int keySize, void* srcData, const unsigned int srcSize,
void*& encData, unsigned int& encSize)
{
LOG_D(TAG, "encryptData()");
if(srcData == nullptr)
{
LOG_E(TAG,"");
return DataEncryption::RESULT_EMPTY_DATA_ERROR;
}
if(srcSize == 0)
{
LOG_E(TAG,"");
return DataEncryption::RESULT_SIZE_ZERO_ERROR;
}
IBuffer^ encrypted;
IBuffer^ buffer;
IBuffer^ iv = nullptr;
String^ algName;
bool cbc = false;
switch (algo)
{
case DataEncryption::ENC_DEFAULT:
algName = "AES_CBC";
cbc = true;
break;
default:
break;
}
// Open the algorithm provider for the algorithm specified on input.
SymmetricKeyAlgorithmProvider^ Algorithm = SymmetricKeyAlgorithmProvider::OpenAlgorithm(algName);
// Generate a symmetric key.
IBuffer^ keymaterial = CryptographicBuffer::GenerateRandom((keySize + 7) / 8);
CryptographicKey^ key;
try
{
key = Algorithm->CreateSymmetricKey(keymaterial);
}
catch(InvalidArgumentException^ e)
{
LOG_E(TAG,"encryptData(): Could not create key.");
return DataEncryption::RESULT_ERROR;
}
// CBC mode needs Initialization vector, here just random data.
// IV property will be set on "Encrypted".
if (cbc)
iv = CryptographicBuffer::GenerateRandom(Algorithm->BlockLength);
// Set the data to encrypt.
IBuffer ^srcDataBuffer = byteArrayToIBufferPtr(static_cast<byte*>(srcData),256);
// Encrypt and create an authenticated tag.
encrypted = CryptographicEngine::Encrypt(key, srcDataBuffer, iv);
//encData = encrypted;
byte *bb = IBufferPtrToByteArray(encrypted);
encData = IBufferPtrToByteArray(encrypted);
encSize = encrypted->Length;
return DataEncryption::RESULT_SUCCESS;
}
int DataEncryption::decryptData(EncryptionAlgorithm algo, int keySize, void* encData, const unsigned int encSize,
void*& decData, unsigned int& decSize)
{
LOG_D(TAG, "decryptData()");
if(encData == nullptr)
{
LOG_E(TAG,"");
return DataEncryption::RESULT_EMPTY_DATA_ERROR;
}
if(encSize == 0)
{
LOG_E(TAG,"");
return DataEncryption::RESULT_SIZE_ZERO_ERROR;
}
IBuffer^ encrypted;
IBuffer^ decrypted;
IBuffer^ iv = nullptr;
String^ algName;
bool cbc = false;
switch (algo)
{
case DataEncryption::ENC_DEFAULT:
algName = "AES_CBC";
cbc = true;
break;
default:
break;
}
// Open the algorithm provider for the algorithm specified on input.
SymmetricKeyAlgorithmProvider^ Algorithm = SymmetricKeyAlgorithmProvider::OpenAlgorithm(algName);
// Generate a symmetric key.
IBuffer^ keymaterial = CryptographicBuffer::GenerateRandom((keySize + 7) / 8);
CryptographicKey^ key;
try
{
key = Algorithm->CreateSymmetricKey(keymaterial);
}
catch(InvalidArgumentException^ e)
{
LOG_E(TAG,"encryptData(): Could not create key.");
return DataEncryption::RESULT_ERROR;
}
// CBC mode needs Initialization vector, here just random data.
// IV property will be set on "Encrypted".
if (cbc)
iv = CryptographicBuffer::GenerateRandom(Algorithm->BlockLength);
// Set the data to decrypt.
byte *cc = static_cast<byte*>(encData);
IBuffer ^encDataBuffer = byteArrayToIBufferPtr(cc,256);
// Decrypt and verify the authenticated tag.
decrypted = CryptographicEngine::Decrypt(key, encDataBuffer, iv);
byte *bb = IBufferPtrToByteArray(decrypted);
decData = IBufferPtrToByteArray(decrypted);
decSize = decrypted->Length;
return DataEncryption::RESULT_SUCCESS;
}
I'm guessing that the problem is with this function:
byte* IBufferPtrToByteArray(IBuffer ^buffer)
{
Array<unsigned char,1U> ^platArray = ref new Array<unsigned char,1U>(256);
CryptographicBuffer::CopyToByteArray(buffer,&platArray);
byte *dest = platArray->Data;
return dest;
}
What you're doing there is allocating a new Platform::Array<byte>^ with 1 reference, then getting a pointer to its internally-managed storage, then returning that pointer-- at which point the Array is being dereferenced and is thus deallocating its underlying storage. Thus the pointer you return refers to freed memory. The next allocation is likely to overwrite those bytes.
What you'll need to do is take the return-by-reference Array<byte>^ from CopyToByteArray() (which creates a new Array, presumably wrapping the bytes of the input IBuffer^, and returns it) and copy that array's contents.
Your end result will function similarly to this snippet from the Readium SDK project, which takes a std::string instance, hashes it using SHA-1, and copies the hash data into a member variable uint8_t _key[KeySize]:
using namespace ::Platform;
using namespace ::Windows::Foundation::Cryptography;
using namespace ::Windows::Foundation::Cryptography::Core;
auto byteArray = ArrayReference<byte>(reinterpret_cast<byte*>(const_cast<char*>(str.data())), str.length());
auto inBuf = CryptographicBuffer::CreateFromByteArray(byteArray);
auto keyBuf = HashAlgorithmProvider::OpenAlgorithm(HashAlgorithmNames::Sha1)->HashData(inBuf);
Array<byte>^ outArray = nullptr;
CryptographicBuffer::CopyToByteArray(keyBuf, &outArray);
memcpy_s(_key, KeySize, outArray->Data, outArray->Length);
The steps:
Create an ArrayReference<byte> corresponding to the bytes in the std::string (no copying).
Pass that to CryptographicBuffer::CreateFromByteArray() to get your IBuffer^. Still no copying of data.
Call your hash/encryption function, passing the IBuffer^ you just made. You get another IBuffer^ in return, which may or may not be using the exact same storage (that's really up to the implementation of the algorithm, I think).
Create a variable of type Array<byte>^. Don't allocate an object, you're going to be given one by reference.
Pass the address of that object into CryptographicBuffer::CopyToByteArray() to receive a copy of your key data.
While that Array^ remains valid, copy its bytes into your native array.