I need to parse a text file stream after downloading and without saving the text file to harddisk or sdcard using tftp protocol. I've the data in payload. Please help.
struct pbuf {
struct pbuf *next;
void *payload;
u16_t tot_len;
u16_t len;
u8_t type;
u8_t flags;
u16_t ref;
};
The following code assigns the opcode and filename. But how to get the contents of the file?
/* Extract the opcode from a TFTP message in a buffer */
tftp_opcode tftp_decode_op(char *buf)
{
return (tftp_opcode)(buf[1]);
}
void tftp_extract_filename(char *fname, char *buf)
{
strcpy(fname, buf + 2);
}
In the TFTP protocol, you first get a write request packet (opcode WRQ), and then the data in separate data packets (opcode DATA). You will need to do something whenever a data packet is received in order to process the contents of the file.
I'd recommend reading up a bit on the TFTP protocol if you haven't done that yet. An overview is at http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol#Protocol_walkthrough and the spec (which is very readable) is at https://www.rfc-editor.org/rfc/rfc1350 .
If your text file not more then some KB then why you not use static buffer to store content of it.In TFTP protocol you must know at which (in source point of view ) point the packet received.So you get packet buffer and size of the packet.So simply copy that received packet buffer in above static buffer and increase offest of static buffer and so on....So at last you will get whole text file content in static buffer.
Related
Background:
I have a .wav file saved on an SD card. I would like to transfer that file to a server using my esp32. I am using node red to handle the server side activities.
Method Employed:
open the file in binary mode.
evaluate the size of the file
decide on a max upload size and allocate a buffer
Read the file and store to the buffer.
use http post to send data to the server.
if file is too large to send in a single buffer then divide the file up and send multiple http posts.
Problem:
I can successfully send text files. when I try to send .wav files the size of the sent wave file increases and the file is corrupted. Analyzing the file is difficult as its not all text, what I have done is open the file in notepad++ to see if I can spot anything. Everything should be the same in theory but several characters are coming up as blank squares in the transferred file and some are coming up as the exact same.
Analysis/Theory:
I am quite lost as to what the issue is. My leading theory is that a wave file is written in int16_t but in order to post the data it needs to be * uint8_t, maybe when the casting of the int16 to a uint8 data is lost, I looked at trying to change a int16_t into two int8_t bytes as done here https://stackoverflow.com/a/53374797/14050333 but had no luck, maybe I'm jumping to conclusions. Any help would be hugely appreciated!
Code:
Full code used to sell text files.
void loop()
{
WiFiClient client;
Serial.println("starting file upload");
IPAddress host(192, 168, 0, 37);
int port = 1880;
if (!client.connect(host, port))
{ // check connection to host if untrue internet connection could be down
Serial.println("couldn't connect to host");
}
HTTPClient http;
const char* serverName = "http://192.168.0.37:1880/sensor_file";
http.begin(client, serverName);
char *fname = "/sdcard/test_text.txt";
FILE *fp = fopen(fname, "rb"); // read in bytes
//get file size
fseek(fp, 0, SEEK_END); //send file pointer to end of file
int file_size = ftell(fp); //get end position of file
fseek(fp, 0, SEEK_SET); //send pointer back to start
int max_upload_size = 10; // array size, larger = less uploads but too large can cause memory issues
int num_of_uploads = file_size / max_upload_size; // figure out how many evenly sized upload chunks we need
int num_of_uploads_mod = file_size % max_upload_size; //find out size of remaining upload chunk if needed
int i;
//upload file in even chunks
if (num_of_uploads > 0)
{
char buff1[max_upload_size+1] = {}; // array to save file too. add 1 for end of array symbol '\n'
for (i = 0; i < num_of_uploads; i++)
{
fread(buff1, sizeof(buff1)-1, 1, fp); // -1 as don't want to count the '\n'
http.addHeader("File_name", "test file"); //header to say what the file name is
int httpResponseCode = http.POST((uint8_t *)buff1, sizeof(buff1)-1); //send data. Datatype is (uint8_t *)
}
}
//upload any remaining data
if (num_of_uploads_mod > 0)
{
int remainder = file_size - num_of_uploads * max_upload_size;
char buff2[remainder+1] = {};
fread(buff2, sizeof(buff2)-1, 1, fp); //read from file and store to buff2
http.addHeader("File_name", "test file");
int httpResponseCode = http.POST((uint8_t *)buff2, sizeof(buff2)-1); //send buff2 to server
}
http.end(); // Close connection
delay(10 * 1000);
}
Adjustments made for .wav files
int remainder = file_size - num_of_uploads * max_upload_size;
int16_t buff2[remainder+1] = {};
fread(buff2, sizeof(buff2)-1, 1, fp); //remainder
http.addHeader("File_name", "test file");
int httpResponseCode = http.POST((uint8_t *)buff2, sizeof(buff2)-1);
Its working!
There were 2 main issues with the code as outlined by heap underrun. The first issue is that I was reading in the wav file as int16_t the correct datatype to use was uint8_t.
Why are you using an array of int16_t-type elements as a buffer? You are reading a file in binary mode, so be it .wav, .jpg, .ttf, or anything else, it's just a sequence of bytes (uint8_t, not int16_t) anyway. Another thing, fread() expects the size of each object to read as the second parameter and the number of objects to read as the third parameter, so, in case of objects being bytes, first define buffer as uint8_t buff1[max_upload_size] = {}; (no need for +1/-1 games), and then fread(buff1, sizeof *buff1, sizeof buff1 / sizeof *buff1, fp);. The same for buff2. –
heap underrun
The second issue was that I did not include a header in the post stream specifying the content type. As it wasn't needed for the text file and when writing the file in node-red it lets you choose the encoding. I didn't think I would need it, however as it turns out I needed to add:
http.addHeader("Content-Type", "application/octet-stream");
Below is the working code for the file upload section:
if (num_of_uploads > 0)
{
uint8_t buff1[max_upload_size] = {};
for (i = 0; i < num_of_uploads; i++)
{
fread(buff1, sizeof *buff1, sizeof buff1 / sizeof *buff1, fp);
http.addHeader("File_name", "test file"); //header to say what the file name is
http.addHeader("Content-Type", "application/octet-stream");
int httpResponseCode = http.POST(buff1, sizeof(buff1));
}
}
if (num_of_uploads_mod > 0)
{
int remainder = file_size - num_of_uploads * max_upload_size;
uint8_t buff2[remainder] = {};
fread(buff2, sizeof *buff2, sizeof buff2 / sizeof *buff2, fp);
http.addHeader("File_name", "test file");
http.addHeader("Content-Type", "application/octet-stream");
int httpResponseCode = http.POST(buff2, sizeof(buff2));
}
On a slightly interesting side note out of curiosity I tried running the above code but with
uint16_t buff1[max_upload_size] = {};
and
http.POST((uint8_t) buff1, sizeof(buff2));
The file uploaded but the size was 2x what it should be, curiously however the file wasn't corrupted, and played the audio as it was recorded. Just thought that was interesting.
I'll close out this answer as the original question was successfully answered. Again thank you for the help, I've been at this literally weeks and you solved my problems in hours!
I have work this for weeks, very hope for your help!!! please forgive my poor english.
First, I think it's necessary to describe the Application Scenario:
what data I want to decompress?----the data is come from the network traffic of the internet. In these traffic, there are some data are compressed by gzip and store in the http or tcp packet, if the data size is huge and large than the maxlength of tcp payload, it will be sliced and transmiss. I can extract the compressed data from these packet, and group these data by the tcp stream. so I can assure that the data extracted from these packets of one specific tcp stream is belong to Same data source. so the data is consist of many compressed data chunk, the Application Scenario require that you need to decompress the data immediately once recieved one packet. For each tcp stream, we maintain a z_stream data structure.
When does the program report an error? ----All of the error is "Z_DATA_ERROR: invalid distance too far back". then I find when the recieved packet is out-of-order or some packet is lossed, the error will happen!
One simple case:
Compressed data is split into multiple data blocks and stored in network data packets(p1, p2, p3, p4, p5, p6, p7), and then transmiss in one specific tcp stream. For each tcp stream, we maintain a z_stream data structure. Obviously, p1 include the gzip header 0x1f 0x8b 0x08...), but due to the Uncertainty in network transmission, the packet recieved may be out-of-order or loss, for example: (p1,p2,p5,p6,p7,p3,p4),the first two packet can decompree normally, but when decompress p5, the error occur(Z_DATA_ERROR).
SO, I have these problem:
Due to the application scenario, I need to decompress the data once recieved one packet with gzip content-encoding. So I want to know if zlib supports such a function----directly decompress a compressed block without having to consider the packet arrival order?
I also test the influce of packet recieved order: If I sort the data in its original order and then decompress it sequentially, it will decompress normally.
Thirdly, Logically speaking, for the packeted recieved order (p1,p2,p5,p6,p7,p3,p4), when decompress these packet sequentially, p1,p2 will decompress successfully, p5,p6,p7 will decompress failed, the next packet recieved is p3, Logically speaking, it should be decompress successfully, but when I test this case, it failed, I don't understand this.
I also found a confusing problem, which does not happen often: if I sort the packet as (p1,p2,p3,p5,p4...), Logically speaking, when decompress p5, it should report an error, buf it decompress sucessfully, I don't understand this.
the following is source code:
/**
* buf: the gzip compressed data that extract form tcp packet
*/
void dowithGzipDataByZlib(z_stream * p_zlib_strm, unsigned char * buf, int buflen)
{
int zlib_status = Z_OK;
int bytes_dc_now = 0;
unsigned char pNowResBuff[4096];
printf("-------\n");
(*p_zlib_strm).avail_in = buflen;
(*p_zlib_strm).next_in = buf;
do {
memset(pNowResBuff,0,4096);
(*p_zlib_strm).avail_out = 4096;
(*p_zlib_strm).next_out = pNowResBuff;
zlib_status = inflate (p_zlib_strm, Z_NO_FLUSH);
printf("inflate status:%d\n",zlib_status);
if(Z_OK != zlib_status && Z_STREAM_END!=zlib_status){
printf("(*p_zlib_strm).avail_in:%d\n",(*p_zlib_strm).avail_in);
printf("err msg:%s\n",p_zlib_strm->msg);
return ;
}
bytes_dc_now = 4096 - (*p_zlib_strm).avail_out;
// printf("bytes_dc_no:")
} while(0 == (*p_zlib_strm).avail_out) ;
printf("(*p_zlib_strm).avail_in:%d\n",(*p_zlib_strm).avail_in);
}
// under the dirpath, there are some compressed data extract from the packets of one specific tcp stream, and store them in "file_basename_%d" file. (%d is the recieve order num: 1,2,3,4...)
void read( char* dirpath, char* file_basename)
{
char filelist[99][255];
int file_count = listDir(dirpath, filelist, 99, 255);
char filepath[255];
z_stream zlib_strm = {0};
zlib_strm.zalloc = Z_NULL;
zlib_strm.zfree = Z_NULL;
zlib_strm.opaque = Z_NULL;
zlib_strm.next_in = Z_NULL;
zlib_strm.avail_in = 0;
inflateInit2 (& zlib_strm, 32 | MAX_WBITS);
FILE* fp;
char buf[2048];
// sort_file_ind: the array store the origin order of the compressed data.
int sort_file_ind[99] = {0,1,2,3,15,16,17,18,19,20,21,4,5,6,7,8,9,10,11,12,13,14};
for(int i=1;i<=file_count-2;i++)
{
memset(filepath,0,sizeof(filepath));
// snprintf(filepath,sizeof(filepath), "%s%s%d",dirpath,file_basename,sort_file_ind[i]);
snprintf(filepath,sizeof(filepath), "%s%s%d",dirpath,file_basename,i);
printf("%s\n",filepath);
fp = fopen(filepath,"r");
if(fp == NULL){
return;
}
fseek(fp, 0, SEEK_END);
int flen = ftell(fp);
fseek(fp, 0, SEEK_SET);
memset(buf,0,sizeof(buf));
int dlen = fread(buf, 1, flen, fp);
if(dlen != flen){
fclose(fp);
return;
}
printf("dlen:%d\n",dlen);
dowithGzipDataByZlib(&zlib_strm,(unsigned char *)buf,dlen);
fclose(fp);
}
}
char * dir = "/data/GzipDC/softDC/DocumentAnalyze/testbyzs/data/119.40.37.65.42050/";
char * base_filename = "119.40.37.65.42050>180.76.22.49.80_1_";
int main()
{
read(dir,base_filename);
return 0;
}
I've asked around and tried many things for days, and I really need someone with knowledge on the subject to weigh in here. Thanks for your time!
I am having a lot of trouble with GRPC when using byte array. This is by .proto
message myType {
int32 format = 1;
bytes data = 2;
}
I am using CPP for Server implementation and Java for Client. Using ByteString in Java is a breeze but cannot deserialize in CPP (byte[] being changed from what was being sent from Java).
buffer is a byte[] byte buffer[<large_size>] And I'm converting the byte array (it's an image) into a smaller byte array, and it's crashing when trying to convert the byte[] received from grpc. The conversion function in CPP is good as I used it with the same image before using GRPC
This is the deserialization code for CPP. Here "req" is a myType object, and buffer is a byte[]
myFormat = req->format();
dataLen = req->data().length();
memcpy(buffer, req->data().c_str(), dataLen);
From what I understand, req->data() is in cpp std::string format
On the client side, you should pass both the parameter and its length.
parameter.set_framemat(mat, 12);
Do check if the length of the array at the server side is not zero. Note that bytes is char array and grpc is de marshalling it as string. So if say the array is filled with null characters the data length comes as zero.
I was trying a similar use case
Proto file
message Parameters {
bytes framemat = 16;
};
Client Snippet
const char mat[12] = {0}
parameter.set_framemat(mat);
stream->Write(parameter);
Server Snippet
std::thread reader([&]() {
::nokia::nas::Parameters request;
while (stream->Read(&request))
{
//get the params
grpc::string mat = request.framemat();
logger->info(" Length of Bytes= {} , mat.length()}
Output was zero!
So I changed the client input to check with some char strings and sure enough the data length was coming as 4 at the server side; because this time it was a valid string and string length matched.
const char mat[12] = "sdsd";
Better way is to specify in the proto the data as well as the data length
message StreamBytes {
bytes data_bytes =1;
int32 data_length = 2;
};
I am capturing some audio from my microphone using SFML.
The data is being stored in samples of type Int16*.
Int16* samples;
My question is. What should I do to this samples to stream it over a socket to be played in another place? I ask in relation of data type. Do I need to convert this Int16 array to another type? Or can I just send this Int16* as it is?
EDIT
void BroadcastRecorder::loadBufferFromSamples()
{
//m_samples is of type vector<Int16*>
if (!m_samples.empty()){
m_buffer.loadFromSamples(&m_samples[0], m_samples.size(), 1, getSampleRate());
m_samples.clear();
}
}
void Broadcaster::Send()
{
//load the buffer with the samples
if(!m_recorder->empty()){
m_recorder->loadBufferFromSamples();
const sf::SoundBuffer& buffer = m_recorder->getBuffer();
size_t dataLength = m_recorder->GetSamplesSize();
wxSocketClient * socket = new wxSocketClient(wxSOCKET_NOWAIT);
socket->Notify(false);
// ------------- DATA----------------------
wxString data = "";
wxString strToPrepend(_("--myboundary\r\nContent-Type: audio/wav\r\n"));
wxString strToAppend(_("\r\n--myboundary\r\n"));
// ------------- HEADER -----------------------
wxString header = "";
header.append("POST ");
header.append("/cgi-bin/operator/transmit");
header.append(" HTTP/1.0\r\n");
header.append("Content-Type: multipart/form-data; boundary=--myboundary\r\n");
header.append("Content-Length: " + wxString::Format(wxT("%i"),(dataLength + strToPrepend.Len() + strToAppend.Len()) ) + "\r\n");
header.append("Authorization: Basic keykeykeykey\r\n");
header.append("\r\n");
//-------------- CONNECTION ---------------
wxString host = _("192.168.50.11");
wxIPV4address * address = new wxIPV4address();
address->Hostname(host);
address->Service(8084);
if (socket->Connect(*address)){
//Write header
socket->Write(header.c_str(),header.Len());
//Write data
socket->Write(strToPrepend.c_str(),strToPrepend.Len());
const sf::Int16* samples = buffer.getSamples();
const char* bytesData = reinterpret_cast<const char*>(samples);
socket->Write(bytesData,dataLength);
socket->Write(strToAppend.c_str(),strToAppend.Len());
socket->Close();
}
delete socket;
delete address;
}
}
I am getting only some noises between gaps.
BTW. The audio is being sent to an IP camera p2 connector.
The data format is just the way your application treats them. After all you send raw bytes over a socket. And you can do it with anything you want
Int16 data;
const char* pBytesOfData = (const char*) &data;
int size = sizeof (Int16);
send( socket, pBytesOfdata, size, flags);
When the bytes arrive on the second end it is up to you to interpret them correctly. Probably you will want again treat them as Int16. You need to have a protocol (common way of communication) to do it right (maybe send size of the data at the begining of the transmission, etc).
You can also take a look on libraries that ease serialization: Boost.Asio and Boost.Serialization.
Firstly, You need to create and bind a socket. Then you have to send the data stored in "samples" to another peer by using socket API. For using socket API to send the data, you need to convert this data to char*. As send API of socket takes input of data you need to send as char*. For more information about sending you can go through this link. This is for windows. For Unix you can check the manpage for send API for unix.
Int16* is a pointer. The samples you get should also have an associated length. Your data will likely be between addresses: [samples, samples + length) (where samples is the address to the first sample).
To play the samples remotely (actual code will depend on what APIs you use):
open socket
in a loop
get samples from your microphone
transmit the data over socket
on the server, you will have to read samples in a loop and send them to whatever sound output API you use.
Sockets work with bytes, so in the end you will send bytes. As long as the way you interpret these bytes on the receiving side matches the data you sent, you can send whatever you want in those bytes.
In this case sending the samples directly without conversion seems the most trivial thing to do, but you will probably need to send the size of the sample before, most likely in a fixed length format, for example:
[size on 4 bytes][sample on `size` bytes]
[] [] [] [][] [] [] [] [] []
I am doing a synchronous read/write using boost-asio. The data is coming in binary format, without boundary, the length information is encoded in the packet format. So it is important to read in with specified size. Can ip::tcp::iostream do that? Can someone provide an example? Thanks.
Simple:
boost::asio::read(socket, buffers, boost::asio::transfer_exactly(your_fixed_size));
I work on a program wich send different data with different size. I use a fixed header of 8 byte to encode the size, then, I add the data :
enum { header_length = 8 }; //const header length
I get the size (m_outbound_data is a std::string == a serialized object)
//give header length
std::ostringstream header_stream
header_stream << std::setw(header_length) //set a field padding for header
<< std::hex //set next val to hexadecimal
<< m_data_out.m_outbound_data.size(); //write size in hexa
m_data_out.m_outbound_header = header_stream.str(); //m_outbound_head == size in hexa in a std::string
//m_outbound_header = [ 8 byte size ]
//m_outbound_data = [ serialized data ]
//write all data in the std::vector and send it
std::vector<boost::asio::const_buffer> buffer;
buffer.push_back(boost::asio::buffer(m_data_out.m_outbound_header));
buffer.push_back(boost::asio::buffer(m_data_out.m_outbound_data));
And for reading, you need to read in 2 time : 1st read 8 byte to get the size, then read the data in a vector and deserialize into object :
struct network_data_in {
char m_inbound_header[header_length]; //size of data to read
std::vector<char> m_inbound_data; // read data
};
I use this struct to get data, call read on the m_inbound_header to fill the buffer with size first, then, in the handle :
//get size of data
std::istringstream is(std::string(m_data_in.m_inbound_header, header_length));
std::size_t m_inbound_datasize = 0;
is >> std::hex >> m_inbound_datasize;
m_data_in.m_inbound_data.resize(m_inbound_datasize); //resize the vector
then call again read with the m_inbound_data on buffer, this result of reading exactly the data sent
In the second handle_read you juste have to deserialize the data :
//extract data
std::string archive_data (&(m_data_in.m_inbound_data[0]),m_data_in.m_inbound_data.size());
std::istringstream archive_stream(archive_data);
boost::archive::text_iarchive archive(archive_stream);
archive >> t; //deserialize
Hope that help you !
TCP is a stream-based protocol. This means that whatever you read is just a stream of bytes.
Let's consider an example: you have a message of a fixed size and you send it over TCP. How can the program at the other end read the entire message? there are two ways, one is to surround you message with control chracters (e.g. STX at start and ETX at end). At the start, the program would discard any chars before STX, then read any other chars into the message buffer until ETX is encountered.
Another way is to encode the message length in a fixed-size header (which apparently is your case). So the best thing you can do is figure out a way to read the message length, parse it and read the remaining bytes accordingly.