Serial.read() skips over serial input - c++

Trying to send serial messages using Arduino Uno and standard IDE. Ran into issue parsing the serial message sent to the device.
See that if I include this line Serial.println("Serial.available() = " + String(Serial.available())); I will be able to read the rest of the message. If this is commented out I will only see the first letter of the message and skip over the rest. Attached image of output that I'm seeing with and without the added line of code.
// the setup routine runs once when you press reset:
void setup() {
// initialize serial communication at 9600 bits per second:
Serial.begin(9600);
while (!Serial) {} // wait for serial to be initialized
Serial.println("Setup called. Serial port ready.");
Serial.println("Waiting for time sync message...");
while (!Serial.available()) {}
processSyncMessage();
}
void processSyncMessage() {
// parse first letter of message
char messageHeader = (char) Serial.read();
switch (messageHeader) {
case TIME_HEADER:
// do processing
break;
default:
Serial.println("Unknown message sent with header: " + String(messageHeader));
// must include this line in order to see the entire message sent
// just calling a println or a Serial.available() doesn't work ????
Serial.println("Serial.available() = " + String(Serial.available()));
Serial.println("---start of message");
for (int r = 0; r != -1; r = Serial.read()) {
Serial.print((char) r);
}
Serial.println();
Serial.println("---end of message");
break;
}
}
Missing Buffer
With printout
Is this somehow related to a buffer? Can I flush it somehow with fflush(SOME_SECRET_BUFFER)?

have you tried Serial.readString() to parse the entire missing characters?

Serial data is transmitted and received one character at a time. At 9600 baud, the transmission rate is approximately one character per millisecond.
The code assumes that once the first character has arrived, all of them have. This is not the case. The addition of the println consumes CPU time, and therefore has the effect of adding a delay. This delay allows the rest of the original message to be received.
A receive function with an appropriate timeout for your application is needed here.

Related

Arduino serial read and check

I am trying to learn some things about arduino serial reading from a bluetooth device. This is the code I found everywhere:
int incomingByte = 0; // for incoming serial data
void setup() {
Serial.begin(9600); // opens serial port, sets data rate to 9600 bps
}
void loop() {
// send data only when you receive data:
if (Serial.available() > 0) {
// read the incoming byte:
incomingByte = Serial.read();
// say what you got:
Serial.print("I received: ");
Serial.println(incomingByte);
}
}
When I send the word "word" from my mobile bluetooth I get 4 lines:
I received: w
I received: o
I received: r
I received: d
which is fine... But here 's my question:
I want to check the received characters as soon as they come into the serial input, so if one of them is the character "r", I would like an extra line to be printed in the Serial Monitor, something like: "wow, that was an r!"
So I add an if statement after the println(incomingByte) and my code is now like that:
Serial.print("I received: ");
Serial.println(incomingByte);
if (incomingByte == "r") {
Serial.println("wow, that was an r!");
}
That code never works, it 's like not having an "r" at all.
Could someone explain to me?
Thanks
I think it is due to the usage of "r" instead of 'r'. The difference is that in first case you have string, which is char[], and in the second you have single char.

Problem with esp8266 sending large JSON Document via MQTT

I developed a little application that read data from a sensor, store them in SPIFFS memory of my wemos D1 mini (esp8266) and then create a JSON Document and send it via MQTT to my topic. The problem is that as long as I send a JSON Doc with 10 object everything works great, but when I increase the size of the doc over 10 object nothing works. Eventually I need to send a JSON doc with 100 object inside.
What have I already done?
I'm using PubSubClient and I already set the MAX_PACKET_SIZE to the correct value
Using arduinojson assistant I found out the size of my JSON Document (8192 bytes)
I tried to use mqtt.fx to test if the problem was the esp8266 or the mqtt broker. Using mqtt.fx I'm able to send a JSON doc with 100 objects
As soon as I increase the size of the JSON doc I get a wdt error from the serial monitor of my arduino IDE.
I search the internet for wdt error but I don't get what they are and how to solve my problem
Last things I already tried to show on the serial monitor the file.txt in the SPIFFS where I store the data and I can store and then read the 100 object
So in the end I think it's an esp8266 problem and not PubSubClient or MQTT. Am I right?
Does anyone of you here ever encountered this problem before or have some other test I can run?
I search the internet for wdt error but I don't get what they are and how to solve my problem
WDT stands for a Watch Dog Timer. https://os.mbed.com/cookbook/WatchDog-Timer#:~:text=A%20watchdog%20timer%20(WDT)%20is,a%20software%20or%20hardware%20fault.
A watchdog timer (WDT) is a hardware timer that automatically generates a system reset if the main program neglects to periodically service it. It is often used to automatically reset an embedded device that hangs because of a software or hardware fault. Some systems may also refer to it as a computer operating properly (COP) timer. Many microcontrollers including the mbed processor have watchdog timer hardware.
Let's paint a better picture with an example. Let's say that you setup a WDT with a time of 10 seconds. Then the WDT starts counting down from 10 seconds. If it reaches 0 the processor will reset. "Feeding" the WDT will reset the countdown to the original value in this case 10 seconds. So if the WDT has counted down to 4 seconds remaining and you feed it, it resets the countdown back to 10 and starts counting down again.
Does anyone of you here ever encountered this problem before or have some other test I can run?
It looks to me like sending a larger JSON object takes a longer period of time than what the WDT is set for. One possibility would be to break up the JSON object into multiple pieces and send it in smaller chunks instead of one large one. This way the time between WDT "feedings" is reduced. I have no idea if this would be possible for you to change. But this should at least give you a better idea of what's happening.
OK in the end the problem was that sending a large JsonDocument triggered the WDT and the only way I found to overcome this problem was, as suggested by adamvz, to create a main file with all the 100 object, then call a function to split that file in 10 smaller one and send each of them over the internet through an HTTP request or Mosquitto.
Supposing you already created the main file in the spiffs memory, then:
This to split the main file:
void WritePacks() {
sourceFile = LittleFS.open("/file.txt", "r");
if (!sourceFile) {
Serial.println(F("Error: file.txt open failed"));
} else {
Serial.println("File open w/ success");
for (byte idx = 0; idx < outputCount; idx++) {
String aLine;
aLine.reserve(capacity);
if (sourceFile.available() == 0) break;
destinationFile = LittleFS.open(outputFileNames[idx], "w");
if (!destinationFile) {
Serial.print(F("can't open destination "));
Serial.println(outputFileNames[idx]);
break;
} else {
int lineCount = 0;
while (sourceFile.available() && (lineCount <= 10)) {
aLine = sourceFile.readStringUntil('\n');
destinationFile.println(aLine); // double check if the '\n' is in the String or not (--> print or println accordingly)
lineCount++;
}
outputIndex = idx;
Serial.println(outputIndex);
destinationFile.close();
}
} // end for
sourceFile.close();
}
}//end WritePacks
This to publish:
//------ HTTP Publish ------
void httpPublish(){
const char * outputFileNames[] = {"/out1.txt", "/out2.txt", "/out3.txt", "/out4.txt", "/out5.txt", "/out6.txt", "/out7.txt", "/out8.txt", "/out9.txt", "/out10.txt"};
const byte outputCount = sizeof outputFileNames / sizeof outputFileNames[0];
byte outputIndex = 0;
File sourceFile;
File destinationFile;
//Serial.println(capacity);
for (byte idx = 0; idx < outputCount; idx++) {
DynamicJsonDocument doc(capacity);
DynamicJsonDocument globalDoc(capacity);
StaticJsonDocument <1024> localDoc;
String aLine;
aLine.reserve(capacity);
destinationFile = LittleFS.open(outputFileNames[idx], "r");
if (!destinationFile) {
Serial.print(F("can't open destination "));
Serial.println(outputFileNames[idx]);
break;
} else {
Serial.print("Reading: ");
Serial.println(outputFileNames[idx]);
//int lineCount = 0;
while (destinationFile.available()) {
aLine = destinationFile.readStringUntil('\n');
DeserializationError error = deserializeJson(localDoc, aLine);
if (!error) globalDoc.add(localDoc);
else{ Serial.println("Error Writing All files");}
}//while
JsonObject Info = doc.createNestedObject("Info");
Info["Battery"] = battery;
Info["ID"] = id;
Info["Latitudine"] = latitudine;
Info["Longitudine"] = longitudine;
JsonArray Data = doc.createNestedArray("Data");
Data.add(globalDoc);
HTTPClient http;
//Send request
http.begin("yourURL");
char buffer[capacity];
size_t n = serializeJson(doc, buffer);
http.POST(buffer);
Serial.println(buffer);
http.end();
destinationFile.close();
}
}// end for
}//end httpPublish

Strange characters after JSON

I have an application that gets the Arduino Nano information and sends it to ESP-01 via UART. The ESP-01 send this to MQTT.
NANO CODE:
#include "DHTesp.h"
#include <ArduinoJson.h>
#include <SoftwareSerial.h>
#define gasSensor A1
#define dhtPin 5
#define rain A2
#define soil A3
#define ldr A4
DHTesp dht;
void setup() {
Serial.begin(115200);
pinMode(gasSensor, INPUT);
pinMode(rain, INPUT);
pinMode(soil, INPUT);
pinMode(ldr, INPUT);
digitalWrite(dhtPin, LOW);
dht.setup(dhtPin, DHTesp::DHT11);
}
void loop() {
delay(dht.getMinimumSamplingPeriod());
float humidity = dht.getHumidity();
float temperature = dht.getTemperature();
DynamicJsonBuffer jBuffer;
JsonObject& measure = jBuffer.createObject();
JsonObject& data = jBuffer.createObject();
measure["gas"] = analogRead(gasSensor);
measure["humidity"] = humidity;
measure["temperature"] = temperature;
measure["heatindex"] = dht.computeHeatIndex(temperature, humidity, false);
measure["rain"] = analogRead(rain);
measure["soil"] = analogRead(soil);
measure["ldr"] = analogRead(ldr);
data["measure"] = measure;
data.printTo(Serial);
}
ESP-01 CODE:
#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <PubSubClient.h>
const char* mqtt_server = "0.0.0.0";
WiFiClient espClient;
PubSubClient client(espClient);
char mystr[100];
void setup() {
Serial.begin(115200);
WiFi.begin("", "");
Serial.print("Connecting");
while (WiFi.status() != WL_CONNECTED) {
delay(1000);
Serial.print(".");
}
Serial.println();
Serial.print("Connected to IP: ");
Serial.println(WiFi.localIP());
client.setServer(mqtt_server, 1883);
}
void loop() {
if (client.connect("ESP")) {
Serial.println("STATUS MQTT-ESP: OK");
while (true) {
Serial.readBytes(mystr, 108);
client.publish("esp", mystr);
delay(1000);
}
} else {
Serial.println("STATUS MQTT-ESP: OFF");
}
}
But, I'm getting strange characters into MQTT:
And, at the connection between NANO <-uart-> Computer, the JSON is normal:
Someone could help me?
Thanks.
I am basing this answer on a bit of assumptions as I don't have an Arduino at hand.
You send your data like this:
data.printTo(Serial);
This sends the string holding the formatted JSON data. This does not include the terminating 0 byte.
Then you receive it like this:
char mystr[100];
...
while(true){
Serial.readBytes(mystr, 108);
client.publish("esp", mystr);
delay(1000);
}
This has multiple errors:
You do not care if you got any bytes at all. The method returns the number of bytes but you do not handle the return value at all. If you got 0 bytes within the timeout value, you will just send the previous message again instead of waiting for valid data.
You cannot put 108 bytes into memory location of 100 bytes.
You put an array of char into the publish method that does not contain any termination. How should that method know how many characters are really part of the string?
Try this:
while(true){
size_t num = Serial.readBytes(mystr, sizeof(mystr)-1);
mystr[num] = 0;
client.publish("esp", mystr);
delay(1000);
}
You do not have any protocol that allows you to detect when a message starts or ends. UART communiation is a byte stream without any boundaries for datagrams. You must ensure that you know what belongs to a message and what does not. This also means that you can never know if you have a complete message in your receive bufffer or an incomplete or even more than one message. It's up to you to detect and split messages. Fixing this is a bit more complex. I can only give some hints what you need to do.
a) Detect message boundaries:
You might send some terminator like \n, 0 or similar after each JSON string and scan for this extra byte on receiver side.
Or you can send the length of the string before you send the string.
You could also just check on receiver side when you have a matching pair of {}. This would not require any change on sender side.
b) Collect messages:
Call your read function as long as it takes until you detect the end of a string.
You might need to use intermediate buffer to collect multiple read buffers.
c) Forward messages:
As soon as you detected the end of a message, forward it via publish function.
Then move the remaining content of your collection buffer to the start of that buffer.
If you immediately find the end of another message, repeat that step.
If you don't find any complete message, continue to collect more data until the next message is complete.

Use Hardware RX pin as an interrupt pin of arduino

I want to use hardware RX pin of Arduino as interrupt pin. If there is any data available on RX pin, an interrupt signal will be generated, call a callback function to read incoming serial data.I don't want my loop() function constant reading on serial port. I am using this code but my interrupt is not triggered.I also tried by removing digitalPintointerrupt() but getting no response.
`#include <SoftwareSerial.h>
const byte interruptPin = 0;//In arduino MEGA RX 19. TX 18
String msg = "";//Incomming message
#define Line_RX 3 //UART RX
#define Line_TX 2 //UART TX
SoftwareSerial mySerial (Line_TX, Line_RX); //initialize software serial
void setup() {
// put your setup code here, to run once:
Serial.begin(19200);
mySerial.begin(19200);
attachInterrupt(digitalPinToInterrupt(interruptPin), serial_read, HIGH);
}//end setup
void loop() {
// put your main code here, to run repeatedly:
}//end loop
void serial_read(){
char _bite;
sei();//Disable hardware interrupts for a moment
while(Serial.available()>0){
delay(1);//Do not delete this delay
if(Serial.available()>0){
_bite = (char)Serial.read();
msg += _bite;
if(_bite == '\n'){
mySerial.print(msg);//Do what you print your message
msg = "";//Clean message for new one
break;
}//end if
}//end if
}//end while
cli();//re-enabling hardware interrupts
}//en
d serial_read`
sei() ENABLES interrupts, while cli() disables them. Your comments suggest you have them backwards. Perhaps there are other problems, but these instructions are certainly not consistent with your intentions.
If you want to get lower-level, consider a pure interrupt-driven design like:
ISR (USART0_UDRE_vect)
{
// Send next byte and increment pointer
UDR0 = *ub_outptr++;
// Pointer wrapping
if (ub_outptr >= UART_buffer + BUFF_SIZE)
ub_outptr = UART_buffer;
// If buffer is empty: disable interrupt
if(--ub_buffcnt == 0)
UCSR0B &= ~(1 << UDRIE0);
}
I know this takes you out of the Arduino library stuff, so this may not be ideal for you. But it works (the example is for sending data, as I have an active project where the microcontroller sends data to an LCD display. Just an example in AVR-GCC C.)
Try my NeoHWSerial. It is a modified version of the Arduino core class, HardwareSerial, the class used for Serial, Serial1, etc. It adds the ability to register a callback for each received character.

Arduino doesn't receive data after reconnecting to USB

So, I've built a basic QT GUI where I want to establish communication with an Arduino Nano through USB. I send a number through the GUI and the Arduino receives the number and processes it.
The communication works fine when I upload the code to Arduino and right afterwards open the GUI and start the process. However, when I disconnect the Arduino from the USB (or restart my PC - I've tried both) and reconnect it to use it with the GUI, the Arduino behaves like it received nothing.
More specifically, in the first case Serial.available() returns "1" as it receives the number properly, but in the latter case it returns "0", so it does nothing.
I made the code as simple as I could trying to track down the issue and the problem continues.
So here is the main QT GUI code:
depth_ = insertDepthEdit->text().toInt(); // user input from GUI
myThread *mThread;
mThread = new myThread(this, depth_);
connect(mThread, SIGNAL(valueRead(QString)), this, SLOT(onTextChange(QString)));
//valueRead is the signal emitted from Arduino
//onTextChange the function that processes the received string
mThread->start();
mThread->wait(100);
mThread->quit();
The Arduino thread code (also QT):
void myThread::run() {
QSerialPort serial;
serial.setPortName("COM3");
serial.setBaudRate(QSerialPort::Baud9600);
serial.setDataBits(QSerialPort::Data8);
serial.setParity(QSerialPort::NoParity);
serial.setStopBits(QSerialPort::OneStop);
serial.setFlowControl(QSerialPort::NoFlowControl);
serial.open(QIODevice::ReadWrite);
if (serial.isOpen() && serial.isWritable()) {
qDebug() << "Ready to write..." << endl;
QByteArray ba(QString::number(depth_).toStdString().c_str());
qDebug() << ba << endl;
serial.write(ba);
if (serial.bytesToWrite() > 0) {
serial.flush();
if (serial.waitForBytesWritten(1000)) {
qDebug() << "data has been sent" << endl;
}
}
if (serial.flush()) {
qDebug() << "ok" << endl;
}
}
else {
qDebug() << "Error";
}
if (serial.isOpen() && serial.isReadable()) {
qDebug() << "Ready to read..." <<endl;
while (serial.waitForReadyRead(5000)) {
QByteArray inByteArray = serial.readLine();
input_ = QString(inByteArray);
qDebug() << input_;
qDebug() << "ok" << endl;
emit valueRead(input_);
}
}
serial.close();
}
And finally the Arduino code:
int c = 0;
const int ledPin = 13;
void setup() {
Serial.begin(9600);
}
void loop() {
Serial.print(Serial.available());
while (Serial.available() > 0) {
digitalWrite(ledPin, HIGH);
delay(5);
c = Serial.read() - '0';
Serial.flush();
}
delay(1000);
digitalWrite(ledPin, LOW);
delay(500);
}
When I upload the code to Arduino, it functions properly no matter if I close the GUI and restart it. The problem happens only if Arduino loses power, e.g: when I disconnect it from USB or restart the PC.
~~~~~~~~~~~~~~~~~~~~~~~~~~EDIT~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
COM port remains the same after reconnecting and Arduino Rx LED flashes normally when I send data through the GUI.
~~~~~~~~~~~~~~~~~~~~~~~~~~EDIT 2~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OK, so, I tried using the code from Arduino Serial documentation and the problem remains. When I upload the code the Arduino receives the character properly and turns the LED on, but once I disconnect it and then connect it back, it does nothing, the LED remains low as it never enters "if".
Here's the code I used:
int incomingByte = 0;
void setup() {
Serial.begin(9600);
}
void loop() {
if (Serial.available() > 0) {
digitalWrite(13, HIGH);
incomingByte = Serial.read();
}
}
~~~~~~~~~~~~~~~~~~~~~~EDIT 3~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
So I have the following 3 scenarios:
Use Scenario A:
Upload code
Run GUI
Send data - It receives properly
Disconnect and reconnect
Run GUI again
Send data - RX blinks but Serial.available returns 0
Use Scenario B:
Upload code
Run Brays
Send data - It receives properly
Disconnect and reconnect
Run Brays again
Send data - It receives properly
Use Scenario C (the most interesting) :
Upload code
Run GUI
Send data - It receives properly
Disconnect and reconnect
Run Brays this time
Send data - It receives properly
Run GUI again after Brays
Send data - It receives properly
I also made the QT GUI code as simple as that but the problem persists:
void myThread::run()
{
QSerialPort *serial = new QSerialPort();
serial->setPortName("COM3");
serial->setBaudRate(QSerialPort::Baud9600);
serial->setDataBits(QSerialPort::Data8);
serial->open(QIODevice::WriteOnly);
if (serial->isOpen() && serial->isWritable())
{
QByteArray ba(QString::number(depth_).toStdString().c_str());
serial->write(ba);
serial->flush();
serial->close();
}
delete serial;
}
~~~~~~~~~~~~~~~~~~~~~~EDIT 4~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
So after much effort and curiosity, I realized that the source of the problem is not the Arduino code but something related to QT or Windows. I added the delays that Jeff recommended and noticed that each time it functioned properly Rx blinked and LED became high as indicated by the code. However, after reconnecting, the problem still remained but I noticed that, this time, immediately after clicking "Send" to send the characters, the LED blinked for some milliseconds (possibly indicating some error??) and then after the 1 second delay the Rx blinked indicating the receipt of data and LED remained LOW as Serial.available remained 0.
So, what I tried next, was to remove one line of code at a time to see what causes the problem. And I ended up with literally blank Arduino code, just empty setup and loop methods, and the following QT GUI code:
void myThread::run()
{
QSerialPort *serial1 = new QSerialPort();
serial1->setPortName("COM5");
serial1->open(QIODevice::WriteOnly);
serial1->close();
}
To summarize, what happens now is:
Upload code to Arduino
Run GUI
Send data
Nothing happens (normal behaviour)
Disconnect and reconnect Arduino to USB
Run GUI
Send data
Arduino LED momentarily blinks once (possibly indicating some kind of error)
OK, so, after hours of debugging I've found what caused the problem.
The root of it was that after reconnecting the Arduino, each time I called serial.open in QT, Arduino did a reset (indicated by the blink of the LED) and by the time it was after the bootloader stage and was running the code, the main program had already passed the serial.write QT command without receiving the data.
So, what I did to solve the problem was to just add a Sleep(uint(2000)); after serial.open in order to let Arduino finish booting and then start sending data.
Thank you all for your help and immediate answers!
In my experience, the issue is not the code in the Arduino. It is because the serial port gets a different name when it is plugged back in.
For example in Linux, originally the port is /dev/ARD0, but when it is disconnected and plugged back in with connections on ARD0, the new plugin is named /dev/ARD1. (In Windows, it might be COM17 then COM18.)
The only way I know to make it become the original port name is to close everything that is connected to it before plugging in again: Close the Arduino IDE, close all programs which have opened the port, etc.
If you use this example for the Arduino Serial documentation do you receive the chars you send?
int incomingByte = 0; // for incoming serial data
void setup() {
Serial.begin(9600); // opens serial port, sets data rate to 9600 bps
}
void loop() {
// send data only when you receive data:
if (Serial.available() > 0) {
// read the incoming byte:
incomingByte = Serial.read();
// say what you got:
Serial.print("I received: ");
Serial.println(incomingByte, DEC);
}
}
Grasping at straws here, replace my comments below a one second delay. Editing on iPhone messed with the format a little, but I believe you can see my intent.
Edit: Also, I think you should not do serial->close inside your loop. I would also try sending a single character repeatedly until we have that working.
void myThread::run()
{
QSerialPort *serial = new QSerialPort();
serial->setPortName("COM3");
serial->setBaudRate(QSerialPort::Baud9600);
serial->setDataBits(QSerialPort::Data8);
serial->open(QIODevice::WriteOnly);
if (serial->isOpen() && serial->isWritable())
{
QByteArray ba(QString::number(depth_).toStdString().c_str());
serial->write("x");
delay 1 second here
serial->flush();
delay 1 second here
}
serial->close();
delay 1 second here
delete serial;
}