How to get Arduino to read serial from Python? - python-2.7

Question

Specific problem
I am trying to communicate between a Python script and my Arduino Pro Mini using serial with a USB cable. In Python I can read what the Arduino writes but the Arduino can either not read what the Python writes or it is reading it in a form that I can't process. The most obvious way this manifests itself is when I try to write what the Arduino has just read from Python, to Python. Nothing gets displayed in Python.
My Arduino sketch appears to work just fine when I work with it in the Serial Monitor, i.e. it can read and write so I suspect it is something at the Python end. What's causing this?
Details
I want to be able test how quickly I can send 64 byte-long pieces of information to my Arduino. To do this I wrote a Python script that could should be able to periodically send various information to my Arduino. I can change the period and thus test what kind of speed I can get.
However, despite the fact that my Arduino sketch works just fine with Serial Monitor, i.e. I can send it information and read information from it, I can't get it to work with Python.
With my Python script I can open the connection and read what the Arduino writes, but for some reason there is a problem with writing from Python. I have tried the following but none of them have worked:
Using an Uno instead of a Pro Mini.
Putting a Serial.println() statement in my get_data() function. Nothing gets printed.
Sending a string with the format "T(number);" and using sscanf() to extract the number. Using an if statement to examine the number and turn an onboard LED on. Did not work.
Sending a string beginning with "T" and using an if statement to see if it is stored in raw_data[]. It isn't.
Writing the data from Python at different times (i.e. using a longer delay).
Writing from Python using device.write(b"mystring") and device.write("mystring".encode())
I'm at a loss for what to try next…
Python (2.7) code
import serial, time, random, string
device = serial.Serial('/dev/tty.usbserial-A105YOZX', baudrate = 38400)
initial_time = time.time()
current_time = time.time()
counter = 0
control = 0
while counter < 1000:
current_time = time.time()
if current_time - initial_time > 0.1:
initial_time = current_time
counter += 1
device.read()
if current_time - initial_time > 1 and control == 0:
device.write(b"Message being sent")
control = 1
Arduino code
#define RAW_LEN 64 //max length of incoming string (buffer size is only 64 bytes)
//variables for incoming data
char raw_data[RAW_LEN];
int raw_data_i = 0; //index
unsigned long my_time;
int control = 0;
int for_print;
int get_data(int i) {
//fills up raw_data[] with serial data. Clears raw_data if it gets too long
if (Serial.available()>0) {
raw_data[i] = Serial.read();
Serial.println(raw_data[i]);
//Serial.println(raw_data[i]);
if (i<RAW_LEN-1){
i++;
}
}
return i;
}
void clear_data() {
//clears raw_data and resets raw_data_i
raw_data_i = 0;
memset(raw_data,'\0',RAW_LEN-1);
}
//print functions
void raw_print(char data[]) {
Serial.println("Received data: ");
//if (raw_data[0] == 'T') {digitalWrite(LED_BUILTIN, HIGH);}
for (int i=0; i< RAW_LEN;i++) {
Serial.println(data[i]);
}
}
void setup() {
Serial.begin(38400);
Serial.println("Restart");
pinMode(LED_BUILTIN, OUTPUT);
digitalWrite(LED_BUILTIN, LOW);
}
void loop() {
my_time = millis();
raw_data_i = get_data(raw_data_i);
//clear raw_data if it gets full
if (raw_data_i == RAW_LEN-1) {
clear_data();
}
if (my_time > 5000 && control == 0) {
Serial.println("Here");
raw_print(raw_data);
control = 1;
}
}

Answer 1

The problem turned out to be in the Python code. In while loop the first if statement is True after 0.1s, and then executes 9 more times before the second if statement is True.
The problem with this is that device.read() just hangs if it does not receive a character. Because I wasn't sending enough characters from the Arduino the Python code paused at the device.read() and never got to the second if statement.
An annoying mistake that took a while to figure out, hopefully this question and answer will save someone else a lot of debugging time.