Using FTDI D2xx and Thorlabs APT communication protocol results in delays on Linux - c++

I am trying to communicate with the Thorlabs TDC001 controllers (apt - dc servo controller) by using the FTDI D2xx driver on Linux. However, when I send writing commands, large delays occur (1-2 seconds) until the command is actually executed on TDC001.
In particular, this can be observed when the connected linear stage is moving and a new position command is sent. It takes 1-2 seconds until the stage actually changes its direction. Also, if I request DCSTATUSUPDATE (which gives position and velocity) and then read out the queue of FTDI, I do not get the right data. Only if I wait 1 second between requesting and reading, I get the (correct) data, but for the past. I added the C++ code for this case.
I need live-data and faster execution of writing commands for closed-loop control.
I'm not sure if the problem is on the side of Thorlabs or FTDI. Everything works, except for the large delays. There are other commands, e.g. MOVE_STOP, which respond immediately. Also, if I send a new position command right after finishing homing, it is executed immediately.
Whenever I ask for FT_GetStatus, there is nothing else in the Tx or Rx queue except the 20 bytes in Rx for DCSTATUSUPDATE.
The references for D2XX and APT communication protocol can be found here:
FTDI Programmer's Guide
Thorlabs APT Communication Protocol
The initialization function:
bool CommunicationFunctions::initializeKeyHandle(string serialnumber){
//INITIALIZATION//
/*
* This function initializes the TDC motor controller and finds its corresponding keyhandle.
*/
keyHandle = NULL;
// To open the device, the vendor and product ID must be set correctly
ftStatus = FT_SetVIDPID(0x403,0xfaf0);
}
//Open device:
const char* tmp = serialnumber.c_str();
int numAttempts=0;
while (keyHandle ==0){
ftStatus = FT_OpenEx(const_cast<char*>(tmp),FT_OPEN_BY_SERIAL_NUMBER, &keyHandle);
if (numAttempts++>20){
cerr << "Device Could Not Be Opened";
return false;
}
}
// Set baud rate to 115200
ftStatus = FT_SetBaudRate(keyHandle,115200);
// 8 data bits, 1 stop bit, no parity
ftStatus = FT_SetDataCharacteristics(keyHandle, FT_BITS_8, FT_STOP_BITS_1, FT_PARITY_NONE);
// Pre purge dwell 50ms.
usleep(50);
// Purge the device.
ftStatus = FT_Purge(keyHandle, FT_PURGE_RX | FT_PURGE_TX);
// Post purge dwell 50ms.
usleep(50);
// Reset device.
ftStatus = FT_ResetDevice(keyHandle);
// Set flow control to RTS/CTS.
ftStatus = FT_SetFlowControl(keyHandle, FT_FLOW_RTS_CTS, 0, 0);
// Set RTS.
ftStatus = FT_SetRts(keyHandle);
return true;
}
How I read out my data:
bool CommunicationFunctions::read_tdc(int32_t* position, uint16_t* velocity){
uint8_t *RxBuffer = new uint8_t[256]();
DWORD RxBytes;
DWORD BytesReceived = 0;
// Manually request status update:
uint8_t req_statusupdate[6] = {0x90,0x04,0x01,0x00,0x50,0x01};
ftStatus = FT_Write(keyHandle, req_statusupdate, (DWORD)6, &written);
if(ftStatus != FT_OK){
cerr << "Command could not be transmitted: Request status update" << endl;
return false;
}
// sleep(1); //**this sleep() would lead to right result, but I don't want this delay**
// Get number of bytes in queue of TDC001
FT_GetQueueStatus(keyHandle,&RxBytes);
// Check if there are bytes in queue before reading them, otherwise do
// not read anything in
if(RxBytes>0){
ftStatus=FT_Read(keyHandle,RxBuffer,RxBytes,&BytesReceived);
if(ftStatus != FT_OK){
cerr << "Read device failed!" << endl;
return false;
}
}
// Check if enough bytes are received, i.e. if signal is right
if(!(BytesReceived >= 6)){
cerr << "Error in bytes received" << endl;
return false;
}
// Look for correct message in RxBuffer and read out velocity and position
getPosVel(position,velocity,RxBuffer);
// Delete receive buffer
delete[] RxBuffer;
RxBuffer = NULL;
return true;
}
If I use read_tdc function after homing and during movement to absolute position, I just get "Homing completed" message in the first attempt. When I try read_tdc again, I get an old value (probably the one from before). I don't understand what happens here. Why does this old data even remain in the queue (latency is 10 ms). Can anybody help me to get faster responses and reactions?

Related

Problem with esp8266 sending large JSON Document via MQTT

I developed a little application that read data from a sensor, store them in SPIFFS memory of my wemos D1 mini (esp8266) and then create a JSON Document and send it via MQTT to my topic. The problem is that as long as I send a JSON Doc with 10 object everything works great, but when I increase the size of the doc over 10 object nothing works. Eventually I need to send a JSON doc with 100 object inside.
What have I already done?
I'm using PubSubClient and I already set the MAX_PACKET_SIZE to the correct value
Using arduinojson assistant I found out the size of my JSON Document (8192 bytes)
I tried to use mqtt.fx to test if the problem was the esp8266 or the mqtt broker. Using mqtt.fx I'm able to send a JSON doc with 100 objects
As soon as I increase the size of the JSON doc I get a wdt error from the serial monitor of my arduino IDE.
I search the internet for wdt error but I don't get what they are and how to solve my problem
Last things I already tried to show on the serial monitor the file.txt in the SPIFFS where I store the data and I can store and then read the 100 object
So in the end I think it's an esp8266 problem and not PubSubClient or MQTT. Am I right?
Does anyone of you here ever encountered this problem before or have some other test I can run?
I search the internet for wdt error but I don't get what they are and how to solve my problem
WDT stands for a Watch Dog Timer. https://os.mbed.com/cookbook/WatchDog-Timer#:~:text=A%20watchdog%20timer%20(WDT)%20is,a%20software%20or%20hardware%20fault.
A watchdog timer (WDT) is a hardware timer that automatically generates a system reset if the main program neglects to periodically service it. It is often used to automatically reset an embedded device that hangs because of a software or hardware fault. Some systems may also refer to it as a computer operating properly (COP) timer. Many microcontrollers including the mbed processor have watchdog timer hardware.
Let's paint a better picture with an example. Let's say that you setup a WDT with a time of 10 seconds. Then the WDT starts counting down from 10 seconds. If it reaches 0 the processor will reset. "Feeding" the WDT will reset the countdown to the original value in this case 10 seconds. So if the WDT has counted down to 4 seconds remaining and you feed it, it resets the countdown back to 10 and starts counting down again.
Does anyone of you here ever encountered this problem before or have some other test I can run?
It looks to me like sending a larger JSON object takes a longer period of time than what the WDT is set for. One possibility would be to break up the JSON object into multiple pieces and send it in smaller chunks instead of one large one. This way the time between WDT "feedings" is reduced. I have no idea if this would be possible for you to change. But this should at least give you a better idea of what's happening.
OK in the end the problem was that sending a large JsonDocument triggered the WDT and the only way I found to overcome this problem was, as suggested by adamvz, to create a main file with all the 100 object, then call a function to split that file in 10 smaller one and send each of them over the internet through an HTTP request or Mosquitto.
Supposing you already created the main file in the spiffs memory, then:
This to split the main file:
void WritePacks() {
sourceFile = LittleFS.open("/file.txt", "r");
if (!sourceFile) {
Serial.println(F("Error: file.txt open failed"));
} else {
Serial.println("File open w/ success");
for (byte idx = 0; idx < outputCount; idx++) {
String aLine;
aLine.reserve(capacity);
if (sourceFile.available() == 0) break;
destinationFile = LittleFS.open(outputFileNames[idx], "w");
if (!destinationFile) {
Serial.print(F("can't open destination "));
Serial.println(outputFileNames[idx]);
break;
} else {
int lineCount = 0;
while (sourceFile.available() && (lineCount <= 10)) {
aLine = sourceFile.readStringUntil('\n');
destinationFile.println(aLine); // double check if the '\n' is in the String or not (--> print or println accordingly)
lineCount++;
}
outputIndex = idx;
Serial.println(outputIndex);
destinationFile.close();
}
} // end for
sourceFile.close();
}
}//end WritePacks
This to publish:
//------ HTTP Publish ------
void httpPublish(){
const char * outputFileNames[] = {"/out1.txt", "/out2.txt", "/out3.txt", "/out4.txt", "/out5.txt", "/out6.txt", "/out7.txt", "/out8.txt", "/out9.txt", "/out10.txt"};
const byte outputCount = sizeof outputFileNames / sizeof outputFileNames[0];
byte outputIndex = 0;
File sourceFile;
File destinationFile;
//Serial.println(capacity);
for (byte idx = 0; idx < outputCount; idx++) {
DynamicJsonDocument doc(capacity);
DynamicJsonDocument globalDoc(capacity);
StaticJsonDocument <1024> localDoc;
String aLine;
aLine.reserve(capacity);
destinationFile = LittleFS.open(outputFileNames[idx], "r");
if (!destinationFile) {
Serial.print(F("can't open destination "));
Serial.println(outputFileNames[idx]);
break;
} else {
Serial.print("Reading: ");
Serial.println(outputFileNames[idx]);
//int lineCount = 0;
while (destinationFile.available()) {
aLine = destinationFile.readStringUntil('\n');
DeserializationError error = deserializeJson(localDoc, aLine);
if (!error) globalDoc.add(localDoc);
else{ Serial.println("Error Writing All files");}
}//while
JsonObject Info = doc.createNestedObject("Info");
Info["Battery"] = battery;
Info["ID"] = id;
Info["Latitudine"] = latitudine;
Info["Longitudine"] = longitudine;
JsonArray Data = doc.createNestedArray("Data");
Data.add(globalDoc);
HTTPClient http;
//Send request
http.begin("yourURL");
char buffer[capacity];
size_t n = serializeJson(doc, buffer);
http.POST(buffer);
Serial.println(buffer);
http.end();
destinationFile.close();
}
}// end for
}//end httpPublish

Serial WriteFile returns before completion

I've been working on a program which dialog with an external device via a serial RS422 bus. The goal is to send commands to the device, which send an answer back.
So far, the code for sending a message look like this:
OVERLAPPED osWrite = {0};
void init()
{
// Create this write operation's OVERLAPPED structure's hEvent.
osWrite.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (osWrite.hEvent == NULL)
// error creating overlapped event handle
std::cout << "Error osWrite.hEvent" << std::endl; // Error in communications; report it.
*hPort = CreateFile("\\\\.\\COM18", (GENERIC_READ | GENERIC_WRITE), FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL);
if (*hPort == INVALID_HANDLE_VALUE) {
std::cout << "Invalid port com handle" << GetLastError() << std::endl;
return;
}
COMMTIMEOUTS commTimeout;
if (GetCommTimeouts(*hPort, &commTimeout)) {
commTimeout.ReadIntervalTimeout = 10;
commTimeout.ReadTotalTimeoutConstant = 10;
commTimeout.ReadTotalTimeoutMultiplier = 10;
commTimeout.WriteTotalTimeoutConstant = 10;
commTimeout.WriteTotalTimeoutMultiplier = 10;
} else
return;
if (!SetCommTimeouts(*hPort, &commTimeout)) {
std::cout << "Error comm timeout" << std::endl;
}
DCB dcb;
if (!GetCommState(*hPort, &dcb)) {
std::cout << "Invalid port com settings" << std::endl;
return;
}
dcb.BaudRate = CBR_115200;
dcb.ByteSize = 8;
dcb.Parity = NOPARITY;
dcb.StopBits = ONESTOPBIT;
SetCommMask(*hPort, EV_RXCHAR);
SetCommState(*hPort, &dcb);
return;
}
DWORD serial_send(HANDLE *hPort, char *msg, int length) {
DWORD dwWritten;
DWORD dwRes;
BOOL fRes;
PurgeComm(*hPort, PURGE_TXCLEAR);
ResetEvent(osWrite.hEvent);
// Issue write.
if (!WriteFile(*hPort, msg, length, &dwWritten, &osWrite)) {
if (GetLastError() != ERROR_IO_PENDING) {
// WriteFile failed, but isn't delayed. Report error and abort.
fRes = FALSE;
} else {
fRes = FALSE;
while (!fRes) {
// Write is pending.
dwRes = WaitForSingleObject(osWrite.hEvent, INFINITE);
switch (dwRes) {
// OVERLAPPED structure's event has been signaled.
case WAIT_OBJECT_0:
if (!GetOverlappedResult(*hPort, &osWrite, &dwWritten, FALSE))
fRes = FALSE;
else
// Write operation completed successfully.
fRes = TRUE;
break;
default:
// An error has occurred in WaitForSingleObject.
// This usually indicates a problem with the
// OVERLAPPED structure's event handle.
fRes = FALSE;
break;
}
}
}
} else {
// WriteFile completed immediately.
fRes = TRUE;
}
return dwWritten;
}
The last function can't return until the write operation is successful. The init() function load without error.
I've used a lot of code from here : https://learn.microsoft.com/en-us/previous-versions/ff802693(v=msdn.10)
Each message is 210 bytes long, and the serial port is running at 115200 bit/s, meaning that I should send a message every ~18.2ms. (210 bytes * 10 bits / 115200)
However, when I measure the time elapsed between 2 messages, I sometimes get a duration much inferior than the expected 18ms (it can go down to 11ms).
Is this a normal behavior for an asynchronous WriteFile + WaitForSingleObject?
What happens if I send another message just after 11ms, will it corrupt the previous message, or does it gets buffered?
I used std::chrono::high_resolution_clock::now() and std::chrono::duration<double, std::milli>(end - start).count() to get the duration of the frame, is it really accurate?
Since Windows is not a real-time OS and is a multi-process & multi-thread OS, time accuracy should not be guaranteed.
If the system is lightly loaded, most of it will work as intended, but not always.
Conversely, the completion of WriteFile() may be notified earlier than it actually is, depending on the hardware and device driver stack configuration.
For example, it may be considered that the process is completed at the time when the data is completely stored in the buffer of the device driver or when the last data is written to the FIFO buffer of the interface chip.
It is better to think that WriteFile() may be completed even if not all the data actually reaches the other party.
It is considered the same as writing file data to the hard disk. Completion of writing to a file on disk is done in the system buffer, and should be written to the actual media at a different time.
If the next serial_send() function is called before all the WriteFile data of the previous time has reached the other party due to bad conditions, there is a possibility that some of the previous transmission data will be discarded.
Because PurgeComm(*hPort, PURGE_TXCLEAR); is called at the beginning of the serial_send() function.
It's not as critical as specifying PURGE_TXABORT, but there is still the possibility of data being discarded with PURGE_TXCLEAR.
PurgeComm function
PURGE_TXABORT 0x0001 Terminates all outstanding overlapped write operations and returns immediately, even if the write operations have not been completed.
PURGE_TXCLEAR 0x0004 Clears the output buffer (if the device driver has one).
If a thread uses PurgeComm to flush an output buffer, the deleted characters are not transmitted. To empty the output buffer while ensuring that the contents are transmitted, call the FlushFileBuffers function (a synchronous operation).
The workaround is to simply not call PurgeComm().
If it is a serial port API, it may be possible to wait for the completion of transmission by specifying/detecting EV_TXEMPTY with SetCommMask()/WaitCommEvent(), but this will only be complicated.
SetCommMask function / WaitCommEvent function
EV_TXEMPTY 0x0004 The last character in the output buffer was sent.
Then your usage of WriteFile() + WaitForSingleObject() + GetOverlappedResult() will eventually work similar to calling WriteFile() synchronously.
Asynchronous operation is not always necessary, but it is better to consider in detail based on what kind of behavior your system requires.

VIDIOC_DQBUF hangs on camera disconnection

My application is using v4l2 running in a separate thread. If a camera gets disconnected then the user is given an appropriate message before terminating the thread cleanly. This works in the vast majority of cases. However, if the execution is inside the VIDIOC_DQBUF ioctl when the camera is disconnected then the ioctl doesn't return causing the entire thread to lock up.
My system is as follows:
Linux Kernel: 4.12.0
OS: Fedora 25
Compiler: gcc-7.1
The following is a simplified example of the problem function.
// Get Raw Buffer from the camera
void v4l2_Processor::get_Raw_Frame(void* buffer)
{
struct v4l2_buffer buf;
memset(&buf, 0, sizeof (buf));
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
// Grab next frame
if (ioctl(m_FD, VIDIOC_DQBUF, &buf) < 0)
{ // If the camera becomes disconnected when the execution is
// in the above ioctl, then the ioctl never returns.
std::cerr << "Error in DQBUF\n";
}
// Queue for next frame
if (ioctl(m_FD, VIDIOC_QBUF, &buf) < 0)
{
std::cerr << "Error in QBUF\n";
}
memcpy(buffer, m_Buffers[buf.index].buff,
m_Buffers[buf.index].buf_length);
}
Can anybody shed any light on why this ioctl locks up and what I might do to solve this problem?
I appreciate any help offered.
Amanda
I am currently having the same issue. However, my entire thread doesn't lock up. The ioctl times out (15s) but thats way too long.
Is there a what to query V4L2 (that wont hang) if video is streaming? or at least change the ioctl timeout ?
UPDATE:
#Amanda you can change the timeout of the dequeue in the v4l2_capture driver source & rebuild the kernel/kernel module
modify the timeout in the dqueue function:
if (!wait_event_interruptible_timeout(cam->enc_queue,
cam->enc_counter != 0,
50 * HZ)) // Modify this constant
Best of luck!

Linux poll on serial transmission end

I'm implementing RS485 on arm developement board using serial port and gpio for data enable.
I'm setting data enable to high before sending and I want it to be set low after transmission is complete.
It can be simply done by writing:
//fd = open("/dev/ttyO2", ...);
DataEnable.Set(true);
write(fd, data, datalen);
tcdrain(fd); //Wait until all data is sent
DataEnable.Set(false);
I wanted to change from blocking-mode to non-blocking and use poll with fd. But I dont see any poll event corresponding to 'transmission complete'.
How can I get notified when all data has been sent?
System: linux
Language: c++
Board: BeagleBone Black
I don't think it's possible. You'll either have to run tcdrain in another thread and have it notify the the main thread, or use timeout on poll and poll to see if the output has been drained.
You can use the TIOCOUTQ ioctl to get the number of bytes in the output buffer and tune the timeout according to baud rate. That should reduce the amount of polling you need to do to just once or twice. Something like:
enum { writing, draining, idle } write_state;
while(1) {
int write_event, timeout = -1;
...
if (write_state == writing) {
poll_fds[poll_len].fd = write_fd;
poll_fds[poll_len].event = POLLOUT;
write_event = poll_len++
} else if (write == draining) {
int outq;
ioctl(write_fd, TIOCOUTQ, &outq);
if (outq == 0) {
DataEnable.Set(false);
write_state = idle;
} else {
// 10 bits per byte, 1000 millisecond in a second
timeout = outq * 10 * 1000 / baud_rate;
if (timeout < 1) {
timeout = 1;
}
}
}
int r = poll(poll_fds, poll_len, timeout);
...
if (write_state == writing && r > 0 && (poll_fds[write_event].revent & POLLOUT)) {
DataEnable.Set(true); // Gets set even if already set.
int n = write(write_fd, write_data, write_datalen);
write_data += n;
write_datalen -= n;
if (write_datalen == 0) {
state = draining;
}
}
}
Stale thread, but I have been working on RS-485 with a 16550-compatible UART under Linux and find
tcdrain works - but it adds a delay of 10 to 20 msec. Seems to be polled
The value returned by TIOCOUTQ seems to count bytes in the OS buffer, but NOT bytes in the UART FIFO, so it may underestimate the delay required if transmission has already started.
I am currently using CLOCK_MONOTONIC to timestamp each send, calculating when the send should be complete, when checking that time against the next send, delaying if necessary. Sucks, but seems to work

Flush queued GPIB responses

Architecture ->GBIP from external interface is connected to target ( linux) system via gpib bus.
Inside Linux box , there is ethernet cable from GPIB to motherboard.
The PIC_GPIB card on external interface is IEEE 488.2
I am sending a query from external interface to linux box.
Few scenarios
1) If I send a query which does not expect a response back , then next query send will work.
2) If I send a query which expect response back , and when I have received the response and read it and then fire next query it works fine.
3) BUT if I send a query from external interface and got response back and I ignore to read the response , then Next query fails.
I am requesting help for scenario 3.
The coding is done on linux side and its a socket programming , which uses linux inbuilt function from unistd.h for read and write.
My investigation : I have found there is a internal memory on gbib card on external interface which stores the value of previous response until we have the read. Generally I use IEEE string utility software to write commands that goes to linux box and read reposne via read button .
Could someone please direct me how to clean input buffer or memory which stores value so that write from external command contiunues without bothering to read it.
My code on linux side has been developed in C++ and socket programming. I have used in bulit write and read function to write and read to the gpib and to json server.
Sample code is shown below
bool GpibClass::ReadWriteFromGPIB()
{
bool check = true;
int n = 0;
char buffer[BUFFER_SIZE];
fd_set read_set;
struct timeval lTimeOut;
// Reset the read mask for the select
FD_ZERO(&read_set);
FD_SET(mGpibFd, &read_set);
FD_SET(mdiffFd, &read_set);
// Set Timeout to check the status of the connection
// when no data is being received
lTimeOut.tv_sec = CONNECTION_STATUS_CHECK_TIMEOUT_SECONDS;
lTimeOut.tv_usec = 0;
cout << "Entered into this function" << endl;
// Look for sockets with available data
if (-1 == select(FD_SETSIZE, &read_set, NULL, NULL, &lTimeOut))
{
cout << "Select failed" << endl;
// We don't know the cause of select's failure.
// Close everything and start from scratch:
CloseConnection(mGpibFd);
CloseConnection(mdifferntServer); // this is different server
check = false;
}
// Check if data is available from GPIB server,
// and if any read and push it to gpib
if(true == check)
{
cout << "Check data from GPIB after select" << endl;
if (FD_ISSET(mGpibFd, &read_set))
{
n = read(mGpibFd, buffer, BUFFER_SIZE);
cout << "Read from GPIB" << n << " bytes" << endl;
if(0 < n)
{
// write it to different server and check if we get response from it
}
else
{
// Something failed on socket read - most likely
// connection dropped. Close socket and retry later
CloseConnection(mGpibFd);
check = false;
}
}
}
// Check if data is available from different server,
// and if any read and push it to gpib
if(true == check)
{
cout << "Check data from diff server after select" << endl;
if (FD_ISSET(mdiffFd, &read_set))
{
n = read(mdiffFd, buffer, BUFFER_SIZE);
cout << "Read from diff servewr " << n << " bytes" << endl;
if (0 < n)
{
// Append, just in case - makes sure data is sent.
// Extra cr/lf shouldn't cause any problem if the json
// server has already added them
strcpy(buffer + n, "\r\n");
write(mGpibFd, buffer, n + 2);
std::cout <<" the buffer sixze = " << buffer << std::endl;
}
else
{
// Something failed on socket read - most likely
// connection dropped. Close socket and retry later
CloseConnection(mdiffFd);
check = false;
}
}
}
return check;
}
You should ordinarily be reading responses after any operation which could generate them.
If you fail to do that, an easy solution would be to read responses in a loop until you have drained the queue to empty.
You can reset the instrument (probably *RST), but you would probably loose other state as well. You will have to check it's documentation to see if there is a command to reset only the response queue. Checking the documentation is always a good idea, because the number of instruments which precisely comply with the spec is dwarfed by the number which augment or omit parts in unique ways.