Unable to open file in bytes from sd card (ESP32) - c++

I want to open a .wav file in bytes from the sd card connected to esp32.
My code is based in this question
I just want upload a file from my sd card to a server but by the moment I can't read the file despites the file exists in the sd card:
Look this lines of code:
char *fname = "/sdcard/test.wav";
FILE *fp = fopen(fname, "rb"); // read in bytes
if(!fp) {
Serial.println("File doens't exist");
}
I don't understand why it File file = SD.open("/test.wav") works but the above code not. According to my reference I should read this file in bytes and not in a normal manner like this => File file = SD.open("/test.wav").
I would like know how I can see the full path of my sd card because the original code uses this:
char *fname = "/sdcard/test_text.txt";
FILE *fp = fopen(fname, "rb");
why that uses this path /sdcard/test_text.txt I don't know exactly. If I try to open by using:
File file = SD.open("/test.wav")
will works but some lines of code after this will throws errors because the file should be read in bytes.
I would like to see your advices guys I will appreciate any idea to fix this problem.
this is the entire code:
#include "Arduino.h"
#include "WiFi.h"
#include "SD.h"
#include "FS.h"
#include "HTTPClient.h"
//SD CARD MODULE
#define SD_CS 5
#define SPI_MOSI 23
#define SPI_MISO 19
#define SPI_SCK 18
// Wifi Credentials
const char* ssid = "XXX";
const char* password = "XXX";
void setup() {
Serial.begin(115200);
WiFi.mode(WIFI_STA);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.println("MY IP address: ");
Serial.println(WiFi.localIP());
if (!SD.begin(SD_CS)) {
Serial.println("SD init failed!");
return;
}
Serial.println("SD init successfull!");
}
void loop()
{
WiFiClient client;
Serial.println("starting file upload");
IPAddress host(192, 168, 18, 8); //server ip
int port = 80;
if (!client.connect(host, port))
{ // check connection to host if untrue internet connection could be down
Serial.println("couldn't connect to host");
}
Serial.println("Connect to host!");
HTTPClient http;
const char* serverName = "http://192.168.18.8/upload/upload.php";
http.begin(client, serverName);
char *fname = "/sdcard/test.wav";
FILE *fp = fopen(fname, "rb"); // read in bytes
if(!fp) {
Serial.println("File doens't exist");
}
//get file size
fseek(fp, 0, SEEK_END); //send file pointer to end of file
int file_size = ftell(fp); //get end position of file
fseek(fp, 0, SEEK_SET); //send pointer back to start
int max_upload_size = 10; // array size, larger = less uploads but too large can cause memory issues
int num_of_uploads = file_size / max_upload_size; // figure out how many evenly sized upload chunks we need
int num_of_uploads_mod = file_size % max_upload_size; //find out size of remaining upload chunk if needed
int i;
//upload file in even chunks
if (num_of_uploads > 0)
{
char buff1[max_upload_size+1] = {}; // array to save file too. add 1 for end of array symbol '\n'
for (i = 0; i < num_of_uploads; i++)
{
fread(buff1, sizeof(buff1)-1, 1, fp); // -1 as don't want to count the '\n'
http.addHeader("file_name", "file_name"); //header to say what the file name is
int httpResponseCode = http.POST((uint8_t *)buff1, sizeof(buff1)-1); //send data. Datatype is (uint8_t *)
}
}
//upload any remaining data
if (num_of_uploads_mod > 0)
{
int remainder = file_size - num_of_uploads * max_upload_size;
uint8_t buff2[remainder] = {};
fread(buff2, sizeof *buff2, sizeof buff2 / sizeof *buff2, fp);
http.addHeader("file_name", "file_name");
http.addHeader("Content-Type", "application/octet-stream");
int httpResponseCode = http.POST(buff2, sizeof(buff2));
}
http.end(); // Close connection
delay(10 * 1000);
}
thanks so much.

Unfortunately, this isn't straight-forward. You can either use the SD component directly or mount the correct file system driver (typically FatFs for an SD card). To use fopen, you need to register the correct driver first, hints can be found here: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html#using-fatfs-with-vfs-and-sd-cards
I haven't tried this myself yet, but the magic function seems to be esp_vfs_fat_sdspi_mount that should enable the use of fopen with the SD card over SPI. The official documentation lacks a real-world example, but you might find something in this thread or via your preferred search engine. For further help, you might want to ask a question on arduino.se instead, where a number of experienced ESP32 experts are present.

I struggled myself with using SD Cards (fat32) on my ESP32.
When trying to open my file using
std::ofstream dataFile("/sdcard/data.json", std::ofstream::out);
It wouldn't create my file. a sys/stat would thus give a -1. Turned out that you really need to use the 8.3 naming convention for FAT32. There's no "MS windows-magic" That abbreviates the file to: sdcard.j~1 etc. So just use a correct file name. not "/sdcard/mylomgfilename.exotic_extention" but just a "/mpoint/12345678.123" filename like "/sdcard/data.txt".

Related

How to create a function: Get a single value from SD card file using the file name as function parameter

I appreciate any guidance or help I can get on this. I am writing a program with values for a PID stored on an SD card so I can change them on the touchscreen without the need to hook it up to my computer. I want a single function that I can call with parameters allowing me to increase or decrease the number and change the file name. The below function is what I have to change whether "pnum.txt" increases or decreases; however I cannot figure out how to make a File work as a parameter.
I have tried making " "pnum.txt" " (with quotes) as a char and as a String, and even though it prints out as it should, it doesn't work when inserted into the function. I have also tried passing the whole SD.open("pnum.txt", FILE_WRITE) and myFile = SD.open("pnum.txt", FILE_WRITE) as a File, but that does something odd - it will create the file when I open it, but it won't write to the file. I'm finding myself just trying the same things over and over, so I clearly have a lack of understanding that I'm not finding anywhere. Thank you again for any help on this!
float incDecValue(float value) {
//This is important, because the libraries are sharing pins
pinMode(XM, OUTPUT);
pinMode(YP, OUTPUT);
value;
SD.remove("pnum.txt");
myFile = SD.open("pnum.txt", FILE_WRITE);
myFile.print(value);
myFile.close();
counter = 0;
myFile = SD.open("pnum.txt");
if (myFile) {
while (myFile.available()) {
testChar[counter] = myFile.read();
counter++;
}
myFile.close();
}
float convertedValue = atof(testChar);
return convertedValue;
}
And I will call it like this.
pValue = incValue(pValue+=.5);
As not exactly knowing what you really want I did the following assumptions:
You want to save a single float value to a file called pnum.txt
You want to do something with that value
You want the processed value to write back to the file pnum.txt (overwriting the content)
Two different functions parametrized each with fileName as input and the value for write
So here a complete sequence (works as posted) you could easily implement into your code. No String class is used and its a one line file read/write only:
/* SD card pid read/write
The circuit:
SD card attached to SPI bus as follows:
** MOSI - pin 11
** MISO - pin 12
** CLK - pin 13
** CS - pin 4 */
#include <SPI.h>
#include <SD.h>
const uint8_t chipSelect = 4;
char dataChar[16] = {'\0'};
void setup() {
// Open serial communications and wait for port to open:
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect.
}
Serial.print("Initializing SD card...");
// see if the card is present and can be initialized:
if (!SD.begin(chipSelect)) {
Serial.println("Card failed or no card present");
return;
}
Serial.println("Card initialized");
}
void loop() {
float pValue = readFileValue ("pnum.txt") + 0.5;
writeFileValue ("pnum.txt", pValue);
}
float readFileValue (const char* fileName) {
/* Open the file. note that only one file can be open at a time,
so you have to close this one before opening another.*/
File dataFile = SD.open(fileName, FILE_READ);
// if the file is available, write to it:
if (dataFile) {
char c;
uint8_t i = 0;
while (dataFile.available()) {
c = dataFile.read(); // Read char by char
if (c != '\n') { // As long no line terminator
dataChar[i] = c;
i++;
}
else {
dataChar[i] = '\0'; // Terminate char array properly
dataFile.close();
break;
}
}
Serial.print("Success writing content: ");
Serial.println(dataChar);
}
else { // If the file isn't open, pop up an error:
Serial.print("Error opening requested file: ");
Serial.println(fileName);
}
float fileVal = atof(dataChar);;
return fileVal;
}
bool writeFileValue (const char* fileName, float fileVal) {
SD.remove(fileName); // Delete the existing file if existing
File dataFile = SD.open(fileName, FILE_WRITE);
// If the file opened okay, write to it
if (dataFile) {
// dtostrf(floatvar, StringLengthIncDecimalPoint, numVarsAfterDecimal, charbuf);
dtostrf(fileVal, 5, 2, dataChar); // Not really needed for this simple issue, but ..
Serial.print("Writing to file: ");
Serial.println(fileName);
dataFile.println(dataChar);
// Close the file
dataFile.close();
Serial.println("Success saving done");
return true;
} else {
// if the file didn't open, print an error:
Serial.println("Error opening file: ");
Serial.println(fileName);
return false;
}
}

How to receive a JPEG image over serial port

So I am trying to send a jpeg image (4Kb) from a raspberry pi to my Mac wirelessly using Xbee Series 1. I have an image on the raspberry pi and can read it into binary format. I've used this binary format to save it into another image file and it creates a copy of the image correctly. That tells me that I am reading it correctly. So I am trying to send that data over a serial port (to be transferred by the xbee's) to my Mac. Side note, Xbee's can only transmit I think 80 bytes of data per packet or something. I don't know how that affects what I'm doing though.
My problem is, I do not know how to read the data and properly store it into a jpeg file itself. Most of the Read() functions I have found require you to enter a length to read and I don't know how to tell how long it is since its just a serial stream coming in.
Here is my code to send the jpeg.
#include "xSerial.hpp"
#include <iostream>
#include <cstdlib>
using namespace std;
int copy_file( const char* srcfilename, const char* dstfilename );
int main(){
copy_file("tylerUseThisImage.jpeg", "copyImage.jpeg");
return 0;
}
int copy_file( const char* srcfilename, const char* dstfilename )
{
long len;
char* buf = NULL;
FILE* fp = NULL;
// Open the source file
fp = fopen( srcfilename, "rb" );
if (!fp) return 0;
// Get its length (in bytes)
if (fseek( fp, 0, SEEK_END ) != 0) // This should typically succeed
{ // (beware the 2Gb limitation, though)
fclose( fp );
return 0;
}
len = ftell( fp );
std::cout << len;
rewind( fp );
// Get a buffer big enough to hold it entirely
buf = (char*)malloc( len );
if (!buf)
{
fclose( fp );
return 0;
}
// Read the entire file into the buffer
if (!fread( buf, len, 1, fp ))
{
free( buf );
fclose( fp );
return 0;
}
fclose( fp );
// Open the destination file
fp = fopen( dstfilename, "wb" );
if (!fp)
{
free( buf );
return 0;
}
// this is where I send data in but over serial port.
//serialWrite() is just the standard write() being used
int fd;
fd = xserialOpen("/dev/ttyUSB0", 9600);
serialWrite(fd, buf, len);
//This is where the file gets copied to another file as a test
// Write the entire buffer to file
if (!fwrite( buf, len, 1, fp ))
{
free( buf );
fclose( fp );
return 0;
}
// All done -- return success
fclose( fp );
free( buf );
return 1;
}
On the receive side I know I need to open up the serial port to read and use some sort of read() but I don't know how that is done. Using a serial library it has some functions to check if serial data is available and return the number of characters available to read.
One question about the number of characters available to read, will that number grow as the serial stream comes over or will it immediately tell the entire length of the data to be read?
But finally, I know after I open the serial port, I need read the data into a buffer and then write that buffer to a file but I have not had any luck. This is what I have tried thus far.
// Loop, getting and printing characters
char temp;
bool readComplete = false;
int bytesRead = 0;
fp = fopen("copyImage11.jpeg", "rwb");
for (;;)
{
if(xserialDataAvail(fd) > 0)
{
bytesRead = serialRead(fd, buf, len);
readComplete = true;
}
if (readComplete)
{
if (!fwrite(buf, bytesRead, 1, fp))
{
free(buf);
fclose(fp);
return 0;
}
fclose(fp);
free(buf);
return 1;
}
}
I don't get errors with my code, it just doesnt create the jpeg file correctly. Maybe I'm not transmitting it right, or maybe I'm not reading/writing to file correctly. Any help would be appreciated. Thanks everyone you rock!
If you are defining your own protocol, then you need to have a method for sending the length first.
I would recommend testing your code by sending short blocks of ascii text to confirm your i/o. Once that is working you can use the ascii to set up the transfer; ie send the length, and have your receiver ready for an expected block.

How to properly delimit multiple images before sending them over a socket

let's say I need to send, for instance, five images from a client to a server over a socket and that I want to do it at once (not sending one and waiting for an ACK).
Questions:
I'd like to know if there are some best practices or guidelines for delimiting the end of each one.
What would be the safest approach for detecting the delimiters and processing each image once in the server? (In C/C++ if possible)
Thanks in advance!
Since images are binary data, it would be difficult to come up with delimiter that cannot be contained in the image. (And ultimately confusing the receiving side)
I would advice you to create a header that would be placed at the beginning of the transmission, or at the beginning of each image.
An example:
struct Header
{
uint32_t ImageLength;
// char ImageName[128];
} __attribute__(packed);
The sender should prepend this before each image and fill in the length correctly. The receiver would then know when the image ends and would expect another Header structure at that position.
The attribute(packed) is a safety, that makes sure the header will have the same alignment even if you compile server and client with different GCC versions. It's recomended in cases where structures are interpreted by different processes.
Data Stream:
Header
Image Data
Header
Image Data
Header
Image Data
...
You can use these function to send files (from client in java) to a server (in C). The idea is to send 4 bytes which indicates the file's size followed by the file content, when all files have been sent, send 4 bytes (all set to 0 zero) to indicate the end of the transfer.
// Compile with Microsoft Visual Studio 2008
// path, if not empty, must be ended with a path separator '/'
// for example: "C:/MyImages/"
int receiveFiles(SOCKET sck, const char *pathDir)
{
int fd;
long fSize=0;
char buffer[8 * 1024];
char filename[MAX_PATH];
int count=0;
// keep on receiving until we get the appropiate signal
// or the socket has an error
while (true)
{
if (recv(sck, buffer, 4, 0) != 4)
{
// socket is closed or has an error
// return what we've received so far
return count;
}
fSize = (int) ((buffer[0] & 0xff) << 24) |
(int) ((buffer[1] & 0xff) << 16) |
(int) ((buffer[2] & 0xff) << 8) |
(int) (buffer[3] & 0xff);
if (fSize == 0)
{
// received final signal
return count;
}
sprintf(filename, "%sIMAGE_%d.img", pathDir, count+1);
fd = _creat(filename, _S_IREAD | _S_IWRITE);
int iReads;
int iRet;
int iLeft=fSize;
while (iLeft > 0)
{
if (iLeft > sizeof(buffer)) iReads = sizeof(buffer);
else iReads=iLeft;
if ((iRet=recv(sck, buffer, iReads, 0)) <= 0)
{
_close(fd);
// you may delete the file or leave it to inspect
// _unlink(filename);
return count; // socket is closed or has an error
}
iLeft-=iRet;
_write(fd, buffer, iRet);
}
count++;
_close(fd);
}
}
The client part
/**
* Send a file to a connected socket.
* <p>
* First it send the file size if 4 bytes then the file's content.
* </p>
* <p>
* Note: File size is limited to a 32bit signed integer, 2GB
* </p>
*
* #param os
* OutputStream of the connected socket
* #param fileName
* The complete file's path of the image to send
* #throws Exception
* #see {#link receiveFile} for an example on how to receive the file from the other side.
*
*/
public void sendFile(OutputStream os, String fileName) throws Exception
{
// File to send
File myFile = new File(fileName);
int fSize = (int) myFile.length();
if (fSize == 0) return; // No empty files
if (fSize < myFile.length())
{
System.out.println("File is too big'");
throw new IOException("File is too big.");
}
// Send the file's size
byte[] bSize = new byte[4];
bSize[0] = (byte) ((fSize & 0xff000000) >> 24);
bSize[1] = (byte) ((fSize & 0x00ff0000) >> 16);
bSize[2] = (byte) ((fSize & 0x0000ff00) >> 8);
bSize[3] = (byte) (fSize & 0x000000ff);
// 4 bytes containing the file size
os.write(bSize, 0, 4);
// In case of memory limitations set this to false
boolean noMemoryLimitation = true;
FileInputStream fis = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(fis);
try
{
if (noMemoryLimitation)
{
// Use to send the whole file in one chunk
byte[] outBuffer = new byte[fSize];
int bRead = bis.read(outBuffer, 0, outBuffer.length);
os.write(outBuffer, 0, bRead);
}
else
{
// Use to send in a small buffer, several chunks
int bRead = 0;
byte[] outBuffer = new byte[8 * 1024];
while ((bRead = bis.read(outBuffer, 0, outBuffer.length)) > 0)
{
os.write(outBuffer, 0, bRead);
}
}
os.flush();
}
finally
{
bis.close();
}
}
To send the files from the client:
try
{
// The file name must be a fully qualified path
sendFile(mySocket.getOutputStream(), "C:/MyImages/orange.png");
sendFile(mySocket.getOutputStream(), "C:/MyImages/lemmon.png");
sendFile(mySocket.getOutputStream(), "C:/MyImages/apple.png");
sendFile(mySocket.getOutputStream(), "C:/MyImages/papaya.png");
// send the end of the transmition
byte[] buff = new byte[4];
buff[0]=0x00;
buff[1]=0x00;
buff[2]=0x00;
buff[3]=0x00;
mySocket.getOutputStream().write(buff, 0, 4);
}
catch (Exception e)
{
e.printStackTrace();
}
If you cannot easily send a header containing the length, use some likely delimiter. If the images are not compressed and consist of bitmap-stype data, maybe 0xFF/0XFFFF/0xFFFFFFF as fully-saturated luminance values are usually rare?
Use an escape-sequence to eliminate any instances of the delimiter that turn up inside your data.
This does mean iterating all the data at both ends, but depending on your data flows, and what is being done anyway, it may be a useful solution :(

Rendering files from C++ Node.js addon

I would like to render files in node.js from C++ addon.
I want to apply some file processing and render the output to the browser via node.js
Here is my C++ Code
std::ifstream in(filename, std::ios::binary);
in.seekg (0, in.end);
int length = in.tellg();
in.seekg (0, in.beg);
char * buffer = new char [length];
in.read (buffer,length);
in.close();
return buffer;
Following is the V8 code to add bindings for node.js, here buffer is the output from the above c++ code.
Local<Function> cb = Local<Function>::Cast(args[1]);
const unsigned argc = 1;
Local<Value> argv[argc] = {Local<Value>::New(String::New(buffer))};
cb->Call(Context::GetCurrent()->Global(), argc, argv);
This code works well for normal text files. I'm getting problem when reading text files which are having unicode characters.
For eg,
Original text file
test start
Billél
last
When receiving in node, I will get
test start
Bill�l
last
Similarly when reading a jpg, png files the output file is different than the original file.
Please help.
I was having problems with this as well. I found an implementation in the V8 examples from Google. The example I found that properly handles UTF8 encoded files is found here:
https://code.google.com/p/v8/source/browse/trunk/samples/shell.cc#218
I adapted the source to this:
const char* ReadFile(const char* fileName, int* fileSize)
{
// reference to c-string version of file
char *fileBuffer = 0;
// attempt to open the file
FILE* fd = fopen(fileName, "rb");
// clear file size
*fileSize = 0;
// file was valid
if(fd != 0)
{
// get size of file
fseek(fd, 0, SEEK_END);
*fileSize = ftell(fd);
rewind(fd);
// allocate file buffer for file contents
fileBuffer = (char*)malloc(*fileSize + 1);
fileBuffer[*fileSize] = 0;
// copy file contents
for (int charCount = 0; charCount < *fileSize;)
{
int charRead = static_cast<int>(fread(&fileBuffer[charCount], 1, *fileSize - charCount, fd));
charCount += charRead;
}
// close the file
fclose(fd);
}
return fileBuffer;
}
Also, make sure when you create a V8 string that you create a String::Utf8Value.
String::Utf8Value v8Utf8String(...);
Then to use the String::Utf8Value as a char* use the following function:
https://code.google.com/p/v8/source/browse/trunk/samples/shell.cc#91

Winpcap saving raw packets not from an adapter

I am trying to build an application that converts my old custom Ethernet logs (bin files) to standard winpcap style logs.
The problem is that I can't seem to find an example of how to opening a pcap_t* without using an adapter (network card). The temp.pkt has not been created.
I have looked thou the examples provided with Winpcap and all of them use a live adapter when dumping packets. This example is the closest \WpdPack\Examples-pcap\savedump\savedump.c is the closest, see example below slightly modified.
#ifdef _MSC_VER
/*
* we do not want the warnings about the old deprecated and unsecure CRT functions
* since these examples can be compiled under *nix as well
*/
#define _CRT_SECURE_NO_WARNINGS
#endif
#include "pcap.h"
int main(int argc, char **argv)
{
pcap_if_t *alldevs;
pcap_if_t *d;
int inum;
int i=0;
pcap_t *adhandle;
char errbuf[PCAP_ERRBUF_SIZE];
pcap_dumper_t *dumpfile;
/* Open the adapter */
if ((adhandle= pcap_open(??????, // name of the device
65536, // portion of the packet to capture.
// 65536 grants that the whole packet will be captured on all the MACs.
1, // promiscuous mode (nonzero means promiscuous)
1000, // read timeout
errbuf // error buffer
)) == NULL)
{
fprintf(stderr,"\nUnable to open the adapter. %s is not supported by WinPcap\n", d->name);
/* Free the device list */
pcap_freealldevs(alldevs);
return -1;
}
/* Open the dump file */
dumpfile = pcap_dump_open(adhandle, argv[1]);
if(dumpfile==NULL) {
fprintf(stderr,"\nError opening output file\n");
return -1;
}
// ---------------------------
struct pcap_pkthdr header;
header.ts.tv_sec = 1 ; /* seconds */
header.ts.tv_usec = 1; /* and microseconds */
header.caplen = 100; /* length of portion present */
header.len = 100 ; /* length this packet (off wire) */
u_char pkt_data[100];
for( int i = 0 ; i < 100 ; i++ ) {
pkt_data[i] = i ;
}
pcap_dump( (u_char *) dumpfile, &header, (u_char *) &pkt_data);
// ---------------------------
/* start the capture */
// pcap_loop(adhandle, 0, packet_handler, (unsigned char *)dumpfile);
pcap_close(adhandle);
return 0;
}
I suggest doing that using pcap_t since using WinPcap is better than writing it yourself.
The following steps is how to do it:
Use pcap_open_dead() function to create a pcap_t. Read the function description here. The linktype for Ethernet is 1.
Use pcap_dump_open() function to create a pcap_dumper_t.
Use pcap_dump() function to write the packet to the dump file.
I hope this would help you.
If all you're doing is converting your own file format to .pcap, you don't need a pcap_t*, you can just use something like:
FILE* create_pcap_file(const char *filename, int linktype)
{
struct pcap_file_header fh;
fh.magic = TCPDUMP_MAGIC;
fh.sigfigs = 0;
fh.version_major = 2;
fh.version_minor = 4;
fh.snaplen = 2<<15;
fh.thiszone = 0;
fh.linktype = linktype;
FILE *file = fopen(filename, "wb");
if(file != NULL) {
if(fwrite(&fh, sizeof(fh), 1, file) != 1) {
fclose(file);
file = NULL;
}
}
return file;
}
int write_pcap_packet(FILE* file,size_t length,const unsigned char *data,const struct timeval *tval)
{
struct pcap_pkthdr pkhdr;
pkhdr.caplen = length;
pkhdr.len = length;
pkhdr.ts = *tval;
if(fwrite(&pkhdr, sizeof(pkhdr), 1, file) != 1) {
return 1;
}
if(fwrite(data, 1, length, file) != length) {
return 2;
}
return 0;
}