We have a Server that sends some data to a client(both written using c++/MFC) using sockets
The data is getting a little too big and I'm looking for solutions to compress it. Basically I need to compress some char Arrays
I am not very familiar with MFC. I looked at zlib but I had a hard time getting anything compiling with my VS project. I am also able to use CLI (I tried using GZipStream and got it to work on C# but when I tried it on c++ I can't manage to get the CLI arrays to play nicely with the c++ char arrays)
Is zlib a mandatory requirement of your project ?
Otherwise, some other programs have simpler header file, which are easier to work with.
For example, this one (tested with GCC and MS's VS) :
int LZ4_compress (char* source, char* dest, int isize);
int LZ4_uncompress (char* source, char* dest, int osize);
Source code : http://code.google.com/p/lz4/
Take a look at http://zlib.net/
Also, there's a guy who did a wrapper around it (for convenience), might be worth checking it out: http://www.firstobject.com/easy-zlib-c++-xml-compression.htm
You can use any 3rd party compression library.
Also this question deals with a similar problem : c++ compress byte array
What error are you getting in trying to compile zlib.
Related
I'm currently writing a library application for my Teensy using the Arduino IDE and so far, it has been straight forward enough to do. I have however come across a bit of an odd compilation error that I just can't seem to get my head around. My library includes the well known IRRemote Library in order for me to create what is almost a wrapper library in order to be more specific into the way I need to interact with the IR Remote library. The issue I'm having is with one of the IR Remote library functions:
void sendRaw(const unsigned int buf[],unsigned int len, unsigned int);
Although this may seem like a straight forward function to use, the issue i have is with the first argument to the function. I have tried all various ways to pass something to this function paramater but whatever I do, it simply will not compile. Currently i have set a variable
unsigned int bufferToSend[5] = {1,2,3,4,5}; // data is just for example purposes
when i try to call
sendRaw(bufferToSend, 5, 38); // Values are just for example purposes
i get a compiler error of
undefined reference to `IRsend::sendRaw(unsigned int const*, unsigned int, unsigned int)'
Interestingly, if i call the same function directly from inside the Arduino .ino file rather than via my library call, the code compiles and runs perfectly well. Am I missing something blatantly obvious?
With the help of 'Some programmer dude', I was able to resolve this issue. As it was quite rightfully pointed out, I wasn't actually linking against the external library from within my library. having worked with library linking previously, I know this is absolutely fundamental, however with Arduino and its nature, it compiles libraries that are linked against "on-the-go" when you include from the Arduino script and therefore i fell victim to thinking it would be the same in a custom library. I came across a very good write-up of my exact problem below that provides an excellent explanation to some of the Arduino "smoke and mirrors" magic and how to get round the issue I've faced here.
Advanced Arduino – Including Multiple Libraries In Your Project
I am doing some arduino development using cpp and h files and I am having some troubles using string with them. Currently I have
#include <string>
at the top of both the cpp and the h file. When I do that it gives me the error:
string: no such file or directory
If I go into the h file and change it to
#include <string.h>
then it gives me the error:
std::string has not been declared
Anytime I use the string I use: std::string to declare it. I am not using namespace std and these files were working together fine before I started to try to use string. I am new to C/C++ so I appreciate any help. Thanks!
In short, there is a way to use std::string with the Arduino.
TL;DR:
link to the arduino STLv1.1.2
NOTE
Please note that currently the harrdwareserialstream class provided by this STL should be considered broken (as per my testing, with version 1.6.5 of the IDE, and possibly anything after 1.0.6). therefore, you can't use
hardwareserialstream << "Hi there person number " << (int)i
and so on. it seems to no longer work due to taking a reference to the serial port it would interact with rather than a pointer - in short, continue using
Serial.print("Hi there person number");
Serial.print((int)i);
Lastly the serial classes don't know what a std::string is, so if using them, give it std::string.c_str() instead
Background
As McEricSir says in the comments, the arduino does provide its own string class, though i have found it to have problems related to memory leakage, which eventually ate all of the memory i had and the program stopped running - though this was in the arduino IDE v 1.0.5, it may have been fixed since then.
I had the same problem, and found someone who had created a version of the STL for the arduino (props to Andy Brown for this) which is a cutdown version of the SGI STL. it provides std::string, std::vector and a large amount of the STL to the arduino.
there are some things to be aware when using it though; if you have a board with very little memory, you will fill it quite quickly using the smart containers and other advanced features.
Using the Library
To use the library, you'll need to read the article, though I'll summarise the main points for you here:
Installation
simply extract the library to (assuming you are using the standard Arduino IDE) hardware\tools\avr\avr\include folder and you are good to go.
Using It
To actually use the new libraries, you need to include 2 additional things as well as the library you wanted.
firstly, you need to include the header iterator BEFORE any libraries that come from this STL - and in every file you reference the STL in.
Secondly, you also need to include the file pnew.cpp to provide an implementation of the new operator for the STL to work with.
Lastly, include any header files as you would normally.
to make use of the types gained from them, don't forget the the std:: namespace notation for them. (std::string and so on)
Bugs with it
Since Andy posted the library, there have been two bugs (that i'm aware of).
The first one Andy himself rectifies and explain in the blog post:
The compiler will spit out a typically cryptic succession of template errors, with the key error being this one:
dependent-name std::basic_string::size_type is parsed as a non-type,
but instantiation yields a type c:/program files (x86)/arduino-1.0/
hardware/tools/avr/lib/gcc/../../avr/include/string:1106: note:
say typename std::basic_string::size_type if a type is meant
Basically the STL was written a long time ago when C++ compilers were a little more forgiving around dependent types inherited from templates. These days they are rightly more strict and you are forced to explicitly say that you mean a type using the typename keyword.
Additionally, he provides the updated version for you to grab.
Lastly, there are reports in the comments about a bug in the newer versions of the IDE pertaining to the vector class where the compiler complains about the use of _M_deallocate without a prepending this->, which you can fix if you search for them inside the vector class
For your convenience
As i use this quite frequently, i've packaged up the current version, which can be found here (this includes both the fixes i have commented on)
Lastly
When using this, make sure to keep an eye on your free memory, and to that end i recommend the excellent class MemoryFree Library found here
on a side note if you #include<string> inside the header you won't need to include it in the relevant .cpp file
Background
I am currently working on a project for which I have written a DLL as an interface between a Windows driver and MATLAB. All of this is working very well, but one thing up until recently it has been lacking is documentation for certain functionality - essentially it allows command strings to be sent to an FPGA, and all of these commands need documenting.
This could be done using a PDF etc. but I wanted also a way to integrate it the documentation into the DLL itself - so functions like 'lookup command' etc. can be used. At any rate I went ahead and implemented this in a way which I am mostly satisfied.
Essentially I have a structure (see below) to which a pointer can be returned from functions so that the documentation can be accessed. The caller provides the address of a pointer to one of these which is then updated with the address of an entry in a constant global array.
typedef struct {
CONST CHAR * commandString;
ULONG commandStringLen;
CONST CHAR * documentationString;
ULONG documentationStringLen;
CONST CHAR * commandParameters;
ULONG commandParametersLen;
} COMMAND_DOCS;
#define STRING_LEN(a) a,(sizeof(a)-1)
#define NEWGROUP "\n "
#define NEWENTRY "\n "
#define NEWLINE "\n"
#define ENDTITLE "\n----------------------------------------\n"
CONST COMMAND_DOCS CommandDocs[] = {
//-----
#define COMMAND_xyz_GROUP_INDEX_START (0)
{ STRING_LEN("ABCD"),
STRING_LEN("Something Command"
ENDTITLE"Low Queue"
NEWLINE "Description:"
NEWGROUP"The .........."
NEWLINE),
STRING_LEN(NEWGROUP"Type x:"
NEWENTRY"No Payload"
NEWLINE)
},
#define COMMAND_xyz_GROUP_LENGTH (1)
//-----
... And so on
};
This results in a load of constant strings being stored in memory and an array of documentation structures which contain pointers to these constants and their lengths as well for good measure. A pointer to the required element in the array is returned as I say. The caller of the library API is then free to make copies if needed or display the strings, whatever.
This is all working nicely at the moment, except for a minor annoyance. Whenever I need to update the documentation, it required me to recompile the DLL - as clearly all of the strings are compiled into it. For me that is not an issue as I can easily compile it, but as I am working at a university developing a research platform for them to use, I want it to be as simple for people to update in the future as I move on to other work. Ideally if documentation needs updating - say new commands get added to the system, I would like the additions to be possible without having to do a recompile.
Question
So my question really is about what the best way to go about doing this is.
At the moment I am thinking along the lines of loading the documentation from a file, either at the time of loading the DLL, or when the search function is called. Currently there are #defines in the array to separate indexes (identify groups of commands), but these could be replaced by variables that are initialised by data from the file.
I could go for something like XML and parse that to fill up the structures, but part of me thinks it would be easier to understand in the outside world if it was something simpler, but then I suppose I would still need some way of identifying the boundaries between entries, etc.
Thoughts?
Note that the DLL is mostly C - all of the APIs are C interfaces, but internally it is C++ as I've been using classes for other parts. I don't mind which is used as long as it is compatible with C interfaces.
I went away from the problem for a while as it wasn't urgent, but have come back to it in the last few weeks. I've found that I need to add even more settings and configurable stuff to the DLL which makes the approach I was using before definitely not useable.
Basically as #PhilWilliams suggested in the comments, I've gone for using XML files for configuring everything. There are now some APIs that have to be called just after loading the library with which the location of the XML file to load is specified. The DLL will then parse the XML files and populate a load of internal structures. Rather than using #defines and constant strings, I now have an array of structs in which pointers to the parsed strings are located along with the indices which were previously #defines.
After looking on StackOverflow and finding various suggestions on simple XML parsers, I've gone for TinyXML2 as it means that I don't have the headache of allocating and freeing memory for the many documentation strings as it handles that internally.
I need to do a prototype that involves some serialization in C++. It is a quick'n'dirty prototype, so I don't need to solve the problem generally, provide good error checking, or anything like that. But at the same time, I do need to be able to serialize strings of arbitrary length and with arbitrary charcters.
Are there some best practices for how to whip up a quick data serialization in C++? Normally I'd just have output records into a text file with one record per line, but my strings may have new lines in them.
You could consider using JSON, notably thru JsonCpp. You could also use libs11n, a full fledged, template friendly, C++ serialization framework.
(If you want a C library for Json, consider jansson).
You might also consider using old XDR or ASN1 technology.
For a quick & dirty prototype, I do recommend the JsonCpp library mentioned above. And using JSON in that case is useful, since it is a textual, nearly-human-friendly, format.
Later you could even perhaps consider going to MongoDb which has a Json-like model.
Checkout serialization with boost:
http://www.boost.org/doc/libs/1_51_0/libs/serialization/doc/index.html
Not dirty at all but definitely quick.
If you do not mind binary data, for each string dump a length (cast to a char*) and then the value of the string to file. It is very easy to read back. POD structs can also be dumped directly by casting to a char*
I have not worked with static libraries before, but now I need to.
Scenario:
I am writing a console app in Unix. I freely use std::string everywhere because it's easy to do so. However, I recently found out that I have to support it in Windows and a third party application would need API's to my code (I will not be sharing source, just the DLL).
With this in mind, can I still use std::string everywhere in my code but then provide them with char * when I code the API's? Would that work?
Yep. Use std::string internally and then just use const char * on the interface functions (which will be converted to std::strings on input.
Why not just provide them with std::string?
It's standard C++, and I'd be very suprised if they didn't support it.
The question is, what your clients will do with that pointer. It should of course be const char*, but if clients will keep and reference it later on, its probably risky to use std::string internally, because as soon as you operate yourself on the strings there is no way to keep std::string from moving memory, as its reference counting mechanism can not work with exported char* pointers. As long as you dont touch the std::string objects, their memory wont move, and the pointer is safe.
There is no standardized C++ binary interface (at least I haven;t heard about it), thus projects with different settings may appear to be unlinkable together. For example, Visual C++ provides a way to enable/disable iterator debug support. This is controlled by macro and size of some data structures depends on it.
If two codes compiled with different settings start to communicate using these data structures, the best thing you can have is linker error. Other alternatives are worse - stable run-time error, release-configuration-only error, etc...
So if you don't want to restrict your users to single correct project settings set and compiler version, use only primitive data for interface. For internal implementation choose what is more convenient.
Adding to Poita_'s response:
consider unicode support
If you ever have to support localization, too, you'll be happy to have done it in the first place
when returning char/wchar_t const *, define the lifetime of the data. The best would be to have a project-wide "unless stated otherwise..." standard
Alternatively, you can return a copy that must be freed through a method exported by your library. (C++ clients can move that into a smart pointer to regain automatic memory management.)
std::string will work in, at the very least, Visual Studio C++ (and others), so why not just use that?