I receive data as a vector<char>, from which I need to create a string. Vector may contain utf-16 characters (i.e. null bytes) and is a fixed size. Actual data is padded with null bytes to this fixed sized. So, for example, I can have the following vector:
\0 a \0 b \0 c \0 d \0 \0 \0 \0
Fixed size is 12 and the vector contains utf-16 string "abcd" padded with 4 null chars to size.
From this, I need to actually extract this string. I already have the code for converting from utf-16 to string, the thing where I got myself confused is find the number of characters (bytes) in the vector without the padding. In the example above, the number is 8.
I started by doing something like:
std::string CrmxFile::StringFromBytes(std::vector<char> data, int fixedsize) {
std::vector<char>iterator it = data.rbegin();
while(it != data.rend() && *it == '\0') {
it++;
}
return std::string(&data[0], fixedsize - (it - data.rbegin());
}
However in the full context, the vector contains a lot of data and I need to do the above manipulation with only a specified part of it. For example, the vector may contain 1000 elements and I need to get the string that starts at position 30 and goes for a max of 12 chars. Of course, I can create another vector and copy the required 21 characters into it before applying the above logic, but I feel that I should be able to do something directly on the given vector. Yet, I can't grasp what iterators I am comparing with what. Any help is appreciated.
Now, this is embarrassing: vector<char>::iterator is obviously a random access iterator, therefore I can decrement it. Hence my method now looks like this:
std::string CrmxFile::StringFromBytes(std::vector<char> data, int fixedsize) {
std::vector<char>::iterator begin = data.begin() + start;
std::vector<char>::iterator end = start + length - 1;
while(it >= begin && *it == '\0') {
it--;
}
if(it >= begin) {
int len = it - begin + 1;
if(IsUtf8Heuristic(begin, begin + len) {
return std::string(begin, begin + len);
}
else { //(heuristically this is utf-16)
len = ((len + 1) >> 1) << 1;
std::string res;
ConvertUtf16To8(begin, begin + len, std::back_inserter(res));
return res;
}
}
else {
return "";
}
}
As I understand the question, you want to extract a part of max fixedsize from data, and erase all trailing zeroes. And from the comments you want the optimal solution.
For me, your code is overly complicated if the data will always be in array form. Use indices, they are more self describing.
std::vector<char> data = ...;
int fixedsize = ...;
int start = ...;
int i = start + fixedsize - 1; // last character that can be in the string
while(i >= start && data[i] == 0) i--; // 'remove' the trailing zeroes
std::string result(&data[start], i - start + 1);
This is the optimal algorithm, there are no 'more optimal' algorithm (there is a micro-optimization that consists in testing with ints rather than chars, ie 4 chars in a row).
Related
Given an input string A, is there a concise way to generate a string B that is lexicographically larger than A, i.e. A < B == true?
My raw solution would be to say:
B = A;
++B.back();
but in general this won't work because:
A might be empty
The last character of A may be close to wraparound, in which case the resulting character will have a smaller value i.e. B < A.
Adding an extra character every time is wasteful and will quickly in unreasonably large strings.
So I was wondering whether there's a standard library function that can help me here, or if there's a strategy that scales nicely when I want to start from an arbitrary string.
You can duplicate A into B then look at the final character. If the final character isn't the final character in your range, then you can simply increment it by one.
Otherwise you can look at last-1, last-2, last-3. If you get to the front of the list of chars, then append to the length.
Here is my dummy solution:
std::string make_greater_string(std::string const &input)
{
std::string ret{std::numeric_limits<
std::string::value_type>::min()};
if (!input.empty())
{
if (std::numeric_limits<std::string::value_type>::max()
== input.back())
{
ret = input + ret;
}
else
{
ret = input;
++ret.back();
}
}
return ret;
}
Ideally I'd hope to avoid the explicit handling of all special cases, and use some facility that can more naturally handle them. Already looking at the answer by #JosephLarson I see that I could increment more that the last character which would improve the range achievable without adding more characters.
And here's the refinement after the suggestions in this post:
std::string make_greater_string(std::string const &input)
{
constexpr char minC = ' ', maxC = '~';
// Working with limits was a pain,
// using ASCII typical limit values instead.
std::string ret{minC};
auto rit = input.rbegin();
while (rit != input.rend())
{
if (maxC == *rit)
{
++rit;
if (rit == input.rend())
{
ret = input + ret;
break;
}
}
else
{
ret = input;
++(*(ret.rbegin() + std::distance(input.rbegin(), rit)));
break;
}
}
return ret;
}
Demo
You can copy the string and append some letters - this will produce a lexicographically larger result.
B = A + "a"
I understand how to find the size using a string type array:
char * shuffleStrings(string theStrings[])
{
int sz = 0;
while(!theStrings[sz].empty())
{
sz++;
}
sz--;
printf("sz is %d\n", sz);
char * shuffled = new char[sz];
return shuffled;
}
One of my questions in the above example also is, why do I have to decrement the size by 1 to find the true number of elements in the array?
So if the code looked like this:
char * shuffleStrings(char * theStrings[])
{
//how can I find the size??
//I tried this and got a weird continuous block of printing
int i = 0;
while(!theStrings)
{
theStrings++;
i++;
}
printf("sz is %d\n", i);
char * shuffled = new char[i];
return shuffled;
}
You should not decrement the counter to get the real size, in the fist snippet. if you have two element and one empty element, the loop will end with value , which is correct.
In the second snippet, you work on a pointer to a pointr. So the while-condition should be *theStrings (supposing that a NULL pointer ist the marker for the end of your table.
Note that in both cases, if the table would not hold the marker for the end of table, you'd risk to go out of bounds. Why not work with vector<string> ? Then you could get the size without any loop, and would not risk to go out of bounds
What you are seeing here is the "termination" character in the string or '\0'
You can see this better when you use a char* array instead of a string.
Here is an example of a size calculator that I have made.
int getSize(const char* s)
{
unsigned int i = 0;
char x = ' ';
while ((x = s[i++]) != '\0');
return i - 1;
}
As you can see, the char* is terminated with a '\0' character to indicate the end of the string. That is the character that you are counting in your algorithm and that is why you are getting the extra character.
As to your second question, seem to want to create a new array with size of all of the strings.
To do this, you could calculate the length of each string and then add them together to create a new array.
I need to compress a string. Can make an assumption that each character in the string doesn`t appear more than 255 times. I need return the compressed string and its length.
Last 2 years I worked with C# and forgot C++. I will be glad to hear your comments about code , algorithm and c++ programming practices
// StringCompressor.h
class StringCompressor
{
public:
StringCompressor();
~StringCompressor();
unsigned long Compress(string str, string* strCompressedPtr);
string DeCompress(string strCompressed);
private:
string m_StrCompressed;
static const char c_MaxLen;
};
// StringCompressor.cpp
#include "StringCompressor.h"
const char StringCompressor::c_MaxLen = 255;
StringCompressor::StringCompressor()
{
}
StringCompressor::~StringCompressor()
{
}
unsigned long StringCompressor::Compress(string str, string* strCompressedPtr)
{
if (str.empty())
{
return 0;
}
char currentChar = str[0];
char count = 1;
for (string::iterator it = str.begin() + 1; it != str.end(); ++it)
{
if (*it == currentChar)
{
count++;
if (count == c_MaxLen)
{
return -1;
}
}
else
{
m_StrCompressed+=currentChar;
m_StrCompressed+=count;
currentChar = *it;
count = 1;
}
}
m_StrCompressed += currentChar;
m_StrCompressed += count;
*strCompressedPtr = m_StrCompressed;
return m_StrCompressed.length();
}
string StringCompressor::DeCompress(string strCompressed)
{
string res;
if (strCompressed.length() % 2 != 0)
{
return res;
}
for (string::iterator it = strCompressed.begin(); it != strCompressed.end(); it+=2)
{
char dup = *(it + 1);
res += string(dup, *it);
}
return res;
}
There can be many improvement:
Do not return -1 for a unsigned long function.
consider use size_t or ssize_t to represent size.
Learn const
m_StrCompressed has bogus state if Compress is called repeatedly. Since those member cannot be reused, you may as well make the function static.
Compressed stuff generally should not be considered string, but byte buffer. Redesign your interface.
Comments! Nobody knows you are doing RLE here.
Bonus: Fallback mechanism if your compression yield larger result. e.g. a flag to denote uncompressed buffer, or just return failure.
I assume efficiency is not major concern here.
A few things:
I'm all for using classes, and perhaps you could do that here in a way that makes more sense. But given the scope of what you are trying to do, this here would be better off as two functions. One for compression, one for decompression. For instance, why are you storing the string in the class as an object and never using it? How does grouping this as a class actually enhance the functionality or make it more reusable?
You should pass your compressed string return as a reference instead of a pointer.
It looks like you are trying to count the number of times characters are repeated in a row and save that. For most common strings this will make the size of your compressed string larger than uncompressed as it takes two bytes to store each non-repeated character.
There are a lot of characters, there are two kinds of bits. If you do this method trying to group repeated bits, you'd be more successful (and that's actually one simple method of lossless compression).
If you are allowed, just use a library like zlib to do compression of arbitrary data types.
I need to write a C/C++ function that would quickly check if string ends with one of ~1000 predefined suffixes. Specifically the string is a hostname and I need to check if it belongs to one of several hundred predefined second-level domains.
This function will be called a lot so it needs to be written as efficiently as possible. Bitwise hacks etc anything goes as long as it turns out fast.
Set of suffixes is predetermined at compile-time and doesn't change.
I am thinking of either implementing a variation of Rabin-Karp or write a tool that would generate a function with nested ifs and switches that would be custom tailored to specific set of suffixes. Since the application in question is 64-bit to speed up comparisons I could store suffixes of up to 8 bytes in length as const sorted array and do binary search within it.
Are there any other reasonable options?
If the suffixes don't contain any expansions/rules (like a regex), you could build a Trie of the suffixes in reverse order, and then match the string based on that. For instance
suffixes:
foo
bar
bao
reverse order suffix trie:
o
-a-b (matches bao)
-o-f (matches foo)
r-a-b (matches bar)
These can then be used to match your string:
"mystringfoo" -> reverse -> "oofgnirtsym" -> trie match -> foo suffix
You mention that you're looking at second-level domain names only, so even without knowing the precise set of matching domains, you could extract the relevant portion of the input string.
Then simply use a hashtable. Dimension it in such a way that there are no collisions, so you don't need buckets; lookups will be exactly O(1). For small hash types (e.g. 32 bits), you'd want to check if the strings really match. For a 64-bit hash, the probability of another domain colliding with one of the hashes in your table is already so low (order 10^-17) that you can probably live with it.
I would reverse all of the suffix strings, build a prefix tree of them and then test the reverse of your IP string against that.
I think that building your own automata would be the most efficient way.. it's a sort of your second solution, according to which, starting from a finite set of suffixes, it generates an automaton fitted for that suffixes.
I think you can easily use flex to do it, taking care of reversing the input or handling in a special way the fact that you are looking just for suffixes (just for efficienty matters)..
By the way using a Rabin-Karp approach would be efficient too since your suffixes will be short. You can fit a hashset with all the suffixes needed and then
take a string
take the suffix
calculate the hash of the suffix
check if suffix is in the table
Just create a 26x26 array of set of domains. e.g. thisArray[0][0] will be the domains that end in 'aa', thisArray[0][1] is all the domains that end in 'ab' and so on...
Once you have that, just search your array for thisArray[2nd last char of hostname][last char of hostname] to get the possible domains. If there's more than one at that stage, just brute force the rest.
I think that the solution should be very different depending on the type of input strings. If the strings are some kind of string class that can be iterated from the end (such as stl strings) it is a lot easier than if they are NULL-terminated C-strings.
String Class
Iterate the string backwards (don't make a reverse copy - use some kind of backward iterator). Build a Trie where each node consists of two 64-bit words, one pattern and one bitmask. Then check 8 characters at a time in each level. The mask is used if you want to match less than 8 characters - e.g. deny "*.org" would give a mask with 32 bits set. The mask is also used as termination criteria.
C strings
Construct an NDFA that matches the strings on a single-pass over them. That way you don't have to first iterate to the end but can instead use it in one pass. An NDFA can be converted to a DFA, which will probably make the implementation more efficient. Both construction of the NDFA and conversion to DFA will probably be so complex that you will have to write tools for it.
After some research and deliberation I've decided to go with trie/finite state machine approach.
The string is parsed starting from the last character going backwards using a TRIE as long as the portion of suffix that was parsed so far can correspond to multiple suffixes. At some point we either hit the first character of one of the possible suffixes which means that we have a match, hit a dead end, which means there are no more possible matches or get into situation where there is only one suffix candidate. In this case we just do compare remainder of the suffix.
Since trie lookups are constant time, worst case complexity is o(maximum suffix length). The function turned out to be pretty fast. On 2.8Ghz Core i5 it can check 33,000,000 strings per second for 2K possible suffixes. 2K suffixes totaling 18 kilobytes, expanded to 320kb trie/state machine table. I guess that I could have stored it more efficiently but this solution seems to work good enough for the time being.
Since suffix list was so large, I didn't want to code it all by hand so I ended up writing C# application that generated C code for the suffix checking function:
public static uint GetFourBytes(string s, int index)
{
byte[] bytes = new byte[4] { 0, 0, 0, 0};
int len = Math.Min(s.Length - index, 4);
Encoding.ASCII.GetBytes(s, index, len, bytes, 0);
return BitConverter.ToUInt32(bytes, 0);
}
public static string ReverseString(string s)
{
char[] chars = s.ToCharArray();
Array.Reverse(chars);
return new string(chars);
}
static StringBuilder trieArray = new StringBuilder();
static int trieArraySize = 0;
static void Main(string[] args)
{
// read all non-empty lines from input file
var suffixes = File
.ReadAllLines(#"suffixes.txt")
.Where(l => !string.IsNullOrEmpty(l));
var reversedSuffixes = suffixes
.Select(s => ReverseString(s));
int start = CreateTrieNode(reversedSuffixes, "");
string outFName = #"checkStringSuffix.debug.h";
if (args.Length != 0 && args[0] == "--release")
{
outFName = #"checkStringSuffix.h";
}
using (StreamWriter wrt = new StreamWriter(outFName))
{
wrt.WriteLine(
"#pragma once\n\n" +
"#define TRIE_NONE -1000000\n"+
"#define TRIE_DONE -2000000\n\n"
);
wrt.WriteLine("const int trieArray[] = {{{0}\n}};", trieArray);
wrt.WriteLine(
"inline bool checkSingleSuffix(const char* str, const char* curr, const int* trie) {\n"+
" int len = trie[0];\n"+
" if (curr - str < len) return false;\n"+
" const char* cmp = (const char*)(trie + 1);\n"+
" while (len-- > 0) {\n"+
" if (*--curr != *cmp++) return false;\n"+
" }\n"+
" return true;\n"+
"}\n\n"+
"bool checkStringSuffix(const char* str, int len) {\n" +
" if (len < " + suffixes.Select(s => s.Length).Min().ToString() + ") return false;\n" +
" const char* curr = (str + len - 1);\n"+
" int currTrie = " + start.ToString() + ";\n"+
" while (curr >= str) {\n" +
" assert(*curr >= 0x20 && *curr <= 0x7f);\n" +
" currTrie = trieArray[currTrie + *curr - 0x20];\n" +
" if (currTrie < 0) {\n" +
" if (currTrie == TRIE_NONE) return false;\n" +
" if (currTrie == TRIE_DONE) return true;\n" +
" return checkSingleSuffix(str, curr, trieArray - currTrie - 1);\n" +
" }\n"+
" --curr;\n"+
" }\n" +
" return false;\n"+
"}\n"
);
}
}
private static int CreateTrieNode(IEnumerable<string> suffixes, string prefix)
{
int retVal = trieArraySize;
if (suffixes.Count() == 1)
{
string theSuffix = suffixes.Single();
trieArray.AppendFormat("\n\t/* {1} - {2} */ {0}, ", theSuffix.Length, trieArraySize, prefix);
++trieArraySize;
for (int i = 0; i < theSuffix.Length; i += 4)
{
trieArray.AppendFormat("0x{0:X}, ", GetFourBytes(theSuffix, i));
++trieArraySize;
}
retVal = -(retVal + 1);
}
else
{
var groupByFirstChar =
from s in suffixes
let first = s[0]
let remainder = s.Substring(1)
group remainder by first;
string[] trieIndexes = new string[0x60];
for (int i = 0; i < trieIndexes.Length; ++i)
{
trieIndexes[i] = "TRIE_NONE";
}
foreach (var g in groupByFirstChar)
{
if (g.Any(s => s == string.Empty))
{
trieIndexes[g.Key - 0x20] = "TRIE_DONE";
continue;
}
trieIndexes[g.Key - 0x20] = CreateTrieNode(g, g.Key + prefix).ToString();
}
trieArray.AppendFormat("\n\t/* {1} - {2} */ {0},", string.Join(", ", trieIndexes), trieArraySize, prefix);
retVal = trieArraySize;
trieArraySize += 0x60;
}
return retVal;
}
So it generates code like this:
inline bool checkSingleSuffix(const char* str, const char* curr, const int* trie) {
int len = trie[0];
if (curr - str < len) return false;
const char* cmp = (const char*)(trie + 1);
while (len-- > 0) {
if (*--curr != *cmp++) return false;
}
return true;
}
bool checkStringSuffix(const char* str, int len) {
if (len < 5) return false;
const char* curr = (str + len - 1);
int currTrie = 81921;
while (curr >= str) {
assert(*curr >= 0x20 && *curr <= 0x7f);
currTrie = trieArray[currTrie + *curr - 0x20];
if (currTrie < 0) {
if (currTrie == TRIE_NONE) return false;
if (currTrie == TRIE_DONE) return true;
return checkSingleSuffix(str, curr, trieArray - currTrie - 1);
}
--curr;
}
return false;
}
Since for my particular set of data len in checkSingleSuffix was never more than 9, I tried to replace the comparison loop with switch (len) and hardcoded comparison routines that compared up to 8 bytes of data at a time but it didn't affect overall performance at all either way.
Thanks for everyone who contributed their ideas!
Hey, if you can get a more descriptive tittle please edit it.
I'm writing a little algorithm that involves checking values in a matrix.
Let's say:
char matrix[100][100];
char *ptr = &matrix[0][0];
imagine i populate the matrix with a couple of values (5 or 6) of 1, like:
matrix[20][35]=1;
matrix[67][34]=1;
How can I know if the binary value of an interval of the matrix is zero, for example (in pseudo code)
if((the value from ptr+100 to ptr+200)==0){ ... // do something
I'm trying to pick up on c/c++ again. There should be a way of picking those one hundred bytes (which are all next to each other) and check if their value is all zeros without having to check on by one.(considering char is one byte)
You can use std::find_if.
bool not_0(char c)
{
return c != 0;
}
char *next = std::find_if(ptr + 100, ptr + 200, not_0);
if (next == ptr + 200)
// all 0's
You can also use binders to remove the free function (although I think binders are hard to read):
char *next = std::find_if(ptr + 100, ptr + 200,
std::bind2nd(std::not_equal_to<char>(), 0));
Dang, I just notice request not to do this byte by byte. find_if will still do byte by byte although it's hidden. You will have to do this 1 by 1 although using a larger type will help. Here's my final version.
template <class T>
bool all_0(const char *begin, const char *end, ssize_t cutoff = 10)
{
if (end - begin < cutoff)
{
const char *next = std::find_if(begin, end,
std::bind2nd(std::not_equal_to<char>(), 0));
return (next == end);
}
else
{
while ((begin < end) && ((reinterpret_cast<uintptr_t>(begin) % sizeof(T)) != 0))
{
if (*begin != '\0')
return false;
++begin;
}
while ((end > begin) && ((reinterpret_cast<uintptr_t>(end) % sizeof(T)) != 0))
{
--end;
if (*end != '\0')
return false;
}
const T *nbegin = reinterpret_cast<const T *>(begin);
const T *nend = reinterpret_cast<const T *>(end);
const T *next = std::find_if(nbegin, nend,
std::bind2nd(std::not_equal_to<T>(), 0));
return (next == nend);
}
}
What this does is first checks to see if the data is long enough to make it worth the more complex algorithm. I'm not 100% sure this is necessary but you can tune what is the minimum necessary.
Assuming the data is long enough it first aligns the begin and end pointers to match the alignment of the type used to do the comparisons. It then uses the new type to check the bulk of the data.
I would recommend using:
all_0<int>(); // 32 bit platforms
all_0<long>(); // 64 bit LP64 platforms (most (all?) Unix platforms)
all_0<long long>() // 64 bit LLP64 platforms (Windows)
There's no built-in language feature to do that, nor is there a standard library function to do it. memcmp() could work, but you'd need a second array of all zeroes to compare against; that array would have to be large, and you'd also eat up unnecessary memory bandwidth in doing the comparison.
Just write the function yourself, it's not that hard. If this truly is the bottleneck of your application (which you should only conclude of profiling), then rewrite that function in assembly.
you tagged this C++, so you can use a pointer as an iterator, and use an stl algorithm. std::max. Then see if the max is 0 or not.
You could cast your pointer as an int * and then check four bytes at a time rather than one.
There's no way to tell whether an array has any value other than zero other than by checking all elements one by one. But if you start with an array that you know has all zeros, then you can maintain a flag that states the array's zero state.
std::vector<int> vec(SIZE);
bool allzeroes = true;
// ...
vec[SIZE/2] = 1;
allzeroes = false;
// ...
if( allzeroes ) {
// ...
}
Reserve element 0 of your array, to be set to all zeros.
Use memcmp to compare the corresponding ranges in the two elements.