What is the best way to make use of a C++ library in R, hopefully preserving the C++ data structures. I'm not at all a C++ user, so I'm not clear on the relative merits of the available approaches. The R-ext manual seems to suggest wrapping every C++ function in C. However, at least four or five other means of incorporating C++ exist.
Two ways are packages w/ similar lineage, the Rcpp (maintained by the prolific overflower Dirk Eddelbuettel) and RcppTemplate packages (both on CRAN), what are the differences between the two?
Another package, rcppbind available, on R forge that claims to take a different approach to binding C++ and R (I'm not knowledgeable to tell).
The package inline available on CRAN, claims to allow inline C/C++ I'm not sure this differs from the built in functionality, aside for allowing the code to be inline w/R.
And, finally RSwig which appears to be in the wild but it is unclear how supported it is, as the author's page hasn't been updated for years.
My question is, what are the relative merits of these different approaches. Which are the most portable and robust, which are the easiest to implement. If you were planning to distribute a package on CRAN which of the methods would you use?
First off, a disclaimer: I use Rcpp all the time. In fact, when (having been renamed by the time from Rcpp) RcppTemplate had already been orphaned and without updates for two years, I started to maintain it under its initial name of Rcpp (under which it had been contributed to RQuantLib). That was about a year ago, and I have made a couple of incremental changes that you can find documented in the ChangeLog.
Now RcppTemplate has very recently come back after a full thirty-five months without any update or fix. It contains interesting new code, but it appears that it is not backwards compatible so I won't use it where I already used Rcpp.
Rcppbind was not very actively maintained whenever I checked. Whit Armstrong also has a templated interface package called rabstraction.
Inline is something completely different: it eases the compile / link cycle by 'embedding' your program as an R character string that then gets compiled, linked, and loaded. I have talked to Oleg about having inline support Rcpp which would be nice.
Swig is interesting too. Joe Wang did great work there and wrapped all of QuantLib for R. But when I last tried it, it no longer worked due to some changes in R internals. According to someone from the Swig team, Joe may still work on it. The goal of Swig is larger libraries anyway. This project could probably do with a revival but it is not without technical challenges.
Another mention should go to RInside which works with Rcpp and lets you embed R inside of C++ applications.
So to sum it up: Rcpp works well for me, especially for small exploratory projects where you just want to add a function or two. It's focus is ease of use, and it allows you to 'hide' some of the R internals that are not always fun to work with. I know of a number of other users whom I have helped on and and off via email. So I would say go for this one.
My 'Intro to HPC with R' tutorials have some examples of Rcpp, RInside and inline.
Edit: So let's look at a concrete example (taken from the 'HPC with R Intro' slides and borrowed from Stephen Milborrow who took it from Venables and Ripley). The task is to enumerate all possible combinations of the determinant of a 2x2 matrix containing only single digits in each position. This can be done in clever vectorised ways (as we discuss in the tutorial slides) or by brute force as follows:
#include <Rcpp.h>
RcppExport SEXP dd_rcpp(SEXP v) {
SEXP rl = R_NilValue; // Use this when there is nothing to be returned.
char* exceptionMesg = NULL; // msg var in case of error
try {
RcppVector<int> vec(v); // vec parameter viewed as vector of ints
int n = vec.size(), i = 0;
if (n != 10000)
throw std::length_error("Wrong vector size");
for (int a = 0; a < 9; a++)
for (int b = 0; b < 9; b++)
for (int c = 0; c < 9; c++)
for (int d = 0; d < 9; d++)
vec(i++) = a*b - c*d;
RcppResultSet rs; // Build result set to be returned as list to R
rs.add("vec", vec); // vec as named element with name 'vec'
rl = rs.getReturnList(); // Get the list to be returned to R.
} catch(std::exception& ex) {
exceptionMesg = copyMessageToR(ex.what());
} catch(...) {
exceptionMesg = copyMessageToR("unknown reason");
}
if (exceptionMesg != NULL)
Rf_error(exceptionMesg);
return rl;
}
If you save this as, say, dd.rcpp.cpp and have Rcpp installed, then simply use
PKG_CPPFLAGS=`Rscript -e 'Rcpp:::CxxFlags()'` \
PKG_LIBS=`Rscript -e 'Rcpp:::LdFlags()'` \
R CMD SHLIB dd.rcpp.cpp
to build a shared library. We use Rscript (or r) to ask Rcpp about its header and library locations. Once built, we can load and use this from R as follows:
dyn.load("dd.rcpp.so")
dd.rcpp <- function() {
x <- integer(10000)
res <- .Call("dd_rcpp", x)
tabulate(res$vec)
}
In the same way, you can send vectors, matrics, ... of various R and C++ data types back end forth with ease. Hope this helps somewhat.
Edit 2 (some five+ years later):
So this answer just got an upvote and hence bubbled up in my queue. A lot of time has passed since I wrote it, and Rcpp has gotten a lot richer in features. So I very quickly wrote this
#include <Rcpp.h>
// [[Rcpp::export]]
Rcpp::IntegerVector dd2(Rcpp::IntegerVector vec) {
int n = vec.size(), i = 0;
if (n != 10000)
throw std::length_error("Wrong vector size");
for (int a = 0; a < 9; a++)
for (int b = 0; b < 9; b++)
for (int c = 0; c < 9; c++)
for (int d = 0; d < 9; d++)
vec(i++) = a*b - c*d;
return vec;
}
/*** R
x <- integer(10000)
tabulate( dd2(x) )
*/
which can be used as follows with the code in a file /tmp/dd.cpp
R> Rcpp::sourceCpp("/tmp/dd.cpp") # on from any other file and path
R> x <- integer(10000)
R> tabulate( dd2(x) )
[1] 87 132 105 155 93 158 91 161 72 104 45 147 41 96
[15] 72 120 36 90 32 87 67 42 26 120 41 36 27 75
[29] 20 62 16 69 19 28 49 45 12 18 11 57 14 48
[43] 10 18 7 12 6 46 23 10 4 10 4 6 3 38
[57] 2 4 2 3 2 2 1 17
R>
Some of the key differences are:
simpler build: just sourceCpp() it; even executes R test code at the end
full-fledged IntegerVector type
exception-handling wrapper automatically added by sourceCpp() code generator
Related
Hello and thanks for helping!
So we've got a list of fireworks containing 1) Number in stock 2) Number of fireworks in this package 3) Diameter which equals the noise and
4) The price.
This is the list:
25 17 10 21
10 15 10 18
5 16 10 19
10 15 12 20
15 9 11 12
10 7 28 23
8 7 16 11
10 6 16 10
25 10 18 25
25 12 18 27
10 5 40 35
60 40 5 27
5 25 30 90
50 1 60 8
Our task is to create a shopping list and buy fireworks so we get the highest possible noise. We've got a knapsack capacity of 1000€. We're also supposed to solve this without using tables (so with dynamic programming instead).
I only use one class called Package which contains the four constraints as shown above.
At first I thought it would make sense to try to write an algorithm for a normal knapsack, so just with the price and the diameter (=weight). I tested it with a different list and it worked perfectly fine. I just iterated through all packages and then again used a nested for loop to find the best constellation (exhaustive search). My next idea was to merge the number of fireworks per package with the diameter, because fireworks can only be bought in a full package.
The only thing left which I still haven't found out is what to do with the amount of packages in stock. Like, with my current algorithm it just buys all packages of a firework until the knapsack is full. But obviously that won't be correct.
void knapsack(vector<Package*> stock){
vector<int> indices, tmp_indices;
int noise,tmp_noise= 0;
int price;
for (unsigned int i = 0; i < stock.size(); i++) {
price = stock[i]->price;
noise = stock[i]->diameter*stock[i]->number_fireworks;
indices.push_back(i);
for (unsigned int j = 0; j < stock.size(); j++) {
if (i != j) {
if (price+stock[j]->price<= BUDGET) {
price+=stock[j]->price;
noise+=stock[j]->diameter*stock[j]->number_fireworks;
indices.push_back(j);
}
}
}
// After second loop we have a new possible constellation
// Check if the previous constellation had a lower value and if so, set it to the current one
if (noise > tmp_noise) {
tmp_noise = noise;
tmp_indices.clear();
// tmp save
for (auto &index : indices) {
tmp_indices.push_back(index);
}
}
price= 0;
noise = 0;
indices.clear();
}
// Best constellation found, Print the shopping list
cout << "\Stock.\tNum\Diameter.\Price\n" << endl;
for(unsigned int i = 0; i < tmp_indices.size(); i++) {
cout << stock[tmp_indices[i]]->stock<< "\t";
cout << stock[tmp_indices[i]]->number_fireworks<< "\t";
cout << stock[tmp_indices[i]]->diameter<< "\t";
cout << stock[tmp_indices[i]]->price<< "\t\n";
}
}
We've been told that we should be able to spend exactly 1000€ to get the correct constellation of fireworks. My idea was to add another for loop to iterate through the amount of available packages, but that didn't really work...
This was our first lesson and I'm a bit desperate, because we have only learned how to solve a knapsack problem with 2 constraints and by using a table R.
Edit: Since one user insisted to get a specific question, here it is: Is the idea of using another loop to include the third constraint correct or is there a better/easier way of doing it? Or is it possible that I need a completely different approach for a knapsack with 3 instead of 2 constraints?
Thanks for help in advance
I know that there was a program like this:
#include <iostream>
#include <string>
int main() {
const std::string alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
std::string temp = "1234567890";
srand(MAGICNUMBER);
for (int i = 0;; ++i) {
for (int j = 0; j < 10; ++j)
temp[j] = alphabet[rand() % alphabet.size()];
std::cout << temp << std::endl;
}
}
Basically, random 10-symbol string generator.
I also know that the 124660967-th generated string was "2lwd9JjVnE". Is there a way to find what the MAGICNUMBER is, or, at least, the next string in the sequence?
Brute-forcing would be painful, given the time it takes to generate one such sequence, but I have some info about the compiler used (if that helps?): it was 64-bit g++ 4.8 for Linux.
UPD. Finding the next item would already be very helpful; can I do that in reasonable amount of time (especially without a seed)?
Yes, given typical rand() implementations this is likely to be possible, fairly easy, even.
rand() is typically a linear congruential generator such that each internal state of the generator is formed from a simple arithmetic equation of the previous state: x1 = (a*x0 + c) % m. You'll need to know the constants a, c and m used by the particular implementation you're targeting, and the method of producing the output value from the state (usually the values are either the entire state, or the upper half of the state). It's also important that the state is typically only 32-bits. A larger state would be more difficult.
So you need to find a state for the pRNG such that the next ten states produce the particular sequence of indices that produce the 10 characters you're looking for: 2lwd9JjVnE. So assuming the entire state is output by rand(), you need to find some 32-bit number x such that:
x % 62 = 54
(x1 = (a*x + c) % m) % 62 == 11
(x2 = (a*x1 + c) % m) % 62 == 22
(x3 = (a*x2 + c) % m) % 62 == 3
(x4 = (a*x3 + c) % m) % 62 == 61
(x5 = (a*x4 + c) % m) % 62 == 35
(x6 = (a*x5 + c) % m) % 62 == 9
(x7 = (a*x6 + c) % m) % 62 == 47
(x8 = (a*x7 + c) % m) % 62 == 13
(x9 = (a*x8 + c) % m) % 62 == 30
This could be done without too much difficulty by trying all 2^32 possible state values (assuming the typical 32-bit state). However, since the constants used were probably chosen to ensure that the RNG runs through a complete 32-bit period, you can simply choose any state at all and run it until you find this sequence.
Either way, once you know the state that produces these values, you then simply have to run the generator backwards for 124660967 * 10 steps in order to find which state was used as the original seed. To do that you'll need to compute the congruence multiplicative inverse of a mod m. Alternatively you could run it forward for (period - 124660967*10) steps.
No, it's almost not possible. As #chux pointed out in their comment the exact implementation isn't specified in the c++ standard.
You'll need to check for all of the sequences that will be generated with all possible seeds. That will run in an unreasonable amount of computing time necessary.
Though if the compiler is well known, and the implementation is open source (as is in your specific case), there could be ways to find out the initial seed value, knowing the specific rand() result for a specific iteration on the call.
If you have access to the program, disassemble it to attempt to learn what the magic number was.
Otherwise the standard doesn't specify anything about storing the srand value so you're stuck with alternate approaches, such as brute-forcing all seeds, or possibly trying to store the sequence of random numbers looking for the ten in a row that generate the string you're interested in.
I need to call a lambda with every int in a range. Is there a standard algorithm that does this?
Ideally something equivalent:
for(auto i = 13; i < 42; ++i)[](int i){/*do something*/}(i);
There's nothing built-in, no.
You can do it yourself with a hand-crafted iterator and std::for_each, or use Boost's counting iterators to help you along:
#include <boost/iterator/counting_iterator.hpp>
#include <algorithm>
#include <iostream>
int main()
{
std::for_each(
boost::counting_iterator<int>(13),
boost::counting_iterator<int>(42),
[](int i){ std::cout << i << ' '; }
);
}
Output:
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
(live demo)
I don't know of anything in the standard library that can produce a range of numbers on demand like you're asking for. But this can be done a couple of different ways using Boost.
Use Boost.Range to generate an integer range and use it with the range version of for_each
boost::for_each(boost::irange(13, 42), [](int i){ std::cout << i << ' '; });
Use boost::counting_iterator from Boost.Iterator and pass those to std::for_each
std::for_each(boost::make_counting_iterator(13),
boost::make_counting_iterator(42),
[](int i){ std::cout << i << ' '; });
Live demo
As the other answers have mentioned if Boost is an option there is a better way to do this. If not, the best way is in the original question:
for(auto i = 13; i < 42; ++i)[](int i){/*do something*/}(i);
However, the future is bright, chris has mentioned proposal N4128 which is suggesting incorporating ranges into the standard alongside iterators.
Now the draft is still in an early state, so there's a lot of clarification that is needed before how this will be used is nailed down. But one of the concepts is that all the STL algorithms would be overloaded to take views which is a thin wrapper providing access to the contained elements, but also an intelligent end position.
Although a more complex example, chosen to showcase the power of view, the author's example in Motivation and Scope uses iota, which is exactly what we want:
int total = accumulate(view::iota(1) |
view::transform([](int x){return x*x;}) |
view::take(10), 0);
For our purposes we'd need to use generate_n in a for_each algorithm:
for_each(view::generate_n(28,[]{static int i = 13; return i++;}),[](int i){/*do something*/});
This would cause generate_n to be called 28 times (13 + 28 = 41), the created view would provide iteration over these numbers, feeding them into our original lambda in the for_each.
chris has suggested that instead of generate_n a modification of iota might do the trick: iota(13, 41) It is key to note that whatever is used must have an end condition, because view lazily calls the generator, until no more items are requested. So this for_each(view::iota(10), [](int i){/*do something*/}); defines an infinite loop.
I'm learning Haskell. My interest is to use it for personal computer experimentation. Right now, I'm trying to see how fast Haskell can get. Many claim parity with C(++), and if that is true, I would be very happy (I should note that I will be using Haskell whether or not it's fast, but fast is still a good thing).
My test program implements π(x) with a very simple algorithm: Primes numbers add 1 to the result. Prime numbers have no integer divisors between 1 and √x. This is not an algorithm battle, this is purely for compiler performance.
Haskell seems to be about 6x slower on my computer, which is fine (still 100x faster than pure Python), but that could be just because I'm a Haskell newbie.
Now, my question: How, without changing the algorithm, can I optimize the Haskell implementation? Is Haskell really on performance parity with C?
Here is my Haskell code:
import System.Environment
-- a simple integer square root
isqrt :: Int -> Int
isqrt = floor . sqrt . fromIntegral
-- primality test
prime :: Int -> Bool
prime x = null [x | q <- [3, 5..isqrt x], rem x q == 0]
main = do
n <- fmap (read . head) getArgs
print $ length $ filter prime (2:[3, 5..n])
Here is my C++ code:
#include <iostream>
#include <cmath>
#include <cstdlib>
using namespace std;
bool isPrime(int);
int main(int argc, char* argv[]) {
int primes = 10000, count = 0;
if (argc > 1) {
primes = atoi(argv[1]);
}
if (isPrime(2)) {
count++;
}
for (int i = 3; i <= primes; i+=2) {
if (isPrime(i)){
count++;
}
}
cout << count << endl;
return 0;
}
bool isPrime(int x){
for (int i = 2; i <= floor(sqrt(x)); i++) {
if (x % i == 0) {
return false;
}
}
return true;
}
Your Haskell version is constructing a lazy list in prime only to test if it is null. This seems to indeed be a bottle neck. The following version runs just as fast as the C++ version on my machine:
prime :: Int -> Bool
prime x = go 3
where
go q | q <= isqrt x = if rem x q == 0 then False else go (q+2)
go _ = True
3.31s when compiled with -O2 vs. 3.18s for C++ with gcc 4.8 and -O3 for n=5000000.
Of course, 'guessing' where the program is slow to optimize it is not a very good approach. Fortunately, Haskell has good profiling tools on board.
Compiling and running with
$ ghc --make primes.hs -O2 -prof -auto-all -fforce-recomp && ./primes 5000000 +RTS -p
gives
# primes.prof
Thu Feb 20 00:49 2014 Time and Allocation Profiling Report (Final)
primes +RTS -p -RTS 5000000
total time = 5.71 secs (5710 ticks # 1000 us, 1 processor)
total alloc = 259,580,976 bytes (excludes profiling overheads)
COST CENTRE MODULE %time %alloc
prime.go Main 96.4 0.0
main Main 2.0 84.6
isqrt Main 0.9 15.4
individual inherited
COST CENTRE MODULE no. entries %time %alloc %time %alloc
MAIN MAIN 45 0 0.0 0.0 100.0 100.0
main Main 91 0 2.0 84.6 100.0 100.0
prime Main 92 2500000 0.7 0.0 98.0 15.4
prime.go Main 93 326103491 96.4 0.0 97.3 15.4
isqrt Main 95 0 0.9 15.4 0.9 15.4
--- >8 ---
which clearly shows that prime is where things get hot. For more information on profiling, I'll refer you to Real World Haskell, Chap 25.
To really understand what is going on, you can look at (one of) GHC's intermediate languages Core, which will show you how the code looks like after optimization. Some good info is at the Haskell wiki. I would not recommend to do that unless necessary, but it is good to know that the possibility exists.
To your other questions:
1) How, without changing the algorithm, can I optimize the Haskell implementation?
Profile, and try to write inner loops so that they don't do any memory allocations and can be made strict by the compiler. Doing so can take some practice and experience.
2) Is Haskell really on performance parity with C?
That depends. GHC is amazing and can often optimize your program very well. If you know what you're doing you can usually get close to the performance of optimized C (100% - 200% of C's speed). That said, these optimizations are not always easy or pretty to the eye and high level Haskell can be slower. But don't forget that you're gaining amazing expressiveness and high level abstractions when using Haskell. It will usually be fast enough for all but the most performance critical applications and even then you can often get pretty close to C with some profiling and performance optimizations.
I dont think that the Haskell version (original and improved by first answer) are equivalent with the C++ version.
The reason is this:
Both only consider every second element (in the prime function), while the C++ version scans every element (only i++ in the isPrime() function.
When i fix this (change i++ to i+=2 in the isPrime() function for C++) i get down to almost 1/3 of the runtime of the optimized Haskell version (2.1s C++ vs 6s Haskell).
The output remains the same for both (of course).
Note that this is no specific opimization of the C++ version ,just an adaptation of the trick already applied in the Haskell version.
I have been trying for a while to use the PoDoFo C++ library to extract text and lines (with their respective coordinates). But I have no way to do this.
This is what I have so far:
#include <iostream>
#include <stdio.h>
#include <vector>
#include <podofo/podofo.h>
using namespace PoDoFo;
using namespace std;
int main( int argc, char* argv[] )
{
const char* filename = "hello.pdf";
PdfVecObjects *x = new PdfVecObjects();
PdfParser parser(x, filename);
parser.ParseFile("hello.pdf");
for (TIVecObjects obj = x->begin(); obj != x->end(); obj++){
PdfObject * a = x->RemoveObject(obj);
// THIS IS MY PROBLEM VVVVVVVVVV
cout << a->Reference().ToString() << endl;
}
return 0;
}
However, this only gives me incredibly basic information (seems to be object number)
DEBUG: Size=12
DEBUG: Reading numbers: 0 12
DEBUG: Reading XRef Section: 0 with 12 Objects.
DEBUG: Size=12
DEBUG: Reading numbers: 0 12
DEBUG: Reading XRef Section: 0 with 12 Objects.
1 0 R
2 0 R
3 0 R
4 0 R
5 0 R
6 0 R
7 0 R
8 0 R
9 0 R
10 0 R
11 0 R
I want to print out the coordinates of an object, and if it's a line or text. If it's text, I would also like to be able to print out the text. Does anyone that knows this library better than I do know what I could do to fix this?
This answer will show you how to extract the text.
To get text positioning information, you will also have to process the following commands:
Tc, Tw, Tz, TL, T*, Tr and Tm.
You definitely need to download the PDF spec from Adobe to get all the details. There is a chapter devoted entirely to text processing. It is well worth your time to print out that chapter as you will be referring to it a lot. Everything you need to know is in there, but it's not always obvious.
You will also need to use a bit of Linear Algebra. Nothing too complicated, though.
Since there are many ways to achieve the same results, it is important to implement all the commands thoroughly, even if the documents you are going to process might not seem to need certain features. For example: I ran across a document which set all text sizes to one point, which threw off all my calculations until I realized it was using the text scaling factor to set the actual font sizes.
Use the PoDoFo tools "podofotxtextract" it gives you x,y coordinate (tool folder of PoDoFo package). Extract text from Pdf.