Speeding up Eigen operator() calls in conjunction with operator+= - c++

Question

I am working on a code that emulates a matrix-vector product. In this code, we are explicitly calculating the vector A x, but we do not have access to the full matrix A, as dim is typically very large such that the full matrix A cannot be stored in memory.
By doing some timings, we have established that the following loop takes about 4 seconds to complete. string is a an instance of a custom class, basically a wrapper around an unsigned long, and matvec is an Eigen::VectorXd.
// Off-diagonal contributions
for (size_t I = 0; I < dim; I++) {
for (size_t p = 0; p < K; p++) {
if (string.isOccupied(p)) { // p in I
for (size_t q = 0; q < p; q++) {
if (!string.isOccupied(q)) {
string.annihilate(p);
string.create(q);
size_t J = string.address();
matvec(I) += value(p,q) * x(J);
matvec(J) += value(p,q) * x(I);
string.annihilate(q);
string.create(p);
}
} // q < p loop
}
} // p loop
string.nextPermutation();
}
In trying to determine the bottleneck of this loop, we tried a variety of things, such as optimizing the annihilate and create calls, but these have little effect. However, when we time the code without the Eigen::VectorXd value updating, i.e.
// Off-diagonal contributions
for (size_t I = 0; I < dim; I++) {
for (size_t p = 0; p < K; p++) {
if (string.isOccupied(p)) { // p in I
for (size_t q = 0; q < p; q++) {
if (!string.isOccupied(q)) {
string.annihilate(p);
string.create(q);
size_t J = string.address();
string.annihilate(q);
string.create(p);
}
} // q < p loop
}
} // p loop
string.nextPermutation();
}
the loop now only takes 0.2 seconds. From this, we have deduced that the bottleneck is actually adding the right values to the resulting vector. see the edit
Is there a faster way to do the matvec(I) += value(p,q) * x(J) calls, using the C++ Eigen library?
EDIT: I have done some better timings, inspired by #PeterT and #chtz's comments, and I find that the += statements take about 30% of all the time spent in the for loop, whereas the other parts all take about an equally proportioned size of the remaining time.
So, the question remains: how can the matvec(I) += value(p,q) * x(J) and matvec(J) += value(p,q) * x(I) be sped up?