AABB Collision response(Push back colliding objects) - c++

I'm having problem with collision detection with my C++ game. I first tried AABB and successfully got it to detect collision but the problem was that I want to be able to push each other so I went with a basic solution to push both objects back half of the distance penetrated. But that only pushed the objects to a 45 degree angle to each other.Link to GIF explaining what i mean
IF YOU DON'T WANT TO READ SKIP THIS PARAGRAPH
So I figured that I need an direction to push the objects in, so I tried to modify some code from a tutorial. The direction is a normalized vector between the center of both objects. And then half of the collision depth is multiplied with the normalized vector so they get pushed in the opposite direction of each other. This worked but it used circular collision(it used radius + radius to calculate collision depth) and this meant that I couldn't use rectangles with different width and height, for instance 32x64 48x32(I had to use 32x32 48x48 64x64 and so on and so forth). This is a big problem because I NEED TO BE ABLE TO USE TEXTURES WITH DIFFERENT WIDTH AND HEIGHT.
So once again a went back to AABB and got it to work with out being able to push each other, so you just come to a complete stop. But the problem here is that when I collide on one axis I can't move in the other. For example, if I hold down UP and LEFT and collide with a wall on the LEFT i come to a complete stop instead of continue going UP where there is no wall.
SO WHAT I WANT HELP WITH IS either AABB collision where objects push each other with half of the distance penetrated in the correct direction or just plain AABB where you come to a complete stop BUT are able to walk in the non colliding direction when colliding on the other axis, for example Holding down up and down would result in this

Here's something to think about AABB collision checking:
Let's say you have two box objects: O1(x,y,width,height) and O2(x,y,width,height), where 'x' and 'y' are the bottom-left corner of the box object.
Let's say you move O1 with a step of S(x,y)
In order to avoid a full stop on collision, you should divide your collision check in two - once a collision for horizontal, and once for a vertical.
So, get the origin differences after moving: diff = vec2(O1.x + S.x - O2.x, O1.y + S.y - O2.y)
Before going any further, you should check if the two objects never overlap:
if( (diff.x < -O1.width || diff.x > O2.width) && (diff.y < -O1.height || diff.y > O2.height) return;
If they never overlap, there's no need to constrain the movement and we just skip the collision.
Then we start constraining our horizontal movement:
If diff.x is smaller than -O1.width, then O1 is on the far left. It can move freely on the horizontal. So S.x remains as is.
Else if diff.x is bigger than O2.width, then O1 is on the far right, and can move freely on the horizontal. So S.x remains as is.
Otherwise, we have a situation where O1 overlaps O2 on the horizontal. So just check if(diff.x<0) S.x -= O1.width+diff.x; and, respectively if(diff.x>=0) S.x += O2.width-diff.x;
Your horizontal movement should be ready in S.x. Now, following the same principle, take care of your vertical movement:
If diff.y is smaller than -O1.height, then O1 is far below. It can move freely on the vertical. So S.y remains as is.
Else if diff.y is bigger than O2.height, then O1 is far up, and can move freely on the vertical. So S.y remains as is.
Otherwise, we have a situation where O1 overlaps O2 on the vertical. So just check if(diff.y<0) S.y -= O1.height+diff.y; and, respectively if(diff.y>=0) S.y += O2.height-diff.y;
In essence, at the end you have a S(x,y) which contains your new 'safe' moving step.
All you have to do, is to move O1 by S, and then repeat the process on next step.

Related

Block movement between rectangles c++ (collison done)

I have two rectangles, one of them stationary. Whenever I try to move one towards another I want it to 'slide' rather than getting stuck without being able to move. I use oriented bounding boxes and the code is fairly tested, but I don't know how to avoid them getting stuck.
Currently what I do is this:
o1 = o;
vector2df vxy = o.V;
o1.Pos += vxy;
if(o1.overlaps(o2)){
o1 = o;
o1 += vector2df(0.f, vxy.y);
if(o1.overlaps(o2)){
vxy.y = 0;
}
o1 = o;
o1 += vector2df(vxy.x, 0.f);
if(o1.overlaps(o2)){
vxy.x = 0;
}
}
o1.V = vxy
Which is quite 'dirty' as it just checks if the movement can be done in either X and Y, disabling it if it collides individually.
The velocity is a float, so later it will just add it.
I had thought doing a while loop until it doesn't collide, leaving the smaller value in which it won't collide, but it either doesn't work or is stuck in an infinite loop...
I'm guessing the first thing to do would be to rewind your moving box's movement so that it doesn't collide anymore with your fixed box. Right now your boxes are probably staying in a colliding state, hence the infinite loop.
Then as for determining the new velocity of your moving box, you probably want to get the direction of the edge from your fixed box your moving box is colliding with and set that as your new velocity.
You also might want at some point to multiply this new velocity by the dot product of your initial velocity and your new direction, so that when the moving box goes perpendicular to the fixed box's edge, it basically stops, but if it's going at a sharp angle it doesn't loose much speed.
All of this describes a vastly oversimplified approach, you might want to read more about basic game engines/physics/collision detection if you want a cleaner result.
Hope this helps.

'Ray' creation for raypicking not fully working

I'm trying to implement a 'raypicker' for selecting objects within my project. I do not fully understand how to implement this, but I understand conceptually how it should work. I've been trying to learn how to do this, but most tutorials I find go way over my head. My current code is based on one of the recent tutorials I found, here.
After several hours of revisions, I believe the problem I'm having with my raypicker is actually the creation of the ray in the first place. If I substitute/hardcode my near/far planes with a coordinate that would undisputably be located within the region of a triangle, the picker identifies it correctly.
My problem is this: my ray creation doesn't seem to fully take my current "camera" or perspective into account, so camera rotation won't affect where my mouse is.
I believe to remedy this I need something like using gluUnProject() or something, but whenever I used this the x,y,z coordinates returned would be incredibly small,
My current ray creation is a mess. I tried to use methods that others proposed initially, but it seemed like whatever method I tried it never worked with my picker/intersection function.
Here's the code for my ray creation:
void oglWidget::mousePressEvent(QMouseEvent *event)
{
QVector3D nearP = QVector3D(event->x()+camX, -event->y()-camY, -1.0);
QVector3D farP = QVector3D(event->x()+camX, -event->y()-camY, 1.0);
int i = -1;
for (int x = 0; x < tileCount; x++)
{
bool rayInter = intersect(nearP, farP, tiles[x]->vertices);
if (rayInter == true)
i = x;
}
if (i != -1)
{
tiles[i]->showSelection();
}
else
{
for (int x = 0; x < tileCount; x++)
tiles[x]->hideSelection();
}
//tiles[0]->showSelection();
}
To repeat, I used to load up the viewport, model & projection matrices, and unproject the mouse coordinates, but within a 1920x1080 window, all I get is values in the range of -2 to 2 for x y & z for each mouse event, which is why I'm trying this method, but this method doesn't work with camera rotation and zoom.
I don't want to do pixel color picking, because who knows I may need this technique later on, and I'd rather not give up after the amount of effort I put in so far
As you seem to have problems constructing your rays, here's how I would do it. This has not been tested directly. You could do it like this, making sure that all vectors are in the same space. If you use multiple model matrices (or stacks thereof) the calculation needs to be repeated separately with each of them.
use pos = gluUnproject(winx, winy, near, ...) to get the position of the mouse coordinate on the near plane in model space; near being the value given to glFrustum() or gluPerspective()
origin of the ray is the camera position in model space: rayorig = inv(modelmat) * camera_in_worldspace
the direction of the ray is the normalized vector from the position from 1. to the ray origin: raydir = normalize(pos - rayorig)
On the website linked they use two points for the ray and they don't seem to normalize the ray direction vector, so this is optional.
Ok, so this is the beginning of my trail of breadcrumbs.
I was somehow having issues with the QT datatypes for the matrices, and the logic pertaining to matrix transformations.
This particular problem in this question resulted from not actually performing any transformations whatsoever.
Steps to solving this problem were:
Converting mouse coordinates into NDC space (within the range of -1 to 1: x/screen width * 2 - 1, y - height / height * 2 - 1)
grabbing the 4x4 matrix for my view matrix (can be the one used when rendering, or re calculated)
In a new vector, have it equal the inverse view matrix multiplied by the inverse projection matrix.
In order to build the ray, I had to do the following:
Take the previously calculated value for the matrices that were multiplied together. This will be multiplied by a vector 4 (array of 4 spots), where it will hold the previously calculated x and y coordinates, as well as -1, then +1.
Then this vector will be divided by the last spot value of the entire vector
Create another vector 4 which was just like the last, but instead of -1, put "1" .
Once again divide that by its last spot value.
Now the coordinates for the ray have been created at the far and near planes, so it can intersect with anything along it in the scene.
I opened a series of questions (because of great uncertainty with my series of problems), so parts of my problem overlap in them too.
In here, I learned that I needed to take the screen height into consideration for switching the origin of the y axis for a Cartesian system, since windows has the y axis start at the top left. Additionally, retrieval of matrices was redundant, but also wrong since they were never declared "properly".
In here, I learned that unProject wasn't working because I was trying to pull the model and view matrices using OpenGL functions, but I never actually set them in the first place, because I built the matrices by hand. I solved that problem in 2 fold: I did the math manually, and I made all the matrices of the same data type (they were mixed data types earlier, leading to issues as well).
And lastly, in here, I learned that my order of operations was slightly off (need to multiply matrices by a vector, not the reverse), that my near plane needs to be -1, not 0, and that the last value of the vector which would be multiplied with the matrices (value "w") needed to be 1.
Credits goes to those individuals who helped me solve these problems:
srobins of facepunch, in this thread
derhass from here, in this question, and this discussion
Take a look at
http://www.realtimerendering.com/intersections.html
Lot of help in determining intersections between various kinds of geometry
http://geomalgorithms.com/code.html also has some c++ functions one of them serves your purpose

Inbetweening a rotation

I have an image (let's say it's a simple rectangle) positioned on the left of my screen, which I can move up and down. When moving it upwards, I use some simple trigonometry to rotate it so that the rectangle "points" towards the upper right corner of the screen. When moving downwards, it points towards the lower left corner of the screen.
Given that my application uses the following coordinate system:
I use the following code to achieve the rotation:
// moving upwards
rotation = -atan2(position.y , res.x - position.x));
// moving downwards
rotation = atan2(res.y - position.y , res.x - position.x));
where res is the reference point and position is the position (upper left corner) of our rectangle image. (For information on atan2(): atan2() on cplusplus.com).
This works just fine: it rotates more when farther away from the reference point (res). However, let's say the image is all the way at the bottom of the screen. If we move it upwards, it will very suddenly rotate. I would like to 'inbetween' this rotation, so that it is smoothened out.
What I mean by suddenly rotating is this:
Let's say the rectangle is not moving in frame n: therefore its rotation is 0 degrees. I then press the up arrow, which makes it calculate the angle. In frame n+1, the angle is 30 degrees (for example). This is ofcourse not very smooth.
Is my question clear? How do I go about this?
You can incrementally change the angle on each frame. For a very "smooth" rotation effect, you can use
target_angle = ...
current_angle += (target_angle - current_angle) * smoothing_factor
where smoothing_factor gives the rate at which current_angle should converge to target_angle. For example, a value of 1 would be instantaneous, a value of 0.1 would probably give a smooth effect.
By doing this you may encounter the wraparound issue whereby something like going from 10 degrees to 350 degrees would go the wrong way. In such a case, use
target_angle = ...
current_angle += diff(target_angle, current_angle) * smoothing_factor
where
diff(a, b) {
return atan2(sin(a - b), cos(a - b))
}
This nice angle difference formula is taken from another question.

Getting the velocity vector from position vectors

I looked at a bunch of similar questions, and I cannot seem to find one that particularly answers my question. I am coding a simple 3d game, and I am trying to allow the player to pick up and move entities around my map. I essentially want to get a velocity vector that will "push" the physics object a distance from the player's eyes, wherever they are looking. Here's an example of this being done in another game (the player is holding a chair entity in front of his eyes).
To do this, I find out the player's eye angles, then get the forward vector from the angles, then calculate the velocity of the object. Here is my working code:
void Player::PickupOtherEntity( Entity& HoldingEntity )
{
QAngle eyeAngles = this->GetPlayerEyeAngles();
Vector3 vecPos = this->GetEyePosition();
Vector3 vecDir = eyeAngles.Forward();
Vector3 holdingEntPos = HoldingEntity.GetLocation();
// update object by holding it a distance away
vecPos.x += vecDir.x * DISTANCE_TO_HOLD;
vecPos.y += vecDir.y * DISTANCE_TO_HOLD;
vecPos.z += vecDir.z * DISTANCE_TO_HOLD;
Vector3 vecVel = vecPos - holdingEntPos;
vecVel = vecVel.Scale(OBJECT_SPEED_TO_MOVE);
// set the entity's velocity as to "push" it to be in front of the player's eyes
// at a distance of DISTANCE_TO_HOLD away
HoldingEntity.SetVelocity(vecVel);
}
All that is great, but I want to convert my math so that I can apply an impulse. Instead of setting a completely new velocity to the object, I want to "add" some velocity to its existing velocity. So supposing I have its current velocity, what kind of math do I need to "add" velocity? This is essentially a game physics question. Thank you!
A very simple implementation could be like this:
velocity(t+delta) = velocity(t) + delta * acceleration(t)
acceleration(t) = force(t) / mass of the object
velocity, acceleration and force are vectors. t, delta and mass scalars.
This only works reasonably well for small and equally spaced deltas. What you are essentially trying to achieve with this is a simulation of bodies using classical mechanics.
An Impulse is technically F∆t for a constant F. Here we might want to assume a∆t instead because mass is irrelevant. If you want to animate an impulse you have to decide what the change in velocity should be and how long it needs to take. It gets complicated real fast.
To be honest an impulse isn't the correct thing to do. Instead it would be preferable to set a constant pick_up_velocity (people don't tend to pick things up using an impulse), and refresh the position each time the object rises up velocity.y, until it reaches the correct level:
while(entPos.y < holdingEntPos.y)
{
entPos.y += pickupVel.y;
//some sort of short delay
}
And as for floating in front of the player's eyes, set an EyeMovementEvent of some sort that also sends the correct change in position to any entity the player is holding.
And if I missed something and that's what you are already doing, remember that when humans apply an impulse, it is generally really high acceleration for a really short time, much less than a frame. You wouldn't see it in-game anyways.
basic Newtonian/D'Alembert physics dictate:
derivate(position)=velocity
derivate(velocity)=acceleration
and also backwards:
integrate(acceleration)=velocity
integrate(velocity)=position
so for your engine you can use:
rectangle summation instead of integration (numerical solution of integral). Define time constant dt [seconds] which is the interval between updates (timer or 1/fps). So the update code (must be periodically called every dt:
vx+=ax*dt;
vy+=ay*dt;
vz+=az*dt;
x+=vx*dt;
y+=vy*dt;
z+=vz*dt;
where:
a{x,y,z} [m/s^2] is actual acceleration (in your case direction vector scaled to a=Force/mass)
v{x,y,z} [m/s] is actual velocity
x,y,z [m] is actual position
These values have to be initialized a,v to zero and x,y,z to init position
all objects/players... have their own variables
full stop is done by v=0; a=0;
driving of objects is done only by change of a
in case of collision mirror v vector by collision normal
and maybe multiply by some k<1.0 (0.95 for example) to account energy loss on impact
You can add gravity or any other force field by adding g vector:
vx+=ax*dt+gx*dt;
vy+=ay*dt+gy*dt;
vz+=az*dt+gz*dt;
also You can add friction and anything else you need
PS. the same goes for angles just use angle/omega/epsilon/I instead of x/a/v/m
to be clear by angles I mean rotation (pitch,yaw,roll) around mass center

Implementing a complex rotation-based camera

I am implementing a 3D engine for spatial visualisation, and am writing a camera with the following navigation features:
Rotate the camera (ie, analogous to rotating your head)
Rotate around an arbitrary 3D point (a point in space, which is probably not in the center of the screen; the camera needs to rotate around this keeping the same relative look direction, ie the look direction changes too. This does not look directly at the chosen rotation point)
Pan in the camera's plane (so move up/down or left/right in the plane orthogonal to the camera's look vector)
The camera is not supposed to roll - that is, 'up' remains up. Because of this I represent the camera with a location and two angles, rotations around the X and Y axes (Z would be roll.) The view matrix is then recalculated using the camera location and these two angles. This works great for pan and rotating the eye, but not for rotating around an arbitrary point. Instead I get the following behaviour:
The eye itself apparently moving further up or down than it should
The eye not moving up or down at all when m_dRotationX is 0 or pi. (Gimbal lock? How can I avoid this?)
The eye's rotation being inverted (changing the rotation makes it look further up when it should look further down, down when it should look further up) when m_dRotationX is between pi and 2pi.
(a) What is causing this 'drift' in rotation?
This may be gimbal lock. If so, the standard answer to this is 'use quaternions to represent rotation', said many times here on SO (1, 2, 3 for example), but unfortunately without concrete details (example. This is the best answer I've found so far; it's rare.) I've struggled to implemented a camera using quaternions combining the above two types of rotations. I am, in fact, building a quaternion using the two rotations, but a commenter below said there was no reason - it's fine to immediately build the matrix.
This occurs when changing the X and Y rotations (which represent the camera look direction) when rotating around a point, but does not occur simply when directly changing the rotations, i.e. rotating the camera around itself. To me, this doesn't make sense. It's the same values.
(b) Would a different approach (quaternions, for example) be better for this camera? If so, how do I implement all three camera navigation features above?
If a different approach would be better, then please consider providing a concrete implemented example of that approach. (I am using DirectX9 and C++, and the D3DX* library the SDK provides.) In this second case, I will add and award a bounty in a couple of days when I can add one to the question. This might sound like I'm jumping the gun, but I'm low on time and need to implement or solve this quickly (this is a commercial project with a tight deadline.) A detailed answer will also improve the SO archives, because most camera answers I've read so far are light on code.
Thanks for your help :)
Some clarifications
Thanks for the comments and answer so far! I'll try to clarify a few things about the problem:
The view matrix is recalculated from the camera position and the two angles whenever one of those things changes. The matrix itself is never accumulated (i.e. updated) - it is recalculated afresh. However, the camera position and the two angle variables are accumulated (whenever the mouse moves, for example, one or both of the angles will have a small amount added or subtracted, based on the number of pixels the mouse moved up-down and/or left-right onscreen.)
Commenter JCooper states I'm suffering from gimbal lock, and I need to:
add another rotation onto your transform that rotates the eyePos to be
completely in the y-z plane before you apply the transformation, and
then another rotation that moves it back afterward. Rotate around the
y axis by the following angle immediately before and after applying
the yaw-pitch-roll matrix (one of the angles will need to be negated;
trying it out is the fastest way to decide which).
double fixAngle = atan2(oEyeTranslated.z,oEyeTranslated.x);
Unfortunately, when implementing this as described, my eye shoots off above the scene at a very fast rate due to one of the rotations. I'm sure my code is simply a bad implementation of this description, but I still need something more concrete. In general, I find unspecific text descriptions of algorithms are less useful than commented, explained implementations. I am adding a bounty for a concrete, working example that integrates with the code below (i.e. with the other navigation methods, too.) This is because I would like to understand the solution, as well as have something that works, and because I need to implement something that works quickly since I am on a tight deadline.
Please, if you answer with a text description of the algorithm, make sure it is detailed enough to implement ('Rotate around Y, then transform, then rotate back' may make sense to you but lacks the details to know what you mean. Good answers are clear, signposted, will allow others to understand even with a different basis, are 'solid weatherproof information boards.')
In turn, I have tried to be clear describing the problem, and if I can make it clearer please let me know.
My current code
To implement the above three navigation features, in a mouse move event moving based on the pixels the cursor has moved:
// Adjust this to change rotation speed when dragging (units are radians per pixel mouse moves)
// This is both rotating the eye, and rotating around a point
static const double dRotatePixelScale = 0.001;
// Adjust this to change pan speed (units are meters per pixel mouse moves)
static const double dPanPixelScale = 0.15;
switch (m_eCurrentNavigation) {
case ENavigation::eRotatePoint: {
// Rotating around m_oRotateAroundPos
const double dX = (double)(m_oLastMousePos.x - roMousePos.x) * dRotatePixelScale * D3DX_PI;
const double dY = (double)(m_oLastMousePos.y - roMousePos.y) * dRotatePixelScale * D3DX_PI;
// To rotate around the point, translate so the point is at (0,0,0) (this makes the point
// the origin so the eye rotates around the origin), rotate, translate back
// However, the camera is represented as an eye plus two (X and Y) rotation angles
// This needs to keep the same relative rotation.
// Rotate the eye around the point
const D3DXVECTOR3 oEyeTranslated = m_oEyePos - m_oRotateAroundPos;
D3DXMATRIX oRotationMatrix;
D3DXMatrixRotationYawPitchRoll(&oRotationMatrix, dX, dY, 0.0);
D3DXVECTOR4 oEyeRotated;
D3DXVec3Transform(&oEyeRotated, &oEyeTranslated, &oRotationMatrix);
m_oEyePos = D3DXVECTOR3(oEyeRotated.x, oEyeRotated.y, oEyeRotated.z) + m_oRotateAroundPos;
// Increment rotation to keep the same relative look angles
RotateXAxis(dX);
RotateYAxis(dY);
break;
}
case ENavigation::ePanPlane: {
const double dX = (double)(m_oLastMousePos.x - roMousePos.x) * dPanPixelScale;
const double dY = (double)(m_oLastMousePos.y - roMousePos.y) * dPanPixelScale;
m_oEyePos += GetXAxis() * dX; // GetX/YAxis reads from the view matrix, so increments correctly
m_oEyePos += GetYAxis() * -dY; // Inverted compared to screen coords
break;
}
case ENavigation::eRotateEye: {
// Rotate in radians around local (camera not scene space) X and Y axes
const double dX = (double)(m_oLastMousePos.x - roMousePos.x) * dRotatePixelScale * D3DX_PI;
const double dY = (double)(m_oLastMousePos.y - roMousePos.y) * dRotatePixelScale * D3DX_PI;
RotateXAxis(dX);
RotateYAxis(dY);
break;
}
The RotateXAxis and RotateYAxis methods are very simple:
void Camera::RotateXAxis(const double dRadians) {
m_dRotationX += dRadians;
m_dRotationX = fmod(m_dRotationX, 2 * D3DX_PI); // Keep in valid circular range
}
void Camera::RotateYAxis(const double dRadians) {
m_dRotationY += dRadians;
// Limit it so you don't rotate around when looking up and down
m_dRotationY = std::min(m_dRotationY, D3DX_PI * 0.49); // Almost fully up
m_dRotationY = std::max(m_dRotationY, D3DX_PI * -0.49); // Almost fully down
}
And to generate the view matrix from this:
void Camera::UpdateView() const {
const D3DXVECTOR3 oEyePos(GetEyePos());
const D3DXVECTOR3 oUpVector(0.0f, 1.0f, 0.0f); // Keep up "up", always.
// Generate a rotation matrix via a quaternion
D3DXQUATERNION oRotationQuat;
D3DXQuaternionRotationYawPitchRoll(&oRotationQuat, m_dRotationX, m_dRotationY, 0.0);
D3DXMATRIX oRotationMatrix;
D3DXMatrixRotationQuaternion(&oRotationMatrix, &oRotationQuat);
// Generate view matrix by looking at a point 1 unit ahead of the eye (transformed by the above
// rotation)
D3DXVECTOR3 oForward(0.0, 0.0, 1.0);
D3DXVECTOR4 oForward4;
D3DXVec3Transform(&oForward4, &oForward, &oRotationMatrix);
D3DXVECTOR3 oTarget = oEyePos + D3DXVECTOR3(oForward4.x, oForward4.y, oForward4.z); // eye pos + look vector = look target position
D3DXMatrixLookAtLH(&m_oViewMatrix, &oEyePos, &oTarget, &oUpVector);
}
It seems to me that "Roll" shouldn't be possible given the way you form your view matrix. Regardless of all the other code (some of which does look a little funny), the call D3DXMatrixLookAtLH(&m_oViewMatrix, &oEyePos, &oTarget, &oUpVector); should create a matrix without roll when given [0,1,0] as an 'Up' vector unless oTarget-oEyePos happens to be parallel to the up vector. This doesn't seem to be the case since you're restricting m_dRotationY to be within (-.49pi,+.49pi).
Perhaps you can clarify how you know that 'roll' is happening. Do you have a ground plane and the horizon line of that ground plane is departing from horizontal?
As an aside, in UpdateView, the D3DXQuaternionRotationYawPitchRoll seems completely unnecessary since you immediately turn around and change it into a matrix. Just use D3DXMatrixRotationYawPitchRoll as you did in the mouse event. Quaternions are used in cameras because they're a convenient way to accumulate rotations happening in eye coordinates. Since you're only using two axes of rotation in a strict order, your way of accumulating angles should be fine. The vector transformation of (0,0,1) isn't really necessary either. The oRotationMatrix should already have those values in the (_31,_32,_33) entries.
Update
Given that it's not roll, here's the problem: you create a rotation matrix to move the eye in world coordinates, but you want the pitch to happen in camera coordinates. Since roll isn't allowed and yaw is performed last, yaw is always the same in both the world and camera frames of reference. Consider the images below:
Your code works fine for local pitch and yaw because those are accomplished in camera coordinates.
But when you rotate around a reference point, you are creating a rotation matrix that is in world coordinates and using that to rotate the camera center. This works okay if the camera's coordinate system happens to line up with the world's. However, if you don't check to see if you're up against the pitch limit before you rotate the camera position, you will get crazy behavior when you hit that limit. The camera will suddenly start to skate around the world--still 'rotating' around the reference point, but no longer changing orientation.
If the camera's axes don't line up with the world's, strange things will happen. In the extreme case, the camera won't move at all because you're trying to make it roll.
The above is what would normally happen, but since you handle the camera orientation separately, the camera doesn't actually roll.
Instead, it stays upright, but you get strange translation going on.
One way to handle this would be to (1)always put the camera into a canonical position and orientation relative to the reference point, (2)make your rotation, and then (3)put it back when you're done (e.g., similar to the way that you translate the reference point to the origin, apply the Yaw-Pitch rotation, and then translate back). Thinking more about it, however, this probably isn't the best way to go.
Update 2
I think that Generic Human's answer is probably the best. The question remains as to how much pitch should be applied if the rotation is off-axis, but for now, we'll ignore that. Maybe it'll give you acceptable results.
The essence of the answer is this: Before mouse movement, your camera is at c1 = m_oEyePos and being oriented by M1 = D3DXMatrixRotationYawPitchRoll(&M_1,m_dRotationX,m_dRotationY,0). Consider the reference point a = m_oRotateAroundPos. From the point of view of the camera, this point is a'=M1(a-c1).
You want to change the orientation of the camera to M2 = D3DXMatrixRotationYawPitchRoll(&M_2,m_dRotationX+dX,m_dRotationY+dY,0). [Important: Since you won't allow m_dRotationY to fall outside of a specific range, you should make sure that dY doesn't violate that constraint.] As the camera changes orientation, you also want its position to rotate around a to a new point c2. This means that a won't change from the perspective of the camera. I.e., M1(a-c1)==M2(a-c2).
So we solve for c2 (remember that the transpose of a rotation matrix is the same as the inverse):
M2TM1(a-c1)==(a-c2) =>
-M2TM1(a-c1)+a==c2
Now if we look at this as a transformation being applied to c1, then we can see that it is first negated, then translated by a, then rotated by M1, then rotated by M2T, negated again, and then translated by a again. These are transformations that graphics libraries are good at and they can all be squished into a single transformation matrix.
#Generic Human deserves credit for the answer, but here's code for it. Of course, you need to implement the function to validate a change in pitch before it's applied, but that's simple. This code probably has a couple typos since I haven't tried to compile:
case ENavigation::eRotatePoint: {
const double dX = (double)(m_oLastMousePos.x - roMousePos.x) * dRotatePixelScale * D3DX_PI;
double dY = (double)(m_oLastMousePos.y - roMousePos.y) * dRotatePixelScale * D3DX_PI;
dY = validatePitch(dY); // dY needs to be kept within bounds so that m_dRotationY is within bounds
D3DXMATRIX oRotationMatrix1; // The camera orientation before mouse-change
D3DXMatrixRotationYawPitchRoll(&oRotationMatrix1, m_dRotationX, m_dRotationY, 0.0);
D3DXMATRIX oRotationMatrix2; // The camera orientation after mouse-change
D3DXMatrixRotationYawPitchRoll(&oRotationMatrix2, m_dRotationX + dX, m_dRotationY + dY, 0.0);
D3DXMATRIX oRotationMatrix2Inv; // The inverse of the orientation
D3DXMatrixTranspose(&oRotationMatrix2Inv,&oRotationMatrix2); // Transpose is the same in this case
D3DXMATRIX oScaleMatrix; // Negative scaling matrix for negating the translation
D3DXMatrixScaling(&oScaleMatrix,-1,-1,-1);
D3DXMATRIX oTranslationMatrix; // Translation by the reference point
D3DXMatrixTranslation(&oTranslationMatrix,
m_oRotateAroundPos.x,m_oRotateAroundPos.y,m_oRotateAroundPos.z);
D3DXMATRIX oTransformMatrix; // The full transform for the eyePos.
// We assume the matrix multiply protects against variable aliasing
D3DXMatrixMultiply(&oTransformMatrix,&oScaleMatrix,&oTranslationMatrix);
D3DXMatrixMultiply(&oTransformMatrix,&oTransformMatrix,&oRotationMatrix1);
D3DXMatrixMultiply(&oTransformMatrix,&oTransformMatrix,&oRotationMatrix2Inv);
D3DXMatrixMultiply(&oTransformMatrix,&oTransformMatrix,&oScaleMatrix);
D3DXMatrixMultiply(&oTransformMatrix,&oTransformMatrix,&oTranslationMatrix);
D3DXVECTOR4 oEyeFinal;
D3DXVec3Transform(&oEyeFinal, &m_oEyePos, &oTransformMatrix);
m_oEyePos = D3DXVECTOR3(oEyeFinal.x, oEyeFinal.y, oEyeFinal.z)
// Increment rotation to keep the same relative look angles
RotateXAxis(dX);
RotateYAxis(dY);
break;
}
I think there is a much simpler solution that lets you sidestep all rotation issues.
Notation: A is the point we want to rotate around, C is the original camera location, M is the original camera rotation matrix that maps global coordinates to the camera's local viewport.
Make a note of the local coordinates of A, which are equal to A' = M × (A - C).
Rotate the camera like you would in normal "eye rotation" mode. Update the view matrix M so that it is modified to M2 and C remains unchanged.
Now we would like to find C2 such that A' = M2 × (A - C2).
This is easily done by the equation C2 = A - M2-1 × A'.
Voilà, the camera has been rotated and because the local coordinates of A are unchanged, A remains at the same location and the same scale and distance.
As an added bonus, the rotation behavior is now consistent between "eye rotation" and "point rotation" mode.
You rotate around the point by repeatedly applying small rotation matrices, this probably cause the drift (small precision errors add up) and I bet you will not really do a perfect circle after some time. Since the angles for the view use simple 1-dimension double, they have much less drift.
A possible fix would be to store a dedicated yaw/pitch and relative position from the point when you enter that view mode, and using those to do the math. This requires a bit more bookkeeping, since you need to update those when moving the camera. Note that it will also make the camera move if the point move, which I think is an improvement.
If I understand correctly, you are satisfied with the rotation component in the final matrix (save for inverted rotation controls in the problem #3), but not with the translation part, is that so?
The problem seems to come from the fact that you treating them differently: you are recalculating the rotation part from scratch every time, but accumulate the translation part (m_oEyePos). Other comments mention precision problems, but it's actually more significant than just FP precision: accumulating rotations from small yaw/pitch values is simply not the same---mathematically---as making one big rotation from the accumulated yaw/pitch. Hence the rotation/translation discrepancy. To fix this, try recalculating eye position from scratch simultaneously with the rotation part, similarly to how you find "oTarget = oEyePos + ...":
oEyePos = m_oRotateAroundPos - dist * D3DXVECTOR3(oForward4.x, oForward4.y, oForward4.z)
dist can be fixed or calculated from the old eye position. That will keep the rotation point in the screen center; in the more general case (which you are interested in), -dist * oForward here should be replaced by the old/initial m_oEyePos - m_oRotateAroundPos multiplied by the old/initial camera rotation to bring it to the camera space (finding a constant offset vector in camera's coordinate system), then multiplied by the inverted new camera rotation to get the new direction in the world.
This will, of course, be subject to gimbal lock when the pitch is straight up or down. You'll need to define precisely what behavior you expect in these cases to solve this part. On the other hand, locking at m_dRotationX=0 or =pi is rather strange (this is yaw, not pitch, right?) and might be related to the above.