How to rotate object around local axis in OpenGL? - opengl

I am working on an ongoing project where I want to align the links of a chain so that it follows the contours of a Bezier curve. I am currently following the steps below.
Drawing the curve.
Use a display list to create one link of the chain.
Use a FOR loop to repeatedly call a function that calculates the angle between two points on the curve, returns the angle and the axis around which the link should be rotated.
Rotate by the angle "a" and translate to new position, place the link at the new position.
Edit: I should also say that the centres of the two half torus must lie on the Bezier curve.
Also I am aware that the method I use to draw the torus I tedious, I will use TRIANGLE_FAN or QUAD_STRIP later on to draw the torus in a more efficient way.
While at first glance this logic looks like it would render the chain properly, the end result is not what I had imagined it to be. Here is a picture of what the chain looks like.
I read that you have to translate the object to the origin before rotation? Would I just call glTranslate(0,0,0) and then follow step 4 from above?
I have included the relevant code from what I have done so far, I would appreciate any suggestions to get me code work properly.
/* this function calculates the angle between two vectors oldPoint and new point contain the x,y,z coordinates of the two points,axisOfRot is used to return the x,y,z coordinates of the rotation axis*/
double getAxisAngle(pointType oldPoint[],
pointType newPoint[],pointType axisOfRot[]){
float tmpPoint[3];
float normA = 0.0,normB = 0.0,AB = 0.0,angle=0.0;
int i;
axisOfRot->x= oldPoint->y * newPoint->z - oldPoint->z * newPoint->y;
axisOfRot->y= oldPoint->z * newPoint->x - oldPoint->x * newPoint->z;
axisOfRot->z= oldPoint->x * newPoint->y - oldPoint->y * newPoint->x;
normA=sqrt(oldPoint->x * oldPoint->x + oldPoint->y * oldPoint->y + oldPoint->z *
oldPoint->z);
normB=sqrt(newPoint->x * newPoint->x + newPoint->y * newPoint->y + newPoint->z *
newPoint->z);
tmpPoint[0] = oldPoint->x * newPoint->x;
tmpPoint[1] = oldPoint->y * newPoint->y;
tmpPoint[2] = oldPoint->z * newPoint->z;
for(i=0;i<=2;i++)
AB+=tmpPoint[i];
AB /= (normA * normB);
return angle = (180/PI)*acos(AB);
}
/* this function calculates and returns the next point on the curve give the 4 initial points for the curve, t is the tension of the curve */
void bezierInterpolation(float t,pointType cPoints[],
pointType newPoint[]){
newPoint->x = pow(1 - t, 3) * cPoints[0].x +3 * pow(1 - t , 2) * t * cPoints[1].x + 3
* pow(1 - t, 1) * pow(t, 2) * cPoints[2].x + pow(t, 3) * cPoints[3].x;
newPoint->y = pow(1 - t, 3) * cPoints[0].y +3 * pow(1 - t , 2) * t * cPoints[1].y + 3
* pow(1 - t, 1) * pow(t, 2) * cPoints[2].y + pow(t, 3) * cPoints[3].y;
newPoint->z = pow(1 - t, 3) * cPoints[0].z +3 * pow(1 - t , 2) * t * cPoints[1].z + 3
* pow(1 - t, 1) * pow(t, 2) * cPoints[2].z + pow(t, 3) * cPoints[3].z;
}
/* the two lists below are used to create a single link in a chain, I realize that creating a half torus using cylinders is a bad idea, I will use GL_STRIP or TRIANGLE_FAN once I get the alignment right
*/
torusList=glGenLists(1);
glNewList(torusList,GL_COMPILE);
for (i=0; i<=180; i++)
{
degInRad = i*DEG2RAD;
glPushMatrix();
glTranslatef(cos(degInRad)*radius,sin(degInRad)*radius,0);
glRotated(90,1,0,0);
gluCylinder(quadric,Diameter/2,Diameter/2,Height/5,10,10);
glPopMatrix();
}
glEndList();
/*! create a list for the link , 2 half torus and 2 columns */
linkList = glGenLists(1);
glNewList(linkList, GL_COMPILE);
glPushMatrix();
glCallList(torusList);
glRotatef(90,1,0,0);
glTranslatef(radius,0,0);
gluCylinder(quadric, Diameter/2, Diameter/2, Height,10,10);
glTranslatef(-(radius*2),0,0);
gluCylinder(quadric, Diameter/2, Diameter/2, Height,10,10);
glTranslatef(radius,0, Height);
glRotatef(90,1,0,0);
glCallList(torusList);
glPopMatrix();
glEndList();
Finally here is the code for creating the three links in the chain
t=0.031;
bezierInterpolation(t,cPoints,newPoint);
a=getAxisAngle(oldPoint,newPoint,axisOfRot);
glTranslatef(newPoint->x,newPoint->y,newPoint->z);
glRotatef(a,axisOfRot->x,axisOfRot->y,axisOfRot->z);
glCallList(DLid);
glRotatef(-a,axisOfRot->x,axisOfRot->y,axisOfRot->z);
glTranslatef(-newPoint->x,-newPoint->y,-newPoint->z);
oldPoint[0]=newPoint[0];
bezierInterpolation(t+=GAP,cPoints,newPoint);
a=getAxisAngle(oldPoint,newPoint,axisOfRot);
glTranslatef(newPoint->x,newPoint->y,newPoint->z);
glRotatef(90,0,1,0);
glRotatef(a,axisOfRot->x,axisOfRot->y,axisOfRot->z);
glCallList(DLid);
glRotatef(-a,axisOfRot->x,axisOfRot->y,axisOfRot->z);
glRotatef(90,0,1,0);
glTranslatef(-newPoint->x,-newPoint->y,-newPoint->z);
oldPoint[0]=newPoint[0];
bezierInterpolation(t+=GAP,cPoints,newPoint);
a=getAxisAngle(oldPoint,newPoint,axisOfRot);
glTranslatef(newPoint->x,newPoint->y,newPoint->z);
glRotatef(-a,axisOfRot->x,axisOfRot->y,axisOfRot->z);
glCallList(DLid);
glRotatef(a,axisOfRot->x,axisOfRot->y,axisOfRot->z);
glTranslatef(-newPoint->x,-newPoint->y,newPoint->z);

One thing to note is that glTranslate function builds on previous translations. I.E. a glTranslatef(0.0,0.0,0.0); won't go to the origin, it will just move the "pen" nowhere. Luckily, the "pen" starts at the origin. if you translate out to 1.0,1.0,1.0 then try a glTranslatef(0.0,0.0,0.0); you will still be drawing at 1.0,1.0,1.0;
Also, you seem to grasp the fact that openGL post-multiplies matricies. To that end, you are correctly "undoing" your matrix operations after a draw. I only see one spot where you could potentially be off here and that is in this statement:
glRotatef(90,0,1,0);
glRotatef(a,axisOfRot->x,axisOfRot->y,axisOfRot->z);
glCallList(DLid);
glRotatef(-a,axisOfRot->x,axisOfRot->y,axisOfRot->z);
glRotatef(90,0,1,0);
Here you correctly undo the second rotation, but the first one you seem to rotate even more around the y axis. the very last glRotatef needs to read glRotatef(-90,0,1,0); if you want to be undoing that rotation.

I looked at your code and assuming that code performing bezierInterp and axis angle is correct. Based on code, I have following suggestions:
The way you are creating a single link looks very costly. As you are using gluCylinder for 180 times. This will generate a lot of vertices for a small link. You can create a single torus and apply scale such that it appears like a link!
Whenever you do any matrix operation, it is good idea to set the mode before. This is important before doing push and pop. In you display list you have push and pop without setting any mode and neither it is set in caller. This is not good practice and will result in lot of bugs/issues. You can remove push and pop from call list and keep only geometry in it.
You have heard advice suggesting to do translation to origin before rotation as translation * rotation! = rotation * translation. So the way you would write your render loop is:
// Set matrix mode
glMatrixMode(GL_MODELVIEW);
for(number of links) {
glLoadIdentity(); // makes model view matrix identity - default location`
glTranslatef(x,y,z); // Translate to a point on beizer curve
glRotatef(..); // Rotate link
glCallList(link); // can be simple torus, only geometry centered at origin
}
Above code renders a link repeated at specified location. Read OpenGL Red book's chapter 3 - Example 3.6 (planetary system) example to understand how you can place each link at different location correctly.

Related

Enemies path following (Space Shooter game)

I am recently working with SFML libraries and I am trying to do a Space Shooter game from scratch. After some time working on it I get something that works fine but I am facing one issue and I do not know exactly how to proceed, so I hope your wisdom can lead me to a good solution. I will try to explain it the best I can:
Enemies following a path: currently in my game, I have enemies that can follow linear paths doing the following:
float vx = (float)m_wayPoints_v[m_wayPointsIndex_ui8].x - (float)m_pos_v.x;
float vy = (float)m_wayPoints_v[m_wayPointsIndex_ui8].y - (float)m_pos_v.y;
float len = sqrt(vx * vx + vy * vy);
//cout << len << endl;
if (len < 2.0f)
{
// Close enough, entity has arrived
//cout << "Has arrived" << endl;
m_wayPointsIndex_ui8++;
if (m_wayPointsIndex_ui8 >= m_wayPoints_v.size())
{
m_wayPointsIndex_ui8 = 0;
}
}
else
{
vx /= len;
vy /= len;
m_pos_v.x += vx * float(m_moveSpeed_ui16) * time;
m_pos_v.y += vy * float(m_moveSpeed_ui16) * time;
}
*m_wayPoints_v is a vector that basically holds the 2d points to be followed.
Related to this small piece of code, I have to say that is sometimes given me problems because getting closer to the next point becomes difficult as the higher the speed of the enemies is.
Is there any other way to be more accurate on path following independtly of the enemy speed? And also related to path following, if I would like to do an introduction of the enemies before each wave movement pattern starts (doing circles, spirals, ellipses or whatever before reaching the final point), for example:
For example, in the picture below:
The black line is the path I want a spaceship to follow before starting the IA pattern (move from left to right and from right to left) which is the red circle.
Is it done hardcoding all and each of the movements or is there any other better solution?
I hope I made myself clear on this...in case I did not, please let me know and I will give more details. Thank you very much in advance!
Way points
You need to add some additional information to the way points and the NPC's position in relationship to the way points.
The code snippet (pseudo code) shows how a set of way points can be created as a linked list. Each way point has a link and a distance to the next way point, and the total distance for this way point.
Then each step you just increase the NPC distance on the set of way points. If that distance is greater than the totalDistance at the next way point, follow the link to the next. You can use a while loop to search for the next way point so you will always be at the correct position no matter what your speed.
Once you are at the correct way point its just a matter of calculating the position the NPC is between the current and next way point.
Define a way point
class WayPoint {
public:
WayPoint(float, float);
float x, y, distanceToNext, totalDistance;
WayPoint next;
WayPoint addNext(WayPoint wp);
}
WayPoint::WayPoint(float px, float py) {
x = px; y = py;
distanceToNext = 0.0f;
totalDistance = 0.0f;
}
WayPoint WayPoint::addNext(WayPoint wp) {
next = wp;
distanceToNext = sqrt((next.x - x) * (next.x - x) + (next.y - y) * (next.y - y));
next.totalDistance = totalDistance + distanceToNext;
return wp;
}
Declaring and linking waypoints
WayPoint a(10.0f, 10.0f);
WayPoint b(100.0f, 400.0f);
WayPoint c(200.0f, 100.0f);
a.addNext(b);
b.addNext(c);
NPC follows way pointy path at any speed
WayPoint currentWayPoint = a;
NPC ship;
ship.distance += ship.speed * time;
while (ship.distance > currentWayPoint.next.totalDistance) {
currentWayPoint = currentWayPoint.next;
}
float unitDist = (ship.distance - currentWayPoint.totalDistance) / currentWayPoint.distanceToNext;
// NOTE to smooth the line following use the ease curve. See Bottom of answer
// float unitDist = sigBell((ship.distance - currentWayPoint.totalDistance) / currentWayPoint.distanceToNext);
ship.pos.x = (currentWayPoint.next.x - currentWayPoint.x) * unitDist + currentWayPoint.x;
ship.pos.y = (currentWayPoint.next.y - currentWayPoint.y) * unitDist + currentWayPoint.y;
Note you can link back to the start but be careful to check when the total distance goes back to zero in the while loop or you will end up in an infinite loop. When you pass zero recalc NPC distance as modulo of last way point totalDistance so you never travel more than one loop of way points to find the next.
eg in while loop if passing last way point
if (currentWayPoint.next.totalDistance == 0.0f) {
ship.distance = mod(ship.distance, currentWayPoint.totalDistance);
}
Smooth paths
Using the above method you can add additional information to the way points.
For example for each way point add a vector that is 90deg off the path to the next.
// 90 degh CW
offX = -(next.y - y) / distanceToNext; // Yes offX = - y
offY = (next.x - x) / distanceToNext; //
offDist = ?; // how far from the line you want to path to go
Then when you calculate the unitDist along the line between to way points you can use that unit dist to smoothly interpolate the offset
float unitDist = (ship.distance - currentWayPoint.totalDistance) / currentWayPoint.distanceToNext;
// very basic ease in and ease out or use sigBell curve
float unitOffset = unitDist < 0.5f ? (unitDist * 2.0f) * (unitDist * 2.0f) : sqrt((unitDist - 0.5f) * 2.0f);
float x = currentWayPoint.offX * currentWayPoint.offDist * unitOffset;
float y = currentWayPoint.offY * currentWayPoint.offDist * unitOffset;
ship.pos.x = (currentWayPoint.next.x - currentWayPoint.x) * unitDist + currentWayPoint.x + x;
ship.pos.y = (currentWayPoint.next.y - currentWayPoint.y) * unitDist + currentWayPoint.y + y;
Now if you add 3 way points with the first offDist a positive distance and the second a negative offDist you will get a path that does smooth curves as you show in the image.
Note that the actual speed of the NPC will change over each way point. The maths to get a constant speed using this method is too heavy to be worth the effort as for small offsets no one will notice. If your offset are too large then rethink your way point layout
Note The above method is a modification of a quadratic bezier curve where the control point is defined as an offset from center between end points
Sigmoid curve
You don't need to add the offsets as you can get some (limited) smoothing along the path by manipulating the unitDist value (See comment in first snippet)
Use the following to function convert unit values into a bell like curve sigBell and a standard ease out in curve. Use argument power to control the slopes of the curves.
float sigmoid(float unit, float power) { // power should be > 0. power 1 is straight line 2 is ease out ease in 0.5 is ease to center ease from center
float u = unit <= 0.0f ? 0.0f : (unit >= 1.0f ? 1.0f: unit); // clamp as float errors will show
float p = pow(u, power);
return p / (p + pow(1.0f - u, power));
}
float sigBell(float unit, float power) {
float u = unit < 0.5f ? unit * 2.0f : 1.0f - (unit - 0.5f) * 2.0f;
return sigmoid(u, power);
}
This doesn't answer your specific question. I'm just curious why you don't use the sfml type sf::Vector2 (or its typedefs 2i, 2u, 2f)? Seems like it would clean up some of your code maybe.
As far as the animation is concerned. You could consider loading the directions for the flight pattern you want into a stack or something. Then pop each position and move your ship to that position and render, repeat.
And if you want a sin-like flight path similar to your picture, you can find an equation similar to the flight path you like. Use desmos or something to make a cool graph that fits your need. Then iterate at w/e interval inputting each iteration into this equation, your results are your position at each iteration.
Well, I think I found one of the problems but I am not sure what the solution can be.
When using the piece of code I posted before, I found that there is a problem when reaching the destination point due to the speed value. Currently to move a space ship fluently, I need to set the speed to 200...which means that in these formulas:
m_pos_v.x += vx * float(m_moveSpeed_ui16) * time;
m_pos_v.y += vy * float(m_moveSpeed_ui16) * time;
The new position might exceed the "2.0f" tolerance so the space ship cannot find the destination point and it gets stuck because the minimum movement that can be done per frame (assuming 60fps) 200 * 1 / 60 = 3.33px. Is there any way this behavior can be avoided?

Projecting line from camera

I'm trying to convert a viewport click onto a world position for an object.
It would be quite simple if all I wanted was to draw a point exactly where the user clicks in the canvas:
void Canvas::getClickPosition(int x, int y, Vector3d(&out)[2]) const
{
Vector4d point4d[2];
Vector2d point2d(x, y);
int w = canvas.width();
int h = canvas.height();
Matrix4d model = m_world * m_camera;
for (int i = 0; i < 2; ++i) {
Vector4d sw(point2d.x() / (0.5 * w) - 1,
point2d.y() / (0.5* h) - 1, i * 1, 1);
point4d[i] = (m_proj * model).inverse() * sw;
out[i] = point4d.block<1, 3>(0, 0);
}
}
The expected behavior is achieved with this simple code.
The problem arises when I try to actually make a line that will look like a one pixel when the user first clicks it. Until the camera is rotated in any direction it should look like it was perfectly shot from the camera and that it has whatever length (doesn't matter).
I tried the obvious:
Vector4d sw(point2d.x() / (0.5 * w) - 1,
point2d.y() / (0.5* h) - 1, 1, 1); // Z is now 1 instead of 0.
The result is, as most of you guys should expect, a line that pursues the vanishing point, at the center of the screen. Therefore, the farther I click from the center, the more the line is twitched from it's expected direction.
What can I do to have a line show as a dot from the click point of view, no matter where at the screen?
EDIT: for better clarity, I'm trying to draw the lines like this:
glBegin(GL_LINES);
line.p1 = m_proj * (m_world * m_camera) * line.p1;
line.p2 = m_proj * (m_world * m_camera) * line.p2;
glVertex3f(line.p1.x(), line.p1.y(), line.p1.z());
glVertex3f(line.p2.x(), line.p2.y(), line.p2.z());
glEnd();
Your initial attempt is actually very close. The only thing you are missing is the perspective divide:
out[i] = point4d.block<1, 3>(0, 0) / point4d.w();
Depending on your projection matrix, you might also need to specify a z-value of -1 for the near plane instead of 0.
And yes, your order of matrices projection * model * view seems strange. But as long as you keep the same order in both procedures, you should get a consistent result.
Make sure that the y-axis of your window coordinate system is pointing upwards. Otherwise, you will get a result that is reflected at the horizontal midline.

Sprite rotation offset doesn't stay where it belongs. (SDL)

So, here is the code for my 2D point class to rotate:
float nx = (x * cos(angle)) - (y * sin(angle));
float ny = (y * cos(angle)) + (x * sin(angle));
x = nx;
y = ny;
x and y are local variables in the point class.
And here is the code for my sprite class's rotation:
//Make clip
SDL_Rect clip;
clip.w = width;
clip.h = height;
clip.x = (width * _frameX) + (sep * (_frameX) + osX);
clip.y = (height * _frameY) + (sep * (_frameY) + osY);
//Make a rotated image
col bgColor = image->format->colorkey;
//Surfaces
img *toEdit = newImage(clip.w, clip.h);
img *toDraw = 0;
//Copy the source into the workspace
drawRect(0, 0, toEdit->w, toEdit->h, toEdit, bgColor);
drawImage(0, 0, image, toEdit, &clip);
//Edit the image
toDraw = SPG_Transform(toEdit, bgColor, angle, xScale, yScale, SPG_NONE);
SDL_SetColorKey(toDraw, SDL_SRCCOLORKEY, bgColor);
//Find new origin and offset by pivot
2DVec *pivot = new xyVec(pvX, pvY);
pivot->rotate(angle);
//Draw and remove the finished image
drawImage(_x - pivot->x - (toDraw->w / 2), _y - pivot->y - (toDraw->h / 2), toDraw, _destination);
//Delete stuff
deleteImage(toEdit);
delete pivot;
deleteImage(toDraw);
The code uses the center of the sprite as the origin. It works fine if I leave the pivot at (0,0), but if I move it somewhere else, the character's shoulder for instance, it starts making the sprite dance around as it spins like a spirograph, instead of the pivot staying on the character's shoulder.
The image rotation function is from SPriG, a library for drawing primitives and transformed images in SDL. Since the pivot is coming from the center of the image, I figure the new size of the clipped surface produced by rotating shouldn't matter.
[EDIT]
I've messed with the code a bit. By slowing it down, I found that for some reason, the vector is rotating 60 times faster than the image, even though I'm not multiplying anything by 60. So, I tried to just divide the input by 60, only now, it's coming out all jerky and not rotating to anything between multiples of 60.
The vector rotation code I found on this very site, and people have repeatedly confirmed that it works, so why does it only rotate in increments of 60?
I haven't touched the source of SPriG in a long time, but I can give you some info.
If SPriG has problems with rotating off of center, it would probably be faster and easier for you to migrate to SDL_gpu (and I suggest SDL 2.0). That way you get a similar API but the performance is much better (it uses the graphics card).
I can guess that the vector does not rotate 60 times faster than the image, but rather more like 57 times faster! This is because you are rotating the vector with sin() and cos(), which accept values in radians. The image is being rotated by an angle in degrees. The conversion factor for radians to degrees is 180/pi, which is about 57. SPriG can use either degrees or radians, but uses degrees by default. Use SPG_EnableRadians(1) to switch that behavior. Alternatively, you can stick to degree measure in your angle variable by multiplying the argument to sin() and cos() by pi/180.

Preserving rotations in OpenGL

I'm drawing an object (say, a cube) in OpenGL that a user can rotate by clicking / dragging the mouse across the window. The cube is drawn like so:
void CubeDrawingArea::redraw()
{
Glib::RefPtr gl_drawable = get_gl_drawable();
gl_drawable->gl_begin(get_gl_context());
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glPushMatrix();
{
glRotated(m_angle, m_rotAxis.x, m_rotAxis.y, m_rotAxis.z);
glCallList(m_cubeID);
}
glPopMatrix();
gl_drawable->swap_buffers();
gl_drawable->gl_end();
}
and rotated with this function:
bool CubeDrawingArea::on_motion_notify_event(GdkEventMotion* motion)
{
if (!m_leftButtonDown)
return true;
_3V cur_pos;
get_trackball_point((int) motion->x, (int) motion->y, cur_pos);
const double dx = cur_pos.x - m_lastTrackPoint.x;
const double dy = cur_pos.y - m_lastTrackPoint.y;
const double dz = cur_pos.z - m_lastTrackPoint.z;
if (dx || dy || dz)
{
// Update angle, axis of rotation, and redraw
m_angle = 90.0 * sqrt((dx * dx) + (dy * dy) + (dz * dz));
// Axis of rotation comes from cross product of last / cur vectors
m_rotAxis.x = (m_lastTrackPoint.y * cur_pos.z) - (m_lastTrackPoint.z * cur_pos.y);
m_rotAxis.y = (m_lastTrackPoint.z * cur_pos.x) - (m_lastTrackPoint.x * cur_pos.z);
m_rotAxis.z = (m_lastTrackPoint.x * cur_pos.y) - (m_lastTrackPoint.y * cur_pos.x);
redraw();
}
return true;
}
There is some GTK+ stuff in there, but it should be pretty obvious what it's for. The get_trackball_point() function projects the window coordinates X Y onto a hemisphere (the virtual "trackball") that is used as a reference point for rotating the object. Anyway, this more or less works, but after I'm done rotating, and I go to rotate again, the cube snaps back to the original position, obviously, since m_angle will be reset back to near 0 the next time I rotate. Is there anyway to avoid this and preserve the rotation?
Yeah, I ran into this problem too.
What you need to do is keep a rotation matrix around that "accumulates" the current state of rotation, and use it in addition to the rotation matrix that comes from the current dragging operation.
Say you have two matrices, lastRotMx and currRotMx. Make them members of CubeDrawingArea if you like.
You haven't shown us this, but I assume that m_lastTrackPoint is initialized whenever the mouse button goes down for dragging. When that happens, copy currRotMx into lastRotMx.
Then in on_motion_notify_event(), after you calculate m_rotAxis and m_angle, create a new rotation matrix draggingRotMx based on m_rotAxis and m_angle; then multiply lastRotMx by draggingRotMx and put the result in currRotMx.
Finally, in redraw(), instead of
glRotated(m_angle, m_rotAxis.x, m_rotAxis.y, m_rotAxis.z);
rotate by currRotMx.
Update: Or instead of all that... I haven't tested this, but I think it would work:
Make cur_pos a class member so it stays around, but it's initialized to zero, as is m_lastTrackPoint.
Then, whenever a new drag motion is started, before you initialize m_lastTrackPoint, let _3V dpos = cur_pos - m_lastTrackPoint (pseudocode).
Finally, when you do initialize m_lastTrackPoint based on the mouse event coords, subtract dpos from it.
That way, your cur_pos will already be offset from m_lastTrackPoint by an amount based on the accumulation of offsets from past arcball drags.
Probably error would accumulate as well, but it should be gradual enough so as not to be noticeable. But I'd want to test it to be sure... composed rotations are tricky enough that I don't trust them without seeing them.
P.S. your username is demotivating. Suggest picking another one.
P.P.S. For those who come later searching for answers to this question, the keywords to search on are "arcball rotation". An definitive article is Ken Shoemake's section in Graphical Gems IV. See also this arcball tutorial for JOGL.

Calculating vertices of a rotated rectangle

I am trying to calculate the vertices of a rotated rectangle (2D).
It's easy enough if the rectangle has not been rotated, I figured that part out.
If the rectangle has been rotated, I thought of two possible ways to calculate the vertices.
Figure out how to transform the vertices from local/object/model space (the ones I figured out below) to world space. I honestly have no clue, and if it is the best way then I feel like I would learn a lot from it if I could figure it out.
Use trig to somehow figure out where the endpoints of the rectangle are relative to the position of the rectangle in world space. This has been the way I have been trying to do up until now, I just haven't figured out how.
Here's the function that calculates the vertices thus far, thanks for any help
void Rect::calculateVertices()
{
if(m_orientation == 0) // if no rotation
{
setVertices(
&Vertex( (m_position.x - (m_width / 2) * m_scaleX), (m_position.y + (m_height / 2) * m_scaleY), m_position.z),
&Vertex( (m_position.x + (m_width / 2) * m_scaleX), (m_position.y + (m_height / 2) * m_scaleY), m_position.z),
&Vertex( (m_position.x + (m_width / 2) * m_scaleX), (m_position.y - (m_height / 2) * m_scaleY), m_position.z),
&Vertex( (m_position.x - (m_width / 2) * m_scaleX), (m_position.y - (m_height / 2) * m_scaleY), m_position.z) );
}
else
{
// if the rectangle has been rotated..
}
//GLfloat theta = RAD_TO_DEG( atan( ((m_width/2) * m_scaleX) / ((m_height / 2) * m_scaleY) ) );
//LOG->writeLn(&theta);
}
I would just transform each point, applying the same rotation matrix to each one. If it's a 2D planar rotation, it would look like this:
x' = x*cos(t) - y*sin(t)
y' = x*sin(t) + y*cos(t)
where (x, y) are the original points, (x', y') are the rotated coordinates, and t is the angle measured in radians from the x-axis. The rotation is counter-clockwise as written.
My recommendation would be to do it out on paper once. Draw a rectangle, calculate the new coordinates, and redraw the rectangle to satisfy yourself that it's correct before you code. Then use this example as a unit test to ensure that you coded it properly.
I think you were on the right track using atan() to return an angle. However you want to pass height divided by width instead of the other way around. That will give you the default (unrotated) angle to the upper-right vertex of the rectangle. You should be able to do the rest like this:
// Get the original/default vertex angles
GLfloat vertex1_theta = RAD_TO_DEG( atan(
(m_height/2 * m_scaleY)
/ (m_width/2 * m_scaleX) ) );
GLfloat vertex2_theta = -vertex1_theta; // lower right vertex
GLfloat vertex3_theta = vertex1_theta - 180; // lower left vertex
GLfloat vertex4_theta = 180 - vertex1_theta; // upper left vertex
// Now get the rotated vertex angles
vertex1_theta += rotation_angle;
vertex2_theta += rotation_angle;
vertex3_theta += rotation_angle;
vertex4_theta += rotation_angle;
//Calculate the distance from the center (same for each vertex)
GLfloat r = sqrt(pow(m_width/2*m_scaleX, 2) + pow(m_height/2*m_scaleY, 2));
/* Calculate each vertex (I'm not familiar with OpenGL, DEG_TO_RAD
* might be a constant instead of a macro)
*/
vertexN_x = m_position.x + cos(DEG_TO_RAD(vertexN_theta)) * r;
vertexN_y = m_position.y + sin(DEG_TO_RAD(vertexN_theta)) * r;
// Now you would draw the rectangle, proceeding from vertex1 to vertex4.
Obviously more longwinded than necessary, for the sake of clarity. Of course, duffymo's solution using a transformation matrix is probably more elegant and efficient :)
EDIT: Now my code should actually work. I changed (width / height) to (height / width) and used a constant radius from the center of the rectangle to calculate the vertices. Working Python (turtle) code at http://pastebin.com/f1c76308c