draw graphics over kinect video image in processing 3.0 - computer-vision

Question

I am working through Greg Borenstein's book "Making Things See" and have figured out how to create a cursor that tracks the movement of the closest thing to the Kinect. Right now the cursor is a simple red ball. So I am able to track my finger over
image(kinect.getVideoImage(), 0, 0)
I have also created buttons that apply a filter to the video image when I put the cursor ball in the area of the button.
Its kind of fun but the novelty has run out so now I want to turn the cursor ball into an animated graphic using particles or something fun like that. This animated graphic should still track my finger and be drawn over the video image.
When I try to write this, the graphic comes out wrong because the video image keeps redrawing over the particles so it doesn't look right.
I was thinking I could use the capture() method to draw the video image under the graphics but I can't figure out how to do it with video from the Kinect.
Does anyone have any ideas on how I could do this? Any help would be greatly appreciated.
A sample of my kinect tracker and filter button is below. If you copy and paste it into processing and have a kinect plugged in it should run. my apologies for the code's lack of eloquence. I'm still learning how to make beautiful code.
Instead of filters, I would like to trigger a new graphic for the red ball maybe apply particles or something.
//my kinect tracker
import org.openkinect.freenect.*;
import org.openkinect.processing.*;
Kinect kinect;
boolean ir = true;
boolean colorDepth = true;
boolean mirror = true;
float closestValue;
float closestX;
float closestY;
// create arrays to store recent closest x- and y-coordinates for averaging
int[] recentXValues = new int[3];
int[] recentYValues = new int[3];
// keep track of which is the current value in the array to be changed
int currentIndex = 0;
float circleButtonX, circleButtonY; // position of circle button
float circleButtonSize; // diameter of circle button
color circleButtonColor; // color of circle button
void setup() {
size(640, 480, P3D);
kinect = new Kinect(this);
kinect.initDepth();
kinect.initVideo();
//kinect.enableIR(ir);
kinect.enableMirror(mirror);
kinect.enableColorDepth(colorDepth);
circleButtonColor = color(0, 0, 255);
}
void draw() {
closestValue = 1700;
int[] depthValues = kinect.getRawDepth();
for(int y = 0; y < 480; y++) {
for(int x = 0; x < 640; x++) {
int i = x + y * 640;
int currentDepthValue = depthValues[i];
if(currentDepthValue > 0 && currentDepthValue < closestValue) {
//save its value
closestValue = currentDepthValue;
recentXValues[currentIndex] = x;
recentYValues[currentIndex] = y;
}
}
}
currentIndex++;
if(currentIndex > 2) {
currentIndex = 0;
}
// closetX and ClosestY become a running average
// with currentX and CurrentY
closestX = (recentXValues[0] + recentXValues[1] + recentXValues[2]) / 3;
closestY = (recentYValues[0] + recentYValues[1] + recentYValues[2]) / 3;
//draw the depth image on the screen
image(kinect.getVideoImage(), 0, 0);
fill(0, 0, 250);
ellipse(75, 75, 100, 100);
ellipse(200, 75, 100, 100);
ellipse(75, 200, 100, 100);
rect(540, 25, 75, 100);
//buttons
fill(255,0,0);
textSize(24);
text("Invert", 40, 85);
text("Blur", 50, 210);
textSize(18);
text("Threshold", 155, 85);
text("Stop", 560, 75);
ellipse(closestX, closestY, 25, 25);
if (closestX > 25 && closestX < 125 && closestY > 25 && closestY < 125) {
filter(INVERT);
};
if (closestX > 150 && closestX < 250 && closestY > 25 && closestY < 125) {
filter(THRESHOLD);
};
if (closestX > 25 && closestX < 125 && closestY > 150 && closestY < 250) {
filter(BLUR, 6);
};
if (closestX > 540 && closestX < 615 && closestY > 25 && closestY < 100) {
noLoop();
loop();
background(0);
};

Answer 1

I believe you're asking how to create a particle trail in Processing. This doesn't really have anything to do with kinect- a simple call to background() would also draw over any of your particles. So you currently have something like this:
void setup(){
size(500, 500);
}
void draw(){
background(0);
ellipse(mouseX, mouseY, 10, 10);
}
You're drawing something to the screen (in your code, a red ball, in my code, an ellipse), but it's cleared away (in your code, the kinect video, in my code, a call to background(0)). You're asking how to make it so the ellipse stays on the screen even after you draw the background.
The answer is this: You need to store the positions of your trail in a data structure and then redraw them every frame.
A simple way to do that is to create an ArrayList of PVector instances. To add a particle to the trail, you just add a PVector to the ArrayList. And then you just iterate over the trail and draw every PVector point.
ArrayList<PVector> trail = new ArrayList<PVector>();
void setup(){
size(500, 500);
}
void draw(){
background(0);
//add to the trail
trail.add(new PVector(mouseX, mouseY));
//draw the trail
for(PVector p : trail){
ellipse(p.x, p.y, 10, 10);
}
}
However, this trail will constantly grow, and you'll eventually run out of memory to hold every point. So to keep your trail from doing that, you need to limit its size:
ArrayList<PVector> trail = new ArrayList<PVector>();
void setup(){
size(500, 500);
}
void draw(){
background(0);
//add to the trail
trail.add(new PVector(mouseX, mouseY));
if(trail.size() > 10){
//trail is too long, remove the oldest point
trail.remove(0);
}
//draw the trail
for(PVector p : trail){
ellipse(p.x, p.y, 10, 10);
}
}
From here you can do fancier things: give each point in your trail a momentum or a color, or fade it out, or decrease its width. But the basics are this: you have to store the particles in a data structure, update that data structure to add or remove particles, and then iterate over that data structure to draw the trail.