I have a movie in the after effects that dosent have a KEY color of background, but a still background.
I want to detect the motion of 2 persons walking in front of this still background and bring them to the front, so I can create effects on the background.
Is that possible to do? Witch Effect do I use?
This works with a technique called difference matting. It never works well even if you have a solid clean plate. your best bet is to rotobrush or just mask them out manually. if you want to try difference matting you can set the footage layer with the people in it to difference blending mode and put a clean plate below it.
you can use the rotobtush that allows to separate elements from a static background. Works better if:
you have a clean background
good quality videos
front object needs to be cutted as big as possible
This method works well with a locked down camera. Depending upon how active people are and how still the background - you can remove the people by cutting and assemble pieces of the background to create a background layer without the people... Once you have a clean plate - then use a difference matte to subtract the background from your clip which will leave anything that moves (read people). This will work well if the hue/values between the background and the people are different.
Doing this in conjuncture with animated masks gives good results and can give you a jump on your rotobrushing to improve your edges.
If all else fails the Mocha plug works well as long as you break your masks up into lots of parts to follow the planes of your actors.
I have never found doing this is a single tool solution.
It is not impossible, but I don't think it could give you the best result.
Anyway, you should mask them out manually. but not keyframe by keyframe. there is a technic for it. for setting the mask keyframes, first, you should set 3 keyframes in first, middle and end of your footage.
after masking three one, you do it the same for first keyframe and the middle one.
it means between the first keyframe and the middle keyframe, you should set a key.
Do the same for middle and end key.
this technic could save your time and make masking with less fault.
by the way, you can use Mocha pro as well for tracking the persons.
Related
I have a video where somebody walks in with a shirt on with text. I want to remove the text on every frame but because the person moves and turns i am looking for a way how i can automate that. so that the program understands which part has to go and does that on every frame. Is there a way of doing this automatically?
Full automation is not likely, but a mix of automation and elbow grease might do the trick. The exact mix depends on the shot.
Motion tracking the text on the footage layer through Mocha (Animation -> Track in mocha ae) will give you position/scale/rotation data that you can then apply to a second layer with a fix implemented - a blur or color layer for example. The elbow grease comes in here, where you will most likely need to roto the fix as the person turns and the text changes shape, etc.
Again, depending on the shot it may be more elbow grease than you want. There is no alternative, however - this is the process vfx artists follow at every level of the industry.
I want to ask about what kind of problems there be if i use this method to extract foreground.
The condition before using this method is that it runs on fixed camera so there're not going to be any movement on camera position.
And what im trying to do is below.
read one frame from camera and set this frame as background image. This is done periodically.
periodically subtract frames that are read afterward to background image above. Then there will be only moving things colored differently from other area
that are same to background image.
then isolate moving object by using grayscale, binarization, thresholding.
iterate above 4 processes.
If i do this, would probability of successfully detect moving object be high? If not... could you tell me why?
If you consider illumination change(gradually or suddenly) in scene, you will see that your method does not work.
There are more robust solutions for these problems. One of these(maybe the best) is Gaussian Mixture Model applied for background subtraction.
You can use BackgroundSubtractorMOG2 (implementation of GMM) in OpenCV library.
Your scheme is quite adequate to cases where the camera is fix and the background is stationary. Indoor and man-controlled scenes are more appropriate to this approach than outdoor and natural scenes .I've contributed to a detection system that worked basically on the same principals you suggested. But of course the details are crucial. A few remarks based on my experience
Your initialization step can cause very slow convergence to a normal state. You set the background to the first frames, and then pieces of background coming behind moving objects will be considered as objects. A better approach is to take the median of N first frames.
Simple subtraction may not be enough in cases of changing light condition etc. You may find a similarity criterion better for your application.
simple thresholding on the difference image may not be enough. A simple approach is to dilate the foreground for the sake of not updating the background on pixels that where accidentally identified as such.
Your step 4 is unclear, I assumed that you mean that you update the foreground only on those places that are identified as background on the last frame. Note that with such a simple approach, pixels that are actually background may be stuck forever with a "foreground" labeling, as you don't update the background under them. There are many possible solutions to this.
There are many ways to solve this problem, and it will really depend on the input images as to which method will be the most appropriate. It may be worth doing some reading on the topic
The method you are suggesting may work, but it's a slightly non-standard approach to this problem. My main concern would be that subtracting several images from the background could lead to saturation and then you may lose some detail of the motion. It may be better to take difference between consecutive images, and then apply the binarization / thresholding to these images.
Another (more complex) approach which has worked for me in the past is to take subregions of the image and then cross-correlate with the new image. The peak in this correlation can be used to identify the direction of movement - it's a useful approach if more then one thing is moving.
It may also be possible to use a combination of the two approaches above for example.
Subtract second image from the first background.
Threshold etc to find the ROI where movement is occurring
Use a pattern matching approach to track subsequent movement focussed on the ROI detected above.
The best approach will depend on you application but there are lots of papers on this topic
I try to detect two people hugging or back-to-back closely by prime sense sensor. Currently I can track two people simultaneously and their skeleton data when they stand at a distance. but if they are hugging or back-to-back closely, their skeleton will be merged into one. can anyone tell me what should i do to detect the actions(hug/backToback) between two people?
Platform: windows 7
OpenNI version: 1.5.7
NITE version: 1.5.2
Thanks.
I'm not sure the OpenNI scene segmentation alone will do, since as you observed yourself, having 2 people close to each other will cause the outlines to merge.
You would need to run your own algorithm around this merging issue yourself.
There are probably a few different ways to tackle this.
Here are a few hacky ideas that to mind at the moment:
Idea #1
Use OpenNI's scene segmentation feature and analyze the user pixels even though they are merged. If you analyze the outline. By looking at the distance between the edges(outer most pixels) and the centroid, you should spot extremities (largest distances), which would be hand/feet and the head. If your blob has two heads probably it's two people hugging.
Idea #2
You know that when users get closer they merge into a single user which means new user/lost user event triggers from OpenNI. On top of this you can keep track of each user's CoM(centre of mass, available in OpenNI). If the distance between the two users decreases a lot and this is immediately followed by a new user event (the merged blob), which has a bigger bounding box than each of the two users, then it's likely you're users are hugging or very close to each other.
Idea #3
You can track the upper body skeleton profile only for each user and detect a "hug" gesture. You can start with an initial crude pose detection where the angle of the arms is within certain threshold values. Instead of detecting pose, you could use DTW to detect a gesture. If at least one of the two users triggers this pose/gesture prior to OpenNI detecting 'fused' users then you might detect a hug.
Idea #4
Using some of the ideas above: the fact that the merged blob will be larger than either of the members and will be detect after the number of users decreases, you can use a hug detecting haar cascade on the rgb pixels belonging to the newly detected merged blob to confirm the hug.
Some of these ideas will be easier to implement than others, but it's important to keep out false positive of. Ideally you would have a simple scene (no complex objects in the background) with decent lighting( artificial cold, preferably anything away from the infrared range (sunlight, incandescent light, etc.)) to make your life easier. I've observed that with complex background sometimes even a single user can get merged with a background object.
I am learning opengl es and am planning to make a program which will have a shape which can be cut into a smaller shape by removing a part of the shape dynamicly. The constraint is I must be able to tell if an object is inside or outside the cut shape.
The option I thought of are:
1) use a stencil buffer made up of just a black and white mask. This way I can also use the same map for collision detection.
2) the other option is to dynamicly change my mind renderd primitive an then tesselating it. This sounds more complex and is currently my least favorite option. It would also make the collision detection more difficult.
PS
I would like the part of the shape removed to be fall of in animation, I am not sure how choosing any of these methods will affect the ease of doing so. Please express your opinion.
What are your thoughts on this?
Keep in mind that I am new to opengl an might be making mistakes without realizing it.
Thanks, Jason
It is generally considered a good idea to issue only write-commands to the graphics card. Basically that is "dont use glGet* commands at all", because the latency of those commands might be somewhat high.
That said option 1) is great if you just want to mask out stuff. As you are trying to make the cut part fall off this is really not an option, as you have to retrieve/reconstruct the vertices of that part.
I don't quite get the "tesselation" part of your second option, but if your primitive is a polygon and your cuts are straight lines, it is easy to calculate the 2 polygons after the cut. In fact the viewport clipping routine in OpenGL does that all the time and there is a lot of literatur, for example http://en.wikipedia.org/wiki/Sutherland-Hodgman
In the long term it is often way better to first build a (non-visual) model of what is going on in the application before visualizing.
The question probably applies to drawing systems in general. I was wondering how the undo functionality is implemented in PS. Does the program take snapshots of the canvas before each operation? If so, wouldn't this lead to huge memory requirements? I've looked into the Command pattern, but I can't quite see how this would be applied to drawing.
Regards,
Menno
It's called the command pattern. It's simple to implement as useful for any sort of editor.
Photoshop applies stacked transformations upon the original image. One opetation one command. It simply unapplies the transformation when you undo. So it just keeps the original and latest versions, but I guess it might cache the last few versions just for performance.
Since some operations will be non-reversable and as you say snapshoting the entire image every time would be out of the question then the only other alternative I can see would be a stack of deltas. A delta being the set of masks containing the modified pixels prior to the operation. Of course many operations may be reversable so their deltas could be optimised.
I'm not sure how Adobe Photoshop implements undo, but the Paint node within Apple Shake compositing application is pretty easy to explain:
Each stoke is stored as a series of points, along with some information like stroke-color, brush-size etc.
When you draw a stoke, the changes are made on the current image.
Every x strokes (10 I think) the current image is cached into memory.
When you undo, it redraws the last ~9 stokes on the previous cached image.
There are two problems with this:
When you undo more than 10 times, it has to recalculate the whole image. With thousands of strokes this can cause a several second pause.
With Shake, you save the setup file, containing the stroke information - not the actual pixel values. Then means you have to recalculate the whole image whenever you reopen the Paint node, or render the image (not nearly as big a problem as the undo thing, however).
Well, there is a third problem, that being Shake is horribly buggy and poorly implemented in many areas, the Paint node beign one of them - so I'm not sure how good an implementation this is, but I can't imagine Photoshop being too dissimilar (albeit far better optimised).
The easiest way I've found to solve this problem, though I don't know how Adobe tackles it, is to use a persistent data structure, like so:
You think of an image as a collection of image tiles, say 64x64 pixels each, and they get garbage collected or reference counted (ex: using shared_ptr in C++).
Now when the user makes changes to an image tile, you create a new version while shallow copying the unmodified tiles:
Everything except those dark tiles are shallow copied upon such a change. And when you do it that way, your entire undo system boils down to this:
before user operation:
store current image in undo stack
on undo/redo:
swap image at top of undo stack with current image
And it becomes super easy like that without requiring the entire image to be stored over and over in each undo entry. As a bonus when users copy and paste layers, it barely takes any more memory unless/until they make changes to that pasted layer. It basically provides you an instancing system for images. As yet another bonus, when a user creates a transparent layer that's, say, 2000x2000 pixels but they only paint a little bit of the image, like say just 100x100 pixels, that also barely takes any memory because the empty/transparent tiles don't have to store any pixels, only a couple of null pointers. It also speeds up compositing with such mostly-transparent layers, because you don't have to alpha blend the empty image tiles and can just skip over them. It also speeds up image filters in those cases as well since they can likewise just skip over the empty tiles.
As for PS actions, that's a bit of a different approach. There you might use some scripting to indicate what actions to perform, but you can couple it with the above to efficiently cache only modified portions of the image. The whole point of this approach is to avoid having to deep copy the entirety of the image over and over and blow up memory usage to cache previous states of an image for undoing without having to fiddle with writing separate undo/redo logic for all kinds of different operations that could occur.
Photoshop uses History to track their actions. These also serve as Undo as you can go back in history at any point. You can set the size of history in preferences.
I also suggest you look into Adobe Version Cue as a tool for retrospect undo or versions, it's built into the suite for that sole purpose. http://en.wikipedia.org/wiki/Adobe_Version_Cue