I have a video where somebody walks in with a shirt on with text. I want to remove the text on every frame but because the person moves and turns i am looking for a way how i can automate that. so that the program understands which part has to go and does that on every frame. Is there a way of doing this automatically?
Full automation is not likely, but a mix of automation and elbow grease might do the trick. The exact mix depends on the shot.
Motion tracking the text on the footage layer through Mocha (Animation -> Track in mocha ae) will give you position/scale/rotation data that you can then apply to a second layer with a fix implemented - a blur or color layer for example. The elbow grease comes in here, where you will most likely need to roto the fix as the person turns and the text changes shape, etc.
Again, depending on the shot it may be more elbow grease than you want. There is no alternative, however - this is the process vfx artists follow at every level of the industry.
Related
I am trying to improve my webcam based OpenCV mouse controller for disabled people (MFC C++ application): https://preability.com/face-controlled-mouse/
The cursor moves, when a person moves her/his head, clicks when smile, etc.
Controller finds face area then use goodFeaturesToTrack, cornerSubPix and calcOpticalFlowPyrLK.
In general, I managed to stabilize cursor if lighting is good.
What I use now:
Evaluating and filtering the direction of the each corner point movement.
Spreading the corner points all over the face area for cv::goodFeaturesToTrack() helped a little too.
EWMA (or Kalman) filter for the cursor position.
I’ve included equalizeHist() for the face ROI. The detector performed much better in low light conditions.
In addition, I tried morphology operations of OpenCV without improvement.
However, the corner points still dance in uneven lighting.
I can see that similar old program eViacam has preprocessing module for webcam Creavision (old too) and the corner points are more stable.
Please advise what can be done with the input Mat? Or how can the video be processed with reasonable CPU loading?
Now I can answer my own question. Christoph Rackwitz gave me some good advice:
don’t track the whole head. track each feature. and don’t use those trackers, they’re too complex. use MOSSE. it’s dumb but very precise, as long as the object (which should be a tiny feature on the face) doesn’t change much.
MOSSE approaches optical flow. methods to calculate optical flow work like MOSSE, except they use simpler math and smaller regions, hence the result is noisier. MOSSE uses a larger area (for a single track/point of course) and more sophisticated math, for a more accurate result.
When MOSSE algorithm tracks “corner points”, cursor moves much more smoothly. There was a slight issue with discrete movement as the object rectangles moved the same number of pixels at the same time. Cursor moved in leaps. So, I had to use filter on each tracked point. Anyway, as you can see in the video, the CPU load did not increase comparing to Lukas-Kanade optical flow algorithm + filtration only cursor position. In good light, the difference is also very noticeable.
https://www.youtube.com/watch?v=WKwuas0GVkA
Lucas-Kanade optical flow:
goodFeaturesToTrack,
cornerSubPix,
calcOpticalFlowPyrLK,
cursor EWMA filter
MOSSE object tracking:
goodFeaturesToTrack,
cornerSubPix,
TrackerMOSSE,
all points EWMA filtration
And of course I had to remember to include the tracking453.lib to Linker when add legacy Tracker. I spent half a day googling the “unresolved external symbol LNK2001 error”. For some reason including a tracker from the core library (cv::Tracker) does not result such compilation error, so it is confusing.
I am using the camera tracker on a clip of roughly 10000 frames 740x360, 50fps, prores codec. There are a lot of trackable features in the clip.
When I use the tracker on only a part of the video the tracking completes in less than a minute. But when I try to camera solve the whole clip solving is not done even after 10 hours (I let it run over night).
I was thinking that maybe there is a hard to solve part of the video, so I cut the clip up into multiple parts and solved each of them separately, they all got solved just fine in a few minutes, so I don't think that is the case.
Do any of you know of similar occurrences or of a way to solve that problem?
Something that helps a lot is using super high shadow/highlight and contrast in a pre-comp so that ae has more clarity between different parts of the shot and therefore tracks better.
Pre-compose your shot (right click and select pre-compose)
Apply Shadow/Highlights using Shadow/Highlights and/or lumetri color
Apply contrast once, twice, or more if needed to get strong edges for ae to see everything and track the shot.
Note: You can remove these effects later, this is just to help ae "see" better to track the shot.
Hmm not sure if I understand but the tracker is not supposed to be over the whole image. You need to chose one specific part that is in a contrast to the surroundings and chose that as the tracking point.
I have a movie in the after effects that dosent have a KEY color of background, but a still background.
I want to detect the motion of 2 persons walking in front of this still background and bring them to the front, so I can create effects on the background.
Is that possible to do? Witch Effect do I use?
This works with a technique called difference matting. It never works well even if you have a solid clean plate. your best bet is to rotobrush or just mask them out manually. if you want to try difference matting you can set the footage layer with the people in it to difference blending mode and put a clean plate below it.
you can use the rotobtush that allows to separate elements from a static background. Works better if:
you have a clean background
good quality videos
front object needs to be cutted as big as possible
This method works well with a locked down camera. Depending upon how active people are and how still the background - you can remove the people by cutting and assemble pieces of the background to create a background layer without the people... Once you have a clean plate - then use a difference matte to subtract the background from your clip which will leave anything that moves (read people). This will work well if the hue/values between the background and the people are different.
Doing this in conjuncture with animated masks gives good results and can give you a jump on your rotobrushing to improve your edges.
If all else fails the Mocha plug works well as long as you break your masks up into lots of parts to follow the planes of your actors.
I have never found doing this is a single tool solution.
It is not impossible, but I don't think it could give you the best result.
Anyway, you should mask them out manually. but not keyframe by keyframe. there is a technic for it. for setting the mask keyframes, first, you should set 3 keyframes in first, middle and end of your footage.
after masking three one, you do it the same for first keyframe and the middle one.
it means between the first keyframe and the middle keyframe, you should set a key.
Do the same for middle and end key.
this technic could save your time and make masking with less fault.
by the way, you can use Mocha pro as well for tracking the persons.
For an experiment, we are looking for a way to automatically compare two hand (mouse-)drawn images. These images are, for instance, drawn on an HTML5 canvas element and we need some way to see whether the pictures roughly match.
So, if someone draws a house, we need to test whether the second drawing looks like the first house. It doesn't matter what exactly is in the image, but we only need to know whether the two images look alike. I.e., we want to know whether the person drawing the picture, can redraw roughly the same picture. The exact orientations of the lines, the size of the image or the position of the picture on the canvas shouldn't matter.
Is there, by any chance, a library or project that does this?
Yes absolutely
With API provided by Face.com and with imagemagick Both are providing
api's
http://www.imagemagick.org/api/compare.php
FaceAPI - Track Faces from a Webcam: http://faceapi.com
it might be useful for your requirement
Ultimately, we didn't find a library that does exactly the same, so we have modified the original project's goals and have crowdsourced the comparison that needed to be done.
You can probably also use Mechanical Turk or something similar for the time being, depending on how much you wish to pay for it.
I think researchers are currently be working on technical solutions to the issue described above.
The question probably applies to drawing systems in general. I was wondering how the undo functionality is implemented in PS. Does the program take snapshots of the canvas before each operation? If so, wouldn't this lead to huge memory requirements? I've looked into the Command pattern, but I can't quite see how this would be applied to drawing.
Regards,
Menno
It's called the command pattern. It's simple to implement as useful for any sort of editor.
Photoshop applies stacked transformations upon the original image. One opetation one command. It simply unapplies the transformation when you undo. So it just keeps the original and latest versions, but I guess it might cache the last few versions just for performance.
Since some operations will be non-reversable and as you say snapshoting the entire image every time would be out of the question then the only other alternative I can see would be a stack of deltas. A delta being the set of masks containing the modified pixels prior to the operation. Of course many operations may be reversable so their deltas could be optimised.
I'm not sure how Adobe Photoshop implements undo, but the Paint node within Apple Shake compositing application is pretty easy to explain:
Each stoke is stored as a series of points, along with some information like stroke-color, brush-size etc.
When you draw a stoke, the changes are made on the current image.
Every x strokes (10 I think) the current image is cached into memory.
When you undo, it redraws the last ~9 stokes on the previous cached image.
There are two problems with this:
When you undo more than 10 times, it has to recalculate the whole image. With thousands of strokes this can cause a several second pause.
With Shake, you save the setup file, containing the stroke information - not the actual pixel values. Then means you have to recalculate the whole image whenever you reopen the Paint node, or render the image (not nearly as big a problem as the undo thing, however).
Well, there is a third problem, that being Shake is horribly buggy and poorly implemented in many areas, the Paint node beign one of them - so I'm not sure how good an implementation this is, but I can't imagine Photoshop being too dissimilar (albeit far better optimised).
The easiest way I've found to solve this problem, though I don't know how Adobe tackles it, is to use a persistent data structure, like so:
You think of an image as a collection of image tiles, say 64x64 pixels each, and they get garbage collected or reference counted (ex: using shared_ptr in C++).
Now when the user makes changes to an image tile, you create a new version while shallow copying the unmodified tiles:
Everything except those dark tiles are shallow copied upon such a change. And when you do it that way, your entire undo system boils down to this:
before user operation:
store current image in undo stack
on undo/redo:
swap image at top of undo stack with current image
And it becomes super easy like that without requiring the entire image to be stored over and over in each undo entry. As a bonus when users copy and paste layers, it barely takes any more memory unless/until they make changes to that pasted layer. It basically provides you an instancing system for images. As yet another bonus, when a user creates a transparent layer that's, say, 2000x2000 pixels but they only paint a little bit of the image, like say just 100x100 pixels, that also barely takes any memory because the empty/transparent tiles don't have to store any pixels, only a couple of null pointers. It also speeds up compositing with such mostly-transparent layers, because you don't have to alpha blend the empty image tiles and can just skip over them. It also speeds up image filters in those cases as well since they can likewise just skip over the empty tiles.
As for PS actions, that's a bit of a different approach. There you might use some scripting to indicate what actions to perform, but you can couple it with the above to efficiently cache only modified portions of the image. The whole point of this approach is to avoid having to deep copy the entirety of the image over and over and blow up memory usage to cache previous states of an image for undoing without having to fiddle with writing separate undo/redo logic for all kinds of different operations that could occur.
Photoshop uses History to track their actions. These also serve as Undo as you can go back in history at any point. You can set the size of history in preferences.
I also suggest you look into Adobe Version Cue as a tool for retrospect undo or versions, it's built into the suite for that sole purpose. http://en.wikipedia.org/wiki/Adobe_Version_Cue