Camera stabilizer
Camera stabilizer made with OpenCV library, supporting streamed video stabilizing as well. The method is similar to Nghia Ho's implementation, but with a little twist, that enables support for streaming, so only previous frames can be used for calculating stabilization.
I am smoothing assuming the shaking can be compensated using rigid transformation, more specifically x, y offset and z rotation are smoothened.
The goodFeaturesToTrack method can be used to find strong corners at a frame (1st parameter), and store them in a vector of 2D points (2nd parameter). The rest of the parameters are respectively: maximum feature count, quality level and minimum distance between features.
Notice how I only work with those features, where match was found. I filter the rest.
Notice that some elements may be moving, as you can see in my example video above, the water moves in the background and the subject also moves a little bit. However, this method can estimate the most probable tranformation even though some features may represent an incorrect offset. I now know the delta translation and rotations between the current and previous frames. I keep adding this to the previous tranformation, resulting in the trajectory.
I am smoothing assuming the shaking can be compensated using rigid transformation, more specifically x, y offset and z rotation are smoothened.
The project
This project used to be my lab work at university. It uses the OpenCV open source library, which is commonly used in computer graphics for computer vision or machine learning. I wrote an extension for an experimental C++ node based video compositor software, which uses the Qt interface.Feature detection
Features in computer graphics are regions of the image with specific traits, which can be identified even after transformations (e. g. tranlation, rotation, scale...). Feature detection is the method of finding such regions. Feature matching is identifying the same feature at another image, after several transformations were made.The goodFeaturesToTrack method can be used to find strong corners at a frame (1st parameter), and store them in a vector of 2D points (2nd parameter). The rest of the parameters are respectively: maximum feature count, quality level and minimum distance between features.
goodFeaturesToTrack(PreviousFrameBW, PreviousFrameCorner, 200, 0.01, 30);
Feature matching
Now that I have our features linked to the previous frame, I want to identify the same on the current frame. This can be done with the calcOpticalFlowPyrLK method. The first three parameters are given, PreviousFrameCorner was just previously calculated. Now I want to identify these features in the current frame, the result will be output to CurrentFrameCorner. statusstores 1 if a feature was matched, and 0 if not found. err logs the errors.
cv::calcOpticalFlowPyrLK(PreviousFrameBW, CurrentFrameBW, PreviousFrameCorner, CurrentFrameCorner, status, err);
for(size_t i=0; i < status.size(); i++) {
if(status[i]) {
PreviousFrameFeatures.push_back(PreviousFrameCorner[i]);
CurrentFrameFeatures.push_back(CurrentFrameCorner[i]);
}
}
Notice how I only work with those features, where match was found. I filter the rest.
Features at previous and current frame
Estimating rigid transform
I have now two lists (PreviousFrameFeatures, CurrentFrameFeatures) of 2D points, and at each index, they represent the feature position on the corresponding frame. We can use this method to get an estimated transformation between the two frames:
CurrentTransformMatrix = estimateRigidTransform(PreviousFrameFeatures, CurrentFrameFeatures, false);
Notice that some elements may be moving, as you can see in my example video above, the water moves in the background and the subject also moves a little bit. However, this method can estimate the most probable tranformation even though some features may represent an incorrect offset. I now know the delta translation and rotations between the current and previous frames. I keep adding this to the previous tranformation, resulting in the trajectory.
Smoothing
If I would just keep fixing the image with the inverse trajectory, the shakes would be gone, but depending on the input image stream, the trajectory eventually might be as big the entire frame would be offseted from the screen. This is why I also linearly push the frame back to the center (can be parametrized after how many frames does it reach the center. Once the center is reached, the trajectory is reset, and started over. This way, the image is always kept in frame, except at high shakes.
Trajectory and smoothened trajectory
The inverse trajectory (red) and the vector pusing towards the center result in a smoothed trajectory (green).Trimming
Even though now our subject is smoothly pushed back to the center, at large delta steps, the edges of the stabilized frame may be visible. This is why I introduced trimming.
With and without trimming
With a specific crop radius (can be parametrized) the part of the image, where the edges are visible can be cropped out. The optimal value for this is the smallest possible, where these edges are not visible.