Meitu and UCAS Algorithm Boosts AI Video Restoration Speed by 10x, Fixing Overexposure Frame-by-Frame

Submitted by BlazeBVD Team

Hey everyone, here’s a new trick to eliminate video flickering (such as sudden white flashes)!

Recall that when watching an old movie or video shot on a mobile phone, the image occasionally exhibits phenomena such as flickering or inconsistent colors.

To address these issues, researchers from Meitu Image Research Institute, University of Chinese Academy of Sciences, and Sichuan University have proposed a new algorithm.

The new algorithm, “BlazeBVD,” can automatically eliminate flickering in videos. It is also extremely fast, reportedly 10 times faster than existing methods.

Even better! This method does not require prior knowledge of the specific type or degree of video flickering.

In other words, it is “blind” and can be applied to a wide variety of videos.

Now, you don’t need to worry even if lighting conditions change during shooting or if camera hardware lags behind.[doge]

The related paper has been accepted by ECCV 2024, a top conference in computer vision.

If you’re interested, let’s take a closer look~

How Does BlazeBVD Eliminate Video Flickering?

First, inspired by the classic STE (Scale-Time Equalization for Deflickering), BlazeBVD introduces a histogram-assisted solution.

An image histogram is defined as the distribution of pixel values and is widely used in image processing to adjust an image’s brightness or contrast.

For example, an image histogram acts like a statistical table, telling us how many pixels of different brightness levels exist in a photo.

△ Image generated by Claude 3.5 Sonnet

1. STE works by analyzing the histogram of each frame in a video, then using a method called Gaussian filtering to smooth these histograms. This initially corrects image frames with sudden shifts in histogram distribution, making the scene appear more stable and reducing flickering.

Although STE is only effective for slight flickering, it validates:

• Histograms are more concise than raw pixel data and can capture brightness changes and flickering in videos more effectively.
• Smoothing histograms can reduce video flickering, making the video look more stable.

Therefore, leveraging STE and histogram cues to improve the quality and speed of blind video deflickering is feasible.

Specifically, BlazeBVD consists of three stages.

Detailed Explanation of BlazeBVD’s Three Stages

Just like a doctor treating a patient, BlazeBVD first examines each frame of the video.

It introduces STE to correct the sequence of histograms for video frames in the illumination space.

Then, it extracts important information from the processed frames, such as which frames flicker most significantly (set of singular frames), where lighting needs adjustment (filtered illumination map), and where overexposure or underexposure occurs (exposure map).

Next, BlazeBVD begins restoration.

On one hand, BlazeBVD uses a tool called the Global Flicker Removal Module (GFRM), which utilizes the previously extracted illumination map to adjust the lighting of the entire video, ensuring that the brightness and color of each frame look natural.

On the other hand, for specific local areas requiring attention, such as regions with overexposure or underexposure, BlazeBVD employs a Local Flicker Removal Module (LFRM). This module uses optical flow information (similar to tracking object movement in the video) to restore details in these areas.

After completing this step, BlazeBVD performs final refinement.

It introduces a lightweight temporal network (TCM), which acts as the video’s “beautifier,” ensuring that each frame transitions smoothly visually without abrupt changes.

To further enhance video consistency, BlazeBVD designs a special scoring system (adaptive mask-weighted loss). This system scores each frame to ensure visual consistency across the entire video, making it look more fluid and natural.

Thus, BlazeBVD completes the entire “diagnosis and treatment” process.

Experimental Results

So, how effective is BlazeBVD?

Let’s directly compare the results of existing methods versus BlazeBVD on the blind video deflickering task:

Here, Deflicker refers to existing methods, GT (Ground Truth) represents the ideal flicker-free video, and KL divergence indicates the difference between the processed video and the ideal flicker-free video. A larger KL value signifies a greater difference.

It can be seen that BlazeBVD effectively restores illumination histograms while avoiding color artifacts and distortion (e.g., the man’s arm in the second column).

Furthermore, here is a quantitative comparison with baseline methods:

BlazeBVD achieves higher scores in PSNR (Peak Signal-to-Noise Ratio, where a higher value indicates better video quality) and SSIM (Structural Similarity Index, where a value closer to 1 indicates better video quality), and a lower score in Ewarp (where a lower value indicates greater video coherence and consistency).

In short, BlazeBVD outperforms existing baseline methods.

To intuitively demonstrate this difference, here is a visual comparison between BlazeBVD and the baseline method:

Ablation experiments also verify the effectiveness of the modules designed in BlazeBVD:

In summary, through comprehensive experiments on synthetic videos, real-world videos, and generated videos, BlazeBVD demonstrates superior qualitative and quantitative results, with inference speeds 10 times faster than state-of-the-art models.

The related paper is currently available. Those interested can learn more.

Paper:
https://arxiv.org/html/2403.06243v1