Geometry transformation-based adaptive in-loop filter

Recently, adaptive in-loop filter (ALF) for image/video coding has attracted increasing attention by its proven capability in improving coding performance. ALF is aiming to minimize the mean square error between original samples and decoded samples by using Wiener-based adaptive filter. Samples in a picture are classified into multiple categories and the samples in each category are then filtered with their associated adaptive filter. The filter coefficients may be signaled or inherited to optimize the tradeoff between the mean square error and the overhead. In this paper, a Geometry transformation-based ALF (GALF) scheme is proposed to further improve the performance of ALF, which introduces geometric transformations, such as rotation, diagonal and vertical flip, to be applied to the samples in filter support region depending on the orientation of the gradient of the reconstructed samples before ALF. With the introduction of geometric transformations, more spatial adaptation is supported without excessive signaling of filter coefficients. The experimental results show that GALF outperforms the existing ALF techniques and it has been adopted by the JEM reference software used as the test platform for future video coding technology exploration in JVET.