BRAIN Journal Interactive Companion

Enhancing Image Inpainting Using Hybrid Architecture Combining CNN-Vision Transformer and GAN

A CNN-ViT-GAN framework joins local feature extraction, global contextual reasoning, and adversarial texture refinement in one reconstruction pipeline.

Simge Coşkun · Ali Hakan Işık

PSNR ↑36.2212dB

SSIM ↑0.9779structure

LPIPS ↓0.0260perception

Trace the architecture

224 × 224Reconstruction signal7 → 3 channels

Overview

Three complementary systems, one completion objective

Missing regions are reconstructed without sacrificing the known image. Gated convolutions encode local structure, a Vision Transformer models distant relationships, and a PatchGAN-like discriminator supplies local realism during training.

Local structure

Four gated convolution blocks learn hierarchical features while reducing spatial resolution.

CNN encoder

Global context

Two Transformer layers with eight attention heads process 196 tokens at the bottleneck.

ViT · 256 dimensions

Perceptual detail

Adversarial and feature-matching objectives guide realistic local textures without replacing reconstruction loss.

PatchGAN-like discriminator

CelebAprocessed subset

224 × 224input resolution

200training epochs

15pretraining epochs

8batch size

AdamWoptimizer

Architecture Workbench

Follow the reconstruction signal

Select a stage to inspect its role, dimensions, and connection to the complete system.

Input assembly

Seven-channel tensor

Known pixels preserved I_comp = M ⊙ I_pred + (1 − M) ⊙ I_input M = 1 missing · M = 0 known

Ablation Atlas

What changes when a component disappears?

Compare every controlled configuration against the complete design. Higher PSNR and SSIM are better; lower LPIPS is better.

Complete system

Proposed design

Balanced reference

Exact quantitative comparison
Configuration	PSNR ↑	SSIM ↑	LPIPS ↓

Qualitative Evidence

One metric cannot describe every reconstruction

Inputs, masks, error maps, and outputs reveal texture and boundary behavior that aggregate scores can miss.

Qualitative comparison of proposed design and four ablation variants, showing input, masked input, reconstruction error map, and reconstructed output — **Qualitative comparison across five configurations** Rectangular and irregular masks expose differences in structural consistency, texture generation, and transition quality.

Metric tension

Without GAN

PSNR, SSIM, and LPIPS improve numerically, while reconstructed regions appear smoother and more conservative in texture synthesis.

Largest quantitative drop

Without ViT

PSNR falls to 34.3591, with weaker global structure and semantic coherence across distant regions.

Perceptual supervision

Without perceptual loss

LPIPS rises to 0.0352, the highest value among the tested configurations.

Objective Simulator

Operate the defined calculations

Enter component values to see how the fixed loss weights shape the generator objective, or convert mean squared error into PSNR for normalized images.

Generator loss composer

Adversarial multiplier α remains explicit.

Adversarial multiplier α 0.00

Total objective L_G0.0000

Live equation Set values to compose the objective.

Configuration comparison

Compare A/B

Save two parameter states and inspect the objective difference.

Configuration ANot saved—

Configuration BNot saved—

Save A and B to calculate the difference.

PSNR calculator

MAX_I = 1 for normalized images.

Mean squared error Logarithmic MSE scale10⁻³

30.00dB

Lower MSE produces higher PSNR.

MSE–PSNR curveHigh fidelity

Reverse calculator

Target PSNR → required MSE

Target PSNR dB

Required MSE3.1623e-4

Illustrative normalized range (0–1). The paper defines the weights, but does not report a universal “normal” raw loss value.

L_holeError measured inside the missing region. It receives full weight because this is the area the model must reconstruct.0.10 +0.20 × L_validError measured on known pixels. The 0.20 weight preserves surrounding context without overpowering the masked region.0.05 = 0.0000

Interpretive Boundary

Strong on the tested setting, bounded beyond it.

Experiments used only a processed CelebA subset.
All evaluations used 224 × 224 images.
Transformer cost may constrain real-time or resource-limited use.
Adversarial learning can increase training sensitivity and instability.
PSNR, SSIM, and LPIPS do not fully capture perceived quality.