CodeFormer - AI Image Models Tool

Overview

CodeFormer is a Transformer-based blind face restoration model introduced in the NeurIPS 2022 paper “Towards Robust Blind Face Restoration with Codebook Lookup TransFormer.” It casts face restoration as a discrete codebook prediction task, using a learned codebook prior and a Transformer prediction network to recover high-frequency facial details and resolve severe degradations while offering a fidelity–quality tradeoff via a controllable parameter. The original paper demonstrates robustness on synthetic and real-world degradations and positions CodeFormer as competitive with state-of-the-art blind face restoration methods. ([arxiv.org](https://arxiv.org/abs/2206.11253)) The project is actively maintained as an open-source codebase with pre-trained checkpoints and multiple inference modes (cropped/aligned faces, whole-image enhancement, video input). Official deployments include a Replicate demo (hosted on NVIDIA L40S hardware) and a Hugging Face Space; the GitHub repository documents features such as face inpainting, colorization, optional background upsampling with Real-ESRGAN, multiple face-detector options (dlib/YOLO), and a fidelity weight (w) to balance identity preservation vs. visual quality. The authors release ~350–370 MB pretrained checkpoints and note an S-Lab (NTU) non-commercial license; community threads report wide adoption and some integration/packaging issues in third-party GUIs (e.g., WebUI/Docker), so expect occasional setup troubleshooting. ([github.com](https://github.com/sczhou/CodeFormer?utm_source=openai))

Key Features

• Transformer-based codebook lookup prior for robust blind face restoration
• Controllable fidelity parameter (w) to trade off identity and visual enhancement
• Whole-image enhancement with optional Real-ESRGAN background upsampling
• Face colorization and face inpainting modes for aligned/cropped faces
• Video input support for frame-by-frame face enhancement
• Multiple face-detector options (dlib, YOLO variants) for low-quality inputs
• Pretrained checkpoints and Docker/Colab examples for local deployment

Example Usage

Example (python):

# Example: run the hosted Replicate model (requires REPLICATE_API_TOKEN)
# Install: pip install replicate

import os
import replicate

# set your API token in env: export REPLICATE_API_TOKEN="your_token"
client = replicate.Client()

# Run the latest sczhou/codeformer model version (hosted on Replicate)
# You can control: codeformer_fidelity (0-1), face_upsample (bool), background_enhance (bool), upscale (int)
with open("input.jpg", "rb") as img_file:
    output = client.run(
        "sczhou/codeformer",
        input={
            "image": img_file,
            "codeformer_fidelity": 0.5,
            "face_upsample": True,
            "background_enhance": True,
            "upscale": 2
        }
    )

print("Output URI:", output)
# The API returns a URI pointing to the restored image. Download and inspect locally.

# Alternative (local run): the repo provides inference scripts such as
# python inference_codeformer.py -w 0.5 --has_aligned --input_path inputs/cropped_faces
# See the official GitHub README for local inference options and arguments.

Pricing

Replicate-hosted runs cost approximately $0.0043 per run (about 232 runs per $1) according to the model's Replicate listing. The code and checkpoints are open-source for local use, but the project uses an S‑Lab (NTU) non-commercial license and the Replicate deployment is indicated for research/demo purposes only.

Benchmarks

Replicate hosted cost per run (approx.): $0.0043 per run (~232 runs per $1) (Source: https://replicate.com/sczhou/codeformer)

Replicate inference latency (typical): ≈5 seconds per prediction (NVIDIA L40S, small input sizes) (Source: https://replicate.com/sczhou/codeformer)

Pretrained checkpoint sizes: codeformer.pth ~359 MB (others ~354–365 MB) (Source: https://github.com/sczhou/CodeFormer/releases)

Leaderboard / ranking: Ranked #1 on Blind Face Restoration (CelebA-Test) in PapersWithCode listing (Source: https://paperswithcode.com/paper/towards-robust-blind-face-restoration-with)

Key Information

• Category: Image Models
• Type: AI Image Models Tool