MirrorCheck: Efficient Adversarial Defense for Vision-Language Models

Our framework is designed to detect adversarial attacks, irrespective of the attacker's level of knowledge. In this scenario, there are two parties:

Attacker: The attacker's goal is to generate an adversarial image $x_{\text{adv}} = x_{\text{clean}} + \delta$ that causes the victim model to produce an incorrect caption or classification. The attack can be targeted, where the generated text matches a predefined adversarial target, or untargeted, where the model is simply forced to misinterpret or misdescribe the input image. In both cases, the perturbation $\delta$ is constrained within an $\ell_\infty$ or $\ell_2$ bounded adversarial budget. The adversary may have full white-box access or operate in a black-box setting.

Defender: The defender aims to correctly classify input images as either clean or adversarial by assessing the consistency between the model's interpretation of the input and a reference image generated from the model's textual output. The defender only assumes black-box access to the victim model and has no access to any ground-truth clean reference image.

Let $\mathcal{F}_\theta(x_{\text{in}}; p) \rightarrow t$ be the victim model (which can be a VLM or classification model producing a description text $t$ from input image $x_{\text{in}}$ under prompt $p$). Let $\mathcal{I}_{\phi}(x) \rightarrow z$ be a pretrained image encoder and let $G_{\psi}(t) \rightarrow x_{\text{gen}}$ denote a pretrained text-conditioned image generation (T2I) model.

The defender first obtains the output caption $t = \mathcal{F}_\theta(x_{\text{in}}; p)$, then synthesizes the reference image $x_{\text{gen}} = G_{\psi}(t)$. Semantic consistency is assessed by computing the cosine similarity between the feature embeddings:

Stochastic MirrorCheck & OTU Noise: To defend against adaptive white-box attackers who optimize perturbations to fool both the VLM and the detector, we introduce:

1. Stochastic Model Zoo selection: Randomly selecting T2I generators (Stable Diffusion, UniDiffuser, ControlNet) and image encoders (CLIP variants, OpenCLIP) per query.
2. One-Time-Use (OTU) Perturbations: Injecting unique Gaussian noise scale $\eta \sim \mathcal{N}(0, \sigma^2 I)$ directly to image encoder parameter weights before each query, creating a random projection that breaks gradient-based optimization attacks.

Adaptive Attacks (BPDA+EoT): We stress-test our defense by modeling a white-box adversary who has full access to the victim VLM, the alignment adapter, and the detection pipeline. Under the Backward Pass Differentiable Approximation (BPDA) and Expectation over Transformation (EoT) optimization, we observe that Stochastic MirrorCheck maintains high defense integrity. Specifically, while a single encoder is slightly susceptible, an ensemble of 7 or more encoders combined with One-Time-Use parameter noise (scale $\ge 5\times10^{-4}$) achieves a detection accuracy of over 98%, defeating the adaptive optimization loop.

Computational Complexity: Running on a single NVIDIA Quadro RTX A6000 GPU, the defense pipeline processes an image in 15 seconds (victim model captioning: 0.2s, T2I generation: 5s, similarity calculation: 10s). By reducing the T2I diffusion timesteps from 50 to 10, the total inference time drops to just 1.2 seconds with only a marginal reduction in detection AUROC, showing a practical path for low-latency production serving.