Investigating Autoregressive Diffusion Models: Internal guidance, training schedules, and black-box model behaviour

Abstract

Autoregressive Diffusion Models (ARDM) combine autoregressive sampling with diffusion-based learning, blending two major approaches in modern generative modelling. Although ARDM already benefits from both autoregressive factorization and diffusion-style denoising, its internal UNet representations remain difficult to interpret, and its training objective is primarily driven by scalar likelihood-based losses. This thesis investigates whether internal guidance and scheduling-based interventions can improve or alter ARDM behaviour under fixed-budget experimental conditions.

The thesis evaluates one baseline ARDM and forty-six modified ARDM variants on CIFAR-10. The experimental families include basic discriminator guidance, advanced discriminator and EMA-reference reasoning guidance, counterfactual feature-credit guidance, and scheduling-based timestep sampling. All final experiments use the same dataset, model architecture, training duration, and evaluation pipeline, and, therefore, the observed differences are attributed to the tested intervention rather than to model scaling or additional data.

The results show that no intervention universally dominates the baseline across all metrics. EMA-reference reasoning guidance achieves the strongest likelihood-oriented results, producing the best evaluation NELBO. Low-weight end-feature discriminator guidance produces the best FID, adaptive scheduling produces the best Inception Score, and scalar counterfactual guidance produces the best SSIM. These findings suggest that ARDM improvement is metric-specific: likelihood, sample quality, structural similarity, and training stability do not always move together. Beyond final metric comparison, the thesis provides controlled probes of internal ARDM behaviour through feature placement, EMA feature matching, counterfactual channel perturbation, gradient-alignment diagnostics, and timestep scheduling. The main contribution is therefore a controlled empirical analysis of how internal guidance and scheduling interventions shape ARDM behaviour, rather than a claim of a single universally superior model.