From Parsimony To Local Sensitivity: Generalization and Robustness In Modern Machine Learning
Abstract
Modern machine learning (ML) systems are approaching widespread deployment in high-stakes, real-world settings. As a result, a rigorous understanding of their capabilities and limitations is essential. This thesis investigates two complementary paradigms of evaluation in supervised classification: generalization and robustness. Generalization captures a model’s ability to extrapolate to unseen data, while robustness concerns its performance under adversarial perturbations. Each offers a partial lens on model reliability, and understanding their interplay is critical for the development of trustworthy ML systems.
A central theme of this thesis is that sensitivity—the rate of change of a model’s output with respect to perturbations in input or parameters—is a unifying quantity that governs both generalization and robustness. We introduce a systematic framework in which identifying structure and parsimony in the interaction between a model and data enables a localized, data-dependent notion of sensitivity. This localized measure of sensitivity provides a more faithful characterization of a model’s effective complexity. Building on it, we develop principled tools for evaluating generalization and adversarial robustness—offering sharper insights than prior approaches based on global sensitivity alone.
Alongside our contributions on sensitivity and evaluation, this thesis also addresses a separate foundational issue in robustness: the limitations of prevailing p-norm threat specifications, which are isotropic, global, and task-agnostic. As a result, they fail to distinguish between safe corruptions that preserve the true label and unsafe ones that alter it. To address this, we propose a novel, theoretically grounded alternative that is anisotropic, local, and task-specific. This framework offers practitioners a flexible, task-aligned approach to probing robustness beyond the constraints of traditional norm-based specifications.
