Access Type

Open Access Dissertation

Date of Award

January 2025

Degree Type

Dissertation

Degree Name

Ph.D.

Department

Electrical and Computer Engineering

First Advisor

Nabil J. Sarhan

Abstract

Deep Learning (DL) models are deployed ubiquitously, as they power a wide range of critical applications, including image classification, fraud detection, autonomous vehicles, robots, and NLP. However, their massive size and huge memory footprint (overparameterization) represent a serious challenge to the efficient deployment of such models, especially in resource-scarce environments such as wearable devices, smartphones, edge devices, and embedded systems. Therefore, model compression techniques are typically used to shrink the model size to the currently available computational and memory budget and to accelerate training and inference without sacrificing model accuracy and performance. Therefore, the DL research community considers model compression an increasingly important field.

DL model compression techniques, such as pruning, regularization, quantization, and knowledge distillation, have recently undergone significant advances. However, the robustness of the compressed DL models is yet to be fully understood and comprehensively addressed. Adversarial robustness and out-of-distribution robustness of compressed DL models have received more attention than other aspects of DL model robustness, such as class imbalance. Class imbalance, a well-known DL research problem, refers to the unbalanced sample distribution in the training dataset is not balanced, as some classes have more samples (majority classes) than other classes (minority classes).

This PhD dissertation analyzes and quantifies the effect of class imbalance in training datasets on the robustness of compressed DL models. We first define empirical robustness and use it as a metric to measure the robustness of compressed DL models against class imbalance. We find that compressed DL models are not robust against class imbalance. We also show how different compression techniques, namely pruning, quantization, and knowledge distillation, have different impacts on the class imbalance robustness of DL models. We also demonstrate the impact of different class imbalance ratios and class imbalance types on the class imbalance robustness of compressed DL models.

We propose and implement a robustness-aware Bayesian compressive sensing-based pruning framework to address the problem discussed above. We estimate the criticality of a subset of the model’s parameters to the performance of each class (per-class F-1 score). We leverage Bayesian learning to stochastically select such parameters (measurements). We then utilize compressive sensing to obtain the criticality of the remainder of the model’s parameters. We train ResNet-20 on imbalanced CIFAR-10 and use our proposed framework to prune the model. Our results demonstrate that this approach preserves model robustness against varying degrees of class imbalance in the training dataset under mild, moderate, and severe pruning ratios.

Download

Share

COinS