Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.
We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.
This chapter explores innovative regularization strategies designed to enhance the resilience and accuracy of deep learning models against adversarial attacks, focusing particularly on graph-based methods. At the forefront is Graph Regularized Adversarial Training (GReAT), a pioneering approach that integrates graph-based regularization with adversarial training to exploit the underlying data structure for improved model robustness. By leveraging relationships among related samples, including augmented ones, GReAT significantly enhances performance. This chapter introduces enhanced versions of GReAT (Augmented GReAT) that incorporate standard data augmentation techniques. These extensions further enrich model training by diversifying the training examples within the graph framework. These methods aim to fortify models against sophisticated adversarial perturbations and improve generalization to new unseen datasets. Our comprehensive evaluations on benchmark datasets such as CIFAR-10 and SVHN demonstrate that these techniques, particularly Augmented GReAT, outperform traditional adversarial training methods. Augmented GReAT shows significant performance gains, achieving up to a 12% increase against PGD attacks on CIFAR-10 and 6.3% on SVHN. This chapter details the underlying theories and implementation of these augmented graph-based regularization techniques and thoroughly analyzes their impact through extensive experimental results. By integrating graph structures and data augmentation into the training process, we showcase significant advancements in neural networks’ robustness and predictive accuracy, setting new benchmarks in the field.
Electrical and Computer Engineering Department, University of Delaware, Newark, DE, USA
Kenneth Barner
Electrical and Computer Engineering Department, University of Delaware, Newark, DE, USA
*Address all correspondence to: sbayram@udel.edu
1. Introduction
Deep learning (DL) has achieved impressive results in image and text classification, speech recognition, image generation, and natural language processing [1, 2, 3]. Supervised learning, where models are trained on labeled datasets, is particularly successful. It aims to predict new, unseen data based on patterns learned from labeled data, with deep neural networks adjusting weights and biases through back-propagation.
Semi-supervised learning combines supervised and unsupervised techniques [4]. Here, models are trained on datasets with both labeled and unlabeled instances, using labeled data to predict new unseen data. This approach is useful when labeled data are limited, but unlabeled data are abundant. Algorithms such as label propagation [5, 6], pseudo-labeling [7], transductive SVMs [8], and self-training [9] are commonly used, with label propagation showing outstanding performance.
Despite their success, deep learning models are vulnerable to adversarial examples, crafted by adding small perturbations to input data that cause misclassification while appearing similar to the original [10, 11, 12]. This vulnerability has spurred significant research on adversarial attacks and defenses. Adversarial attacks such as PGD [13], CW [11], BIM [14], and others demonstrate the susceptibility of AI systems, especially in image classification [11, 14, 15, 16, 17, 18, 19, 20].
Data augmentation, a technique to improve model generalization by generating additional training samples, includes transformations such as rotations, scaling, and cropping [21]. Adversarial examples can also be seen as a form of data augmentation [22]. By incorporating adversarial perturbations, models can be trained to recognize and withstand these modified inputs, thus enhancing their robustness.
Adversarial defense mechanisms fall into three categories. The first focuses on pre-processing techniques to remove adversarial perturbations, such as feature denoising [23], Fourier filtering [24], and random resizing/padding [25]. The second focuses on neural network architecture and training processes, including modifications of the activation function [26], distillation [27], and new loss functions [28]. The third combines these approaches.
The authors of [29] introduce TAAT and ATA, which leverage topological information to maintain consistency between natural and adversarial examples, along with a Knowledge-Guided training scheme for stability and efficiency. TRADES [30] decomposes prediction errors into natural and boundary errors, using a differentiable upper bound for its defense method.
Adversarial logit pairing (ALP) [31] improves accuracy on adversarial examples by encouraging logit similarity, while TLA training [32] uses metric learning to enhance robustness. Adversarial Contrastive Learning (ACL) [33] integrates adversarial perturbations into a contrastive learning framework for robustness and label efficiency.
Unlabeled samples are effective in improving deep learning performance [5, 34, 35] and adversarial robustness [36]. Motivated by this, earlier, we proposed Graph-Regularized Adversarial Training (GReAT) to enhance robustness [37]. GReAT uses structural information from input data, constructing a graph with clean data and adversarial neighborhoods to incorporate structural information into training.
In this chapter, we extend GReAT by incorporating standard data augmentation techniques, enriching the diversity of training examples within the graph framework. This Augmented GReAT approach aims to improve model robustness against adversarial attacks further while enhancing generalization to new, unseen datasets.
We evaluated Augmented GReAT on TensorFlow’s flower dataset [38], CIFAR-10 [39], and SVHN [40] datasets, comparing it with state-of-the-art methods. The results show that Augmented GReAT consistently outperforms the baselines in accuracy and robustness. Our contributions include:
Integrating graph structure and standard data augmentation into adversarial training;
Enhancing feature extraction by incorporating neighboring information and augmented samples;
Leveraging training sample structures and augmented data to enhance learning capabilities;
Demonstrating significant advantages over state-of-the-art methods in robustness and generalization.
This section discusses the relevant background and related work, focusing on data augmentation, deep learning and semi-supervised learning, adversarial learning, and graph-based semi-supervised learning.
2.1 Data augmentation techniques
Data augmentation involves artificially increasing the size of a dataset by applying various transformations to existing samples. Data augmentation techniques, such as geometric transformations, noise addition, and color adjustments, enhance training data diversity and model generalization. Geometric transformations alter spatial properties (e.g., scaling, flipping, rotations, translation), color transformations modify color properties (e.g., brightness, contrast, color shifting, saturation), and noise injection adds noise (e.g., Gaussian, salt-and-pepper, speckle, Poisson) to improve robustness. This technique enhances the generalization and robustness of machine learning models by exposing them to a wider range of data variations.
2.2 Deep learning and semi-supervised learning
Deep learning models operate as complex non-linear mapping systems between inputs and outputs. They consist of several layers of neurons linked by activation functions. These networks extract features from input data and use them to predict labels. Training neural networks involves extensive labeled datasets, allowing them to learn and improve their performance over time through backpropagation algorithms.
The traditional DL prediction process can be described by the following equation:
Y=fXθb,E1
where X is the input data passed into the neural network function f, θ are the weights of the neural connections, and b are the biases. The output Y is the prediction generated by the neural network after processing the input data through its layers of neurons.
Semi-supervised learning leverages both labeled and unlabeled data to train the model. The prediction process for semi-supervised learning is expressed as:
Y=fXlXulθb,E2
where Xl represents the labeled data and Xul denotes the unlabeled data. The weights θ and biases b are similar to those used in supervised learning. The output Y is the model’s prediction after processing labeled and unlabeled data through its neuron layers. In semi-supervised learning, a label assignment procedure is typically applied to the unlabeled data, using a smoothing function or similarity metrics to assign labels from the most similar labeled samples to the unlabeled ones [6].
2.3 Adversarial learning
Adversarial learning is a key area of research in machine learning, focused on enhancing model robustness against intentional perturbations. These perturbations, often subtle, can significantly impair model performance, leading to erroneous predictions with potentially severe consequences in areas such as autonomous driving, healthcare, and finance [10, 13].
The main goal of adversarial learning is to create models that can withstand adversarial attacks, which exploit model vulnerabilities [10].
2.3.1 Adversarial examples as data augmentation
Adversarial examples are artificial data points that introduce slight perturbations to natural instances to mislead a model into making incorrect predictions. These examples are crucial for understanding and improving model robustness, often used as a form of data augmentation during training.
Given a natural data instance x with a true label y, an adversarial example x′ is generated to be almost identical to x according to a specific distance metric, but leads the model to predict an incorrect label, such that fx′≠y [41]. Formally, an adversarial example is defined as:
x′:Dx′x<ε,fx′≠y,E3
where D⋅⋅ denotes a distance metric, such as the ∥⋅∥2 norm, and ε is a small threshold.
The Fast Gradient Sign Method (FGSM) [10] is a popular technique to generate adversarial examples. It perturbs the input data by adjusting each pixel in the direction of the gradient of the loss function with respect to the input, scaled by a small factor. This process is formalized as:
x′=x+ε⋅sign∇xℓθxy,E4
where ∇xℓθxy represents the gradient of the loss function ℓ with respect to the input x, and ε is the magnitude of the perturbation.
Another effective method is Projected Gradient Descent (PGD) [13], which iteratively applies small perturbations and projects the resulting adversarial example back onto the valid input space defined by a norm-ball constraint:
xt+1′=Projx+ℬxt′+α⋅sign∇x′ℓθx′y,E5
where α is the step size, and Projx+ℬ denotes the projection onto the norm-ball ℬ centered at x.
Adversarial training is formalized as a min-max optimization problem aimed at finding the optimal model parameters θ that minimize the loss over the worst-case adversarial examples:
θ∗=argminθ∈Θ1L∑i=1LmaxDxi′xi<εℓadvθxi′yi.E6
This approach enhances the model’s defense against known attacks and its robustness to previously unseen perturbations, making adversarial training a cornerstone of defense strategies in adversarial learning.
2.4 Graph-based semi-supervised learning
Graph-based semi-supervised learning effectively utilizes both labeled and unlabeled data to train deep learning (DL) models [34, 35, 42, 43, 44, 45]. This approach combines a small set of labeled data with a larger set of unlabeled data to learn the underlying graph structure of the dataset. In this context, a graph is represented as G=VEW, where V are the data points (vertices), E are the edges connecting these points, and W is the edge weight matrix that captures the similarity between the data points. The edges of a graph are constructed on the basis of a similarity metric. Graph-based semi-supervised learning aims to use this graph structure and the labeled data to infer labels for the unlabeled data points. This is typically achieved by propagating labels from labeled to unlabeled data points through the similarity graph that spans the entire dataset [6, 45, 46, 47].
In graph-based semi-supervised learning, label propagation is a widely used technique for classifying nodes within a graph when only a few nodes are labeled. This method starts with the labeled nodes and propagates their labels to neighboring nodes. The labels are iteratively spread until the process converges, resulting in a fully labeled graph. The loss function for graph-based semi-supervised learning is given by [6]:
∑i=1Lℓθxiyi+λ∑i,jwi,j∥hxi−hxj∥2,E7
where the first term denotes the standard supervised loss, and the second term represents the neighborhood loss penalty. Here, wi,j captures the similarity between different instances, and λ controls the influence of neighborhood regularization. When λ=0, the loss term reduces to the standard supervised loss. The penalty magnitude depends on the similarity between instance xi and its neighbors. Additionally, h is a lookup table containing all samples and similarity weights, which can be obtained through a closed-form solution, as described in [47].
The authors of [42] proposed replacing the lookup tables with sample embeddings, extending the regularization term in Eq. (7) to λ∑i,jαi,j∥gθxi−gθxj∥2, where gθ⋅ denotes the embeddings of the sample generated by a neural network. Transitioning from h to g imposes stronger constraints on the neural network, as observed in [6].
We extend Eq. (7) by replacing the lookup table term with an embedding-based regularization component that defines a general neighbor similarity metric:
∑i=1Lℓθxiyi+λ∑i=1wiDgθxiNgθxi.E8
In Eq. (8), N denotes the neighbors of a given sample xi, wi represents the edge weights between the sample xi and its neighbors, and D represents the distance metric between embeddings.
This framework provides a more flexible and general approach to integrating neighbor similarity metrics into graph-based semi-supervised learning, thereby enhancing the model’s capacity to generalize effectively from labeled and unlabeled data.
In this section, we integrate the adversarial learning process [13] into the graph-based semi-supervised learning framework [6, 42, 45] to take advantage of both adversarial training and semi-supervised learning. The main framework of GReAT augmented with standard data augmentation is shown in Figure 1.
Figure 1.
Augmented GReAT framework.
The feature space encompasses clean samples and adversarial examples created through adversarial regularization and neighbor similarities. Clean samples consist of original labeled training samples and standard augmented samples that we can show as x=xorg+xaug, where xorg are original samples and xaug are augmented samples. The feature space is crucial for identifying nearest-neighbor samples. When we feed a batch of input samples to the neural network, it includes the original samples and their corresponding neighbors. In the final layer of the neural network, we derive a sample embedding for each of these samples.
The training objective for regularization includes two components: the supervised loss and the neighbor loss, which accounts for neighbor-related loss. In other words, it considers the impact of neighbors on the overall training objective. Thus,
ℒGReAT=ℒadv+λℒN,E9
where ℒadv represents the supervised loss from training labels of clean samples and their adversarially perturbed versions, and ℒN represents the neighbor loss, which includes the loss from the clean training samples and adversarially perturbed samples.
We consider similar instances as neighbors of sample x in the graph regularized semi-supervised learning case. In our case, we consider an adversarial example, x′, in addition to a neighbor of sample x. Next, we extend Eq. (8) by including adversarial and adversarial neighbor losses as new regularizer terms. Formally, the unpacked form of Eq. (9) is:
In the above equation, Nx represents neighbors of sample x. The neighbors could be clean or adversarially perturbed samples. Thus, Nx′ represents the neighbors of adversarial example x′. Its neighbors could be clean samples and adversarial examples. Specifically, Nadvx represents the adversarial neighbor of the sample x. The adversarial neighbors have the same label as the original sample x, similar to the standard adversarial training.
We obtain adversarial examples using PGD as described in [13]. Note that the α11,α22,α3 hyperparameters determine the contributions of different neighborhood types, which are shown in Figure 2 as sub-graph types. The α terms can be tuned according to performance on clean and adversarially perturbed testing inputs. The pseudo-code of GReAT method is given in the Algorithm 1. Furthermore, a detailed explanation of the embedding of neighbor nodes and graph construction between clean and adversarial examples is shown in Section 3.3.
Algorithm 1: Graph Regularized Adversarial Training (GReAT).
Input: Labeled data Xl, unlabeled data Xul, model parameters Θ, hyperparameters α11, α22, α3, λ Output: Trained model Θ∗ Train classifier fXlΘ on labeled data using supervised loss ℓ Generate adversarial examples Xadv Propagate labels of Xl to Xul using label propagation on graph Construct graph G with nodes Xl,Xul,Xadv Compute neighbor set Nx, Nx′, Nadvx for each sample x, x′ Train model using loss ℒGReAT defined in Eq. (10) returnΘ∗
Figure 2.
Samples in embedding space. The figure on the left represents all the samples in the validation data set. The figure on the right shows some clean samples and their augmented neighbors.
3.2 Related previous methods
Creating graph embeddings using deep neural networks (DNNs) is a well-known method, [42]. Furthermore, the propagation of unlabeled graph embeddings using transductive methods, [6, 46], is efficient and well-studied. Neural Graph Machines (NGMs), [45], are a commonly used example of label propagation and graph embeddings, along with supervised learning. The proposed training objective takes advantage of these frameworks and provides more robust image classifiers. Therefore, the training objective can be considered a combination of nonlinear label propagation and a graph-regularized version of adversarial training.
3.3 Graph construction
We use a pre-trained model, DenseNet121, [48], to generate image embeddings as a feature extractor. The pre-trained model has weights obtained by training on ImageNet. The pre-trained model is more complex than the model we use to train and test the proposed regularization algorithm in our simulations. Numerous studies show that deep DNNs are better feature extractors than shallow networks [49, 50]. Another significant advantage of using larger pre-trained models is to reduce computational costs while obtaining high-quality embeddings. The process of creating embeddings is illustrated in Figure 3.
Figure 3.
Densenet121 for generating image embeddings.
Generating appropriate inputs to the neural network plays an important role in making the correct predictions. As noted above, we use a pre-trained DL model to create node embeddings. We generate embeddings of clean and augmented samples to obtain the neighborhood relationship between clean and augmented samples. The overall graph construction process is shown in Figure 4.
Figure 4.
Graph creation from embedding of clean and adversarial examples.
Similarly, generating embeddings of the same size is crucial to measuring the similarities between samples. Since the size of the embeddings is the same, we can visualize clean and augmented samples in the embedding space using the [51] t-distributed stochastic neighbor embedding (t-SNE) method.
In Figure 2, we use t-SNE (t-distributed stochastic neighbor embedding) to create a visual representation of the validation data set obtained from TensorFlow’s flower dataset. The primary purpose of this visualization is to provide insight into the distribution and relationships among the data points. The figure’s left panel displays all the samples that constitute the validation data set. It is important to note that this data set encompasses samples of five distinct classes. Each class represents a specific category or data type within the dataset, and the samples within each class share certain common characteristics or features.
Visualizing the embeddings highlights a strong connection between individual samples and their neighbors, effectively distinguishing between various classes. We use strong neighborhood connections to learn better and create more robust models. Consequently, we use these node embeddings as input features to the neural network by creating an adjacency embedding matrix, as shown in Figure 5. In particular, we use the label propagation method [7] to propagate the information from the labeled data points to the unlabeled instances, which improves the model performance on both clean and augmented samples.
Figure 5.
A: A sample with two neighbors showing their sub-graph and feature inputs. Blue nodes represent clean samples, and red nodes represent augmented samples. B–G show how clean samples and augmented samples may link on the graph structure.
Sample sub-graph of training instances are shown in Figure 5. These examples might be labeled or unlabeled since we generate embeddings for each sample and create the graph based on the similarity between embeddings. Note that a labeled sample may have one neighbor or none, for instance, if the similarity measure of the embeddings cannot pass the similarity threshold. In that case, the labeled sample goes through the neural network as a regular input without graph regularization. A visual example of a sub-graph is demonstrated in Figure 6.
Figure 6.
Node 85 and its neighbors (threshold: 0.8).
The sub-graph in Figure 6 shows a node and its most similar neighbors. We limit the number of neighbors for each sample (node) for the training set before the training because it directly affects the training set’s size and computation time. In this example, all neighbor nodes are classified as the main node. However, there could be some cases in which there are misclassified neighbors. Figure 7 shows the image of Node 85 and its top 5 nearest neighbors. As we can see, there are different augmented samples among Node 85’s nearest neighbors.
Figure 7.
Visualization of Node 85 image and its top 5 nearest neighbors.
3.4 Optimization process
The training starts by taking a minibatch of samples and their corresponding edges. Instead of utilizing the entire dataset in one batch, the procedure randomly picks a subset of edges for each iteration, introducing variability and randomness, which benefits the learning process. In addition, to further enhance training, edges are selected from a nearby region to increase the likelihood of including certain edges. This approach helps minimize noise and accelerates the learning process. Like other benchmark models [29], the Stochastic Gradient Descent (SGD) algorithm is employed to update network weights using the cross-entropy loss function. Additionally, we include the GReAT model trained with the Adam optimizer in our benchmark. Notably, we observe that the use of the Adam optimizer yields a superior performance compared to SGD in our experiments.
Note that the overall open form of the cost function in the following is equivalent to Eq. (8). The cost function incorporates the cost of supervised loss from labeled clean and labeled adversarial examples and neighbor losses. That is, the cost includes different neighbor types/edges, as shown in Figure 5. Formally,
where wa,wb,wc,wd represent the similarity weights between the samples and their neighbors calculated by cosine similarity measurement.
The similarity weights for each sample and its neighbors are potentially unique, ranging from zero to one. A sample and its neighbor are considered dissimilar if the similarity weight is close to zero. To calculate the neighbor loss, we use D to represent the distance between a sample and its neighbor, using the norms L1 and L2 as distance metrics. The hyperparameters α11,α12,α21, and α22 determine the contributions of different types of edges. In our simulations, we set all α values to one to include all edges in the training process. This new objective function enables using SGD with clean and adversarial examples and their neighbors in mini-batch training.
3.5 Complexity analysis
The proposed method integrates graph regularization into its training process, applying it to both labeled and unlabeled data instances within the graph, including benign and adversarial examples. The computational complexity of each training epoch depends on the number of edges in the graph, denoted as Ec. To evaluate the complexity of the training, we can express it as OcountEc. It is important to note that Ec is directly proportional to several factors. Firstly, it scales with the number of neighboring data points considered, indicating that more neighbors will increase the complexity. Secondly, it is influenced by a parameter that determines the selection of the most similar neighbors, further affecting the computational load. The step size used for adversarial regularization is also linked to Ec.
We conducted experiments to demonstrate the performance of the proposed Augmented GReAT method. Each experiment is performed on clean datasets with a fixed number of epochs and training steps. The typical hyperparameters are kept constant to ensure fair comparisons with other state-of-the-art methods. The base CNN model is first trained and then regularized using the proposed loss function. We use a copy of the base model to obtain the regularized model each time, preserving the original base model. Once the models are trained, we test each model on the same clean and adversarially perturbed test data to measure their generalization and robustness performances.
4.1 Datasets
Data sets from the Canadian Institute for Advanced Research (CIFAR-10) [39], Street View House Numbers (SVHN) [40], and Flowers [38] are used to evaluate the methods. The CIFAR-10 dataset consists of 60,000 images with 10 classes, and each class contains fixed-size 32×32 three-channel RGB images. To further assess the robust generalizability of our proposed method, we performed evaluations using the SVHN dataset, which comprises 73,257 training samples and 26,032 testing samples. The Flowers dataset contains 3670 images with five classes, each consisting of high-resolution RGB images. The image sizes are not fixed in the Flowers dataset, so resizing is required as one of the pre-processing steps.
The class distributions of the images are balanced in both datasets. We divide each dataset into train-validation-test sets in an 80%-10%-10% split, respectively. For our simulations, we exclusively use the flowers dataset for the ablation study, as existing benchmark methods do not incorporate flowers. Within the ablation study, we reduce the training set to 20% and 50% to observe model performance with fewer labeled samples.
4.2 Pre-processing steps
Several essential pre-processing steps are necessary to prepare batches for training. After generating image embeddings, we compute the similarity between each embedding and organize training batches based on this similarity metric.
4.2.1 Similarity measurement
Identifying the nearest neighbors for a given sample requires assessing the similarity among the embeddings. Various metrics, such as Euclidean distance, cosine similarity, and Structural Similarity Index Measure (SSIM), are available. We opt for cosine similarity due to its effectiveness in quantifying similarity among image embeddings in a multidimensional space. Cosine similarity is formally defined as:
Cosxixj=xi⋅xj∥xi∥∗∥xj∥.E12
4.2.2 Similarity weights
Similarity weights range from 0 to 1, depending on the angle between pairs of vectors. Overlapping embeddings exhibit a weight of 1 when their angle is zero. Conversely, orthogonal vectors result in dissimilarity, with a weight of 0. A predefined similarity threshold is utilized to identify neighbors, excluding embeddings below this threshold from consideration.
4.2.3 Training batches
Once the graph structure is established with clean samples and adversarial examples, training batches are generated for input into the neural network model. Each batch includes samples, their identified neighbors, and adversarial neighbors. The number of neighbors per sample is predetermined, although alternative strategies may be employed. In our simulations, we set this number to 2.
4.2.4 Standard augmented samples generation
For the CIFAR-10 dataset, pixel values are normalized between 0 and 1. Augmentation techniques include random erasing, left-to-right flips, and Gaussian noise. The random erasing parameters are set to minval = 0.02 and maxval = 0.2 for the erasure area, and minval = 10 and maxval = 20 for the mask. Gaussian noise parameters have a mean of 0.0 and a standard deviation of 0.01.
4.2.5 Adversarial examples generation
During training, we adopt the conventional ℓ∞ threat model with a radius of 8/255. Adversarial examples are generated using the PGD attack, iterating 10 steps with a step size of 2/255 across all datasets.
4.3 Network architecture
We used ResNet-18 [52] as the default baseline model architecture for our experiments. We employ the SGD optimizer to train all models with a momentum of 0.9 and the weight decay set to 0.0005. The training batch size is set to 128. We maintain consistency by using the same baseline model architecture and training parameters across all methods for a fair comparison. The Adam optimizer is configured with a learning rate of 0.001.
In this ablation study, we compare the performance of models trained on augmented and non-augmented datasets under different training scenarios using the CIFAR-10 dataset, with 5% for training, 2% for validation, and 2% for testing, and employing the ResNet-18 model for training. The model trained with data augmentation (base (aug) model) demonstrates improved generalization and robustness over the non-augmented model (base (no aug) model), as discussed in the context of standard training. Additionally, incorporating graph regularization with augmentation (graph reg (aug)) further enhances performance, preserving relationships in the feature space and improving resistance to adversarial attacks. The choice of using a smaller training dataset allows us to observe the effects of augmented versus non-augmented training more distinctly. These results are depicted in Figure 8, which shows accuracy performances with increasing perturbation sizes. They are detailed in Table 1, which provides numerical accuracies for clean and adversarially perturbed test sets with 0.2 perturbation size.
Figure 8.
Accuracy vs. attack strength.
Testing
Base (no aug)
Base aug
Graph reg (no aug)
Graph reg (aug)
Base test
37.8
47.2
49.0
52.3
Adversarial test
18.5
18.9
20.2
25.9
Table 1.
Accuracy results for models.
Bold accuracies show the best and the second best performances for each column (test method).
5.1.1 Standard training: augmented versus not augmented models
In standard training, the model trained with the augmented dataset (base (aug) model) outperforms the model trained without augmentation (base (no aug) model) as expected. Augmentation increases the diversity of the training data and helps the model generalize better to unseen data by learning more robust features, reducing overfitting, and improving overall performance on the test set.
5.1.2 Graph regularization: augmented versus not augmented models
When applying graph regularization during training, the augmented model with regularization (graph reg (aug)) shows superior performance compared to both the non-augmented models and the augmented model without regularization. Graph regularization helps preserve the relationships between samples in the feature space, which enhances the model’s ability to generalize by maintaining smoothness in the data manifold. The combination of augmentation and regularization further boosts this effect, leading to better generalization and robustness.
5.1.3 Robustness: augmented versus not augmented models
In terms of robustness against adversarially perturbed test samples, the graph reg (aug) model shows the strongest resistance against variations and perturbations in the test data as shown in Figure 8. The integration of data augmentation introduces a variety of scenarios during training, which helps the model become less sensitive to changes in input data. Graph regularization further enhances this robustness by enforcing smoothness and consistency in the learned feature space. Consequently, the graph reg (aug) model demonstrates superior robustness compared to both the base (aug) and base (no aug) models.
5.2 Comparison with state-of-the-art models
We compare the robust accuracy of our proposed method with various baseline models under different attack methods, specifically FGSM and PGD-100, applied to the CIFAR-10 and SVHN datasets. The evaluations use the ℓ∞ norm with ε=8/255. All models are based on the ResNet-18 architecture. The best checkpoint is selected based on the achievement of the highest robust accuracy on the test set.
Table 2 presents the robustness results for the CIFAR-10 dataset, highlighting the performance of various methods in defending against adversarial attacks from FGSM and PGD-100. Although the original GReAT method outperforms other approaches in terms of robust accuracy, its natural accuracy is slightly lower than that of some other methods. However, the new Augmented GReAT method closes this gap and surpasses other methods in natural accuracy and robustness. Specifically, Augmented GReAT (ADAM) achieves the highest scores, demonstrating the effectiveness of integrating standard data augmentation techniques with GReAT in enhancing the robustness and overall performance of CIFAR-10 classification models against adversarial attacks.
Robust Benchmark under linf type attack for CIFAR-10.
Bold accuracies show the best and the second best performances for each column (test method).
Table 3 displays the natural accuracy and robustness results for the SVHN dataset, showing the accuracy of various methods when faced with adversarial attacks from FGSM and PGD-100. The original GReAT method significantly improves robust accuracy compared to other methods while maintaining comparable natural accuracy. The new Augmented GReAT method further enhances performance, achieving the highest scores in both natural and robust accuracy.
This chapter presented an enhanced version of the Augmented Graph Regularized Adversarial Training Method (Augmented GReAT) to improve classifier robustness by incorporating standard data augmentation techniques. We leveraged classical adversarial training with graph regularization to further enhance the robustness of deep learning classifiers. We bolstered the model’s capacity to withstand adversarial attacks by employing graph-based constraints to regularize the training process. Our comprehensive evaluations, shown in Tables 2 and 3, demonstrate that the Augmented GReAT method achieves superior performance in both natural and robust accuracy compared to other state-of-the-art methods. Specifically, Augmented GReAT (ADAM) outperforms all methods, showcasing the effectiveness of combining graph structures with data augmentation in enhancing the robustness and generalization of deep learning classifiers. This strategy is a valuable tool in adversarial training, significantly advancing neural networks’ robustness and predictive accuracy against sophisticated adversarial perturbations.
References
1.Liu C, Dong Y, Xiang W, Yang X, Su H, Zhu J, et al. A comprehensive study on robustness of image classification models. Benchmarking and Rethinking. 2023. Available from: https://arxiv.org/abs/2302.14301
2.Maurıcio J, Domingues I, Bernardino J. Comparing vision transformers and convolutional neural networks for image classification: A literature review. Applied Sciences. 2023;13(9):306-322. Available from: https://www.mdpi.com/2076-3417/13/9/5521
3.Taye MM. Understanding of machine learning with deep learning: Architectures, workflow, applications and future directions. Computers. 2023;12(5):26. Available from: https://www.mdpi.com/2073-431X/12/5/91
4.Li S, Kou P, Ma M, Yang H, Huang S, Yang Z. Application of semi-supervised learning in image classification: Research on fusion of labeled and unlabeled data. IEEE Access. 2024;12:27331-27343
5.Bengio Y, Delalleau O, Le Roux N. Label propagation and quadratic criterion. In: Semi-Supervised Learning. Cambridge, Massachusetts, USA: The MIT Press; 2006. DOI: 10.7551/mitpress/6173.003.0016
6.Yang Z, Cohen WW, Salakhutdinov R. Revisiting Semi-Supervised Learning with Graph Embeddings. arXiv; 2016. ArXiv:1603.08861 [cs]. Available from: http://arxiv.org/abs/1603.08861
7.Lee DH. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: ICML 2013 Workshop: Challenges in Representation Learning (WREPL). Atlanta, USA: ICML Conference; 2013
8.Joachims T. Transductive inference for text classification using support vector machines. In: ICML ‘99: Proceedings of the Sixteenth International Conference on Machine Learning. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.; 1999. pp. 200-209
9.Amini MR, Feofanov V, Pauletto L, Devijver E, Maximov Y. Self-Training: A Survey. arXiv; 2022. Available from: https://arxiv.org/abs/2202.12040
10.Goodfellow I, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. In: International Conference on Learning Representations. San Diego, CA, USA: ICLR Conference; 2015. Available from: http://arxiv.org/abs/1412.6572
11.Carlini N, Wagner D. Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (SP). San Jose, CA, USA: IEEE, SP; 2017. pp. 39-57
12.Nguyen A, Yosinski J, Clune J. Deep Neural Networks Are Easily Fooled: High Confidence Predictions for Unrecognizable Images. Boston, Massachusetts, USA: CVPR; 2015
13.Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A. Towards Deep Learning Models Resistant to Adversarial Attacks. Vancouver, BC, Canada: ICLR; 2018
14.Kurakin A, Goodfellow IJ, Bengio S. Adversarial Examples in the Physical World. CoRR. International Joint Conferences on Artificial Intelligence Organization (IJCAI); 2016. pp. 4925-4926. Available from: http://arxiv.org/abs/1607.02533
15.Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, et al. Intriguing Properties of Neural Networks. Banff, AB, Canada: ICLR; 2014
16.Biggio B, Roli F. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition. 2018;84:317-331. DOI: 10.1016/j.patcog.2018.07.023
17.Moosavi-Dezfooli SM, Fawzi A, Frossard P. DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. Las Vegas, NV, USA: CVPR; 2016
18.Sharif M, Bhagavatula S, Bauer L, Reiter MK. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. CCS ’16. New York, NY, USA: Association for Computing Machinery; 2016. pp. 1528-1540. DOI: 10.1145/2976749.2978392
19.Eykholt K, Evtimov I, Fernandes E, Li B, Rahmati A, Xiao C, et al. Robust Physical-World Attacks on Deep Learning Models. Salt Lake City, Utah, USA: CVPR; 2018
20.Bayram S, Barner K. A Black-Box Attack on Optical Character Recognition Systems. arXiv. 2022. Available from: https://arxiv.org/abs/2208.14302
21.Shorten C, Khoshgoftaar TM. A survey on image data augmentation for deep learning. Journal of Big Data. 2019;6:1-48. Available from: https://api.semanticscholar.org/CorpusID:195811894
22.Luo R, Wang Y, Wang Y. Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learning. Kigali, Rwanda: ICLR; 2023. Available from: https://arxiv.org/abs/2303.01289
23.Xie C, Wu Y, Lvd M, Yuille AL, He K. Feature denoising for improving adversarial robustness. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: CVPR; 2019. pp. 501-509
24.Bafna M, Murtagh J, Vyas N. Thwarting Adversarial Examples: An L_0-RobustSparse Fourier Transform. arXiv preprint arXiv:181205013. 2018
25.Xie C, Wang J, Zhang Z, Ren Z, Yuille A. Mitigating adversarial effects through randomization. arXiv preprint arXiv:171101991. 2017
26.Wang B, Lin AT, Zhu W, Yin P, Bertozzi AL, Osher SJ. Adversarial defense via data dependent activation function and total variation minimization. arXiv preprint arXiv:180908516. 2018
27.Papernot N, McDaniel P, Wu X, Jha S, Swami A. Distillation as a defense to adversarial perturbations against deep neural networks. In: 2016 IEEE Symposium on Security and Privacy (SP). San Jose, CA, USA: IEEE; 2016. pp. 582-597
28.Chen HY, Liang JH, Chang SC, Pan JY, Chen YT, Wei W, et al. Improving adversarial robustness via guided complement entropy. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul, Korea: ICCV; 2019. pp. 4881-4889
29.Kuang H, Liu H, Lin X, Ji R. Defense against adversarial attacks using topology aligning adversarial training. IEEE Transactions on Information Forensics and Security. 2024;19:1-1
30.Zhang H, Yu Y, Jiao J, Xing EP, Ghaoui LE, Jordan MI. Theoretically principled trade-off between robustness and accuracy. In: ICML. Vol. 97 of Proceedings of Machine Learning Research. Long Beach, California, USA: PMLR; 2019. pp. 7472-7482
31.Kannan H, Kurakin A, Goodfellow I. Adversarial Logit Pairing. Montreal, Canada: NIPS; 2018
32.Mao C, Zhong Z, Yang J, Vondrick C, Ray B. Metric learning for adversarial robustness. In: NeurIPS. Vancouver, Canada: NIPS; 2019. pp. 478-489
33.Jiang Z, Chen T, Chen T, Wang Z. Robust Pre-Training by Adversarial Contrastive Learning. Virtual: NIPS; 2020
34.Zhou D, Huang J, Schölkopf B. Learning from labeled and unlabeled data on a directed graph. In: Proceedings of the 22nd International Conference on Machine learning - ICML ’05; Bonn, Germany. Bonn, Germany: ICML; 2005. pp. 1036-1043. Available from: http://portal.acm.org/citation.cfm?doid=1102351.1102482
35.Belkin M, Niyogi P, Sindhwani V. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research. 2006;7(85):2399-2434. Available from: http://jmlr.org/papers/v7/belkin06a.html
36.Carmon Y, Raghunathan A, Schmidt L, Duchi JC, Liang PS. Unlabeled data improves adversarial robustness. In: Advances in Neural Information Processing Systems. Vol. 32. Vancouver, Canada: Curran Associates, Inc.; 2019
37.Bayram S, Barner K. GReAT: A graph regularized adversarial training method. IEEE Access. 2024;12:63130-63141
38.TensorFlow. Flowers; 2019. Available from: http://download.tensorflow.org/example-images/flower-photos.tgz
39.Krizhevsky A, Nair V, Hinton G. CIFAR10 Dataset. Canada: Canadian Institute for Advanced Research; 2009
40.Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng AY. Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning. Granada, Spain: NIPS; 2011. Available from: http://ufldl.stanford.edu/housenumbers
41.Ren K, Zheng T, Qin Z, Liu X. Adversarial attacks and defenses in deep learning. Engineering. 2020;6(3):346-360
42.Weston J, Ratle F, Collobert R. Deep learning via semi-supervised embedding. In: Proceedings of the 25th International Conference on Machine Learning. ICML ’08. New York, NY, USA: Association for Computing Machinery; 2008. pp. 1168-1175. DOI: 10.1145/1390156.1390303
43.Agarwal N, Liu H, Murthy S, Sen A, Wang X. A social identity approach to identify familiar strangers in a social network. Proceedings of the International AAAI Conference on Web and Social Media. 2009;3(1):2-9. Available from: https://ojs.aaai.org/index.php/ICWSM/article/view/13946
44.Jacob Y, Denoyer L, Gallinari P. Learning latent representations of nodes for classifying in heterogeneous social networks. In: Proceedings of the 7th ACM International Conference on Web Search and Data Mining. New York, New York, USA: ACM; 2014. pp. 373-382. Available from: https://dl.acm.org/doi/10.1145/2556195.2556225
45.Bui TD, Ravi S, Ramavajjala V. Neural graph learning: Training neural networks using graphs. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. Marina Del Rey, CA, USA: ACM; 2018. pp. 64-71. Available from: https://dl.acm.org/doi/10.1145/3159652.3159731
46.Zhu X, Lafferty J, Rosenfeld R. Semi-Supervised Learning with Graphs. USA: Carnegie Mellon University; 2005. p. AAI3179046
47.Zhou D, Bousquet O, Lal TN, Weston J, Schölkopf B. Learning with Local and Global Consistency. San Francisco, CA, USA: ACM Digital Library; 2004. p. 8
48.Huang G, Liu Z, van der Maaten L, Weinberger KQ. Densely Connected Convolutional Networks. arXiv. 2016. Available from: https://arxiv.org/abs/1608.06993
49.Krizhevsky A. Learning Multiple Layers of Features from Tiny Images. Canada: University of Toronto; 2009
50.Mhaskar H, Liao Q, Poggio TA. When and why are deep networks better than shallow ones? In: Singh SP, Markovitch S, editors. AAAI. San Francisco, California, USA: AAAI Press; 2017. pp. 2343-2349. Available from: http://dblp.uni-trier.de/db/conf/aaai/aaai2017.htmlMhaskarLP17
51.van der Maaten L, Hinton G. Visualizing data using t-SNE. Journal of Machine Learning Research. 2008;9:2579-2605. Available from: http://www.jmlr.org/papers/v9/vandermaaten08a.html
52.He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. Las Vegas, USA: CVPR; 2016
Written By
Samet Bayram and Kenneth Barner
Submitted: 18 July 2024Reviewed: 26 July 2024Published: 03 October 2024
Open access peer-reviewed chapter - ONLINE FIRST
