SENSITIVITY Experiments ON ONE-CLASS Support Vector Machine

M. Koda*, T. Takabatake, T. Akana, 
University of Tsukuba, Japan
koda@sk.tsukuba.ac.jp

One-Class Support Vector Machine (OC-SVM) is a most popular method in outlier detection or anomaly detection. It is based on the kernel trick and standard SVM procedures, and, in a quadratic programming formulation, uses a trade-off parameter that defines the ratio between number of samples in data class and that of outlier class, i.e., class of support vectors. The sensitivity of the trade-off parameter involved in the OC-SVM is investigated with respect to noisy mislabelled data, and some original modification of the method is proposed.

1.	Introduction
In binary classification (e.g., operating or failed), support vector machines (SVMs) have attracted much attention recently because of their excellent quality in various real-world applications [1]. Utilizing the kernel trick that is a method of converting a linear classification algorithm into a non-linear one by replacing a dot product in high dimensional feature space with kernels, SVM can find the maximum-margin hyperplane that separates two clouds of data points at equally located distance. SVM is reputed to achieve high performance, not suffering from the curse of dimensionality even when the number of training samples is small compared to the feature vector dimensionality. The great advantage of SVM is that the algorithm often does not suffer from overfitting and enhance a generalization capability. In order to use kernels as dot product, note that the kernel function must be symmetric and positive semi-definite.
A powerful extension of SVM to one-class problems is referred to as One-Class SVM (OC-SVM) [2]. Using an appropriate kernel function, OC-SVM first maps the data points into a high dimensional feature space, and then finds the hyperplane that separates, with maximum margin, the feature vectors from the origin of the transformed space. The kernel most commonly used in SVM and OC-SVM is the Gaussian radial basis function. The OC-SVM interprets the origin of the transformed feature space as the second class and feature vectors that are classified as belonging to the second class are regarded as outliers or anomalies. In the risk management applications, OC-SVM is typically used for classification of anomalies and malfunctions occurring in nuclear components and systems from measured operational data [3].
In the quadratic programming formulation of OC-SVM, there is a parameter that controls a trade-off between maximizing the distance of the hyperplane from the origin, and containing most of the training samples in the domain created by the hyperplane. The paper explores sensitivity investigations of this trade-off parameter involved in the OC-SVM against noisy mislabelled data since the parameter significantly influences the quality of classification. In addition, we propose some original modification of the method based on the sensitivity experiments.

2.	Sensitivity Analysis in OC-SVM
Let xbe a measurement vector. Using the Gaussian radial basis function as kernel

where  denotes the standard deviation, OC-SVM can be formulated as the following quadratic programming problem:

where  is the appropriate mapping function, w and  denote the weight vector and threshold level, respectively, (0,1) is the trade-off parameter, and  are slack variables that penalize the objective function with allowing some of the feature vectors to be located in between the origin and desired hyperplane. Recall that the parameter  theoretically defines the ratio between number of samples in data class and that of outlier class, i.e., class of support vectors.
In order to perform the anomaly detection, a dataset of known anomalies is used to train OC-SVM with the proper label, say, +1. When all the training data are correctly labelled, the resulting model can recognize whether the test data belongs to the known or unknown anomaly class. However, if the class label is noisy, i.e., some data are mislabelled, it often becomes crucial to discriminate true (failed states) and false anomalies since there is a number of possible situations with false-positive and false-negative cases. For system reliability evaluation, the correctly classified failed states provide most valuable information. In the present study, the sensitivity is defined by the percentage of correctly classified anomalous events belonging to the known and/or unknown anomaly class with varying a value of the trade-off parameter .

3.	Sensitivity Experiments
For experimental investigations, we take Spam E-mail Database from UCI Repository of machine learning databases [4]. Only preliminary results are presented here with appropriate pre-processing of Spam E-mail Database, where OC-SVM is used to eliminate possible mislabelled data. In Fig. 1, we plot the classification accuracy against the level of mislabel with varying a value of involved in the OC-SVM used as a pre-processor.

Fig. 1 depicts that there may be some optimal value of  against the level of anomaly or percentage of mislabelled data. Although it is too early to draw any definite conclusions, OC-SVM may be useful for the one class anomaly recognition tasks. But, at the same time, preliminary results warn us against its uncritical use. In addition, some original modification of the method is proposed based on conceptual and experimental investigation details of which will be presented at the conference.

References

[1] Vapnik V: Statistical Learning Theory. Wiley, New York, 1998 

[2] Scholkopf B, Smola AJ, Williamson RC, Bartlett PL: New support vector algorithms. Neural Computation 12, 1207-1245(2000) 

[3] Rocco S, C.M., Zio, E.: A support vector machine integrated system for the classification of operation anomalies in nuclear components and systems. Reliability Engineering and System Safety 92, 593-600(2007) 

[4] Newman DJ, Hettich S, Blake CL, Merz CJ: UCI Repository of machine learning databases [http://www.ics.uci.edu/~mlearn/MLRepository.html]. Irvine, CA: University of California, 1998