Statistical Noise Reduction for Robust Human Activity Recognition Song-Mi Lee, Heeryon Cho, and Sang Min Yoon

Abstract— Noise and variability in accelerometer data collected using smart devices obscure accurate human activity recognition. In order to tackle the degradation of the triaxial accelerometer data caused by noise and individual user differences, we propose a statistical noise reduction method using total variation minimization to attenuate the noise mixed in the magnitude feature vector generated from triaxial accelerometer data. The experimental results using Random Forest classifier prove that our noise removal approach is constructive in significantly improving the human activity recognition performance.

I. INTRODUCTION Human activity recognition using portable and wearable smart devices such as smartphones and smartwatches is an important yet challenging research topic for smart environments, healthcare, and home security. We aim to develop a model that is capable of recognizing a multiple set of basic human activities under real-world conditions using data collected by a single triaxial accelerometer built into a smart device. A triaxial accelerometer is a sensor that returns an estimate of acceleration along the x, y, and z axes from which velocity and displacement can also be measured. Activity recognition is formulated as a supervised classification problem, whose training data is obtained by instructing the human subjects to perform the activities to be classified [1]. Lester et al. [2] developed an automatic physical activities recognition system in a controlled environment using accelerometers and microphones. A wavelet-based activity classification method using one or more accelerometers was proposed by Mannini and Sabitini [3], in which the dynamic motion component is separated from the gravity components, leading to a 98.4% of accuracy. Foerster and Fahrenberg [4] collected data from subjects using five accelerometers and built a hierarchical classification model in order to identify different body postures and movements. There is a clear limitation in robust human activity recognition that does not reduce the effect of noise added during data sensing process. Traditional noise reduction approaches like Gaussian filter and adaptive filter in the frequency domain are popularly used because filter-based noise reduction approaches are very fast, but the original signal can be also smoothed while removing noise. We were supported by the Institute for Information and Communications Technology Promotion (IITP) grants (No. 2014-0-00501 & No. 2017-0-00205) and by the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B04932889, NRF-2015R1A5A7037615) and the Korean government (MSIP) (NRF2017R1A2B4011015). S.-M. Lee, H. Cho, and S.M. Yoon are with College of Computer Science, Kookmin University, 77, Jeongneung-ro, Seongbuk-gu, Seoul 02707 Korea.

[email protected]

In this paper, we present an effective human activity recognition system which estimates the true signal in the noisy accelerometer data using total variation (TV) minimization [5]. Our system removes noise while retaining the original signal by statistically analyzing the movement of the signal, and it uses Random Forest classifier [6] to recognize activities. Figure 1 shows the overall process of our approach for robust human activity recognition; the process combines feature vector transformation, noise reduction using TV minimization, and Random Forest-based human activity recognition. The main contributions of this work are summarized as follows: (1) human activity dataset construction for fair comparison and analysis, (2) TV-based noise removal of accelerometer signal to effectively recognize the human activity, and (3) human activity recognition performance evaluation using Random Forest and Convolutional Neural Networks. This paper proceeds as follows: In Section II, we survey the previous approaches for human activity recognition with a focus on data acquisition methods, feature processing techniques, and classifier learning algorithms. We then explain the technical details of our TV minimization-based noise removal approach for human activity recognition using a smartphone accelerometer in Section III. Experimental results of our noise removal approach applied to Random Forest and Convolutional Neural Networks are compared using the triaxial accelerometer data that we collected, and another experimental result using an open human activity recognition dataset are presented in Section IV. Finally, we conclude this paper in Section V. II. RELATED WORK A. Data Acquisition Methods In recent years, widespread usage of mobile phones such as smartphones have enabled easy collection of users’ timeseries motion data through the device embedded sensors. While initial researches have used various standalone onbody sensors to collect motion data, more recent works have taken advantage of phone embedded sensors since it is more accessible, feasible, practical, and less intrusive than the standalone wearable sensors that usually need to be mounted or attached to the body. The predominantly used sensor types in mobile phones are inertial sensors, namely, motion sensors (accelerometers) and rotation sensors (gyroscopes), which measure the human movement in three-dimensions. Among the two inertial sensors, the most widely used sensor in human activity recognition is the triaxial accelerometer that measures the acceleration along the three orthogonal axes. These axes capture the horizontal (x-axis), vertical

Fig. 1.

Overall process of noise reduction-based activity recognition.

(y-axis), and forward/backward (z-axis) movement of the user. Many research combine the accelerometer inputs with other sensor readings, whereas some research use only the accelerometer data for activity recognition. We look at the various researches that employ a single triaxial accelerometer in smartphones or mobile phones for data collection. Various researches that use a single triaxial accelerometer for data collection investigate different sampling rates, placements of the mobile phone for data collection, and activities for detection: Khan et al. [7] placed the smartphone in five different positions of the human subjects, i.e., shirt’s pocket, jean’s front-left or front-right pocket, jean’s rear pocket, and coat’s inner pocket, and recorded five activities, sitting, walking, walk-upstairs, walk-downstairs, and running, at 45Hz.; Kwapisz et al. [8] instructed the human subjects to carry the Android phone in the front leg pocket of their pants and asked them to perform six activities, walk, jog, ascend stairs, descend stairs, sit, and stand, while sampling the triaxial acceleration at 20Hz.; Bayat et al. [9] did not specify the carry position of the smartphone, but sampled six activities, walking, climbing down stairs, climbing up stairs, sitting down, standing up, and falling, while recording the accelerometer data at 30Hz; Brezmes et al. [10] guided the human subjects to maintain the position of the smartphone either in hand or in pocket while performing the six activities, running, slow walking, fast walking, aerobic dancing, walking upstairs, and walking downstairs, when collecting the accelerometer data at 100Hz.; Sun et al. [11] instructed human subjects to position smartphones in right/left/front/back pocket of their trousers or left/right pocket of their jackets and collected the seven activities data, stationary, walking, running, bicycling, ascending stairs, descending stairs, and driving, at 10Hz.; Zhang et al. [12] ordered the human subjects to wear the smartphone on the left side of the waist in a horizontal orientation by mounting the phone on a belt and collected six activities, sitting, standing, lying, walking, posture transition, and gentle motion, at 1Hz.; Yang et al. [13] did not specify the phone position, but collected six activities, sitting, standing, walking, running, driving, and bicycling, from human subjects at a sampling rate of 36Hz.; Lee et al. [14] gathered three activities, walking, running, and staying, from graduate students at sampling frequency of 1Hz, but did not specify the carry position of the smartphone.; Lockhart et al. [15] instructed the human subjects to carry the smartphone in their pocket while performing six activities, walking, jogging, stair

climbing (up and down), sitting, standing, and lying down, at a sampling rate of 20Hz. B. Preprocessing & Feature Processing Techniques When preparing the dataset for learning the activity classifier, the collected raw sensor data are segmented using a fixed-size window by sliding the window across the timeseries sensor data. The generated training and test data can contain either the disjoint segments or overlaps of the raw sensor data. For example, different segmenting of accelerometer data was tried in the past researches: 128 samples of accelerometer data sliced per 1.28 seconds with 50% or 64 samples overlap [9]; 90 samples of accelerometer data sliced per 2 seconds with no overlapping between consecutive windows [7]; and 100 samples of accelerometer, gyroscope, and magnetometer data sliced per 2 seconds with 50% overlap [16]. Generally, the larger the size of the data segments, or the longer the duration of the activity data, better performing activity classifier may be constructed since more descriptive activity features can be incorporated for learning; however, larger data segments may also mean the inclusion of two or more different activities, i.e., the inclusion of transitional states, which can lead to confusion in classifier learning. Hence, both the descriptiveness of data and the possibility of mixed-activity inclusion must be considered when segmenting the raw sensor data. Based on the segmented raw data, many researches have generated additional features by calculating the various statistical features such as mean, standard deviation, maximum, minimum, etc. of the fixed-length samples, and the time domain features such as the signal vector magnitude of the triaxial acceleration values, the frequency domain features such as Fourier Transform (FT) and Discrete Cosine Transform (DCT), the time and frequency features such as Discrete Wavelet Transform (DWT) of the raw sensor data and so forth. Wang et al. proposed a hybridized filter and wrapper-based feature selection method to select a subset of discriminant features in order to remove irrelevant and redundant features [17]. C. Classifier Learning Algorithms Previous researches have investigated various supervised machine learning algorithms to effectively learn multi-class activity classifiers. Guinness compared a wide range of supervised learning techniques for activity classification including decision trees, support vector machines, naive Bayes

classifiers, Bayesian networks, logistic regression, artificial neural networks and several instance-based classifiers [18]. Shoaib et al. [19] surveyed thirty different activity recognition studies that used various classification algorithms including KNN, DHMM, hierarchical decision tree, hierarchical SVM, K-medoids clustering and so forth. In recent years, with the remarkable development in deep learning methods, many studies have employed deep neural network algorithms in human activity recognition. As a result, deep learning techniques has paved its way into the user activity and context recognition domain. Many state-of-theart activity recognition algorithms now use various deep neural network algorithms to recognize human activities. Prior to the adoption of deep learning methods, shallow neural network classifiers such as Multi-Layer Perceptron (MLP) was investigated as the representative neural network algorithm for activity recognition. For example, Kwapisz et al. [8] compared the recognition performance of a decision tree, logistic regression, and MLP, and found that MLP performed the best. Dernbach et al. [20] compared the performance of MLP and other approaches such as Naive Bayes and tree-based methods on simple activites (e.g., biking, climbing, stairs, driving, lying, running, sitting, standing, and walking) and complex activities (e.g., cleaning kitchen, cooking, medication, sweeping, washing hands, and watering plants), and found that MLP performed the best. Bayat et al. [9] investigated the recognition performance of MLP as an individual classifier and also as a part of a combination of different classifiers. As an individual classifier, MLP performed the best among SVM, Random Forest, logistic model trees, logistic regression, and additive logistic regression. More recently, Convolutional Neural Network (CNN)-based approaches have been investigated in activity recognition for its advantages in capturing local dependencies of activity signals and preserving the feature scale invariance [21]. Existing works that use CNN include [21], [22], [23]. Some approaches combine Long-Short-Term Memory (LSTM) recurrent neural networks with CNN. III. HUMAN ACTIVITY RECOGNITION WITH NOISE REMOVAL USING TOTAL VARIATION MINIMIZATION The accelerometer sensor measures the magnitude of the x, y, and z components of the smartphone’s acceleration. Therefore, the x, y, and z values can change according to the smartphone’s movement, which in turn corresponds to the user’s action. However, since the acceleration data include a rotation component, such rotation component might interfere with the correct activity recognition. In order to keep the rotational interference to a minimum, we transform the raw x, y, and z acceleration data into vector magnitude data by calculating the Euclidean norm at time t as follows: p (1) u(t) = x(t)2 + y(t)2 + z(t)2 . The above equation transforms the x, y, and z component values to a single representative value while reducing the error possibly generated by the rotation component. The

Fig. 2.

Vector magnitude (top) and noise-reduced (down) signals.

vector magnitude time series data are then divided into three activity data based on the ground truth labels. The observed u(t) from Eq. (1), which denote the Euclidean norm of the accelerometer data, can be directly used for human activity recognition, but the position of the smartphone (e.g., hand, pocket, bag, etc.) and the individual or personal differences in the given actions (e.g., walk, run, and staying still) can act as noise that mask the ideal signal for activity recognition. In order to remove such noise mixed in the acceleration data, we apply a total variation (TV) minimization-based noise reduction to the vector magnitude data from Eq. (1) by statistically analyzing the movement of the signal. Suppose that the observed data u(t) is a discrete sampling from the accelerometer data, and the signal of time length N is contaminated by noise introduced during signal acquisition procedure. Then, the magnitude data can be written as u(tn ) = v(tn ) + n , n = 1, .., N , where v(tn ) is the value of an underlying function v, and n indicates the additive noise at time t. The normal signal estimation method to reconstruct an original signal from a noise-degraded signal is formulated as follows. From a given noisy input discrete signal u(t) defined on T = {t1 , t2 , . . . , tN }, the Total Variation (TV) minimization technique generates an estimated signal vˆ which retains the movement of the data and removes the noisy input signal u by solving the following minimization problem: vˆ = argmin

N X

w:T →R n=1

2

(w(tn ) − u(tn )) +λ

N −1 X n=1

|w(tn+1 ) − w(tn )| .

(2) The TV minimization model is particularly useful for signal reconstruction because the minimization of the TV norm rapidly eliminates small scaled structure from the signal. The incorporation of the TV minimization term in Eq. (2) reduces the influence of outliers which are mainly due

b=1

IV. EXPERIMENTS A. Performance Evaluation Using Collected Dataset The accelerometer data indicating three human activities (i.e., running, walking, and staying still) were collected via the accelerometer sensor embedded in the smartphones. The acceleration on x, y, and z axes were recorded at 1Hz frequency along with the ground truth activity labels. A total of 2,377, 3,588, and 3,934 seconds of running, walking, and staying still time series activity data were collected respectively. The raw accelerometer data were then transformed into twenty-dimension magnitude feature vectors with each dimension representing one-second vector magnitude of x, y, and z axes’ acceleration. A total of 4,953 training and 1,743 test data were constructed with equal number of running, walking, and staying still data for both training (1,641 each) and test (581 each) data. We then created a separate noise-reduced training and test dataset by applying TV minimization to the existing training and test data. When applying the TV minimization, the λ parameter in Eq (2) was set as six (λ = 6) in all our experiments using the collected dataset. In order to evaluate the effectiveness of the TV minimization noise reduction on human activity data, we employed two distinct machine learning algorithms, Random Forest (RF) [6] and one-dimensional Convolutional Neural Network (CNN) [24] in our activity recognition evaluation experiments. The CNN consisted of one convolutional layer, one max-pooling layer, one fully-connected layer with dropout applied, and one softmax layer that outputs the probability of the three activities. Table I compares the performance of CNN on the vector magnitude data (CNN only) and the TV minimizationapplied data (TV-CNN). We see that for feature learning

TABLE I P ERFORMANCE OF CNN: CNN ONLY VS . TV-CNN

Actual Class

Actual Class

CNN only Activity Run Walk Still Precision TV-CNN Activity Run Walk Still Precision

Predicted Class Run Walk Still 527 54 0 18 560 3 0 52 529 96.70% 84.08% 99.44% Predicted Class Run Walk Still 537 44 0 1 573 7 0 361 220 99.81% 58.59% 96.92%

Recall 90.71% 96.39% 91.05% 92.71% Recall 92.43% 98.62% 62.13% 76.31%

TABLE II P ERFORMANCE OF R ANDOM F OREST: RF ONLY VS . TV-RF

Actual Class

RF only Activity Run Walk Still Precision TV-RF Activity Run Walk Still Precision

Actual Class

to uneven signal acquisition error. The second term in Eq. (2) is called the total variation norm which preserves the sharp discontinuities in the signal while removing noise. Figure 2 shows vector magnitude time series acceleration signals (top) and noise-reduced signals using TV minimization (bottom) using Eq. (2). The noise removed signal vˆ is then used as an input signal for human activity recognition using Random Forest (RF) which is a general term for classifier combination of L tree-structured classifiers h(v, θk ), k = 1, ..., L, where θk are independent and identically distributed random vectors. Given a training set X = vˆ1 , ..., vˆm with responses Y = y1 , ..., ym , bagging repeatedly selects a random sample with replacement from the training set and fits the trees to these samples. For b = 1, ..., B, m training samples are sampled with replacement from X and Y to form Xb and Yb training sets. Then, the decision tree hb is trained on Xb and Yb . After training, prediction for the test magnitude vector of the accelerometer data vtest can be obtained by averaging the predictions from all the individual decision trees on vtest as B 1 X hb (vtest ). (3) h= B

Predicted Class Run Walk Still 522 57 2 33 530 18 0 80 501 94.05% 79.46% 96.16% Predicted Class Run Walk Still 581 0 0 0 581 0 0 0 581 100% 100% 100%

Recall 89.85% 91.22% 86.23% 89.10% Recall 100% 100% 100% 100%

algorithms such as CNN, applying the TV minimization adversely affects the recognition performance. The precision of the walking activity dropped from 84.08% to 58.59% and the recall of the staying still activity drastically decreased from 91.05% to 62.13%. Meanwhile, both the precision and recall of the running activity increased somewhat. These results suggest that the convolutional and max-pooling layers of CNN had difficulty capturing the features of the noisereduced signals, and that these layers can better learn from the more jagged raw signals (e.g., Fig. 2 top). In contrast to the results of CNN-based recognition, the performance of the TV minimization-applied data on RF (TV-RF) is significantly better than the simple vector magnitude data on RF (RF only) as shown in Table II. The Random Forest classifier on the test data achieved 100% recognition accuracy when the TV minimization was applied to the raw activity signals. Because the Random Forest classifier is a collection of decision trees, and each decision tree performs recursive partitioning based on the metrics for partitioning, the TV minimization of the raw signals has transformed the signal more amenable to the metrics, and in turn positively contributed to the effective partitioning of the subsets. It is notable that the recognition of walking activity was the most difficult for both CNN and RF as evidenced by the worst precision shown in Table I (both CNN only & TV-CNN) and Table II (RF only). We think this is due to the intermediate position of the walking signal; the walking signal can be viewed as a transitional signal that bridges the running and staying still signals, and since the walking signal lies nearby the running and staying still signals at

both ends, the less distinct running and staying signals are mistaken as walking signal and vice versa. This misclassified walking signal constitutes the largest portion of the classification error. By introducing the TV minimization-based noise reduction on activity data, we were not only able to reduce the confounding noise caused by the smart device carriers’ individual differences and smart devices’ positional variances, but also demarcate the running, walking, and staying still signals more clearly than the original vector magnitude data. B. Performance Evaluation Using Open Dataset We also evaluated our approach using an open human activity recognition dataset which contained seven features generated from the GPS and accelerometer data [18].1 A total of 8,498 data instances with seven activity labels, i.e., static, moving slowly, walking, running, driving a car, riding a bus, and riding a train, were given in the dataset. While [18] showed 97.5% activity recognition accuracy with a Random Forest classifier, we were unable to replicate the result because the open dataset lacked one entire feature given in the paper (i.e., 1HzPeak) and also had many missing values. Since our objective is to validate the effectiveness of our statistical noise reduction approach using a Random Forest classifier, we compared our TV minimization noise reduction approach to the raw data approach. Here, we set the denoising weight as λ = 1 and the number of Random Forest tree size as 50. We conducted thirty trials of 10-fold cross validation and calculated the mean accuracy of the three hundred test results for each dataset. Our noise reduction approach exhibited higher accuracy of 85.7% whereas the raw data approach showed 85.4%. The Wilcoxon signedrank test showed that the accuracy difference was significant (p = 0.008) verifying the effectiveness of our approach. V. C ONCLUSION We presented an effective TV minimization-based noise reduction approach for human activity recognition and evaluated the activity recognition performance of our approach using RF and CNN. A significant performance increase in RF-based activity recognition was observed for our TV minimization-based noise reduction approach. However, the performance of TV-based approach on CNN was worse than the non-TV-based approach hinting that the representation learning algorithms such as CNN are unsuitable for noise reduction-based human activity recognition. TV minimization has the potential to reduce the noise generated by the users’ individual differences in actions and the positional variations of the smart devices. Moreover, TV minimization approach has the effect of more clearly demarcating the running, walking, and staying still signals in a way that is suitable for RF. R EFERENCES [1] F. Attal, S. Mohammed, M. Dedabrishvili, F. Chamroukhi, L. Oukhellou, and Y. Amirat, Physical human activity recognition using wearable wensors, Sensors, vol. 15, no. 12, pp. 31314–31338, Dec. 2015. 1 https://gist.github.com/robguinness/4667980

[2] J. Lester, T. Choudhury, and G. Borriello, A practical approach to recognizing physical activities, in 2006 Proc. Int. Conf. Pervasive Computing, pp. 1–16. [3] F. Foerster and J. Fahrenberg, Motion pattern and posture: Correctly assessed by calibrated accelerometers, Research Methods, Instruments, & Computers, vol. 32, no. 3, pp. 450–457, Sept. 2000. [4] A. Mannini and A. M. Sabatini, Machine learning methods for classifying human physical activity from on-body accelerometers, Sensors, vol. 10, no. 2, pp. 1154–1175, Feb. 2010. [5] A. Chambolle, An algorithm for total variation minimization and applications, Journal of Mathematical Imaging and Vision, vol. 20, no. 1-2, pp. 89–97, Jan. 2014. [6] L. Breiman, Random forests, Machine Learning, vol. 45, no. 1, pp. 5–32, Oct. 2001. [7] A. M. Khan, Y. K. Lee, S. Y. Lee, and T. S. Kim, Human activity recognition via an accelerometer-enabled-smartphone using kernel discriminant analysis, in 2010 Proc. Int. Conf. Future Information Technology, pp. 1–6. [8] J. R. Kwapisz, G. M. Weiss, and S. A. Moore, Activity recognition using cell phone accelerometers, ACM SIGKDD Explorations Newsletter, vol. 12, no. 2, pp. 74–82, Dec. 2011. [9] A. Bayat, M. Pomplun, and D. A. Tran, A study on human activity recognition using accelerometer data from smartphones, in Proc. Procedia Computer Science, vol. 34, , pp. 450–457, 2014. [10] T. Brezmes, J.-I. Gorricho, and J. Cotrina, Activity recognition from accelerometer data on a mobile phone, in 2009 Proc. Int. Work-Conf. Artificial Neural Networks, pp. 796–799. [11] L. Sun, D. Zhang, B. Li, B. Guo, and S. Li, Activity recognition on an accelerometer embedded mobile phone with varying positions and orientations, in 2010 Proc. Int. Conf. Ubiquitous Intelligence and Computing, pp. 548–562. [12] S. Zhang, P. McCullagh, C. Nugent, and H. Zheng, Activity monitoring using a smart phone’s accelerometer with hierarchical classification, in 2010 Proc. Int. Conf. on Intelligent Environments, pp. 158–163. [13] J. Yang, Toward physical activity diary: Motion recognition using simple acceleration features with mobile phones, in 2009 Proc. Int. Workshop Interactive Multimedia for Consumer Electronics, pp. 1– 10. [14] S.-M. Lee, S. M. Yoon, and H. Cho, Human activity recognition from accelerometer data using convolutional neural network, in 2017 Proc. IEEE Int. Conf. Big Data and Smart Computing, pp. 131–134. [15] J. W. Lockhart and G. M. Weiss, The benefits of personalized smartphone-based activity recognition models, in Proc. SIAM Int. Conf. Data Mining, pp. 614–622. [16] M. Shoaib, H. Scholten, and P. J. M. Havinga, Towards physical activity recognition using smartphone sensors, in 2013 Proc. IEEE Int. Conf. Ubiquitous Intelligence and Computing, pp. 80–87. [17] A. Wang, G. Chen, J. Yang, S. Zhao, and C. Y. Chang, A comparative study on human activity recognition using inertial sensors in a smartphone, IEEE Sensors Journal, vol. 16, no. 11, pp. 4566–4578, June 2016. [18] R. E. Guinness, Beyond where to how: a machine learning approach for sensing mobility contexts using smartphone sensors, Sensors, vol. 15, no. 5, pp. 9962–9985, Apr. 2015. [19] M. Shoaib, S. Bosch, O. D. Incel, H. Scholten, and P. J. Havinga, A survey of online activity recognition using mobile phones, Sensors, vol. 15, no. 1, pp. 2059–2085, Jan. 2015. [20] S. Dernbach, B. Das, N. C. Krishnan, B. L. Thomas, and D. J. Cook, Simple and complex activity recognition through smart phones, in 2012 Proc. Int. Conf. Intelligent Environments, pp. 214–221. [21] M. Zeng, L. T. Nguyen, B. Yu, O. J. Mengshoel, J. Zhu, P. Wu, and J. Zhang, Convolutional neural networks for human activity recognition using mobile sensors, in 2014 Proc. Int. Conf. Mobile Computing, Applications and Services, pp.197–205. [22] J. B. Yang, M. N. Nguyen, P. P. San, X. L. Li, and S. Krishnaswamy, Deep convolutional neural networks on multichannel time series for human activity recognition, in 2015 Proc. Int. Joint Conf. Artificial Intelligence, pp. 995–4001. [23] C. A. Ronao and S.-B. Cho, Recognizing human activities from smartphone sensors using hierarchical continuous hidden Markov models, International Journal of Distributed Sensor Networks, vol. 13, no. 1, pp. 1–16, Jan. 2017. [24] A. Karpathy, G. Toderici, and S. Shetty, Large-scale video classification using convolution neural networks, in 2014 Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1725–1732.