The 11th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI 2014) Nov. 12 – 15, 2014 at Double Tree Hotel by Hilton, Kuala Lumpur, Malaysia

Robust Self-localization of Ground Vehicles Using Artificial Landmark Jae-Yeong Lee and Wonpil Yu Electronics and Telecommunications Research Institute, Daejeon 305-700, Korea ([email protected], [email protected]) Abstract - Stable and accuracy localization has crucial importance on the success of autonomous navigation of robotic ground vehicles. In this paper, we present a landmark-based sensor fusion localization method that utilizes artificial landmarks, a cheap GPS sensor, and wheel odometry. We describe overall architecture of the proposed localization system and technical details of landmark-based localization algorithm. The developed landmark-based localization system gives centimeters localization accuracy and real time performance more than 100 Hz. The experimental results on extensive real video sequences confirms the effectiveness and reliability of the method, showing successful detection of most landmarks with almost no false detection. Keywords - localization, artificial landmark, sensor fusion.

1. Introduction Stable localization has crucial importance on the success of autonomous navigation of robotic vehicles. In urban environment, most common approach for this problem is to use sensor fusion of a GPS sensor and other supplementary sensors like wheel odometry, INS, gyro, or digital compass [1], [2], [3]. However, the problem of GPS-based approach is that the final localization accuracy is highly dependent on the accuracy of the used GPS sensors but the price of GPS sensor with high accuracy of centimeters degree is too expensive to be applicable for practical applications. In case of cheap GPS sensors the localization accuracy is about ±10 meters and generally not acceptable for navigation purpose. Another problem of GPS-based approach is that even with high accuracy GPS sensors the sensor output becomes unstable and may fluctuate especially near tall buildings or under forest. Vision-based localization is another possible approach and gives quite good results for some applications [4], [5]. However it is well known that it is very hard to meet robustness against the change of the environment such as weather variation, day and night variation, view point change of camera, and so on. In this paper, we present a landmark-based sensor fusion localization method for outdoor navigation that utilizes artificial landmarks, a cheap GPS sensor, and wheel odometry. Artificial landmarks are adopted to overcome the aforementioned limit of vision-based approaches and to give reliable localization performance. The main purpose of our work is to provide reliable location information of centimeters accuracy without relying on expensive sensors like high accuracy DGPSs (Differential Global Positioning Systems) and thus provide a practical solu-

Camera Image

Landmark Detection

Landmark Validation Landmark Map

Landmark Identification Mark-based Localization

GPS Wheel Odometry

EKF Localization

Fig. 1 System architecture of the developed localization system.

tion for the localization of robotic vehicles in urban environment. This paper can be viewed as an extension of [6]. In [6], the authors presented a basic idea of using artificial landmarks for outdoor localization in urban environment and described overall process and sensor fusion architecture to build a localization system. In this paper, we focus on the technical details of the developed localization system and describe method for landmark detection, validation, identification of the detected landmarks, and final EKFbased localization system. The paper is organized as follows. In Sect.2 we describe overall system architecture of the proposed localization system briefly. In Sect. 3 we describe vision algorithm to detect landmarks from an image. A geometric validation method for reducing false detections is also described. Landmark identification and sensor fusion method for final localization is described in Sect. 4. Experimental results is presented in Sect. 5 and conclusion is given in Sect. 6.

2. System Overview Figure 1 shows overall system architecture of the developed landmark-based localization system. Artificial landmarks are attached or printed on the road along the navigation path of a ground vehicle and a front camera mounted on the vehicle captures images continuously. Input camera images are processed by landmark detection module, giving landmark candidates. Falsely detected landmark candidates then are filtered out by checking if

Fig. 2 Developed landmarks. the camera pose computed from the detected candidate is within a valid range or not. The landmark identification module determines the ID of each detected landmark based on the landmark map and current estimate of vehicle location from EKF localization module. Once landmarks are identified, the vehicle location is computed from the world coordinate of the detected landmark. The estimated location from landmark then is given as an input to EKF localization module and filtered with GPS and wheel odometry, giving final location of a vehicle.

3. Landmark Detection 3.1 Landmark Figure 2 shows developed artificial landmarks which consist of type A and type B. Type A is a ’U’ shape mark and type B consists of a pair of bars. The landmarks are designed to be well-matched with real markers on the road environment and have 60 cm × 60 cm size and 15 cm line width. For type A landmark we can define an origin of the mark uniquely and therefore it gives us an unique camera pose once it is detected. However, type B landmark has ambiguity of orientation of 180 degree and thus gives two possible camera poses. Despite this ambiguity type B landmark is designed to use because its shape is more similar and harmonious with real lane markers. The developed landmarks have been designed to be as simple as possible for the robustness of vision processing and easy detection at a long distance. For this purpose the developed landmarks do not contain identification information in their pattern and thus additional identification process is required. The identification step is described in Subsect. 4.1 . 3.2 Candidate detection Image processing steps required for the detection of landmark candidates are depicted in Fig. 3. Firstly an input image is thresholded by a locally adaptive thresholding where local threshold value of each image pixel is determined based on the intensity mean of the local neighbors of the pixel, giving a binary image. From the binary image, image contours are extracted and each connected contours is approximated by a polygon. We next check shape consistency of the approximated polygons to identify true landmark candidates. In order to check if an approximated polygon has valid

Fig. 4 Verification of landmark candidate by checking shape of approximated polygon. shape similar with landmark we first compute a minimum bounding quadrangle enclosing the polygon. Examples of computed bounding quadrangles are shown by blue quadrangles in Fig. 4. And then we estimate a homography transformation H from the bounding quadrangle to a predefined rectangle and transforms all other vertexes of the polygon by using the estimated homography H. Next we check if each of the transformed vertexes is wellmapped into the right vertex position in the predefined rectangle. With this shape checking most of the nonlandmark candidates are successfully filtered out. Decision on the type of landmark candidates (type A, type B) is made based on the ratio of the areas of the bounding quadrangle and the polygon. 3.3 Geometric validation It is most important to minimize false detections of landmark for successful application of localization system. However, it is generally hard to remove false detections completely using only visual appearance because there may be lots of natural patterns similar with the landmark in natural scene. To overcome this difficulty we introduce a pose-based validation step. The proposed validation method is as follows. We first estimate relative 6 DOF (degree of freedom) camera pose (x, y, z, pitch, roll, yaw) from each landmark candidate [7]. It is well known that we can estimate camera pose uniquely from four correspondences between world coordinates and image coordinates of an object. We then check if the estimated camera pose is within a valid range or not and filter out false detections whose pose is out of valid range. Note that estimated height, pitch, and roll of a camera can be used as powerful cues for the validation because a camera is mounted on the vehicle with fixed alignment. The proposed geometric validation method is simple but very effective and is able to remove most of

Fig. 3 Detection of landmark candidates. (a) Input image. (b) Binary image by adoptive thresholding. (c) Contour extraction. (d) Polygon approximation.

Fig. 5 An example of landmark-based localization system for robotic shuttle service built in National Science Museum, Daejoen Korea. Total 22 landmarks are used to cover a navigation path which is about 500 meter length.

Fig. 6 EKF sensor fusion architecture of landmark localization, GPS, and wheel odometry.

to compute D. 4.2 EKF sensor fusion

probable false detections effectively.

4. Localization 4.1 Landmark identification With only using single landmark the localization coverage is limited within camera view. Therefore in real applications it is better to use multiple landmarks such that landmarks are attached or printed on the road in some interval along the navigation path of a ground vehicle (Fig. 5. In that case an inheriting problem is to identify each detected landmark because we use the landmarks that have no identification information except for type difference. The identification method used in this paper is as follows. Let L1 , L2 , ..., Ln be all the landmarks used in a localization system. The world coordinate of each landmark is assumed to be known and stored a landmark map. Let L be an unknown landmark detected from current input image. We first compute camera poses for all possible identifications by assuming that L is Li for i = 1, ..., n. Let P be a predicted camera pose and Pi be the camera pose computed from the assumption that L is Li . The predicted camera pose P can be obtained from the localization system by setting it as the latest localization output. Then, L is identified as the landmark that gives the closest Pi to P as follows: L = arg min D(P, Pi ), Li

(1)

where D(·) denotes a distance measure between two camera pose P and Pi . We use Mahalanobis distance

Although the landmark-based localization gives exact localization with centimeters accuracy, the coverage of localization is limited within camera view. Therefore it is more efficient to combine the landmark-based localization with other supplementary localization sensors like GPS and wheel odometry to provide seamless localization. Another necessity for the sensor fusion is that seamless localization is required to identify landmarks when multiple landmarks are used in the system. Although it is possible to build a localization system using only landmarks if we place landmarks very closely, it is more efficient to use a sensor fusion approach if possible. Figure 6 shows the EKF(extended Kalman fiter) sensor fusion architecture. As shown in the figure, we use the localization result from landmark detection to reset EKF state with the landmark localization result instead of embedding landmark localization into ordinary EKF update loop. It is because it is better to set the current location of vehicle with the landmark localization result whenever a landmark is detected since landmark localization gives centimeters accuracy.

5. Experiment 5.1 Localization accuracy In our previous work [6] we presented experimental results on the localization accuracy of the proposed landmark-based localization. The previous experiment was performed by measuring position and orientation errors of the estimate camera poses for predefined sixteen

Seq. # Frames # GT Recall 2,203 495 0.66 #1 2,203 585 0.89 #2 516 218 0.87 #3 2,203 680 0.84 #4 1,104 197 0.65 #5 2,203 754 0.79 #6 666 79 0.52 #7 2,222 467 0.61 #8 2,222 561 0.82 #9 1,270 222 0.61 #10 Total 16,812 4,258 0.76

# False # False with G.V. 18 0 6 0 27 0 2 0 3 0 19 0 2 1 54 0 17 0 15 0 163 1

Table 2 Detection rate and false alarm rate of the developed landmark detection system with and without geometric validation. Fig. 7 Experimental setting for the evaluation of localization accuracy . Distance (meter) 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 Average

Position Error (meter) 0.003 0.014 0.035 0.030 0.043 0.056 0.086 0.124 0.120 0.131 0.085 0.066

Orientation Error (degree) 0.0 0.3 0.7 0.4 0.6 0.6 1.0 1.5 1.3 0.9 0.9 0.7

Table 1 Localization accuracy with respect to the distance from camera to landmark.

landmarks positions and orientations, giving 3.3 centimeters of displacement error and 0.3 degree of orientation error on average. However, it was not evaluated previously to what distance the algorithm can detect landmarks and how localization accuracy varies depending on the distance. The experimental setting for measuring detection range and localization accuracy of the landmark-based localization system is similar with [6]. A camera is placed at a fixed position of 1.4 meter height from the ground and a landmark was moved on the ground at regular interval of 0.5 meter starting from 1.0 meter distance from the camera (Fig. 7). For each landmark position localization errors were evaluated more than 20 times and averaged. Table 1 shows the experimental result. In the experiment we were able to detect successfully the landmark until 6 meter distance from the camera when using VGA image resolution. As for localization accuracy we can observe that the localization error has a tendency of increasing as the distance between a camera and a landmark increases, showing 6.6 centimeter of displacement error and 0.7 degree of orientation error on average.

The processing time for detecting landmarks and estimating camera pose on a full VGA image was 132 fps on average on i7 CPU. 5.2 Detection rate and false alarm rate The second experiment evaluates the detection rate and false alarm rate of the developed landmark-based localization system on real video sequences. The experiment was performed for two different configurations. One configuration is to detect landmark using only appearance information of the landmark (pure vision-based detection) and the other configuration is to filter out false detections by geometric validation which is described in Subsect. 3.3 . The test dataset consists of 10 video sequences which were recorded from on-board camera of a robot shuttle at different time along the navigation path of robotic shuttle service built in National Scient Museum (Fig. 5). Each video sequence consists of a few thousand of image frames and the total number is 16,812 image frames. The camera model is PGR CMLN-13S2C and the resolution of video frames is 640 × 480. Ground truth positions of landmarks in test dataset were labeled manually. Table 2 shows true detection rate (Recall), false detections without geometric validation (False), and false detections with geometric validation (False with G.V.) for each test sequence. The experimental results show that the system was able to detect 76 percent of landmarks in the test dataset successfully on average. Most of the missed detections were occurred when the camera are far away from the landmarks. Note that the 76 percent of detection rate is sufficient for most sensor fusion applications where localization error is corrected whenever a landmark is detected. The total number of false detections from 10 video sequences was 163 with pure vision-based detection without geometric validation. However with the geometric validation the false detections were dramatically reduced and we had only one false detection through whole processing of the 16,812 image frames. Another point is that the true detection rate did not decrease with the geometric

[3]

[4]

[5]

Fig. 8 Landmark detection examples. Yellow colored detections denote candidate detections that are filtered out successfully by geometric validation. The last figure (right bottom) show the false detection that failed to be filtered out with geometric validation. validation and was preserved the same. Figure 8 shows several examples of landmark detection result.

6. Conclusion In this paper we presented a vision-based sensor fusion localization method for ground vehicles in urban environment. The developed localization system utilizes two types of artificial landmarks attached on the ground and gives centimeters localization accuracy at 132 fps without aid of any expensive location sensors. Another appealing point of the developed system is that the system is able to detect landmarks with nearly zero false detections with the proposed geometric validation, giving stable localization for practical applications. The experimental results on extensive dataset confirms the effectiveness and reliability of the proposed system.

Acknowledgement This work was supported by the R&D program of the Korea Ministry of Trade, Industry and Energy (MOTIE) and the Korea Evaluation Institute of Industrial Technology (KEIT). (The Development of Low-cost Autonomous Navigation Systems for a Robot Vehicle in Urban Environment, 10035354).

References [1]

Borenstein, J., Everett, H. R., and Feng, L. (1996). Where am I? Sensors and methods for mobile robot positioning. University of Michigan, 119(120), 15. [2] Chae, H., Choi, S., Yu, W., and Cho, J. (2010, October). Autonomous navigation of mobile robot based on DGPS/INS sensor fusion by EKF in semi-outdoor structured environment. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on (pp. 1222-1227). IEEE.

[6]

[7]

Munkhzul Enkhtur, Seong Yun Cho, and KyongHo Kim, ”Modified Unscented Kalman Filter for a Multirate INS/GPS Integrated Navigation System,” ETRI Journal, vol. 35, no. 5, pp. 943-946, Oct. 2013. Se, S., Lowe, D. G., and Little, J. J. (2005). Visionbased global localization and mapping for mobile robots. Robotics, IEEE Transactions on, 21(3), 364375. Brubaker, M. A., Geiger, A., and Urtasun, R. (2013, June). Lost! leveraging the crowd for probabilistic visual self-localization. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (pp. 3057-3064). IEEE. Jae-Yeong Lee, Heesung Chae, and Wonpil Yu, ”Development of Mark-based Localization System for Outdoor Navigation,” 9th Korea Robotics Society Annual Conference (KRoC’14), 2014. V. Lepetit, F. Moreno-Noguer and P. Fua. EPnP: An Accurate O(n) Solution to the PnP Problem, in International Journal Of Computer Vision, vol. 81, p. 155-166, 2009.