IEEE COMMUNICATIONS LETTERS, VOL. 14, NO. 5, MAY 2010

1

Improving IEEE 1588v2 Clock Performance through Controlled Packet Departures Brent Mochizuki, Student Member, IEEE, and Ilija Hadˇzi´c, Member, IEEE

Abstract—Packet delay variation (PDV) is the dominant impairment in packet-based synchronization systems. One way to mitigate its effect is to apply advanced filtering techniques on phase error information derived from packet arrival events. In this letter we consider an alternative approach in which the backpressure to the background traffic source is coordinated with timing packet generation such that the PDV is completely eliminated. Although the proposed method is limited to tree network topologies, the results are notable due to complete elimination of the PDV noise even if the background traffic load approaches 100%.

S

SYNC traffic M

S S

Background traffic

Fig. 1. Simplified mobile backhaul network topology with background and synchronization traffic generated at the root node.

Index Terms—Synchronization, Ethernet, IEEE 1588v2, PDV.

I. I NTRODUCTION

W

E consider a network of packet switches organized in a tree topology shown in Fig. 1. Slaves (S) located at the leaves synchronize their frequencies to the master (M) located at the root using the Precision Time Protocol (PTP), specified by the IEEE 1588v2 standard [1]. The master node broadcasts (or multicasts) SYNC packets (timing packets defined by the standard) to the slaves. Each slave uses the departure timestamp contained in the SYNC packet and a locally generated arrival timestamp to derive the phase error that is used to discipline the slave’s local oscillator using a phase-locked loop (PLL). Background packets (data traffic) are also injected at the root node and follow the same path towards the leaves. These packets interfere at each packet switch with SYNC packets. A practical example that corresponds to this topology is a mobile backhaul network in which the leaves are the base stations and the root is the switching center that controls the network, handles the base station traffic, and provides synchronization service for the network [2]. We have previously shown [3] that in this type of network, although the background traffic is injected at only one point (the root), SYNC packets can incur significant delay variation. This remains the case even if all switches assign the highest queuing priority to the timing traffic. Because the phase noise in the PLL is directly related to the packet delay variation (PDV), keeping the PDV low is crucial for the stability of the recovered clock [4]. A common method is to send SYNC packets at a higher rate than the PLL sampling rate and to

Manuscript received January 4, 2010. The associate editor coordinating the review of this letter and approving it for publication was S. Pierre. B. Mochizuki is currently with the University of Illinois. The work described in this letter was performed in its entirety while he was with AlcatelLucent, Bell Labs. I. Hadˇzi´c is with Alcatel-Lucent, Bell Labs (e-mail: [email protected]). The authors thank Martin Carroll and Dennis Morgan of Alcatel-Lucent for their helpful comments. Digital Object Identifier 10.1109/LCOMM.2010.05.100014

use a packet filter to select only the packets that were least affected by the queuing delay. The most widely used filter is based on the sample-minimum of the difference between the arrival and departure timestamps [5], [6]. This approach, however, does not guarantee stable clock recovery and, in the network topology considered here, yields significant performance degradation even for relatively low traffic loads [3]. Alternatively, one can use the transparent clock feature of PTP [1], but such an approach requires that switches modify the SYNC packet payload, which might not be practical. II. C OORDINATED BACKPRESSURE AND SYNC PACKET D EPARTURES Assuming that the background traffic source can be backpressured [7], it is possible to reserve idle “headroom” before each SYNC packet such that the packet does not incur any queuing delay in the network. Fig. 2 shows how a SYNC packet is delayed through the network. Because of the commonly used store-and-forward architecture of packet switches, a long packet can easily inflict an additional delay on a short packet that follows it [8]. The figure depicts a situation in which the SYNC packet (gray) is delayed by 𝐷 time units before the third hop, because a background packet (white) is still being transmitted after the SYNC packet was (fully) received and stored in the queue. Delay 𝐷 is a random variable that depends on the background packet distribution size, the number of hops in the path, and the network load. In [3] we reported the distribution of 𝐷 and showed that it is not at all amenable to sample-minimum packet filtering even for modest traffic loads (e.g., 40%). Fig. 3 shows how the SYNC packet delay can be minimized if sufficient idle time is reserved between its start and the end of the preceding background packet. This “headroom” ensures that the SYNC packet does not encounter a busy transmission channel; note that the gap between the two packets narrows along the path, but never enough to cause additional queuing delays. To achieve a favorable transmission pattern, the synchronization master (SYNC packet source)

c 2010 IEEE 1089-7798/10$25.00 ⃝

2

IEEE COMMUNICATIONS LETTERS, VOL. 14, NO. 5, MAY 2010

Sbg

t bg

Ssync

t sync

D

background; for specific packets we substitute the subscripts “sync” and “bg”. Substituting (2) into (1) and solving for 𝑠sync yields

Hop 1

𝑠sync = 𝑠bg +

Hop 2

𝑖=1

(𝑖)

(𝑁 ) (𝑡bg − 𝑡(𝑖) sync ) + 𝑡ipg + 𝑡sync .

(3)

The above expression can be simplified if we assume that all links in the network are of the same rate — in other words, that the transmission delay is the same for all links. Hence,

Hop3 Fig. 2.

𝑁 ∑

𝑠sync = 𝑠bg + 𝑁 (𝑡bg − 𝑡sync ) + 𝑡ipg + 𝑡sync .

SYNC packet delay without stalling the background traffic.

(4)

III. O PTIMIZING THE G AP T IMER Hop 1 Hop 2 Hop 3 r

Fig. 3.

r

bg

sync

SYNC packet delay with stalling the background traffic.

must also be the node that multiplexes SYNC and background packets. The multiplexing state machine of the master stores the background packets in the local buffer and transmits them normally whenever there is no SYNC packet to be transmitted. As a scheduled SYNC packet transmission time approaches, the state machine finishes the background packet in progress, stops any transmission, and starts the gap timer. After the gap timer expires, the multiplexing state machine sends the SYNC packet and resumes normal background traffic transmission. During the gap time, a backlog may build in the local buffer, so it may be necessary to occasionally backpressure the background traffic source. To avoid wasting network bandwidth, the gap time should be set to the smallest value that is guaranteed to yield the situation depicted in Fig. 3 — the SYNC end-of-packet arrival time at the slave, 𝑟sync , should relate to that of the background packet, 𝑟bg , as 𝑟sync = 𝑟bg + 𝑡ipg +

) 𝑡(𝑁 sync ,

(1)

(𝑁 )

where 𝑡sync is the time required to transmit the SYNC packet over the last (𝑁 th) hop and 𝑡ipg is the inter-packet gap — the idle time between consecutive packets, mandated by many packet transmission protocols (e.g., Ethernet). If we assume no queuing delay, the store-and-forward nature of a packet switch is captured by the following relationship: 𝑟pkt = 𝑠pkt +

𝑁 ∑ 𝑖=1

(𝑖)

(𝑖)

(𝑖)

(𝑖)

(𝑡pkt + 𝜏pkt ),

(2)

where 𝑡pkt and 𝜏pkt are the transmission and propagation times over the 𝑖th hop, 𝑁 is the number of hops in the path, and 𝑠pkt is the transmission start time of a packet at the root node. We use subscript “pkt” to indicate any packet, SYNC or

A constant gap time can be calculated from (4) and configured when the network is deployed, using the values for 𝑁 and 𝑡bg that correspond to the longest path through the network and the maximum packet size allowed by the protocol. We use a more sophisticated and network bandwidth conserving approach that, for each SYNC packet, calculates a gap time that is just large enough for the channel to clear. Given a background packet in isolation, the earliest time that the following SYNC packet can be safely transmitted is given by (4). This equation does not account for the case when there are two background packets transmitted such that the second packet is delayed along the network path, which in turn inflicts additional delays on a SYNC packet that may follow. When this case occurs, the 𝑠sync must be postponed by the transmission time of the background packet, plus the inter-packet gap time. Hence, whenever the multiplexing state machine observes a background packet, it recalculates the safe transmission time of a SYNC packet as ( 𝑠sync [𝑝 + 1] = max 𝑠sync [𝑝] + 𝑡bg [𝑝] + 𝑡ipg , ) (5) 𝑠bg [𝑝] + 𝑁 (𝑡bg [𝑝] − 𝑡sync ) + 𝑡ipg + 𝑡sync . Note that 𝑠sync [𝑝] merely represents the time when the SYNC packet can be transmitted. When the transmission actually occurs also depends on the configured SYNC packet rate, which together with (5) determines the gap time value. IV. R ESULTS We implemented the multiplexing state machine in an FPGA and integrated it with our general implementation of the IEEE 1588v2 master. The system had two gigabit Ethernet ports, one to interface with the background traffic source and one to interface to the network; SYNC packets were generated and combined with the background packet flow inside the FPGA. We constructed a network path consisting of 15 commodity Ethernet switches and used a Smartbits SMB-6000B system with a LAN-3320A module to generate the background traffic. We generated load patterns of various packet-size distributions and load levels, collected the network transit time measurements for SYNC packets over 10-minute periods (about 40000 SYNC packets at a 64 packets/second rate), and constructed an empirical probability density function (PDF) for each test case. The results for one test case are shown in Figs. 4 and 5. The background traffic was 400 Mb/s (i.e. 40% utilization for Gigabit Ethernet network) with the

ˇ C: ´ IMPROVING IEEE 1588V2 CLOCK PERFORMANCE THROUGH CONTROLLED PACKET DEPARTURES MOCHIZUKI and HADZI 4000 phase error [ns]

PDF [1/µs]

0.015

0.01

0.005

0 80

Fig. 4.

3

2000 0 −2000 −4000

100

120

140

160 180 200 packet delay [µs]

220

240

260

280

0

5000

10000

15000

time [s]

Fig. 6.

SYNC packet delay PDF (uncontrolled departures).

Recovered clock performance.

PDF [1/µs]

3

2

1

0 28.6

Fig. 5.

28.8

29

29.2

29.4 29.6 29.8 packet delay [µs]

30

30.2

30.4

30.6

SYNC packet delay PDF (coordinated departures).

packet size uniformly distributed between 64 and 1500 bytes; the SYNC packet size was 64 bytes. Fig. 4 shows the case in which SYNC packet flow is generated without any coordination with the background traffic. Packet delays over a 16-hop network range between 98.5𝜇s and 274.4𝜇s, with a mean delay of 215.5𝜇s, and a standard deviation of 30.5𝜇s. When the packet-multiplexing state machine is turned on, the delay range reduces to between 28.7𝜇s and 30𝜇s and the standard deviation narrows to 170ns. The residual Gaussian-like PDV is attributed to packet processing time in the switches and is practically irrelevant for the recovered clock performance. The constant component of the delay is attributed to transmission and cable propagation delays in the network. As anticipated, the PDV resulting from queuing has been completely eliminated. To further validate our results, we characterized the recovered clock performance using our implementation of the IEEE 1588v2 slave. Fig. 6 shows the accumulated phase error (time integral of frequency error, also known as the time-interval error) of the recovered clock. The dashed line represents the result for the delay distribution of Fig. 4, whereas the solid line represents the case of Fig. 5. As a final note, we address the network bandwidth penalty concern. Namely, stalling the background traffic flow means that the network cannot be fully utilized. It is of interest to calculate the maximum achievable utilization when the packet-multiplexing state machine is turned on. For the packet scheduling of (5) and fixed background packet size, the maximum background throughput is ) ( 𝜂 = 1 − 𝑅sync ⋅ (𝑁 − 1)(𝑡bg − 𝑡sync ) + 𝑡sync + 𝑡ipg , (6) where 𝑅sync is the SYNC packet transmission rate. Sweeping (6) with some typical values (e.g., 64–1500 byte background packet size range, 16 hops, 64 SYNC/s rate, 1Gb/s links, etc.), leads to the conclusion that the utilization is 98% or higher — a negligible penalty for the achieved performance. We have experimentally confirmed this estimate by configuring the Smartbits LAN-3320A card to fully saturate the link, but to

obey backpressure requests (PAUSE packets) received by our master system. For fixed background packet size, the resulting rate generated by the LAN-3320A card was consistent with the predictions of (6). For variable background packet size, calculating the utilization is more difficult and the expression depends on the packet size distribution, so we only report an experimental result. We used uniformly distributed packet size between 64 and 1518 bytes and measured a throughput of 953.6Mb/s. The maximum achievable throughput was 975Mb/s (781 byte mean packet size, 1Gb/s transmission rate, 12-byte inter-packet gap, and 8-byte preamble), so the utilization was 97.8%. V. C ONCLUSIONS We have shown that, for a certain class of networks, a relatively simple modification to the SYNC packet departure process can completely eliminate the PDV from the frequency tracking PLL. The proposed scheme is compatible with existing standards and is simple to implement. Once the PDV is eliminated, the slave can use every SYNC packet to discipline the oscillator (i.e., the sample-minimum packet filter used by typical clock recovery algorithms can be turned off), which increases the PLL bandwidth by two orders of magnitude. This approach significantly improves the ability to track the local oscillator variations, which, in the absence of PDV, is the main contributor to output wander. R EFERENCES [1] “Draft standard for a precision clock synchronization protocol for networked measurement and control systems,” IEEE Draft P1588/D2.2, Dec. 2007. [2] M. Gasparroni, “Synchronisation in future VF mobile networks,” in The 5th Int. Telecom. Sync. Forum (ITSF), Nov. 2007. [3] I. Hadˇzi´c and D. R. Morgan, “On packet selection criteria for clock recovery,” in Proc. IEEE Int. Symp. Precision Clock Synchronization Meas., Contr., Commun. (ISPCS), Oct. 2009, pp. 35–40. [4] R. Subrahmanyan, “Implementation considerations for IEEE 1588v2 applications in telecommunications,” in Proc. IEEE Int. Symp. Precision Clock Synchronization Meas., Contr., Commun. (ISPCS), Oct. 2007, pp. 148–154. [5] D. Veitch, J. Ridoux, and S. B.Korada, “Robust synchronization of absolute and difference clocks over networks,” IEEE/ACM Trans. Networking, vol. 17, no. 2, pp. 417–430, Apr. 2009. [6] G. Dowd and G. Zampetti, “Characterizing network synchronization potential with the minTDEV statistic,” in Proc. IEEE Int. Symp. on Precision Clock Synchronization for Meas., Control and Commun., Sep. 2008, pp. 98–104. [7] “Carrier sense multiple access with collision detection (CSMA/CD) access method and physical layer specification—annex 31b, MAC control PAUSE operation,” IEEE Std. 802.3-2005, pp. 763–772, Dec. 2005. [8] T. Frost, “Topology-dependent blocking mechanisms in packet networks,” contribution to ITU-T, SG15, Q13, standards meeting, Sep. 2007, ITU Working Document WD49.