Time and Timestamping in Softwarized Environments Tal Mizrahi, Yoram Moses Technion – Israel Institute of Technology
April 2016
The First Synchronized Clocks
Railway switch
The TimedSDN Project
2
Networks: Communications vs. Trains
SDN Controller
Train Control Center
Network switch
Railway switch The TimedSDN Project
3
Timed
at a Glance Controller
A protocol allowing the controller to schedule network updates
Synchronized clocks
Switches The TimedSDN Project
4
Accurate Synchronization: It’s Already Here Precision Time Protocol (PTP) [IEEE 1588 2008] ~1μsec accuracy
Mobile backhaul
Industrial automation
But what about Software Defined Networks?
Power substations
Open Networking Foundation (ONF) SDN product directory
Financial applications
~70%
~70% of the SDNenabled silicons have native PTP support.
China Mobile: over 1,000,000 PTP-enabled base stations https://mailarchive.ietf.org/arch/attach/tictoc/pdfsY1ADO.pdf The TimedSDN Project
5
Timed
at a Glance Controller
Challenge: A protocol allowing the controller to schedule network updates
Synchronized clocks
Switches The TimedSDN Project
6
The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization
Project
Flow Swaps TIME4 [INFOCOM, ‘16]
Multi-phase updates [SOSR, ‘15]
Data Plane Timestamping [SWFAN, ‘16]
ONECLOCK to Rule Them All [NOMS, ‘16]
OpenFlow Scheduled Bundles
NETCONF Time Capability
[INFOCOM ’16, OpenFlow 1.5]
[NOMS ’16, RFC7758]
TIMEFLIP
ONECLOCK
[INFOCOM ‘15]
[NOMS ‘16]
REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project
7
The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization
Project
Flow Swaps TIME4 [INFOCOM, ‘16]
Multi-phase updates [SOSR, ‘15]
Data Plane Timestamping [SWFAN, ‘16]
ONECLOCK to Rule Them All [NOMS, ‘16]
OpenFlow Scheduled Bundles
NETCONF Time Capability
[INFOCOM ’16, OpenFlow 1.5]
[NOMS ’16, RFC7758]
TIMEFLIP
ONECLOCK
[INFOCOM ‘15]
[NOMS ‘16]
REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project
8
Time is Optimal for Flow Swaps
[INFOCOM ‘16]
A key benefit of SDN: Dynamic path allocation based on network load
old new
o1
...
controller o2
The TimedSDN Project
9
Time is Optimal for Flow Swaps
[INFOCOM ‘16]
A key benefit of SDN: Dynamic path allocation based on network load If O1 , O2 are not updated at the same time…
old new
Temporary congestion.
o1
...
controller o2
O2 is updated first. The TimedSDN Project
10
Time is Optimal for Flow Swaps
[INFOCOM ‘16]
A key benefit of SDN: Dynamic path allocation based on network load
TIME4
old new
If O1 , O2 are updated at the same time: No congestion!
o1
...
controller o2
The TimedSDN Project
11
The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization
Project
Flow Swaps TIME4 [INFOCOM, ‘16]
Multi-phase updates [SOSR, ‘15]
Data Plane Timestamping [SWFAN, ‘16]
ONECLOCK to Rule Them All [NOMS, ‘16]
OpenFlow Scheduled Bundles
NETCONF Time Capability
[INFOCOM ’16, OpenFlow 1.5]
[NOMS ’16, RFC7758]
TIMEFLIP
ONECLOCK
[INFOCOM ‘15]
[NOMS ‘16]
REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project
12
Example: Consistent Path Update before after
Sequential update approaches: [Dionysus ‘14] [zUpdate ‘13] [Vanbever et al. ‘11] [Francois et al. ’07]
0. 1. 2. 3.
The ‘before’ configuration. Controller updates S1. Controller updates S2. Controller updates S3.
X
S3
S4
S3
S4
S2
S1
S2
S1
0
wait…
S3
S4
S2
S1
1
2 The TimedSDN Project
wait…
S3
S4
S2
S1
S3
S4
S2
S1
3 13
Simultaneous Updates? En-route packets run into a ‘black hole’. Not consistent!
S3
S4
S3
S4
S3
S4
S3
S4
S2
S1
S2
S1
S2
S1
S2
S1
1
2 The TimedSDN Project
3 14
Timed Multi-phase Consistent Updates [SOSR ‘16] - The controller sends timed update messages to S1, S2, S3. - Scheduled updates occur at times T1, T2, T3.
Controller does not need to wait between steps!
S3
S4
S3
S4
S3
S4
S3
S4
S2
S1
S2
S1
S2
S1
S2
S1
T1
The TimedSDN Project
T2
T3
15
The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization
Project
Flow Swaps TIME4 [INFOCOM, ‘16]
Multi-phase updates [SOSR, ‘15]
Data Plane Timestamping [SWFAN, ‘16]
ONECLOCK to Rule Them All [NOMS, ‘16]
OpenFlow Scheduled Bundles
NETCONF Time Capability
[INFOCOM ’16, OpenFlow 1.5]
[NOMS ’16, RFC7758]
TIMEFLIP
ONECLOCK
[INFOCOM ‘15]
[NOMS ‘16]
REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project
16
Using Timestamps instead of The advantages of timestamping are well-known, e.g., [Lamport ‘78]. Current work: Data Plane Timestamping (DPT) in SDN. ingress switch
Timestamp is • Pushed by ingress switch. • Removed by egress switch.
egress switch
VNF
X
External Network
vSwitch
Timestamp can be used in processing of intermediate devices. The TimedSDN Project
Timestamped Domain 17
DPT Header Example: NSH Encapsulation • All traffic is encapsulated with an NSH. • NSH may include a timestamp [NSH-Timestamp]. [NSH-Timestamp] is under discussion in the SFC working group of the IETF.
ingress switch
egress switch
VNF
External Network
vSwitch
Timestamped Domain • •
NSH is used for Service Function Chaining (SFC). Defined by the SFC working group of the IETF.
Network Service Header (NSH) encapsulation
[NSH-Timestamp] R. Browne, A. Chilikin, B. Ryan, T. Mizrahi, and Y. Moses, “Network service header timestamping,” draft-browne-sfc-nsh-timestamp, work in progress, IETF, 2015. The TimedSDN Project
18
Timestamps Ranges SDN switch (OpenFlow / P4) matchaction procedure SDN Switch
SDN Switch DPT Processing
Flow Table
field1 field2 field3
Flow Table
…
packet header
action
timestamp field2 field3 field4 … packet header
action
Each Eachbit bithas hasaternary ternaryvalue: value:00//11//**
Timestamp-based ranges [TimeFlip, INFOCOM ‘15] Extremal range (T≥T0)
T0 0
Periodic range
... timestamp value
0
The TimedSDN Project
... timestamp value 19
Timestamp Format Network Time Protocol (NTP) timestamp format: Time.Sec
Time.Frac
Seconds
Second Fraction
32 bits
32 bits
The TimedSDN Project
20
Timestamps Ranges: Examples
packet header
01 1
SDN Switch
SDN Switch
DPT Processing
DPT Processing
Flow Table
Flow Table
timestamp field1 field2 field3 … action timestamp field1 field2 field3 …
*…* *…*
*…* *…*
Time.Sec
Time.Frac
timestamp field2 field3 field4 … packet header
231>T≥230 T≥231
*…*
Time.Sec
0
*…*
Time.Frac
Periodic range
Extremal range: T≥230 230
1
action
... timestamp value
0
... timestamp value
1 second The TimedSDN Project
21
DPT Use Case 1: Network Telemetry Delay measurement Loss measurement
S1
S2
T1
Time
T2
One-way delay: T2-T1 Obviously DPT can be used for delay measurement. How can DPT be used for loss measurement? The TimedSDN Project
22
Color-based Loss Measurement • [Coloring]: All data packets from S1 to S2 include a 1-bit ‘color’. • Measurement is performed separately for each block. S1 inserts 1-bit color into packet header.
S1 CS11
S1 toggles color.
S1 and S2 periodically export counters to central point, e.g., SDN controller. Controller can compute: Packet loss = (CS12-CS11)-(CR12-CR11)
CR11 CS01 CR01
CS12
Time
S1 and S2 maintain per-color counters.
S2
CR12
[Coloring] M. Chen, L. Zheng, G. Mirsky, G. Fioccola, and T. Mizrahi, “IP Flow Performance Measurement Framework,” draft-chen-ippm-coloringThe TimedSDN Project based-ipfpm-framework, work in progress, 2016.
23
Color-based Delay Measurement Delay measurement:
S1
S2
T1
Time
T2
T3 T4
Controller can compute: Two-way delay = (T4-T1)-(T3-T2) The TimedSDN Project
24
Color-based Measurement
Q: How does an SDN switch toggle the color? A: Controller periodically updates S1 coloring policy... Overhead on the controller…
S1 CS11
S2 CR11
CS01 CR01
Time
CS12
The TimedSDN Project
CR12
25
DPT-based Coloring • Choose 1 bit from the timestamp. • Use this bit as the color. • Each switch has two match-action rules: • Timestamp bit=0 • Timestamp bit=1
‘1’
• Each block: a constant time interval. • The controller is not involved in the color toggling.
The TimedSDN Project
Time.Sec
Time.Frac
Seconds
Second Fraction
color ‘0’ ‘1’
‘0’
timestamp ... value
26
DPT-based Coloring DPT allows for both delay and loss measurement. No other fields are necessary! ‘1’
Time.Sec
Time.Frac
Seconds
Second Fraction
color ‘0’ ‘1’
‘0’
timestamp ... value
A 1-bit timestamp is sufficient!
DPT-based coloring reduces the load on controller.
The TimedSDN Project
27
DPT Use Case 2: Consistent Updates Multicast tree update • • •
Two-phase update [Reitblatt et al. ’12]. S2 attaches version tags to packets. • ‘before’ / ‘after’. • Indicate distribution tree. Guarantees per-packet consistency.
The TimedSDN Project
S3
S4
S2
S1
before after
28
Two-Phase Consistent Updates 1. Controller sends ‘after’ configuration to S1,S2 (subject to ‘after’ version tag). 2. Controller updates S2 to start sending ‘after’ version tag.
S3
S4
S3
S4
S3
S4
S2
S1
S2
S1
S2
S1
1 The TimedSDN Project
2 29
DPT-based One-Phase Consistent Updates 1. Controller sends ‘after’ configuration to S1,S2 (subject to Timestamp≥Tthreshold). 2. Controller updates S2 to start sending ‘after’ version tag. Switches automatically start using the ‘after’ path when Timestamp≥Tthreshold
The DPT is the version tag. S3
S4
S3
S4
S2
S1
S2
S1
DPT
S3
S4
S2
S1
1 The TimedSDN Project
30
DPT-based One-Phase Consistent Updates DPT reduces the load on controller.
DPT reduces the management overhead of per-flow version tags.
The TimedSDN Project
31
DPT Use Case 3: Elephant Flow Load Balancing
src
20
S3 10
S1
10 A 20 20 S2 S5 dst B 10 S 10 4
20 Gbps elephant flow. Stateless load balancing: - Equal Cost Multiple Paths (ECMP) 50% utilization. - Random Packet Spraying (RPS) [Dixit et al. ‘13]. Stateful methods, e.g., Round Robin. The TimedSDN Project
32
DPT-based Load Balancing Push DPT to all packets.
src
20
S1
S3 10
10 A 20 20 S2 S5 dst B 10 S 10 4
20 Gbps elephant flow.
DPT-based load balancing:
...
A
B
A
B
... timestamp value
toggle interval The TimedSDN Project
33
DPT-based Load Balancing
src
20
S3 10
S1
10 A 20 20 S2 S5 dst B 10 S 10 4
How does the controller compute the toggle interval? • L = maximal packet length. ... A B • BW = total bandwidth of the two links. • Toggle interval = L / BW Toggle interval Independent of the flow rate ! The TimedSDN Project
A
B
... timestamp value 34
Goodput of DPT-based Load Balancing • Load balancing using 1-bit from the DPT. • DPT: higher goodput than RPS / ECMP.
A
Time.Sec
Time.Frac
Seconds
Second Fraction
B
A
B
Toggle interval = L / BW 1 msec
timestamp ... value
A B The TimedSDN Project
35
The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization
Project
Flow Swaps TIME4 [INFOCOM, ‘16]
Multi-phase updates [SOSR, ‘15]
Data Plane Timestamping [SWFAN, ‘16]
ONECLOCK to Rule Them All [NOMS, ‘16]
OpenFlow Scheduled Bundles
NETCONF Time Capability
[INFOCOM ’16, OpenFlow 1.5]
[NOMS ’16, RFC7758]
TIMEFLIP
ONECLOCK
[INFOCOM ‘15]
[NOMS ‘16]
REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project
36
OneClock to Rule Them All
[NOMS ‘16]
Many networked applications require time - Sensors - Actuators - IoT devices
- Switches - Routers - Toasters Client
Client
T T Server 1 Server 2
...
T Server n
T T Server 1 Server 2
Coordinated operation
...
T Server n
Coordinated snapshot The TimedSDN Project
37
Summary
Summary Controller A protocol allowing the controller to schedule network updates
Synchronized clocks
Switches
The Timed
Project
http://tx.technion.ac.il/~dew/TimedSDN.html The TimedSDN Project
39
Thanks!
References [1]
T. Mizrahi, Y. Moses, “Software Defined Networks: It’s About Time", IEEE INFOCOM, 2016.
[2]
T. Mizrahi, Y. Moses, “OneClock to Rule Them All: Using Time in Networked Applications”, IEEE/IFIP Network Operations and Management Symposium (NOMS), 2016.
[3]
T. Mizrahi, E. Saat, Y. Moses, "Timed Consistent Network Updates", ACM SIGCOMM Symposium on SDN Research (SOSR), 2015. (http://tx.technion.ac.il/~dew/TimedConsistentSOSR.pdf)
[4]
T. Mizrahi, O. Rottenstreich, Y. Moses, "TimeFlip: Scheduling Network Updates with Timestamp-based TCAM Ranges", IEEE INFOCOM, 2015. (http://tx.technion.ac.il/~dew/TimeFlipINFOCOM.pdf)
[5]
T. Mizrahi, Y. Moses, "Time-based Updates in Software Defined Networks", the second workshop on hot topics in software defined networks (HotSDN), 2013. (http://tx.technion.ac.il/~dew/TimeSDN.pdf)
[6]
T. Mizrahi, Y. Moses, “Using ReversePTP to Distribute Time in Software Defined Networks”, International IEEE Symposium on Precision Clock Synchronization for Measurement, Control and Communication, (ISPCS), 2014. (http://tx.technion.ac.il/~dew/ISPCSReversePTPFinal.pdf)
[7]
T. Mizrahi, Y. Moses, "ReversePTP: A Software Defined Networking Approach to Clock Synchronization", in the third workshop on hot topics in software defined networks (HotSDN), 2014. (http://tx.technion.ac.il/~dew/ReversePTPHotSDN.pdf)
[8]
T. Mizrahi, Y. Moses, "Time4: Time for SDN", arXiv preprint arXiv:1505.03421, 2016. (http://tx.technion.ac.il/~dew/Time4TR.pdf)
[9]
Open Networking Foundation, OpenFlow switch specification, Version 1.5.0, 2015. (https://www.opennetworking.org/images/stories/downloads/sdn-resources/onf-specifications/openflow/openflow-switch-v1.5.0.noipr.pdf)
[10]
Open Networking Foundation, OpenFlow extensions 1.3.x package 2, 2015. (https://www.opennetworking.org/images/stories/downloads/sdn-resources/onf-specifications/openflow/openflow-extensions-1.3.x-pack2-noipr.zip) The TimedSDN Project
41
