Download mung-class-jan05 - Princeton University

Traffic Engineering for ISP Networks
Jennifer Rexford
Internet and Networking Systems
AT&T Labs - Research; Florham Park, NJ
http://www.research.att.com/~jrex
Who am I, and Why am I Here?
 Who
am I?
– Princeton EE class of ‘91, PhD in ‘96 from U. Michigan
– AT&T Research from 1996, in the IP Network
Measurement and Engineering Department
– Joining the Princeton CS faculty in February 2005
 Why
am I here?
– Working closely with operators of AT&T’s IP network
– Creating tools for them to engineer the flow of traffic
– Applying optimization techniques to real problems
Outline
 Background
– Internet architecture
– Interdomain and intradomain routing
– Internet service provider backbone
 Traffic
engineering
– Optimizing routing to prevailing traffic
– Local search to select the integer link weights
 Inferring
the traffic matrix
– Working backwards from link loads
 Conclusion
and ongoing work
Internet Architecture
 Divided
into Autonomous Systems
– Distinct regions of administrative control
– Routers and links managed by an institution
– Service provider, company, university, …
 Hierarchy
of Autonomous Systems
– Large, tier-1 provider with nationwide backbone
– Medium-sized regional provider with smaller backbone
– Small network run by a single company or university
 Interaction
between Autonomous Systems
– Internal topology is not shared between ASes
– … but, neighboring ASes interact to coordinate routing
Autonomous Systems (ASes)
Path: 6, 5, 4, 3, 2, 1
4
3
5
2
7
1
6
Web server
Client
Interdomain Routing: Border Gateway Protocol
 ASes
exchange info about who they can reach
– IP prefix: block of destination IP addresses
– AS path: sequence of ASes along the path
 Policies
configured by the AS’s network operator
– Path selection: which of the paths to use?
– Path export: which neighbors to tell?
“I can reach 12.34.158.0/24
via AS 1”
“I can reach 12.34.158.0/24”
2
1
data traffic
12.34.158.5
3
data traffic
Internet Service Provider Backbone
modem banks,
neighboring providers
business customers,
web/e-mail servers
How should traffic be routed through the ISP backbone?
Interior Gateway Protocol (Within an AS)
 Routers
flood information to learn the topology
– Routers determine “next hop” to reach other routers…
– By computing shortest paths based on the link weights
 Link
weights configured by the network operator
2
3
2
1
1
1
3
5
4
3
Heuristics for Setting the Link Weights
 Proportional
to physical distance
– Cross-country links have higher weights than local ones
– Minimizes end-to-end propagation delay
 Inversely
proportional to link capacity
– Smaller weights for higher-bandwidth links
– Attracts more traffic to links with more capacity
 Tuned
based on the offered traffic
– Network-wide optimization of weights based on traffic
– Directly minimizes key metrics like max link utilization
Why Are the Link Weights Static?
 Strawman
alternative: load-sensitive routing
– Link metrics based on traffic load
– Flood dynamic metrics as they change
– Adapt automatically to changes in offered load
 Reasons
why this is typically not done
– Delay-based routing unsuccessful in the early days
– Oscillation as routers adapt to out-of-date information
– Most Internet transfers are very short-lived
 Research
and standards work continues…
– … but operators have to do what they can today
Our Approach: Measure, Model, and Control
Network-wide
“what if” model
Offered
Topology/
traffic
Configuration
measure
Changes to
the network
control
Operational network
Traffic Engineering in an ISP Backbone
 Topology
– Connectivity and capacity of routers and links
 Traffic
matrix
– Offered load between points in the network
 Link
weights
– Configurable parameters for Interior Gateway Protocol
 Performance
objective
– Balanced load, low latency, service level agreements …
 Question:
Given the topology and traffic matrix in
an IP network, which link weights should be used?
Key Ingredients of Our Approach
 Instrumentation
– Topology: monitoring of the routing protocols
– Traffic matrix: widely deployed traffic measurement
 Network-wide
models
– Representations of topology and traffic
– “What-if” models of shortest-path routing
 Network
optimization
– Efficient algorithms to find good configurations
– Operational experience to identify key constraints
Formalizing the Optimization Problem
 Input:
graph G(R,L)
– R is the set of routers
– L is the set of unidirectional links
– cl is the capacity of link l
 Input:
traffic matrix
– Mi,j is traffic load from router i to j
 Output:
setting of the link weights
– wl is weight on unidirectional link l
– Pi,j,l is fraction of traffic from i to j traversing link l
Multiple Shortest Path With Even Splitting
0.25
0.25
0.5
1.0
0.25
0.5
1.0
0.25
0.5
0.5
Values of Pi,j,l
Defining the Objective Function
 Computing
the link utilization
– Link load: ul = Si,j Mi,j Pi,j,l
– Utilization: ul/cl
 Objective
functions
– minl(ul/cl)
f(x)
– minl(S f(ul/cl))
x
1
Complexity of the Optimization Problem
 NP-complete
optimization problem
– No efficient algorithm to find the link weights
– Even for the simple convex objective functions
 Why
can’t we just do multi-commodity flow?
– E.g., solve the multi-commodity flow problem…
– … and the link weights pop out as the dual
– Because IP routers cannot split arbitrarily over ties
 What
are the implications?
– Have to resort to searching through weight settings
Optimization Based on Local Search
 Start
with an initial setting of the link weights
– E.g., same integer weight on every link
– E.g., weights inversely proportional to link capacity
– E.g., existing weights in the operational network
 Compute
the objective function
– Compute the all-pairs shortest paths to get Pi,j,l
– Apply the traffic matrix Mi,j to get link loads ul
– Evaluate the objective function from the ul/cl
 Generate
a new setting of the link weights
repeat
Making the Search Efficient
 Avoid
repeating the same weight setting
– Keep track of past values of the weight setting
– … or keep a small signature (e.g., a hash) of past values
– Do not evaluate a weight setting if signatures match
 Avoid
computing the shortest paths from scratch
– Explore weight settings that changes just one weight
– Apply fast incremental shortest-path algorithms
 Limit
the number of unique values of link weights
– Do not explore all 216 possible values for each weight
 Stop
early, before exploring the whole search space
Incorporating Operational Realities
 Minimize
number of changes to the network
– Changing just 1 or 2 link weights is often enough
 Tolerate
failure of network equipment
– Weights settings usually remain good after failure
– … or can be fixed by changing one or two weights
 Limit
dependence on measurement accuracy
– Good weights remain good, despite random noise
 Limit
frequency of changes to the weights
– Joint optimization for day and night traffic matrices
Application to AT&T’s Backbone Network
 Performance
of the optimized weights
– Search finds a good solution within a few minutes
– Much better than link capacity or physical distance
– Competitive with multi-commodity flow solution
 How AT&T changes
the link weights
– Maintenance done every night from midnight to 6am
– Predict effects of removing link(s) from the network
– Reoptimize the link weights to avoid congestion
– Configure new weights before disabling equipment
Example from My Visit to Our Operations Center
 Amtrak
repairing/moving part of the train track
– Need to move some of the fiber optic cables
– Or, heightened risk of the cables being cut
– Amtrak notifies us of the time the work will be done
 AT&T engineers
model the effects
– Determine which IP links go over the affected fiber
– Pretend the network no longer has these links
– Evaluate the new shortest paths and traffic flow
– Identify whether link loads will be too high
Example Continued
 If
load will be too high
– Reoptimize the weights on the remaining links
– Schedule the time for the new weights to be configured
– Roll back to the old weight setting after Amtrak is done
 Same
process applied to other cases
– Assessing the network’s risk to possible failures
– Planning for maintenance of existing equipment
– Adapting the link weights to installation of new links
– Adapting the link weights in response to traffic shifts
Conclusion of This Part of the Talk
 IP networks
do not adapt on their own
– Routers compute shortest paths based on static weights
 Service
providers need to adapt the weights
– Due to failures, congestion, or planned maintenance
 Leads
to an interesting optimization problems
– Optimize link weights based on topology and traffic
 Optimization
problem in NP-complete
– Forces the use of efficient local-search techniques
 Results
of the local search are useful
– Near-optimal solutions that minimize disruptions
Computing the Traffic Matrix Mi,j
 Hard
to measure the traffic matrix
– IP networks transmit data as individual packets
– Routers do not keep traffic statistics, except link
utilization on (say) a five-minute time scale
 Need
to infer the traffic matrix Mi,j from
– Current topology G(R,L)
– Current routing Pi,j,l
– Current link load ul
– Link capacity cl
Inference: Network Tomography
From link counts to the traffic matrix
Sources
5Mbps
3Mbps
4Mbps
4Mbps
Destinations
Tomography: Formalizing the Problem
 Ingress-egress
pairs
– p is a ingress-egress pair of nodes (i,j)
– xp is the (unknown) traffic volume for this pair Mi,j
 Routing
– Plp is proportion of p’s traffic that traverses l
 Links
in the network
– l is a unidirectional edge
– ul is the observed traffic volume on this link
 Relationship:
u = Px (work backwards to get x)
Tomography: One Observation Not Enough
 Linear
system of n nodes is underdetermined
– Number of links e is around O(n)
– Number of ingress-egress pairs c is O(n2)
– Dimension of solution sub-space at least c - e
 Multiple
observations are needed
– k independent observations (over time)
– Stochastic model with Poisson iid counts
– Maximum likelihood estimation to infer matrix
 Doesn’t
work all that well in practice…
Approach Used at AT&T: Tomo-gravity
 Gravitational
assumption
– Ingress point a has traffic via
– Egress point b has traffic veb
– Pair (a,b) has traffic proportional to via * veb
9
6
3
20
10
Approach Used at AT&T: Tomo-gravity
 Problem
with gravity model
– Gravity model ignores the load on the inside links
– Gravity assumption isn’t always 100% correct
– Resulting traffic matrix might not satisfy the link loads
 Combining
the two techniques
– Gravity: find a traffic matrix using the gravity model
– Tomography: find the family of traffic matrices
consistent with all link load statistics
– Tomo-gravity: find the tomography solution that is
closest to the output of the gravity model
 Works
extremely well (and fast) in practice
Conclusions
 Managing
IP networks is challenging
– Routers don’t adapt on their own to congestion
– Routers don’t reveal much information about traffic
 Measurement
provides a network-wide view
– Topology
– Traffic matrix
 Optimization
enables the network to adapt
– Inferring the traffic matrix from the link loads
– Optimizing the link weights based on the traffic matrix