Exchange Point Engineering

A BOF led by Bill Manning, ISI

Editor's Note: I was in the other bof, so both Kent and Michael took notes at my request. I was not sure that both would be able to do this and I am delighted that both did. Both sets of notes are presented here. My thanks to both of them.
Stan Barber

Table of Contents

• Michael Dillon's Notes
• Kent England's Notes
• February 1997 NANOG Home Page

Michael Dillon's Notes

This was one of two BOF tracks. Stan Barber took notes on the other track and not having perfected the art of being in two places at once he asked for a volunteer to record this track :-)

Bill asked for a show of hands; approx. 1/2 the audience is ISP's, 1/2 is smart alecks, and 1/2 is non-ISP's. Approx 20% are at one or more IXP's (Internet eXchange Points).

What are IXP's?

• neutral meetpoints for traffic exchange
• a "commons" where lots of people can connect
• general open to anyone

• CIX was the earliest public exchange
• GIX (Global IX) came next. It was Peter Lothberg's idea and was originally located at DC because that was the cheapest place to bring in circuits.
• Then the MAE's were started by MFS with MAE-East
• The NSF created the NAP's as part of the decommissioning of the NSFNet
• And then many others appeared

• what is your market niche?
• what is your 1 year plan? 3 years? 5 years?
• who do you want to talk to, are they at the IXP's?
• how much will it cost? Lots!

• is it a good thing?
• is it for you?
• be prepared for planning, execution and support of routing

• Yes.
• This is the way the Internet grows
• there are new market opportunities in operating IXP's

• The eased into it with CIX first, then MAE East when it became available
• made some mistakes
• once it became obvious the IXP model would be prevalent it was easier to plan
• decide who to peer with
• are these peers at the IXP?
• will they peer with you?

• having six lanes to a brick wall is bad and that's what you get if you connect to an IXP without peering
• threshold requirements for peering are rising
• they would buy links to 5 NAP's to meet peering requirements even if it was over capacity

• There are transit issues that you have to worry about
• you can't just listen to the IXP sales rep
• route reflection -- you can steal services from IXP participants or have them stolen by others
• operational coordination is required
• it is more difficult to coordinate internal ISP changes if they are visible to IXP participants. At MAE-West this requires that you call 60 people
• shared media
• insufficient bandwidth -- links are generally bigger than the switching capacity of the IXP

• localise traffic -- you don't have 5 coast to coast round trips to go across the hall
• MLPA's (Multi-Lateral Peering Agreements)
• don't depend on long-haul carriers
• there are economies of scale
• you can develop confederations or cooperatives
• shared buying of long-haul circuits
• strengthen and build local infrastructure
• it used to be cheaper to cross the US border 6 times to get packets from Mexico's east coast to its west coast

• Addressing. Read and understand the CIDR FAQ and the PIER FAQ
• some discussion of hole-punching in CIDR blocks ensued and it was determined that it is silly for providers to forbid hole-punching. And unless there is specific language in your contract prohibiting it, you can announce more specific prefixes from an aggregate block supplied by an upstream provider
• however, announcing a more specific route via only one provider will result in asymmetric incoming traffic
• there is a difference between switching providers and staying with a provider but multihoming to additional providers
• ASN's
• Peering and Transit
• you have to run BGP4 and you cannot inject IGP information into your BGP announcements, i.e. don't redistribute OSPF, RIP etc. into BGP
• Route maps are important
• Flap dampening -- use it and be aware of how it can affect you
• Route blocking -- at minimum RFC1918 routes should be filtered
• Media issues -- FDDI, 100baseTx, ATM...
• Internal Infrastructure
• leased lines, frame relay, X.25 (ugh!)

• First you have to have too much money and need to throw it away
• 90% are ISO level 2 but some IXP's (CIX, Singapore ...) are level 3
• How is the IXP managed?, what are the responsibilities of each particpant?, what requirements to participate?, what are the rules to disconnect?, is the IXP incorporated?
• Layer 2 IXP's are the easiest
• with layer 3 the IXP operations people are more heavily involved
• AOL clients want access to Bill's Bait and Sushi shop archive of pictures of raw meat. Fish under 18 are bait!
• Ethernet -- can be used in smaller IXP's
• FDDI
• Frame Relay or X.25 (ATM is X.25 on steroids) These can be used to build an exchange that does not use a common physical space. Participants need only connect to the cloud and the IXP operator sets up PVC's as needed
• Switched media
• various exotic media
• mixing technologies is done but will bring up additional issues with buffering etc.

• reporting is often bad or missing. It is desirable that IXP operators report regularly to participants
• common space usage, physical issues, wire layout, racks, etc.
• error detection, who is responsible?, how do you report them?, what is the time between a problem report and getting a trouble ticket to the time when someone responds who is capable of fixing the problem?, what is the mean time to respond?, the ETA?, currently with MFS the mean time to respond is a minimum of 20 minutes.

• is the IXP public like the MAE's or private like the FIX's?
• common rules
• who does the flow analysis. Note that in some places this is illegal, in other places it is required and in some it is almost impossible.

• What to do when something goes wrong?
• Who fixes the media?
• How do you correct problems with participants such as pointing default at a non-transit peer?

• get your IXP connection installed
• arrange peering -- MLPA's or Bilateral (wild west) peering agreements.

• NIC/NOC coordination -- who do you call? Do you have the phone numbers you need?
• If you can't contact people, unplug them
• sometimes people claim that phone numbers are 7/24 but they go to an answering machine. This is not useful.
• ISP's have got to give accurate contact info
• do a whois, the phone number listed might be their NOC number
• there is too much obsolete whois/rwhois info.
• test the contact number before installing circuits. Make them fix their NOC first, then plug them in.
• tell everyone when your number or your employer changes
• check your own whois info
• whois #'s get too many calls regarding legal issues and not enough technical ones
• dig -x IP.add.re.ss -- threaten to release IP address if they don't update inverse mapping
• Do IXP's maintain spares for purchase in time of need? Some do, some allow you to store your own spares and some allow no storage space
• monitoring -- can anyone call in a problem?

• Who pays insurance?
• Who is liable?
• What about taxes and fees?

• When you have large traffic flows. Monitor your egress traffic
• transit carriers. Is it appropriate to do transit at an IXP?
• not enough bandwidth. Adding more Gigaswitch ports just won't cut it.

• private exchange points
• some sort of perr/peer bypass. Can you do it at the IXP location?
• forward thinking IXP operators will allow this

• Route Server has the global policy
• routers at an IXP that use the route server are merely forwarding agents
• if two routers lose direct connectivity (broken ATM PVC) but still have connectivity with the route server then you have a problem
• if the RS database gets wierd data then you are susceptible
• What if AS A and AS B use the route server at 2 IXP's and they lost A<->B connectivity at only one IXP?

Return to the top of the page.

Ken England's Notes

Discussion of issues involved in connecting to an IXP.

What business are you in? Do you need a connection?

Routing at Exchange Points

• Is this a good thing?
• Do I need it?

IXPs are one of the ways that the Internet grows.

Audience member stood up to describe process for connecting to IXP. While getting a connection is straightforward, getting the peering is more difficult.

Petach from Internex mentioned how the hurdle for peering has steadily been raised.

back to Bill:

Some of the Cons of going to an exchange

• transit doesn't come with exchange point connection
• route reflection (pointing of default)
• operational coordination can be difficult (change management)
• shared media issues
• insufficient bandwidth

Some of the Pros of connecting to an exchange

• localize traffic (less traffic on long haul circuits)
• MLPA (multi-lateral peering agreement)
• doesn't depend on long haul carriers

Economies of Scale

• develop confederations/cooperatives (more true outside US)
• focus on strengthening/building local infrastructure

Planning Issues

• Addressing
  • read CIDR FAQ
  • read PIER FAQ

There was a discussion of niceties of punching holes in cidr blocks. While it can be done, it isn't "nice" to do it without permission of block owner.

There was a discussion of contracts that prevent/allow taking sub-blocks when migrating from one ISP to another and the advantages of this.

Peering and Transit

• you will use BGP
• route maps are important
• dampening route flaps is a good idea
• route blocking may be required

Media Issues

• ISO 2 or ISO 3 (MAE or NAP v CIX or Singapore routing)
• management of exchange
• connection policy
• enforcement of policies
• legal standing of exchange

Media

• ISO level 2 is easiest (FDDI)
• ISO level 3 (IP) requires exchange operator to manage routing

Technology

• Ethernet
• FDDI
• FR/X.25 (meaning ATM)
• Switches
• Exotics

Management Issues

• reporting of exchange performance
• common space usage for co-lo exchanges (how is space managed?)
• error detection (time required to detect and correct problems)

Connection Policy

• public or private exchange
• common rules
• flow analysis (traffic stats)

Enforcement

• What to do when something goes wrong
• Correcting problems with the media
• Correcting problems with a participant (default pointing, etc)

Execution

• the connection
• MPLAs and bi-laterals

Support

• NIC-NOC coordination (need good points of contact from clients)
• monitoring

There was a discussion of whois contacts and how much out of date information exists in NSI databases.

There was a comment regarding use of RADB for better contact information. Someone said use "dig -x ip-addr" to find noc contacts

Q: Do I need to maintain my own spares or use exchange point spares? A: Depends on operator

Legal Standing

• what about insurance coverage for fire, etc?
• taxes and fees?
• liabilities

When to Leave an Exchange Point

• large traffic flows (move to private interconnects)
• transit carriers (can I use exchange for transit?)
• not enough bandwidth

Where to Go from Here?

• private peer to peer connections
• exotic high speed experiments (high speed shared media)

Return to the top of the page.