Interconnection,
Peering,
and
Settlements
Abstract
Over the past century the telephony industry has developed
a relatively sophisticated set of mechanisms for undertaking cost distribution
across multiple service providers. The domain of operation of these models of
interprovider interaction extends from those of two-party local transactions up
through multiparty international transactions.
The Internet industry presents a number of interesting
counterpoints to this observation. The number of Internet service providers is
now in the tens of thousands, operating within a business space that is
predominantly deregulated. The great number of service providers and the sparse
mesh of interconnection lead to a complex environment of interaction. Any
particular Internet transaction commonly extends not only across the
originating and terminating providers, but involves two or three transit
providers as well. It is not uncommon to observe transit paths that entail over
10 service providers. To support this relatively complex environment of
interconnection, the Internet industry makes use of only the most basic
financial systems of cost distribution, most commonly based on the bilateral
relationships of customer/provider and mutual peering.
Similarly, the Internet industry uses a relatively small
set of physical mechanisms for supporting interconnection, concentrating on the
model of a co-location environment with a local LAN (local area network)
switch.
While such simple engineering and financial models do
manage to support a very diverse Internet provider industry and also manage to
support a diverse set of applications for a very large user base, some
inevitable problems have arisen from this model.
This paper examines the various engineering models that are
used to support Internet provider interaction, looking at the evolution of the
Internet exchange concept of the research Internet of the 1980s into the
various forms of interprovider exchange evident in today's Internet.
Above this engineering layer is placed a level of financial
interaction between providers, commonly termed "financial
settlement." The paper will examine the various models of settlement
commonly used in the communications industry, and then examine their
applicability to the Internet environment. The requirements of a financial
settlement will be examined, as will the relationship between retail service
models and settlement models. The conclusion drawn in the paper is that the
zero-dollar peering relationship and the customer/provider relationship are the
only models that are stable within the Internet environment, and other models
of financial interaction pose excessive risk to one or both interconnecting
parties. This polarization of the interconnection environment into just two
models is an important feature of today's Internet industry.
Such a conclusion is not without its consequences in terms
of supportable services in the Internet. For example, widespread deployment of
end-to-end quality of service is highly unlikely in such an environment, given
that there is no stable mechanism of cost distribution to support the transit
of elevated-quality packets. The conclusion also has a number of business
outcomes, not the least of which is the long-term inability of such an Internet
environment to support a highly diverse provider environment, and the current
trend of aggregation within the Internet provider industry is seen as a natural
outcome of the current polarized interprovider peering environment. The paper
will briefly examine these outcomes and look at the likely directions of the
Internet provider industry as a consequence.
To provide some motivation for this issue of ISP
interconnection, it is first appropriate to examine the nature of the
environment. The regulatory framework that defined the traditional structure of
other communications enterprises such as telephony or postal services was
largely absent in the evolution of the Internet service industry. The resultant
service industry for the Internet is most accurately characterized as an
outcome of business and technology interaction, rather than a planned outcome
of some regulatory process. In this section we will examine this interaction
between business and technology within the ISP environment.
A natural outcome of the Internet model is that the
effective control of the retail service environment rests with a network client
of an access service rather than with the access service provider, as such a
client of an ISP access service has the discretionary ability to resell the
access service to third-party clients. In this environment, reselling and
wholesaling were very natural developments, with or without the explicit
concurrence of the provider ISP. The provider ISP may see this reselling as an
additional channel to market for its own Internet carriage services, and may
adopt a positive stance by actively encouraging resellers into the market as a
means of overall market stimulus, while tapping into the marketing, sales, and
support resources of these reselling entities to continue to drive the volumes
of the underlying Internet carriage service portfolio. The low barriers to
entry to the wholesale market provide a means of increasing the scope of the
operation, as, to lift business cash-flow levels, the business enters into
wholesale agreements that effectively resell the carriage components of the
operation without the bundling of other services normally associated with the
retail operation. This process allows the ISP to gain higher volumes of
carriage capacity, that in turn allow the ISP to gain access to lower unit
costs of carriage.
Figure 1. ISP roles and
relationships.
Given that a retail operation can readily become a
wholesale provider to third-party resellers at the effective discretion of the
original retail client, is a wholesale transit ISP restricted from undertaking
retail operations? Again, there is no such natural restriction from a technical
or business perspective. An Internet carriage service is a commodity service
that does not allow for a significant level of intrinsic product
discrimination. The relative low level of value added by a wholesale service
operation implies a low unit rate of financial return for that operation. This
low unit rate of financial return, together with an inability to effectively
competitively discriminate the wholesale product, induces a wholesale provider
into the retail sector as a means of improving the financial performance of the
service operation. The overall result is that many ISPs operate both as a
client and as a provider. Few, if any, reasonable technical-based
characterizations draw a clear and unambiguous distinction between a client and
service provider when access services to networks are considered. A campus
network may be a client of one or more service providers, while the network is
also a service provider to campus users. Indeed most networks in a similar
situation take on the dual role of client and provider, and the ability to
resell an access service can extend to almost arbitrary depths of the reselling
hierarchy. From this technical perspective, very few natural divisions of the
market support a stable segmentation into exclusively wholesale and exclusively
retail market sectors. The overall structure of roles is indicated in Figure 1.
The resultant business environment is one characterized by
a reasonable degree of fluidity, in which no clear delineation of relative
roles or markets exists. The ISP market environment is, therefore, one of
competitive market forces in which each ISP tends to create a retail market
presence. However, no ISP can operate in isolation. Each client has the
expectation of universal and comprehensive reachability, such that any client
of any other Internet ISP can reach the client, and the client can reach a
client of any other ISP. The client of an ISP is not undertaking a service
contract that limits connectivity only to other clients of the same ISP. As no
provider can claim ubiquity of access, every provider relies on every other
provider to complete the user-provided picture of comprehensive connectivity.
Because of this dependent relationship, an individual provider's effort to
provide substantially superior service quality may have little overall impact
on the totality of client-delivered service quality. In a best-effort public
Internet, the service quality becomes something that can be impacted negatively
by poor local engineering but cannot be uniformly improved beyond the quality
provided by the network's peers, and their peers in turn. Internet wholesale
carriage services in such an environment are constrained to be a commodity
service, in which scant opportunity exists for service-based differentiation.
In the absence of service quality as an effective service discriminator, the wholesale
activity becomes a price-based service with low levels of added value, or in
other words a commodity market.
The implication in terms of ISP positioning is that the
retail operation, rather than the wholesale activity, is the major area where
the ISP can provide discriminating service quality. Within the retail
operation, the ISP can offer a wide variety of services with a set of
associated service levels, and base a market positioning on factors other than
commodity carriage pricing.
Accordingly, the environment of interconnection between
ISPs does not break down into a well-ordered hierarchical model of a set of
wholesale carriage providers and associated retail service providers. The
environment currently is one with a wide diversity of retail-oriented
providers, where each provider may operate both as a retail service operator,
and a wholesale carriage provider to other retailers.
One of the significant issues that arises here is whether
an objective determination can be made of whether an ISP is a peer to, or a
client of, another ISP. This is a critical question, as, if a completely
objective determination cannot be readily made, the question then becomes one
of who is responsible for making a subjective determination, and on what basis.
This question is an inevitable outcome of the reselling
environment, where the reseller starts to make multiple upstream service
contracts with a growing number of downstream clients of the reselling service.
At this point, the business profile of the original reseller is little
distinguished from that of the original provider. The original reseller sees no
unique value being offered by the original upstream provider and may conclude
that it is in fact adding value to the original upstream provider by offering
the upstream provider high volume carriage and close access to the reseller's
client base. From the perspective of the original reseller, the roles have
changed, and the reseller is now perceived as a peer ISP to the original
upstream ISP provider.
This assertion of role reversal is perhaps most significant
when the generic interconnection environment is one of a zero sum financial
settlement, in which the successful assertion by a client of a change from
client to peer status results in the dropping of client service revenue without
any net change in the cost base of the provider's operation. The party making
the successful assertion of peer interconnection sees the opposite, with an
immediate drop in the cost of the ISP operation with no net revenue change.
The traditional public regulatory resolution of such
matters has been through an administrative process of "licensed"
communications service providers, who become peer entities through a process of
administrative fiat. In this model, an ISP would become a licensed service
provider through the payment of license fees to a communications regulatory
body. The license then allows the service enterprise access to interconnection
arrangements with other licensed providers. The determination of peer or client
is now quite simple: a client is an entity that operates without such a carrier
license, and a peer is one that has been granted such an instrument. However,
such regulated environments are quite artificial in their delineation of the
entities that operate within a market, and this regulatory process often acts
as a strong disincentive to large-scale private investment, thereby placing the
burden of underwriting the funding of service industries into the public
sector. The regulatory environment is changing worldwide to shift the burden of
communications infrastructure investment from the public sector, or from a
uniquely positioned small segment of the private sector, to an environment that
encourages widespread private investment. The Internet industry is at the
leading edge of this trend, and the ISP domain typically operates within a
deregulated valued-added communications service provider regulatory
environment. Individual licenses are replaced with generic class licenses or
similar deregulated structures in which formal applications or payments of
license fees to operate in this domain are unnecessary. In such deregulated
environments no authoritative external entity makes the decision as to whether
the relationship between two ISPs is that of a provider and client or that of
peers.
If no public regulatory body wants to make such a
determination, is there a comparable industry body that can undertake such a
role? The early attempts of the Commercial Internet eXchange (CIX) arrangements
in the United States in the early 1990s were based on a description of the
infrastructure of each party, in which acknowledgments of peer capability were
based on the operation of a national transit infrastructure of a minimum
specified capability. This specification of peering within the CIX was
subsequently modified so that CIX peer status for an ISP was simply based on
payment of the CIX Association membership fee.
This CIX model was not one that intrinsically admitted
bilateral peer relationships. The relationship was a multilateral one, in which
each ISP executed a single agreement with the CIX Association and then
effectively had the ability to peer with all other association member networks.
The consequence of this multilateral arrangement is that the peering
settlements can be regarded as an instance of zero sum financial settlement
peering, using a single threshold pricing structure.
Other industry models use a functional peer specification.
For example, if the ISP attaches to a nominated physical exchange structure,
then the ISP is in a position to open bilateral negotiations with any other ISP
also directly attached to the exchange structure. This model is inherently more
flexible, as the bilateral exchange structure enables each represented ISP to
make its own determination of whether to agree to a peer relationship or not
with any other co-located ISP. This model also enables each bilateral peer
arrangement to be executed individually, admitting the possibility of a wider
diversity of financial settlement arrangements.
The bottom line is that a true peer relationship is based
on the supposition that either party can terminate the interconnection
relationship and that the other party does not consider such an action a
competitively hostile act. If one party has a high reliance on the
interconnection arrangement and the other does not, then the most stable
business outcome is that this reliance is expressed in terms of a service
contract with the other party, and a provider/client relationship is established.
If a balance of mutual requirement exists between both parties, then a stable
basis for a peer interconnection relationship also exists. Such a statement has
no intrinsic metrics that allow the requirements to be quantified. Peering in
such an environment is best expressed as the balance of perceptions, in which
each party perceives an acceptable approximation of equal benefit in the
interconnection relationship in their own terms.
This conclusion leads to the various tiers of accepted
peering that are evident in the Internet today. Local ISPs see a rationale to
view local competing ISPs as peers, and they still admit the need to purchase
trunk transit services from one or more upstream ISPs under terms of a client
contract with the trunk provider ISP. Trunk ISPs see an acceptable rationale in
peering with ISPs with a similar role profile in trunk transit but perceive an
inequality of relationship with local ISPs. The conclusion drawn here is that
the structure of the Internet is one where there is a strong business pressure
to create a rich mesh of interconnection at various levels, and the
architecture of interconnection structures is an important feature of the
overall architecture of the public Internet.
One of the physical properties of electromagnetic
propagation is that the power required to transmit an electromagnetic pulse
over a distance varies in accordance with this distance. The shorter the
distance between the transmitter and the receiver, the lower the transmission
power budget required; closer is cheaper.
This statement holds true not only for electrical power
budgets but also for data protocol efficiency. Minimizing the delay between the
sender and receiver allows the protocol to operate faster and operate more
efficiently as well; closer is faster, and closer is more efficient.
These observations imply that distinct and measurable
advantages are gained by localizing data traffic, that is by ensuring that the
physical path traversed by the packets passed between the sender and the
receiver is kept as physically short as possible. These advantages are
realizable in terms of service performance, efficiency, and service cost. How
then are such considerations of locality factored into the structure of the
Internet?
A strictly hierarchical model of Internet structure is one
in which a small number of global ISP transit operators is at the
"top"; a second tier is of national ISP operators; and a third tier
consists of local ISPs. At each tier the ISPs are clients of the tier above, as
shown in Figure 2. If this hierarchical model were strictly adhered to, traffic
between two local ISPs would be forced to transit a national ISP, and traffic
between two national ISPs would transit a global ISP, even if both national
ISPs operated within the same country. In the worst case, traffic between two
local ISPs would need to transit a national ISP, and then a global ISP from one
hierarchy, then a second global ISP, and a second national ISP from an adjacent
hierarchy in order to reach the other local ISP. If the two global providers
interconnect at a remote location, the transit path of the traffic between
these two local ISPs could be very long indeed.
As noted above, such extended paths are inefficient and
costly, and such costs are ultimately part of the cost component of the price
of Internet access. In an open competitive market, strong pressure always is
applied to reduce costs. Within a hierarchical ISP environment, strong pressure
is applied for the two national providers, who operate within the same market
domain, to modify this strict hierarchy and directly interconnect their
networks. Such a local interconnection allows the two networks to service their
mutual connectivity requirements without payment of transit costs to their
respective global transit ISP providers. At the local level is a similar
incentive for the local ISPs to reduce their cost base, and a local
interconnection with other local ISPs would allow local traffic to be exchanged
without the payment of transit costs to the respective transit providers.
Figure 2. A purely
hierarchical structure for the Internet.
Although constructing a general interconnection regime
based on point-to-point bilateral connections is possible, this approach does
not exhibit good scaling properties. Between N providers, who want to
interconnect, the outcome of such a model of single interconnecting circuits is
(N2 - N) / 2 circuits and (N2 - N)
/ 2 routing interconnections, as indicated in Figure 3. Given that
interconnections exhibit the greatest leverage within geographical local
situations, simplifying this picture within the structure of a local exchange
is possible. In this scenario each provider draws a single circuit to the local
exchange and then executes interconnections at this exchange location. Between N
providers who want to interconnect, the same functionality of complete
interconnection can be constructed using only N point-to-point circuits.
Figure 3. Fully meshed
peering.
One model of an exchange is to build the exchange itself as
a router, as indicated in Figure 4. Each provider's circuit terminates on the
exchange router, and each provider's routing system peers with the routing
process on the exchange router. This structure also simplifies the routing
configuration, so that full interconnection of N providers is effected
with N routing peer sessions. This simplification does allow greater
levels of scaling in the interconnection architecture.
However, the exchange router model does become an active
component of the interconnect peering policy environment. In effect, each
provider must execute a multilateral interconnection peering with all of the
other connected providers. Selectively interconnecting with a subset of the
providers present at such a router-based exchange is not easily achieved. In
addition, this type of exchange must execute its own routing policy. When two
or more providers are advertising a route to the same destination, the exchange
router must execute a policy decision as to which provider's route is loaded in
the router's forwarding table, making a policy choice of transit provider on
behalf of all other exchange-connected providers.
Because the exchange is now an active policy element in the
interconnection environment, the exchange is no longer completely neutral to
all participants. This imposition on the providers may be seen as unacceptable,
in that some of their ability to devise and execute an external transit policy
is usurped by the exchange operator's policies.
Figure 4. An exchange
router.
Typically, providers have a higher expectation of
flexibility of policy determination from exchange structures than this base
level of functionality as provided by an exchange router. Providers want the
flexibility to execute interconnections on a bilateral basis at the exchange,
and to make policy decisions as to which provider to prefer when the same
destination is advertised by multiple providers. They require the exchange to
be neutral with respect to such individual routing policy decisions.
The modification to the inter-provider exchange structure
is to use a local layer 2 switch (or local area network (LAN)) as the exchange
element. In this model a participating provider draws a circuit to the exchange
and locates a dedicated router on the exchange LAN. This structure is indicated
in Figure 5. Each provider executes a bilateral peering agreement with another
provider by initiating a router peering session with the other party's router.
When the same network destination is advertised by multiple peers, the provider
can execute a policy-based preference as to which peer's route will be loaded
in the local forwarding table. Such a structure preserves the cost efficiency
of using N circuits to effect interconnection at the N provider exchange, while
admitting the important policy flexibility provided by up to (N2
- N)/2 potential routing peer sessions.
Figure 5. An exchange LAN.
Early inter-provider exchanges were based on an Ethernet
LAN as the common interconnection element. This physical structure was simple,
and not all that robust under the pressures of growth as the LAN become
congested. Subsequent refinements to the model have included the use of Ethernet
switches as a higher capacity LAN, and the use of Fiber Distributed Data
Interface (FDDI) rings, switched FDDI hubs, fast Ethernet hubs, and switched
fast Ethernet hubs. Exchanges are very high traffic concentration points, and
the desire to manage ever higher traffic volumes has lead to the adoption of
gigabit Ethernet switches as the current evolutionary technology step within
such exchanges.
The model of the exchange co-location accommodates a model
of diversity of access media, in which the provider's co-located router
undertakes the media translation between the access link protocol and the
common exchange protocol.
The local traffic exchange hub does represent a critical
point of failure within the local Internet topology. Accordingly, the exchange
should be engineered in the most resilient fashion possible, using standards
associated with a premium quality data center. This structure may include
multiple power utility connections, uninterruptible power supplies, multiple
trunk fiber connections, and excellent site security measures.
The exchange should operate neutrally with respect to every
participating ISP, with the interests of all the exchange clients in mind.
Therefore, exchange facilities, which are operated by an entity that is not
also a local or trunk ISP, enjoy higher levels of trust from the clients of the
exchange.
There are also some drawbacks to an exchange, and a
commonly cited example is that of imposed transit. If an exchange participant
directs a default route to another exchange router, then, in the absence of
defensive mechanisms, the target router will carry the imposed transit traffic
even when there is no routing peering or business agreement between the two
ISPs. Exchange located routers do require careful configuration management to
ensure that route peering and associated transit traffic matches the currently
executed interconnection agreements.
Distributed exchange models also have been deployed in
various locations. This deployment can be as simple as a metropolitan FDDI
extension, in which the exchange comes to the provider's location rather than
the reverse, as indicated in Figure 6. Other models that use an Asynchronous
Transfer Mode (ATM)-based switching fabric, using LAN Emulation (LANE) to mimic
the layer 2 exchange switch functionality, also have been deployed. Distributed
exchange models attempt to address the significant cost of operating a single
co-location environment with a high degree of resilience and security, but do
so at a cost of enforcing the use of a uniform access technology between every
distributed exchange participant.
Figure 6. A distributed
exchange.
However, the major challenge of such distributed models is
that of switching speed. Switching requires some element of contention
resolution, in which two ingress data elements that are addressed to a common
egress path require the switch to detect the resource contention and then
resolve it by serializing the egress. Switching, therefore, requires signaling,
in which the switching element must inform the ingress element of switch
contention. To increase the throughput of the switch, the latency of this
signaling must be reduced. The dictates of increased switching speed have the
corollary of requiring the switch to exist within the confines of a single
location, if exchange performance is a paramount concern.
Besides speed, we must consider the cost shift. In a
distributed exchange model, the exchange operator operates the set of access
circuits that form the distributed exchange. This process increases costs to
providers, while it prevents the provider from using a specific access
technology that matches their business requirements of cost and supportable
traffic volume. Not surprisingly, to date the most prevalent form of exchange
remains the third-party hosted co-location model. This model admits a high
degree of diversity in access technologies, while still providing the substrate
of an interconnection environment that can operate at high speed and therefore
manage high traffic volumes.
The co-location environment is often broadened to include
other functions, in addition to a pure routing and traffic exchange role. For a
high-volume content provider, the exchange location offers minimal transit
distance to a large user population distributed across multiple local service
providers, as well as allowing the content provider to exercise a choice in
selecting a non-local transit provider.
The exchange operator can also add value to the exchange
environment by providing additional functions and services, as well as
terminating providers' routers and large-volume content services. The exchange
location within the overall network topology is an ideal location for hosting
multicast services, as the location is quite optimal in terms of multicast
carriage efficiency. Similarly, Usenet trunk feed systems can exploit the local
hub created by the exchange. The overall architecture of a co-location
environment that permits value-added services, which can productively use the
unique environment created at an exchange, is indicated in Figure 7.
Figure 7. Exchange-located
service platforms.
The role of the exchange was broadened with the
introduction of the network access point (NAP) in the National Science
Foundation (NSF)-proposed post-NSFNET architecture of 1995.
The NAP was seen to undertake two roles: the role of an
exchange provider between regional ISPs that want to execute bilateral peering
arrangements and the role of a transit purchase venue, in which regional ISPs
could execute purchase agreements with one or more of a set of trunk carriage
ISPs also connected at the NAP. The access point concept was intended to
describe access to the trunk transit service. This mixed role of both local
exchange and transit operations leads to considerable operational complexity,
in terms of the transit providers being able to execute a clear business
agreement. What is the bandwidth of the purchased service in terms of
requirements for trunk transit, versus the access requirements for exchange
traffic? If a local ISP purchases a transit service at one of the NAPs, does
that imply that the trunk provider is then obligated to present all the ISP's
routes at remote NAPs as a peer? How can a trunk provider distinguish between
traffic presented to it on behalf of a remote client versus traffic presented
to it by a local service client?
We also should consider the issue that the quality of the
purchased transit service is colored by the quality of the service provided by
the NAP operator. Although the quality of the transit provider's network may
remain constant, and the quality of the local ISP's network and ISP's NAP
access circuit may be acceptable, the quality of the transit service may be negatively
impacted by the quality of the NAP transit itself.
One common solution is to use the NAP co-location facility
to execute transit purchase agreements and then use so-called backdoor
connections for the transit service provision role. This usage restricts the
NAP exchange network to a theoretically more simple local exchange role. Such a
configuration is illustrated in Figure 8.
Figure 8. Peering and
transit purchase.
For the ISP industry, a number of attributes are considered
highly desirable for an exchange facility. The common model of an Internet
exchange includes many, if not all, of the following elements:
- Operated by a neutral
party that is not an ISP (to ensure fairness and neutrality in the
operation of the exchange)
- Constructed in a robust
and secure fashion
- Located in areas of high
density of Internet market space
- Able to scale in size
- Operate in a fiscally
sound and stable business fashion
A continuing concern exists about the performance of
exchanges and the consequent issue of quality of services that traverse the
exchange. Many of these concerns stem from an exchange business model that may
not be adequately robust under pressures of growth from participating ISPs.
The exchange business models typically are based on a
flat-fee structure. The most basic model uses a fee structure based on the
number of rack units used by the ISP to co-locate equipment at the exchange.
When an exchange participant increases the amount of traffic presented over an
access interface, under a flat-fee structure, this increased level of traffic
is not accompanied by any increase in exchange fees. However, the greater
traffic volumes do imply that the exchange itself is faced with a greater
traffic load. This greater load places pressure on the exchange operator to
deploy further equipment to augment the switching capacity, without any
corresponding increase in revenue levels to the operator.
For an exchange operator to base tariffs on the access
bandwidths is not altogether feasible, given that such access facilities are
leased by the participating ISPs and the access bandwidth may not be known to
the exchange operator. Nor is using a traffic-based funding model possible
given that an exchange operator should refrain from monitoring individual ISP
traffic across the exchange, given the unique position of the exchange
operator. Accordingly, the exchange operator has to devise a fiscally prudent
tariff structure at the outset that enables the exchange operator to
accommodate large-scale traffic growth, while maintaining the highest possible
traffic throughput levels.
Alternatively there are business models in which the
exchange is structured as a cooperative entity between a number of ISPs. In
these models the exchange is a nonprofit common asset of the cooperative body.
This model is widely used, but also one that is prone to the economic condition
of the Tragedy of the Commons. It is in everyone's interest to maximize
exploitation of the exchange, while no single member wants to underwrite the
financial responsibility for ensuring that the quality of the exchange itself
is maintained.
The conclusion that can be drawn is that the exchange is an
important component of Internet infrastructure, and the quality of the exchange
is of paramount importance if it is to be of any relevance to ISPs. Using an
independent exchange operator whose income is derived from the utility of the
exchange is one way of ensuring that the exchange is managed proficiently and
that the service quality is maintained for the ISP clients of the exchange.
Enhancing the Internet infrastructure is quantified by the
following objectives:
- Extension of
reachability.
- Enhancement of policy
matching by ISPs.
- Localization of
connectivity.
- Backup arrangements for
reliability of operation.
- Increasing capacity of
connectivity.
- Enhanced operational
stability.
- Creation of a rational
structure of the connection environment to allow scalable structuring of
the address and routing space in order to accommodate orderly growth.
We have reached a critical point within the evolution of
the Internet. The natural reaction of the various network service entities in
response to the increasing number of ISPs will be to increase the complexity of
the interconnection structure to preserve various direct connectivity
requirements. Today, we are in the uncomfortable position of increasingly
complex inter-provider connectivity environment which is stressing the
capability of available technologies and equipment. The inability to reach
stable cost distribution models in a transit arrangement creates an environment
in which each ISP attempts to optimize its position by undertaking as many
direct 1:1 connections with peer ISPs as it possibly can. Some of these connections
are managed via the exchange structure. Many more are implemented as direct
links between the two entities. Given the relative crudity of the inter-AS
routing policy tools that we use today, this structure must be a source of some
considerable concern. The result of a combination of an increasingly complex
mesh of inter-AS connections, together with very poor tools to manage the
resultant routing space, is an increase in the overall instability of the
Internet environment. In terms of meeting critical immediate objectives,
however, such dire general predictions do not act as an effective deterrent to
these actions.
The result is a situation in which the inter-AS space is
the critical component of the Internet. This space can be viewed correctly as
the demilitarized zone within the politics of today's ISP-based Internet. In
the absence of any coherent policy, or even a commonly accepted set of
practices, the lack of administration of this space is a source of paramount
concern.
We have examined the business drivers behind the adoption
of the exchange model as the common basis of interconnection, and also examined
the advantages and pitfalls associated with the operation of such exchanges
within the public Internet. In continuing our examination of the technology and
business considerations that are significant within the subject of Internet
Service Provider (ISP) interconnection, we will now focus on the topic from a
predominately business perspective.
Any large multi-provider distributed service sector has to
address the issue of cost distribution at some stage in its evolution. Cost
distribution is the means by which various providers can participate in the
delivery of a service to a customer who purchases a service from a single
provider, and each provider can be compensated for their costs in an equitable
structure of inter-provider financial settlement.
As an example, when an airline ticket is purchased from one
air service provider, various other providers and service enterprises may play
a role in the delivery of the service. The customer does not separately pay the
service fee of each airport baggage handler, caterer, or other form of service
provider. The customer's original fare, paid to the original service provider,
is distributed by the service provider to other providers who incurred cost in
providing components of the total service. These costs are incurred through
sets of service contracts, and are the subject of various forms of inter-provider
financial settlements, all of which are invisible to the customer.
The Internet is in a very similar situation. Some 50,000
constituent networks must interconnect in one fashion or another to provide
comprehensive end-to-end service to each client. In supporting a data
transaction between two clients, the two parties often are not clients of the
same network. Indeed, the two client service networks often do not directly
interconnect, and one or more additional networks must act in a transit
provider role to service the transaction. Within the Internet environment, how
do all the service parties to a transaction, who incur cost in supporting the
transaction, receive compensation for their cost? What is the cost distribution
model of the Internet?
Here, we examine the basis for Internet inter-provider cost
distribution models and then look at the business models currently used in the
inter-provider Internet environment. This area commonly is termed financial
settlement, a term the Internet has borrowed from the telephony industry.
What exactly is being exchanged between two ISPs that want
to interconnect? In the sense of the meaning of currency as the circulating
medium, the question is: What is the currency of interconnection? The technical
answer to the question is routing advertisements. When two parties exchange
routing entries, the outcome is that traffic flows in response to the flow of
routing advertisements. The route advertisement and traffic flows move in
opposite directions, as indicated in Figure 9.
Figure 9. Routing and
traffic flows.
Within the routing environment of an ISP there are a number
of different classes of routes, with the classification based predominately on
the way in which the route has been acquired by the ISP:
- Client routes are
passed into the ISP's routing domain by virtue of a service contract with
the client. The routes may be statically configured at the edge of the
ISP's network, learned by a Border Gateway Protocol (BGP) session with the
client, or part of an ISP pool of addresses that are dynamically assigned
to the client as part of the dial-up session.
- Internal ISP routes
fall into a number of additional categories. Some routes correspond to
client services operated by the ISP, solely for access to the clients of
the ISP, such as Web caches, point of presence (POP) mail servers, and
game servers. Some routes correspond to ISP-operated client services that
require Internet-wide access, such as Domain Name System (DNS) forwarders
and Simple Mail Transfer Protocol (SMTP) relay hosts. Lastly are internal
services with no visibility outside the ISP network, such as Simple
Network Management Protocol (SNMP) network management platforms.
- Upstream routes
are learned from upstream ISPs as part of a transit service contract the
ISP has executed with the upstream provider.
- Peer routes are
learned from exchanges or private interconnections, corresponding to
routers exported from the interconnected ISP.
How then should the ISP export routes so that the inbound
traffic flow matches the outbound flows implied by this route structure? The
route export policy is generally structured along the following lines:
- Clients. All
available routes in the preceding four categories, with the exception of
internal ISP service functions, should be passed to clients, either in the
form of a default route or as explicit route entries passed via a BGP
session.
- Upstream providers.
All client routes and all internal ISP routes corresponding to
Internet-wide services should be passed to upstream providers. Some
clients may want further restrictions placed on their routes being
advertised in such a fashion. The ability for a client to specify such
caveats on the routing structure, and the mechanism used by the ISP to
allow this to occur, should be clearly indicated in the service contract.
- Peer ISPs. All
client routes and all ISP routes corresponding to Internet-wide service
should be passed to peer ISPs. Again the client may want to place a
restriction on such an advertisement of their routes as a qualification to
the ISP's own route export policy.
This structure is shown in Figure 10.
Figure 10. External
routing interaction.
The implicit outcome of this routing policy structure is
that the ISP does not act in a transit role to peer ISPs and does not permit
peer-to-peer transit or peer-to-upstream transit. Peer ISPs have visibility
only to clients of the ISP. From the service visibility perspective,
client-only services are not visible to peer ISPs or upstream ISPs, and, therefore,
value-added client services are implicitly visible only to clients and only
when they access the service through a client channel.
Financial settlements have been a continual topic of
discussion within the domain of Internet interconnection. To look at the
Internet settlement environment, let's first look at the use of inter-provider
financial settlements within the international telephony service industry.
Then, we will look at the application of these generic principles to the
Internet environment.
Within the traditional telephony model, inter-provider
peering takes place within one of three general models:
The first, and highly prevalent, international peering
model is that of bilateral settlements. A call minute is the unit of settlement
accounting. A call is originated by a local client, and the local client's
service provider charges the client for the duration of the entire end-to-end
call. The call may pass through, or transit, a number of providers, and then
terminate within the network of the remote client's local provider. The cost
distribution mechanism of settlements is handled bilaterally. In the most
general case of this settlement model the originating provider pays the next
hop provider to cover the costs of termination of the call. The next hop
provider then either terminates the call within the local network, or
undertakes a settlement with the next hop provider to terminate the call. The
general telephony trunk model does not admit many multi-party transit
arrangements. The majority of telephony settlements are associated with trunk
calls that involve only two providers: the originating and terminating
providers. Within this technology model, the bilateral settlement becomes
easier, as the model simplifies to the case where the terminating provider
charges the originating provider a per-call minute cost within an accounting
rate that has been bilaterally agreed between the two parties. As both parties
can charge each other using the same accounting currency, the ultimate
financial settlement is based on the net outcome of the two sets of call minute
transactions with the two call minute termination accounting rates applied to
these calls. (There is no requirement for the termination rates for the two
parties to be set at the same level.) Each provider invoices the originating
end user for the entire call duration, and the financial settlements provide
the accounting balance intended to ensure equity of cost distribution in
supporting the costs of the calls made between the two providers. Where there
is equity of call accounting rates between the two providers, the bilateral
inter-provider financial settlements are used in accordance with originating
call minute imbalance, in which the provider hosting the greater number of
originating call minutes pays the other party according to a bilaterally
negotiated rate as the mechanism of cost distribution between the two
providers.
This accounting settlement issue is one of the drivers
behind the increasing interest in Voice-over IP solutions, because typically no
accounting rate settlement component exists in such solutions, and the call
termination charges are cost-based, without bilateral price setting. In those
cases where accounting rates have come to dominate the provider's call costs,
Voice-over IP is perceived as an effective lever to bypass the accounting rate
structure and introduce a new price point for call termination in the market
concerned.
The second model, rarely used in telephony interconnection,
is that of Sender Keep All (SKA), in which each service provider invoices their
originating client's user for the end-to-end services, but no financial
settlement is made across the bilateral interconnection structure. Within the
bilateral settlement model, SKA can be regarded as a boundary case of bilateral
settlements, where both parties simply deem the outcome of the call accounting
process to be absolutely equal, and consequently no financial settlement is
payable by either party as an outcome of the interconnection.
The third model is that of transit fees, in which the one
party invoices the other party for services provided. For example, this
arrangement is commonly used as the basis of the long-distance provider local
access provider interconnection arrangements. Again, this can be viewed as a
boundary case of a general bilateral settlement model , where in this case the
parties agree to apply call accounting in only one direction, rather than
bilaterally.
The international telephony settlement model is by no means
stable, and currently significant pressure is being placed on the international
accounting arrangements to move away from bilaterally negotiated uniform call
accounting rates to rates separately negotiated for calls in each direction of
a bilateral interconnection. Simultaneously, communications deregulation within
many national environments is changing the transit fee model, as local
providers extend their network into the long-distance area and commence
interconnection arrangements with similar entities. Criticism also has been
directed at the bilaterally negotiated settlement rates, because of the
observation that in many cases the accounting rates are not cost-based rates
but are based on a desire to create a revenue stream from accounting
settlements.
A number of critical differences exist between the
telephony models of interconnection and the Internet environment, which have
confounded all attempts to cleanly map telephony interconnection models into
the Internet environment.
Internet Settlement Accounting by the packet. Internet interconnection accounting is a packet-based
accounting issue, because there is no "call minute" in the Internet
architecture. Therefore, the most visible difference between the two
environments is the replacement of the call with the packet as the currency
unit of interconnection. Although we can argue that a Transmission Control
Protocol (TCP) session has much in common with a call, this concept of an
originating TCP call minute is not always readily identified within the packet
forwarding fabric, and accordingly it is not readily apparent that this is a
workable settlement unit. Unlike a telephony call, no concept of state
initiation exists to pass a call request through a network and lock down a
network transit path in response to a call response. The network undergoes no
state change in response to a TCP session, and therefore, no means is readily
available to the operator to identify that a call has been initiated, and by
which party. Of course the use of UDP, and various forms of tunneling traffic,
also confound any such TCP call minute accounting mechanism.
Packets may be dropped.
When a packet is passed across an interconnection from one provider to another,
no firm guarantee is given by the second provider that the packet will
definitely be delivered to the destination. The second provider, or subsequent
providers in the transit path, may drop the packet for quite legitimate
reasons, and will remain within the protocol specification in so doing. Indeed,
the TCP protocol uses packet drop as a rate-control signal. For the efficient
operation of the TCP protocol, some level of packet drop is a useful and
anticipated event. However, if a packet is used as the accounting unit in a
general cost distribution environment, should the provider who receives and
subsequently drops the packet be able to claim an accounting credit within the
interconnection? The logical response is that such accounting credits should
apply only to successfully delivered packets, but such an accounting structure
is highly challenging to implement accurately within the Internet environment.
Routing and traffic flow are not always paired. Packet forwarding is not a verified operation. A provider
may choose to forward a packet to a second provider without reference to the
particular routes the second provider is advertising to the first party. A
packet also may be forwarded to the second provider with a source address that
is not being advertised to the second provider. Given that the generic Internet
architecture strives for robustness under extreme conditions, attempts to
forward a packet to its addressed destination are undertaken irrespective of
how the packet may have arrived at this location in the first place, and
irrespective of how a packet with reverse header IP addresses will transit the
network.
Comprehensive routing information is not uniformly
available. Complete information is not
available to the Internet regarding the status and reachability of every
possible Internet address. Only as a packet is forwarded closer to the
addressed destination does more complete information regarding the status of
the destination address become apparent to the provider. Accordingly a packet
may have incurred some cost of delivery before its ultimate undeliverability
becomes evident. An intermediate transit provider can never be completely
assured that a packet is deliverable.
