CENIC CalREN-2 Status Report

March 16, 1998

The purpose of this document is to describe the status of various activities related to the design and development of the CENIC CalREN-2 network. It is my intention to update it bi-weekly or more often if events warrant. Clearly the status changes rapidly during this implementation period so please send me a note if there is anything that needs clarification or more information.

Technical Planning

Planning for the topology and equipment for Phase 1 is complete. Details can be found on the web (NOTE: those pages are not yet "public"). Specific parts lists for each CalREN-2 node will be sent out as soon as possible, probably by the ISI CalREN-2 project team (see below).

As quickly as possible we will design the interconnection with CSU at their WestEd site in Los Alamitos. Pacific Bell engineering is working on it now. Access to the rest of CSUnet, including the 3 NSF awardee campuses, will be via that node.

Network Infrastructure

The core of the network, on the metro area SONET rings, is IP Packet-over-SONET (POS) OC-12c. The second OC-12c path around the rings will be used for ATM-based services. Sites connected via a single OC-12c SONET circuit will see IP over ATM and other ATM-based services over the same SONET channel.

Connections from each ring to other CalREN-2 nodes will have to be at a single location because of cost. We realize this is a vulnerability and it can be changed later if necessary (and funding is made available). In the South, USC is the connection point. In the North it is at Berkeley.

Connection to the vBNS will occur through Berkeley in the North and through UCSD/SDSC in the South. Both will be OC-12c SONET over which we'll configure ATM VC's (see below).

There are gateways for connection of external service providers or other research networks (ESNET, NREN) at 2 places on each ring:

• Stanford and Berkeley in the North
  
• ISI and Caltech in the South

Each can accommodate two DS-3 plus four DS-1 plus three OC-3 POS connections. An RFP for ISP services in preparation by a BAC/TAC subcommittee.

Connections to additional CENIC Associates will be designed on a case by case basis. Every GSR has at least one empty slot, and at the interconnect nodes (Berkeley, USC) there are at least 2 empty slots.

Service Interface

Each CalREN-2 institution will have a CalREN-2 IP services port and a CalREN-2 ATM services port. The IP services interface will be one of:

• a single OC-12c IP POS port, or
• a single OC-12c IP over ATM port (which must attach to a campus ATM switch), or
• 4-ports of OC-3 IP POS.

In the case of an IP over ATM interface, virtual circuits and BGR peering will be configured between the CalREN-2 node router and up to 4 campus routers.

ATM services will be available via the CalREN-2 ATM switches. Policy for how these will be allocated and managed have not been defined. Both production and "experimental" services will be supported. T1 circuit emulation is a supported "production" service but the line cards must be paid for separately by each institution that wants it (all UC sites have this). The ATM interface may be any one of:

• a single OC-12 ATM port, or
• four OC-3 ports.

vBNS Access

Access to the vBNS must be limited to DS-3 or OC-3 equivalent bandwidth from each institution. This will be done by establishing an ATM VC between each "vBNS qualified institution" and the vBNS border router. This VC will be configured as UBR with a Peak Cell Rate (PCR) equivalent to an OC-3 (366,793 c/s) or DS-3 (106,132 c/s). These VC's will all run over an OC-12c SONET channel between CalREN-2 ATM switches and finally between a CalREN-2 switch and a vBNS switch. An OC-12 ATM interface on vBNS routers will terminate each VC.

At CalREN-2 nodes that include a CalREN-2 router, the vBNS VC will terminate in that router. At other CalREN-2 nodes, the VC must terminate in a campus border router. The vBNS routers must peer individually with each node's router (either CalREN-2 or campus) and such routes will not be propagated. (Note: this does not solve the policy constrained routing problem.)

Routing design, including any policy constraints, multicast, etc., will be done by the ISI project team.

Equipment

The CalREN-2 routing engine is the Cisco GSR 12000 series and the ATM switching platform is the Cisco LS1010. All routers are 12008's except for the one at Berkeley which needed additional slots. Every node has a Cisco 2522 and Catalyst 1924 switch for out of band access to console ports, etc.

All equipment has been ordered; it is all coming from Cisco. The LS1010's, 7507's, 2522's, and 1924's have arrived (unfortunately) and are all in John Hess's office here at UCOP. They will be retrieved by Cisco and shipped to a warehouse in Palo Alto for staging. The tentative plan is that each node site configuration will be completely set up and configured before the equipment is shipped to the site.

Communications Services

All the segments of our topology for Phase 1 are determined. We are finalizing the contracts and developing schedules. As discussed, we are not including the North-South link at this time.

PacBell SONET

We are trying to finalize one last issue with PacBell (regarding the GTE SONET segments in LA). We expect to have the contract signed by the end of the week. Unfortunately all this delay has pushed the schedule out to sometime in May, or maybe later.

As soon as possible we will complete an Addendum to attach CSU WestEd to the LA ring.

Qwest SONET

We have in hand the contract documents for the Qwest circuits and hope to have that signed at the same time we sign the Pac Bell contract. Qwest service delivery will be synchronized with Pac Bell's.

SONET OC-12c to vBNS

We have an offer from MFS/WorldCom to provide OC-12c SONET between the vBNS POP in Hayward and the Northern ring. We expect the contract documents to arrive this week.

SONET access at SDSC is entirely within the SDSC machine room so no circuit order is required.

We will also need to complete contract documents with MCI for the vBNS service. There are still some outstanding issues we are discussing with them because our connection design is quite different from any of the others. Also, the OC-12 ATM requirement will necessitate an upgrade of the vBNS routers at Hayward and SDSC.

ISP connections

A BAC/TAC subcommittee is working on an RFP for commodity Internet access. Several options are being discussed, including:

• an exclusive agreement to provide Internet access for all CalREN-2 sites
• a "one of many" agreement that would provide multi-homing for those sites that want it

The form of the arrangement might be peering (mutual transit, no fees), contractual with CENIC, or contractual with individual institutions under a model where the ISP brings circuits to our GigaPOPs and sells services to whomever wants it.

Financial arrangements might be flat rate but most likely would be usage based, especially if multi-homing is involved.

Other connections

No progress has been made with ESNET or NASA NREN.

We are encouraging PacBell and Cisco to join the network as soon as we have a mechanism for additional Associate connections.

In addition, we will need ISDN or Frame Relay circuits to each node site for Out-of-Band (OOB) access to console ports, and possibly for instrumentation data collection.

Installation

ISI has the contract to plan and manage the installation. The effective date is March 1, ending June 30 but with a possible one month extension. Logistical issues include:

• Address and name space design and acquisition
  
• Routing design and configuration requirements
  
• Equipment lists for each site
  
• Equipment mounting and installation details
  
• Determining needed cables, etc.
  
• Defining power and other siting requirements, building on work that Russ and the TAC have done.
  
• Defining specific configuration details for each piece
  
• Defining requirements for OOB access circuits to each node site

The design of the OOB router setup and configuration is being done by Ken Lindahl at Berkeley on behalf of the TAC.

Operations

The TAC is working on an RFP for Network Operations. It is a working assumption that campus technical staff will be asked to participate in the on-site operation of the CalREN-2 equipment, at the very least. However, some set of people must be responsible for the overall stability of the system, including monitoring and problem resolution. Engineering and change control is an option that might be folded into Operations as well. The TAC will remain an advisory body to both CENIC and to the Operations group.

David Wasley
Director of CENIC Projects