Networking updates:
CHECO

Marla Meehl

UCAR/FRGP/BiSON/WRN/RMCMOA/RMRISC Manager

NETS/FRGP/BiSON

2016-04-27

Acronym soup

  • University Corporation for Atmospheric Research (UCAR; http://www.ucar.edu)
  • National Center for Atmospheric Research (NCAR)
  • Front Range GigaPoP (FRGP)
  • Bi-State Optical Network (BiSON)
  • Western Regional Network (WRN)
  • Rocky Mountain Cyberinfrastructure Mentoring and Outreach Alliance(RMCMOA)
  • Rocky Mountain Regional Internet Security Collaboration (RMRISC)
  • Questions and discussions throughout as needed – and as for acronym soup ingredients, please ask if I miss one or check decoder page at http://bit.ly/acronym-soup

3

Current FRGP voting participants - 11

  • Colorado School of Mines
  • Colorado State University - Fort Collins
  • National Oceanic and Atmospheric Administration - Boulder
  • State of Colorado
  • University Corporation for Atmospheric Research (UCAR/NCAR)
  • University of Colorado - Boulder
  • University of Colorado - Colorado Springs
  • University of Colorado - Denver
  • University of Denver
  • University of Northern Colorado
  • University of Wyoming

FRGP participants – non-voting - 20

  • Auraria Higher Education Center
  • Colorado College
  • Colorado Community College System
  • City of Boulder
  • Colorado Department of Higher Education
  • Colorado Mountain College
  • Colorado State University - Pueblo
  • Colorado Telehealth Network
  • Community College of Denver
  • Educational Access Gateway Learning Environment Network

  • Johnson & Wales University
  • Fort Lewis College
  • I2-USDA
  • JeffcoSD
  • Metropolitan State University of Denver
  • University Information Systems (CU system)
  • NEON
  • NREL
  • State of Wyoming
  • University NAVSTAR Consortium (UNAVCO)

5

Front Range GigaPoP (FRGP)

  • http://www.frgp.net
  • Continued and expanded LANDER support and partnership
  • PerfSonar in the FRGP continues to expand and be a valuable tool
  • 31 participants
    • CU-CS, CC, CSU-P, JeffcoSD - fully active on BiSON and FRGP at 10G
    • USGS - in discussions on fiber and TIC options
    • SHEPC - in discussions on fiber build
    • City of Golden - in discussions on fiber connectivity
    • Pikes Peak Library District (PPLD) - pending approval

FRGP Cont’d

  • 910 15th
        • Adding NOAA TIC colo - May 2016
        • CDOT fiber spliced to expand BiSON ring to Golden
        • Level3 910 5 year colo agreement including cross connects signed
  • Level3 fiber leases
        • All paths renewed for 10 years (Boulder/Denver, DREAM, Boulder/Longmont, Center Green)
  • Continued steady growth in all traffic (peering, commodity, I2, caching, etc.)
  • Hiring two Network Engineer IIs

FRGP Cont’d

  • Peering
    • Google 2x10G running full so may increase in the fall
    • Comcast - little progress on 10G and peering
    • TR/CPS – still largest single FRGP usage
    • Akamai - continues high utilization
      • Upgrading servers
    • Netflix - continues high utilization
    • Coresite/Any2 – upgraded to 2x10G
      • Direct peering with Syringa
    • Open to other peering opportunities
  • Member port maximum and 95th percentiles
    • Adding aggregate port size comparison

9

10

Statistics 2015

Bi-State Optical Network (BiSON)

  • Total BiSON Participants = 9: CC (.167), CSM (1), CSU-FC (1), CSU-P (.167), CU-B (1), CU-CS (.167), FRGP (.5), I2-USDA (.5), JeffcoSD (1), NREL (.5), NOAA (1), UCAR (1), UW (1)
  • Expanded BiSON to:
    • Jefferson County School District – Full ring complete
    • CSM – Full ring complete
    • CU-UIS – in design discussions
  • ADVA 10G Research Wave for CSU-P - pending installation
  • UCAR NWSC 100G Wave on order
  • CDOT Higher Ed agreement in review
    • US 36, US 93, I25 North, I25 CSU Ag, I25 South/US 50

12

13

Internet2

  • Marla co-chairing I2 Diversity Initiative – a session planned at I2 Global Summit
    • Survey out to CIOs
    • Working on expanding efffort - webinars, workshops, blogs
    • Awards for Global Summit announced - CSM awarded
  • Non-Member NET+ Program
    • Webinar recording
    • Slides

Internet2 Tempe Future Strategies Meeting

  • This was a meeting of the Internet2 connector/network member principals to engage in a discussion about the strategic direction of Internet2 Network Services
  • Level setting about what the community faces in the next two years
  • What the community will need through 2023
  • Marla and Jeff attended
  • The following themes were identified from RON papers
    • Improve Network Security and Operations
    • Member Engagement and Membership Expansion
    • Enhance research support
    • Expand the network and Improve general broadband access
    • Deliver shared/collaborative services

WRN

  • WTC and WPC calls as needed
  • 100G I2 ports in LA, Seattle, and Chicago
  • Investigating further commodity sharing/aggregation
    • Considering full 10G to Level3
    • Make sure TSIC and Level3 can back each other up
  • Pacific Wave expanded to include WRN
    • Pete attended meeting to discuss expansion

17

  • Quilt continues to grow
    • The Quilt now at 46 members
      • 36 participants
      • 10 affiliates
  • Next CIS RFP 2016
    • Authorized Quilt Providers (AQPs) from the 2016 CIS RFP:
      • CenturyLink
      • Cogent
      • Level 3
      • NTT America
      • TATA Communications
      • Telia Carrier (formerly called TeliaSonera International Carrier)
      • Verizon

Quilt items

  • Quilt strategic relationships
    • Convened multiple meetings in Washington DC among Quilt leaders and FCC
    • On-going relationship building with NSF
    • FCC, NTIA, and NSF all had reps attend the last Quilt member meeting
    • ESnet and NSF colocated meetings with Quilt in Austin in September 2015 - very successful
    • Quilt security working group
      • Quilt submitted regional security proposal to NSF
        • Best practices
  • Partnered with Zayo to operate and manage network
  • FRGP working in partnership on fiber and service opportunities
    • Decommissioned Deproduction/Openmedia Foundation
    • Transition ECBOCES to EN
    • Zayo fiber opportunities
    • E-rate

Rocky Mountain Cyberinfrastructure Mentoring and Outreach Alliance (RMCMOA)

  • NSF CC*IIE Grant
  • PIs/Organizations: Colorado State University (Pat Burns), UCAR (Marla Meehl), University of Colorado (Thomas Hauser) , University of Utah (Joe Breen), IRON (Michael Guryan – Senior Personnel)

RMCMOA Cont’d

  • Third RMCMOA workshop held in conjunction with Westnet Meeting, 1/5/16
    • Agenda with presentations
    • Well attended
    • Well received per the survey
  • Final RMCMOA Workshop 8/9 - 8/11, CSU in conjunction with RMACC - technical engineering focus
  • Submitted SC ‘15 women funding supplement – WINS (more on that later)
  • “ACI’s plan remains to have another CC* solicitation in 2016. I regret not being able to discuss an accurate estimate of a release timeframe, and I think the community will be asked to have just a little more patience with the process.”

CC*DNI Related Proposals Submitted - 3/24/15

  • Boise State - CI Engineer - AWARDED
  • Boise State - Network Infrastructure - AWARDED
  • CSM - CI Engineer - AWARDED
  • CSU-P - Network Infrastructure - AWARDED
  • NMSU - CI Engineer - AWARDED
  • Operating Innovative Networks (OIN) - ESnet, I2, IU - AWARDED
  • UH Mauna Loa - AWARDED
  • UNM - Regional - AWARDED
  • UW - CI Engineer - AWARDED
  • CU-Boulder/Wyoming/NCAR DIBBS - NOT AWARDED

Women in IT Networking at SC (WINS)

  • Submitted 3 year follow-on proposal to NSF
  • Unsolicited
  • Fund at least 5 women each year for 3 years
  • Encourage awardees to attend in following years
    • Attempt to acquire SC or vendor funding for returning awardees
  • Build a pipeline that becomes self-sustaining
  • Invitations to submit coming out soon
    • Please consider having women in your organization apply

Rocky Mountain Regional Internet Security Collaboration (RMRISC)

  • NSF CICI Regional Security Proposal
  • PIs/Organizations: Colorado State University (Scott Baily/Christos Papadopoulos), UCAR(Marla Meehl/Scot Colburn/Paul Dial/John Hernandez), University of Colorado (Thomas Hauser/Erin Shelton) , University of Utah (Joe Breen), University of Wyoming (Dane Skow/Matt Kelly)
  • Partners: CSM, ESnet, I2, The Quilt, UETN

Rocky Mountain Regional Internet Security Collaboration (RMRISC)

  • During the two-year award term, the project team will conduct at least one regional workshop per year focused on securing network infrastructure.
  • The project team will lead or partner with The Quilt, the Energy Science Network (ESnet), and Internet2 to present to the region quarterly focused security webinars. The project team will provide resources for research and education sites by developing online materials covering security best-practices, tools and other resources.
  • On request, the project team will conduct up to eight security reviews and assessments during the project.
  • The project team will also work closely with researchers at CSU to provide a real world testbed for deploying newly developed data analytics tools to detect, alert, and divert attacks.

Cheyenne (NWSC-2) HPC System Details

  • Cheyenne will be built by Silicon Graphics International Corp. (SGI) in conjunction with centralized file system and data storage components provided by DataDirect Networks (DDN)
  • 5.34 petaflops Cluster
  • 4,032 2-socket nodes with 2.3 GHz Intel® processors
  • Total of 145,152 cores (36 cores/node)
  • 864 nodes have 128 GB mem and 3,168 have 64 GB of memory
  • 313 TB total memory
  • 100 Gbps EDR (Enhanced Data Rate) Infiniband interconnect
  • Linux operating system
  • Workload Manager
  • Intel Parallel Studio XE compiler suite
  • Cluster Management
  • Fabric Manager

28

UCAR CONFIDENTIAL

GLADE (NWSC-2) PFS Details

Key features of the new data storage system:

 

  • The NCAR Globally Accessible Data Environment
  • Over 20 petabytes of usable file system space
    • Can be expanded to over 40 petabytes by adding drives
  • 200 gigabytes per second  aggregate I/O bandwidth
  • 3,360 x 8 TB NL SAS drives
  • 48 x 800 GB mixed-use SSD drives for metadata
  • 24 NSD (Network Shared Disk) servers
  • Red Hat Enterprise Linux operating system
  • IBM GPFS file system  

29

UCAR CONFIDENTIAL

GLADE (NWSC-2) PFS Details

  • The new data storage system will be integrated with NCAR’s existing GLADE file system
  • It will provide an initial total capacity of over 36 petabytes (20 PB new and 16 PB existing GLADE space), expandable to over 56 petabytes with the addition of extra drives
  • The new disk system will transfer data at the rate of 200 gigabytes per second, more than twice as fast as the current file system’s rate of 90 gigabytes per second

30

UCAR CONFIDENTIAL

Yellowstone

1.50 PFLOPS peak

Partner Sites

XSEDE Sites

Data Transfer

Services

Science Gateways

RDA, ESG

Remote Vis

10Gb Ethernet

NCAR HPSS Archive

160 PB capacity

51 PB current usage

~12 PB/yr growth

Geyser, Caldera,

Pronghorn

DAV clusters

GLADE

Central disk resource

16.4 PB, 90 GB/s

Current NWSC HPC Architecture

High Bandwidth Low Latency HPC and I/O Networks

FDR InfiniBand and 10Gb Ethernet

31

UCAR CONFIDENTIAL

Our current (familiar) environment is a 1.5 petaflop system (Yellowstone), with the 16 petabyte GLADE file system, backed up by the HPSS archive.

Partner Sites

XSEDE Sites

Data Transfer

Services

Science Gateways

RDA, ESG

Remote Vis

10Gb Ethernet

NCAR HPSS Archive

240 PB capacity

66 PB expected usage

~42 PB/yr growth

Geyser, Caldera,

Pronghorn

DAV clusters

GLADE

Central disk resource

36 PB, 200 GB/s

Yellowstone

1.50 PFLOPS peak

NWSC HPC Architecture on January 2017

Cheyenne (NWSC-2)

~5.34 PFLOPS peak

High Bandwidth Low Latency HPC and I/O Networks

EDR / FDR InfiniBand and 40Gb Ethernet

32

UCAR CONFIDENTIAL

By January 2017, we plan to put into production the NWSC-2 systems:

   a. The new ~5 Petaflop NWSC-2 supercomputer

   b. A ~20 petabyte augmentation of GLADE - with a 200 GB/sec aggregate

       I/O bandwidth to the NWSC-2 supercomputer (over 2x what's currently

       available with Yellowstone)

One year overlap with Yellowstone

Other Changes:

* Added EDR InfiniBand network

* Upgraded 10Gb ethernet network to 40Gb ethernet

* Increased storage capacity, increased storage bandwidth

Juniper QFX10008 Switch -

Multi-Purpose Unified LAN Ethernet (MULE) Switch

  • Investigated other vendors and chose Juniper because:
    • Largely Juniper shop
    • Supports 288 100GE ports in the 8-slot QFX10008
      • The QFX100016 supports 16 slots, but that's more than we needed
    • Happy with buffer capacity, which should give over 50 mS of line-rate buffer on all ports
    • The QSFP28 optic should be more standard, dense, and flexible than the menagerie of optics offered by other vendors.

33

UCAR CONFIDENTIAL

Juniper QFX10008 Switch -

Multi-Purpose Unified LAN Ethernet (MULE) Switch

  • MULE serves as a bridge between EDR and FDR Infiniband networks
  • QFX10008 modular switch with 8 slot capacity can host 240 100GE or 288 40GE ports
  • Modules support QSFP and QSFP28 optics
  • Modules: 30 x 100GE (QSFP28) or 36 x 40GE (QSFP) / 12 x 100GE or 60 x 10G (SFP+)
  • Will use three 36x40GE modules and QSFP 40GE optics with four CWDM 10G lanes

34

UCAR CONFIDENTIAL

35

UCAR CONFIDENTIAL

36

UCAR CONFIDENTIAL

Miscellaneous

  • UCAR Presidential search still in-progress
  • Implemented ARUBA wireless AP system
    • Investigating clearpass for cert assignment
      • NETS plans to use Aruba's ClearPass (TM) in several roles:
        • a Certificate Authority that issues X.509 certificates to authenticate devices using EAP/TLS. This is best documented in the Clearpass-Guest-User-Guide-6.5.0, in the "Onboard" section.
        • a RADIUS server that allows 802.1X authentication to Aruba Instant wireless Access Points via EAP/TLS. Configuration for this is described in the ClearPass Policy Manager 6.5 User Guide.
        • CPPM, as ClearPass Policy Manager is often called, can be an OSCP (Online Certificate Status Protocol) server, so certificates can be revoked with relative ease.

Discussion

  • Any follow up questions or feedback?
  • Other items?
20160427.CHECO.Regional.update.Marla.Meehl - Google Slides