JT-NM Tested Program August 2019

RIEDEL Communications, Wuppertal, Germany

August 19 - 23, 2019

Table of content

3 Introduction to the program

4 Purpose of the program

5 The Badges

6 What we will do? High level overview

7-8 The Badges FAQ

9 Getting the badges

10-11 Getting the badges FAQ

12 Prerequisites to attend

13 Acceptable PCAP capturing

14-15 Attending rules FAQ

16 Final report and results

17 High level provisional logistics

18-20 Supplies and consumables policy

21 Call for Action

Introduction to the program

JT-NM Tested Program returns to the IP Showcase. The JT-NM continues to partner with vendors and users to provide information that aids the transition to IP. As the industry’s use of IP matures, the JT-NM Tested program offers prospective purchasers of IP based equipment greater, more documented insight into how vendor equipment conforms to the SMPTE standards and AMWA NMOS specifications. Sponsored by the JT-NM and administered by the EBU and IRT, two top European technical bodies, vendors who submit equipment for evaluation will have the opportunity to list that equipment in a JT-NM Tested catalog which will be made publicly available at the IP Showcase booth and on-line. This catalog will provide transparency, describing the test criteria and testing methodology, as well as the hardware and software versions of the products that were tested. This program is not a certification program; rather it is a snapshot in time of how vendor equipment conforms to key parts of SMPTE standards and AMWA NMOS specifications, providing prospective purchasers and users with a reference as they begin their equipment evaluation and qualification process. Because this is a partnership between the JT-NM and the industry, and because this is a voluntary program, vendors will have the opportunity to decline to have the results for a specific device made public, and/or they may choose to withdraw from the program entirely after seeing test results. Should a company choose to withdraw a product, or should that company decide to withdraw from the testing program completely, JT-NM will not disclose those corresponding results, and participating companies will be legally required to keep the results of other participants confidential.

Purpose of the program

• Promotion of SMPTE ST 2110, JT-NM TR-1001-1 and AMWA NMOS Product Capabilities via Catalog/Promotion (“Tested Program” administered by the JT-NM)
• Tested products will receive:
• A badge(s) to display in one’s own booth
• Inclusion in a catalog of tested products
• Promotion via:
• IP Showcase article placement in the IBC 2019 timeframe
• Prominent display/education on the program in the IP Showcase booth
• An on-line catalog listing tested products, accessible via ipshowcase.org

The Badges

• JT-NM Tested August 2019 Program will include two sets of tests
• Vendors will have an opportunity to receive two badges:
• “JT-NM Tested SMPTE ST 2110 August 2019” badge
• Scope, criteria and test plan will be mostly the same as JT-NM Tested March 2019 Program
• “JT-NM Tested NMOS/TR-1001-1 August 2019” badge
• Will have a separate scope, criteria and test plan
• Will cover parts of JT-NM TR-1001-1, including AMWA NMOS
• Will cover AMWA NMOS IS-08

What we will do?

High level overview, more info below

• We will be testing SMPTE ST 2110, AMWA NMOS and JT-NM TR-1001-1 capable Senders and Receivers
• We will test Released Products, no engineering samples or prototypes
• There must be some kind of proof of a product release
• Products scheduled to be released by IBC 2019 are acceptable
• We perform basic tests for critical functionalities of selected parts of the standards
• There will be a test plan per badge
• We are testing compliance to the standards and specifications, not the interoperability
• We reserve the right to make minor modifications to the test plans as required to make it an appropriate test
• The technical team will determine a set of reference senders and receivers
• Recognition will be given for supporting equipment vendors such as GMs, Switches and Test Equipment
• Product information and code (software or firmware) versions will be recorded during the official test
• During the official testing any change of the settings/configurable parameters in the product is allowed
• The code (software or firmware) used during the testing has to be in a stable state, generally accessible to end clients (Business-2-Client (B2C), or Business-2-Business (B2B) for OEMs) by IBC 2019
• The code may not be changed once testing process has started
• Retesting a failed product may not be possible because of limited time to run the tests
• The scope of the Tested event will cover only edge devices. Categories of products out of scope of the event: test and measurement, PTP clocks, network equipment, broadcast and network controllers

The Badges FAQ

• What happens to my JT-NM Tested March 2019 (NAB 2019) badge?
• You may display your JT-NM Tested March 2019 badge per the agreement that you signed before receiving the badge.
• The new catalog will be published with a reference to the previous catalog. However only products tested against August 2019 test plans will be mentioned in the new catalog.
• The new badges will bear a date of testing (August 2019) without any link to trade shows.
• If I already passed the March 2019 Program and got the JT-NM Tested badge, do I need to retest at the August 2019 Program?
• Yes, if you want your product’s results to be published in the new catalog presented at IBC 2019 IP Showcase.

The Badges FAQ continued

• Do I have to undergo the ST 2110 testing in order to participate in NMOS/TR-1001-1 testing?
• Yes. The basic ST 2110 capabilities are required in TR-1001-1. The two test plans overlap and there are some tests relevant to TR-1001-1 already present in the SMPTE ST 2110 test plan. However, if a product does not have any TR-1001-1 functionality (e.g. DHCP, DNS), the tests will be adapted accordingly.
• How are the pass/fail/not attempted marked? Same as NAB?
• No. Clear differentiation between “not applicable” and “not passed” is introduced in order to be fair to vendors with products with limited functionality (e.g. audio-only products will have video-related tests marked as “not applicable”, a video product failing to decode a video payload will be marked as “not passed”). Therefore, the results will be marked as follows: Green - test passed, Blank - test is not applicable, Grey - test not passed, or not tested.

Getting the badges

• In order to have results published in the catalog, vendors/products must attend and pass the JT-NM Tested August 2019 Program event during the week of August 19th at the RIEDEL Communications Facility in Wuppertal, Germany.
• Only products physically present will be tested and may be included in the catalog and in promotion materials.
• A product can be a hardware, software, or an OEM solution with a development platform.
• All devices/software submitted for testing must be at the product level. Proof of product will be required at device/software sign-up. Prototypes and engineering versions will not be allowed.
• Same code versions have to be used for all tests.

Getting the badges FAQ

• Do I need to pass all the tests to get the badge? Some of the tests? Some minimum set of tests?
• All tests applicable to a product’s capabilities will be performed and results published. The product’s results may be published regardless of the amount of tests the product has successfully passed.
• Can I select which tests I participate in or do I need to participate in all the tests?
• You can opt out from a particular test, however if that test is applicable to the product (e.g. functionality is listed in the manual), it will be marked as “not tested”, not as “not applicable”.
• I am an OEM supplier. If I get a badge, can my OEM partners use and display it?
• No. We don’t know how your OEM partner integrated your solution and if the same performance is carried forward to the final product.
• Some of my products share the same IP core. Do I have to test each of those products if I want a badge for all of them?
• Yes. The implementation around an IP core may affect its performance.
• My company is an infrastructure or test and measurement equipment vendor. Can I participate in the JT-NM Tested August 2019 event?
• The JT-NM Tested August 2019 event is limited to the edge devices only (e.g. gateways, playout servers, cameras, mixing consoles, etc.). In order to be as consistent as possible, and in order to make it fair and equal to all participants, the network, PTP, test and measurement components are each provided by a single vendor. Please refer to "Equipment used during testing" in the JT-NM Tested August 2019 Test Plan.
• My product is the broadcast controller. Can I submit it for testing?
• Unfortunately - no. The JT-NM Tested August 2019 event is limited to the edge devices only.

Getting the badges FAQ - continued

• What are the test plans and pass/fail criteria?
• Test plans are published as parts of the final catalog and are also available to vendors in advance. Test plans provide context to the test results and also explain how the tests were executed, what aspects of what functionalities were tested and what the pass criteria were.
• Is retesting allowed during the event?
• We do not guarantee that there will be any opportunities for retesting. If retesting is available, it will be done as time permits, with a maximum of three attempts per product. The decision about whether retesting is available for any given product rests with the testing team and the JT-NM Tested Experts Group. Vendors will be given 20 minutes of testing time per product per attempt.
• Do I have to test a released product? What defines a released product?
• Yes. For the product - a press release or a live web page. For the code - proof of availability to end customer, either B2C or B2B.
• Are code changes permitted during the event prior to the testing slot?
• Yes, before the time of testing. At the time of testing Hardware/Firmware/Software versions are locked and recorded. In any case, software prototypes, beta software, development snapshots, any "special" versions will not be accepted for testing. The version of the code used for testing has to be generally available by IBC 2019.

Prerequisites to attend

• For JT-NM Tested August 2019 program vendors must meet the following prerequisites in order to submit products for testing:
• Vendors must familiarize themselves with the test plans and criteria. Vendors are expected to independently execute all the relevant tests before the face-2-face event.
• For Tx products: Vendors must execute the TX-related sections of the SMPTE ST 2110 test plan at their lab or an external lab. Vendors must capture a pcap file of a TX stream and upload it to the defined location. Vendors must also upload the pcap file to EBU LIST, download a JSON report and upload it to the defined location.
• For product applying for NMOS/TR-1001-1 badge: Vendors must pre-test using NMOS test suite at their lab or an external lab and upload the resulting JSON files.
• A special folder for the pcap file and JSON reports will be created for each vendor.
• A set of guidelines on how to create an acceptable pcap capture and how to operate the EBU LIST will be provided in advance.
• Only products that meet these prerequisites will be tested at the face-2-face event.
• The final results will solely be based on the results of the face-2-face event and not the prerequisite tests results.
• Version of software used for pretesting may be different from the version used at face-2-face event.

Acceptable PCAP capturing and pretesting

PCAP Capturing:

• Acceptable Methods for pcap capture:
• Commercially available test equipment: Packetstorm CRS, Tektronix PRISM, Bridge Technologies VB440.
• If you do not possess such equipment:
• “JT-NM Tested August 2019 - PCAP Capturing Guide” in the documents repository:
• https://drive.google.com/open?id=18UM-F354cMAEvgakVzcuxvdVHIOXVOk7

SDP file:

• Please upload an SDP file corresponding to your PCAP capture

PCAP Analysis:

• Capture at least 1 second of media traffic. Do not compress your PCAP before analyzing it with EBU LIST
• Do not use PCAPNG - it is not supported
• Guide explaining how to use the EBU LIST docker container and how to export the results JSON are available in the documents share folder:
• https://drive.google.com/open?id=18UM-F354cMAEvgakVzcuxvdVHIOXVOk7
• NMOS capabilities have to be tested with AMWA NMOS test suite and results should be also exported in JSON format

PCAP and results uploading:

• Compress your PCAP before uploading it to the FTP server
• Rename you PCAP and JSON files before uploading. Use exactly the same Vendor Name and Model used in the spreadsheet (JT-NM Testing Event Riedel August 2019)
• VendorName_Model_v1.pcap
• VendorName_Model_2110_v1.json
• VendorName_Model_2110-XX_v1.sdp
• VendorName_Model_NMOS_IS04_v1.json (for tests 2.1 to 2.4)
• VendorName_Model_NMOS_IS05_v1.json (for tests 3.1 to 3.6)
• VendorName_Model_NMOS_IS05_Integration_v1.json (for test 3.7)
• VendorName_Model_NMOS_IS08_v1.json (for tests 5.1 to 5.3)
• VendorName_Model_NMOS_IS08_Integration_v1.json (for test 5.4)
• Files with names that do not make sense will be ignored
• FTP server details:
• 6085040acaa1.sn.mynetname.net
• user: jtnm
• password: jtnmtestedaugust2019isawesome
• Upload your files to the JTNM-TESTED-PRETESTS folder
• You will not be able to see folder content
• You will not see your file after successful upload
• You will not be able to overwrite the file, so change the filename appropriately
• Deadline to upload PCAPs and JSONs – 9th of August 2019

Attending rules FAQ

• Do I have to send a product?
• Yes. Only products physically present will be tested and may be included in the catalog and in promotion materials.
• Do I have to send someone from my company?
• Yes, someone needs to represent your company at the event. You may ask for your company to be represented by a consultant or by a person from another company. However, having a person with a lot of experience in setting up and operating your product will increase the chances of achieving desired results. Reminder, you will only have a 20-minute test slot, so there is little time for someone unfamiliar to figure out how your product works during testing.

Attending rules FAQ - continued

• Do I have to attend the entire week or can I only come for the day my equipment is being tested?
• Plan to attend for entire week. There will be no strict schedule with days assigned to particular vendors as things can change during the event. Equipment connecting and disconnecting leads to network instability, therefore plan your return travel on Saturday, since equipment removal will not be allowed before the official end of the event (late on Friday). When your equipment is not in the formal test pods, it will be connected to the "zoo" area with all the other vendors. This is an excellent opportunity for ad-hoc testing of your equipment.
• What is the basic schedule?
• August 17th-18th - infrastructure setup days. Equipment installation is not allowed during those days.
• August 19th - equipment installation and start of testing. Deadline for the equipment to be installed and online - end of Monday.
• August 20th - 23rd - testing. No removal of equipment before the official end of the event.

Final report and results

• The final results from the face-2-face event will be compared to the prerequisite tests results by the testing team.
• The final published results will be based solely on the results of the face-2-face event tests results.
• To evaluate the effectiveness of the self testing, the pretesting and the face-2-face results may be compared, analyzed and an anonymized summary report may be generated by the testing team.
• The initial test results will be given to the participants during the event for their review.
• Final test result will be sent to participants not later than August 27th.
• Published test results will be grouped by application categories (e.g. audio-related tests, ancillary data related tests)
• The options to opt out from the final report is provided in the MOU, section 3. (c)
• The detailed rules on usage of the badge and the results will be provided separately

High level provisional logistics

• Event:
• August 19th – 23rd
• Location: RIEDEL Communications, Wuppertal, Germany
• Time: 8:00 – 19:00
• Morning meetings: 8:30

Supplies and consumables policy. What will be provided on site?

The JT-NM Tested Infrastructure group on site will provide:

• Racks with square holes (6 x of 80 centimeters deep, and 4 x of 60 centimeters deep)
• Rack screws and cage nuts
• Power distribution units to the racks (standard European schuko connector - https://en.wikipedia.org/wiki/Schuko)
• Baseband distribution connection points with standard female BNC connectors for video, XLR for audio and LC fiber for MADI (connectors will be confirmed by the infrastructure team). Without cabling.
• Network infrastructure (switches) without transceivers
• Top of rack SDI multiviewers and displays (TBC)
• UHD SDI Router (TBC)
• Tables and chairs
• Power strips with schuko sockets on the tables
• Wi-Fi access to the Internet and to the management network
• Screwdrivers for equipment racking
• Desks and chairs
• 9 Monitors above the racks. Monitor inputs will be routed from the SDI router
• Breathable air

​

Supplies and consumables policy. What will not be provided on site?

​

The JT-NM Tested Infrastructure group on site will not provide:

Any items not listed on the prior slide, including:

• Rack shelves or rails of any kind
• Optical transceivers of any kind, neither for switches nor for end equipment
• Cabling (fiber, copper, coax, SDI, XLR, MADI, etc.) of any kind
• AC/power cabling for end devices
• Keyboards, Mice, KVMs, PC screens/monitors, HDMI cables
• Speakers, headphones
• Cable adapters
• Power adapters (e.g. UK to schuko, US to schuko)
• Cable ties
• Shipping boxes, tape
• Shipping labels
• Additional SDI monitors
• List may be extended

RIEDEL will not be able to provide any missing materials for you on site

Supplies and Consumables - What you supply

Everything required to connect, configure, and rack your equipment. Only 19” racks are supplied. Please bring spares in case of damage or troubleshooting.

​

Power Cords:

In Rack Power Distribution Units and Tables will be outfitted with CEE7/7 Shuko female power receptacles.

​

Fiber and Ethernet Patch Cords (5 to 10 meters):

LC to LC, OM4, 50/125 Type, Multi Mode, Duplex

MTP Female 12 Fibers Type B Plenum (OFNP) OM4 (OM3) 50/125

Direct Attached Cables (DAC)

Active Optical Cables (AOC)

Cat5E or Cat6 Ethernet Patch Cable (for Management Network as well)

​

Rack Shelves, Mounting Brackets, Rack Rails for 19” Racks

​

Audio Cables: XLR, ¼”, 3.5mm, etc. (5 to 10 meters)

SDI Cables: BNC to the SDI routing equipment (5 to 10 meters)

​

Transceivers - 850nm 50µ MMF 10GBASE-SR, 25GBASE-SR, 40GBASE-SR, etc.

Both Sides Required!

​

Packing & Shipping Materials

AWG European Power Cord - CEE7/7 to IEC320 C13

Connector 1: CEE 7/7 Male

Connector 2: C13 IEC

​

​

Connector 1: CEE7/7 (Schuko)

Connector 2: IEC 60320 C19

​

​

​

Call to Action

• MOU deadline is the 28th of June, 2019
• List of hotel suggestions will be distributed via the mailing list.
• Complete Google Spreadsheet for each product that you want officially tested: Brand, Model, HW revision, SW revision, Type of Product, Sender and/or Receiver
• Contact Bob Ruhl bob.ruhl1@verizon.net to receive access to the Google Spreadsheet
• There will be a deadline for completing Google Spreadsheet
• We will not accept any additional products after this date -- Spreadsheet will be locked
• Meeting invite for teleconferences will be sent out.

Questions to the Participants

• Do you need any assistance filling out the Devices Tested tab?
• Since the deadline is still a few weeks away, are you willing to fill in some information preliminarily for infrastructure planning?
• Have you identified who in your company will be the Opt Out Representative?
• Do you have any questions about what equipment (cables, etc.) you need to provide?
• The infrastructure group is currently looking into sourcing the UHD SDI 12G router. If your company is willing to help out - please reach out to us.