VES Technology Committee WFH best practices and tips & tricks
The following is advice gathered and collected by the members of the VES Technology committee or submitted by members of the community.
It has been gathered here to help the VFX community by sharing some technical solutions to common problems we may encounter in trying to prepare and adapt to Work from home workflows that are for security and technical reasons difficult in our industry.
These are in no means to be interpreted as an endorsement of one solution over another and/or are guaranteed to work in every scenario.
If you want to contribute further advice, please either comment on the document itself or send an email to ves-tech-cov19@googlegroups.com
We hope that these are helpful. Stay healthy!
The VES Technology Committee
Addison Bath, Nick Cannon, Francois Chardavoine, HP Duiker, Ray Feeney, Darin Grant, Barbara Ford Grant, Steve May, Jean-Francois Pannisset, Sam Richards, Michele Sciolette, Sebastian Sylwan.
Guidance from Studios, Vendors and Facilities 2
Where to discuss issues with your peers 3
What are the (secure) remote desktop solutions? 3
What are the bandwidth comparisons? 8
What information to give employees before WFH? 8
Secure File Transfer Solutions 8
Application Specific Information 9
Special Mitigation offers from vendors: 9
Guidance from Studios, Vendors and Facilities
The guidelines in this section have been provided to the VES Tech Committee by various studios (clients), Vendors, and Facilities as their suggested practices. In order to disseminate as much information as possible, they are provided here without specific endorsement.
- Working Remote with Foundry Tools
- Working Remotely with SideFX Houdini
- Autodesk COVID-19 Resource Guide
- Adobe COVID-19 Response
- Netflix Remote Editorial Memo (work in progress document, will be updated, link should remain valid for new versions)
- Shotgun Community Posts on client WFH experiences
- NVIDIA Working Remotely: Connecting to Your Office Workstation
- MPA Best Practice Guidelines to Consider for Remote Content Handling
In the interest of allowing as much information to be shared by vendors as possible, we have created a new section at the end of this document to allow vendors to link to their own sites/pages/etc. As a result, we have pulled those sections out of the rest of this document. We thank them for their contributions to the efforts to safely work from home.
Where to discuss issues with your peers
- StudioSysadmins Slack #discuss-dr channel (you may need to post to the following thread on the SSA mailing list at http://studiosysadmins.com/board/threadview/6296/ or ask someone to invite you). If you are a current member, you can invite a colleague with: /invite_people <email_adddress>
- Reddit thread: help request from small studio
What are the (secure) remote desktop solutions?
- Server / Workstation: hardware solution (Remote Workstation Card) on Linux or Windows, can be used on Mac with external PCIe cardcage (see solution from Amulet Hotkey). Software solution for Windows and Linux: Cloud Access Software Graphics Edition (needed for GPU support), bundled as Teradici All Access Cloud Access + subscription.
- Client: ZeroClient hardware solution (multiple vendors), connect monitors and peripherals directly to ZeroClient hardware. Software Client (no charge) for Windows, Linux, and macOS.
- Teradici's data stream is encrypted (if enabled). Check with the clients if they would authorize its use without having to have a VPN connection on top (which hinders performance). Depending on your network topology this may be secure enough for them.
- One free license ships with each HP Z workstation, licenses can be purchased for other workstations.
- Cons: Client Sender Software running only on Windows and Linux (no macOS), the receiver options are OSX, Windows and Linux.
- Recent versions (7.7, 2020.0.0) of the RGS Receiver on macOS have been unable to start on certain machines. A workaround appears to be to increase the maximum value of socket buffers which can be done in /etc/sysctl.conf :
kern.ipc.maxsockbuf=8388608 - Renamed and Updated to be HP Remote Boost, there are 90 day free trials available of the agent software and the centralised management solution:
- Additional tips regarding using Wacom devices with RGS provided by the Vendor are located here.
- HP RGS currently only supports 44.1KHz audio, if you need support for 48KHz audio to use Autodesk Flame, you can request a patch from the HP support forum to enable 48KHz audio for Flame 2020. The patch script can be retrieved from the following locations:
SFTP Access : sftp -o Port=2222 flamergs@ftp.usa.hp.com
sftp -P 2222 flamergs@ftp.usa.hp.com
HTTPS Access: https://ftp.usa.hp.com/hprc
Drop Box Host: ftp.usa.hp.com (15.73.40.56, Failover: 15.73.244.52)
Login: flamergs
Password: k=WM8fc_
This fix has been reportedly included in version 20.0.1 of HP Remote Boost
- ThinLinc
- TGX - Similar to RGS, Sender only available on windows and linux, receiver options are OSX, Windows and Linux.
- Windows Remote Desktop
- Server / Workstation: Windows only (all versions), requires Windows 10 Pro or Enterprise, single session only (unless you run Windows 10 Enterprise on Azure, which supports “Windows 10 Enterprise multi-session”. Windows Server supports 2 sessions, more if you add Terminal Services CALs. Remote Desktop access typically enabled per user based on Active Directory group membership.
- Windows Client: Windows client built in.
- macOS client: Microsoft Remote Desktop 10 (Apple macOS App Store) or Microsoft Remote Desktop 10 Beta
- Linux Client: Remmina, rdesktop
- On NVIDIA GPUS OpenGL acceleration over Remote Desktop connection was only supported with Quadro cards: NVIDIA has now released support for GeForce GPUs as well
- Nomachine - can do OSX both client and server.
- BeBop - virtualized, secure, remote post production offerings
Reviewing content remotely
Remote review
- HP RGS - Multiple people connecting to the same session
- RV - Using sync.
- FFMpeg - Broadcasting screen to another screen. (Possibly use VLC)
- Cloud Based
Remote Streaming Solutions
When you need to stream the SDI output of an editorial, color grading, VFX review system to users at home.
- Sohonet Clearview Flex: on premise hardware box encodes SDI output, streams can be viewed in browsers, mobile devices, AppleTV. https://www.sohonet.com/clearview-flex/
- TECHSTREAM: Similar to the above
PCoIP Tips & Tricks
- Turn off Dithering in Nvidia Control panel / Xorg config file: this will greatly diminish the amount of bandwidth required / increase image quality for a given amount of bandwidth.
- Pressure sensitivity for Wacom only works for Linux and Windows agents
- Despite documentation stating otherwise, Win10 -> Win10 Cintiqs are supported
- Adding this setting will help with the flickering menus issue that people have pointed out:
- pcoip.use_host_autorepeat 1
- 90 day license for software sender and software client. Need to ask a sales rep about this special flash sale / availability.
- Notes on deploying the latest Cloud Access Manager infrastructure:
- You will likely want the "Cloud Access +" licenses since that's what you need to support GPU-accelerated software agents and/or latest Remote Workstation Card / Zero Client Firmware (which unlocks UHD/30 support on DisplayPort hardware)
- Licensing is either on prem (runs on CentOS / RHEL) or cloud / service. Cloud based licensing is wrapped around Flexera FlexLM (didn't know they offered cloud-based licensing, that's useful)
- Management is done from the Cloud Access Manager at cam.teradici.com You login with either G-Suite or Azure credentials (you need to interact with Teradici support if you don't have those and need to use a personal Google account), create a "Deployment" (a container for your workstations and users) tied to the license string you purchase from Teradici. You then create a "Connector", which is an on-prem server that sits between your workstations and the web console. That step gives you a registration token you will use when deploying the local Connector.
- On prem you need a 2 CPU Ubuntu 18.04 server / VM where you will install the connector (CAC - Cloud Access Connector). A static binary installer is provided: you run the installer by providing your license key, the registration token provided by the web console, AD credentials for a service account, and the DNs of your AD containers where you have the users and computers which will be participating. The installer will install Docker and runs 8 separate containers, including a Sumologic collector that goes to a Teradici Sumologic instance for health monitoring (they document that).
- Once this is done you can go back to the Web console and start adding workstations and users (from the mirrored AD OUs) and assign users to workstations.
- The CAC server also exports a similar web interface, and it seems to be possible to do this directly from the CAC rather than the Teradici-hosted console. Presumably that's to support entirely self-hosted workflows.
- The web console has the ability to deploy and manage workstations to GCP and Azure, I haven't explored that yet.
- Assuming external users have VPN access (there is a way to not use a VPN, I believe it's called the "Gateway" but I haven't explored that yet), they would point the Teradici Software Client to the CAC server (haven't experimented yet with a hardware Zero Client) and login with their AD credentials: if they have been assigned to a single workstation, they will be connected to it directly. If they have been assigned to multiple workstations, they will get a selection menu to pick which workstation to connect to.
- Note: Using Teradici over VPN versus the external Cloud Access manager removes the requirement that all network traffic flows through the CAC as stated below)
- Alternatively Software Client users can connect to the external Cloud Access Manager (cam.teradici.com) using their AD credentials: they will then be able to connect to internal systems without needing a VPN connection, bandwidth is limited to 400 mbit/sec (limitation of the CAC). You can increase this limit by running multiple CACs behind a load balancer (have not tried this yet). Can be useful if firewall becomes bottleneck processing VPN connections.
- On Windows / Linux machines without a Remote Workstation Card, you would install the Graphics Agent for Windows: you give it your license token at installation time, and it will pull licenses at remote connection time. If you have more than one network interface you can specify which interface the PCoIP traffic will go over.
- On Linux machines the pcoip_agent is acting as your Display Manager, but contrary to kdm / gdm it doesn't start the X server until the user has connected. So when no connection is active, X isn't running. There's an issue with the current RPM installer (2020.01.1) which fails to recognize kdm as a possible running Display Manager, and you end up with pcoip_agent and kdm fighting. So before installing the agent on Linux, use “systemctl stop kdm; systemctl disable kdm” to stop and disable kdm before installing the RPM.
- An issue on Linux / Flame systems: when pcoip_agent wants to run the X server, it creates a snippet of xorg.conf configuration in /tmp to turn off dithering with the NVIDIA driver, but on a system running an older version of CentOS than CentOS 7.6, the syntax of that snippet isn't recognized and the X server refuses to start. Support for that syntax was introduced into Xorg server 1.20RC1, and CentOS 7.6 includes Xorg 1.20.1 with that support. CentOS 7.6 is thus required to run any of the Teradici software agents (the Remote Workstation Card Agent or the Graphics Agent) version 2020 and newer.
- On Linux systems with the Remote Workstation Card running firmware 2020.01, you install both the 2020.01.1 host software (which includes the kernel driver to talk to the RWC and the Teradici control panel to control it) and the Remote Workstation Card Agent, which is really the Standard Agent configured in "RWC" mode. This allows you to integrate systems with the RWC card into the Cloud Access Manager infrastructure: you add the system to your console (using its own hostname, not the one for the separate interface of the RWC), and you can login through the CAC the same way you login to a system running a software-based Agent. The same two caveats as above exists: you need to disable kdm before installing the Remote Workstation Card Agent if that's the Display Manager you are running, and older versions of X / NVIDIA driver may be stymied by the xorg.conf config snippet to disable dithering.
- You can roll back the RWC firmware to version 5.1, the last version that supports direct connection from a Zero Client / Software Client and remove the RWC Agent if you just want to stick to direct connections / can't get the RWC agent to work. The 2020.01.1 host software for Linux is compatible with the 5.1 RWC firmware. You would then connect directly to the RWC network interface. But I've had little luck connecting with the 2020.01.1 Software Client on Mac: you can connect but the app segfaults after 30 seconds. Version 19.08.6 seems to work.
- Teradici Software Client users reporting black screens on login to systems with Remote Workstation Cards: unless you are running the latest 2020.01.1 firmware on those cards (which no longer supports direct connection and requires some kind of connection broker), the Remote Workstation Card is limited to Dual Link DVI resolution, i.e. 2560x1600, even when the GPU is connected to the RWC over DisplayPort. If a user tries to connect while running the Software Client on a higher resolution display, they are likely to get either a smaller image window inside their full screen window, or a completely black display. Workarounds are:
- run in windowed mode, keeping the window size below 2560x1600
- lower the display resolution (System Preferences -> Displays -> Resolution: Scaled on a mac)
- upgrade to firmware 2020.01.1 which supports up to UHD/30 if you can figure out how to run the Remote Workstation Agent on your system (I haven't yet) and can broker connections (Leostream, VMWare Horizon, Cloud Access Manager...). Haven't tested this last one myself.
- Autodesk has announced official support for Flame 2020.3.1 on CentOS 7.6 with DKU 15.0.0, allowing the use of latest Teradici software versions in a supported environment. https://knowledge.autodesk.com/search-result/caas/sfdcarticles/sfdcarticles/flame-sysreqs.html
- To use the Graphics Agent on a Windows Workstation with a NVIDIA GTX / RTX (i.e. non Quadro) card and no monitor attached, you need a HDMI dongle to fake the presence of a monitor. These are readily available online. You must disconnect any other monitors with GTX/RTX, in addition to adding the Display Emulator plug. This is to avoid screen resolution issues when connecting to e.g. a 4K display on the host from a 2K display on the client side. More information at https://help.teradici.com/s/article/3770
- Aloys Baillet has created a Pulse Secure / Teradici / nVidia docker image for Linux PCoIP use (our setup at Animal Logic)
- Teradici minisite with helpful info aggregated http://www.teradici.com/remote-work/ and specific documentation on understanding and deploying PCoIP infrastructure can be found at: https://www.teradici.com/web-help/work-from-home/
- Teradici article about using Remote Workstation Cards over VPN (they don’t work with non-VPN access through CACs): https://help.teradici.com/s/article/1487
- By default the PCoIP Software Client on macOS will remap the Command/Apple keys to the neighboring Control keys, making it impossible for the host system to differentiate between these modifier keys. Although currently not possible from the UI, you can disable this functionality by modifying the plist for the application:
defaults write "com.teradici.Teradici PCoIP Client.plist" remap_cmd_to_ctrl 0;
Followed by a reboot of the Mac. After that the Software Client should send different keycodes for the Command/Apple and Control keys. - ILP (Important Looking Pirates) has open sourced a customizable connection broker for Teradici connections: https://github.com/ilpvfx/interstate_love_song
What works on Mac? VNC?
Amulet Hotkey for hardware Teradici solution (external PCIe card cage to house Remote Workstation PCIe card).
What are the bandwidth comparisons?
Using single monitor 1920x1200 as a baseline resolution
- Teradici (NVidia dithering setting off): 1Mbps for operations such as nodegraph manipulations/scripting to 35Mbps for fullscreen frames playback
- HP RGS: <1Mbps node graph manipulation to 12Mbps for full screen playback with audio
What information to give employees before WFH?
Barring anything else being posted, please review the guidelines shared by Netflix.
Secure File Transfer Solutions
- Aspera (self vs cloud hosted)
- Signiant
- BeBop Rocket
VPN solutions
- OpenVPN
- Wireguard
- Clients for Windows, macOS, Linux, iOS, Android at https://www.wireguard.com/install/
Application Specific Information
- Using Substance Painter over Windows Remote Desktop (RDP)
- Nuke on Windows will not start up when connecting via a Remote Desktop session: this claims that “GPU access over RDP” can be enabled in Windows 10 with a GPO. This may require NVIDIA Quadro cards. -https://community.esri.com/thread/225251-enabling-gpu-rendering-on-windows-server-2016-windows-10-rdp
- Adobe Creative Cloud Deployment Best Practices on Virtualized Infrastructure
- To take advantage of the NVIDIA DesignWorks package allowing OpenGL on GeForce GPUs over Windows Remote Desktop connections, Maya 2019.3.1 and 2020 require setting the environment variable MAYA_ALLOW_OPENGL_REMOTE_SESSION=1 https://knowledge.autodesk.com/support/maya/troubleshooting/caas/simplecontent/content/using-maya-windows-remote-desktop-connection.html
Special Mitigation offers from vendors:
- If you have HP servers the Advanced ILO license is free until the end of the year.
Vendor Supplied Information
BeBop Technology -
- Virtualized high-end remote workstations more powerful than any machine on-premises or any laptop
- Replaces on-premises hardware with instantly scalable infrastructure to fit any project
- Only hardware needed is a computer and a modest internet connection
- Access to industry-standard tools for VFX, editing, titling, design, 3D modeling & animation, compositing, motion graphics, and compliance, including Adobe Creative Cloud applications, Foundry products including Nuke, and Autodesk software including Maya
- Expandable high-speed storage
- Supports your existing accounts with the major Cloud Service Providers (Microsoft Azure, Amazon Web Services, Google Cloud Platform) or an account fully managed by BeBop
- BeBop Rocket for seamless, automated file movement to and from on-premises for onboarding existing workflows and projects into the cloud
- Remotely collaborate and review/approve Over The Shoulder securely, in real-time
- Subscription provides unlimited access to the BeBop Platform, including OTS & Rocket
- Secure, complete, affordable, and viable Business Continuity and Disaster Recovery solutions
- Comprehensive cost and use reports and analysis
- 24/7 support
- Highest level of security available, including 2-factor authentication
- High-capacity cloud rendering service, running on both AWS and GCP compute environments.
- Offloads renders to the cloud by detecting and uploading all scene dependencies and running all frames on separate cloud VMs in parallel.
- Setup < 30 min: Conductor plugin installs into all supported and detected DCC packages on the workstation.
- Will need higher bandwidth connection for efficient uploading, but no latency requirements, as all required data is transferred to the cloud service for the scene to run.
- Per-minute licensing for major DCC/render software packages managed by the service, but artists will need a single local license (studio checkout or other mechanism) to invoke scene for submission.
- A project-based payroll platform, tailored to entertainment. In addition, to be union compliant, it supports direct deposit so employers can pay their teams quickly and securely without any physical contact.