1 of 18

What's new with Wikimedia Cloud Services

[[User:BDavis (WMF)]]

WikiConference North America, October 2024

2 of 18

What is Wikimedia Cloud Services?

01

3 of 18

WMCS core projects

Cloud VPS

Infrastructure as a Service

Jupyter notebooks

Web interface / JupyterHub

Toolforge

Platform as a Service

Database SQL

Web query interface

4 of 18

WMCS data services

Wikimedia Dumps

Backups of wiki content

Cloud Elastic

CirrusSearch replicas

ToolsDB

Shared database for tools

Wiki Replicas

Production database replicas

One of the main features that sets WMCS apart from other cloud computing services you might use is our access to data from the Wikimedia projects.

The Wiki Replicas databases are near-realtime copies of the MediaWiki databases of the Wikimedia project wikis. They can be queried from Toolforge tools or with the Quarry and Superset web interfaces.

Wikimedia Dumps offers a range of data downloads including full text dumps and other datasets. Tooforge tools get direct NFS access to the Dumps datafiles. Other Cloud VPS projects can request access if needed.

The Cloud Elastic servers offer full text searches of Wikimedia project wiki content using the same data schema used by CirrusSearch on the wikis themselves. These replicas allow running searches across multiple wikis and have longer query timeouts than their production counterparts.

ToolsDB is managed database hosting for Toolforge tools. Every Toolforge tool can create databases in ToolsDB and these databases can be read by other tools if that is desired.

5 of 18

Cloud VPS

02

6 of 18

Cloud VPS

192 projects
789 virtual machines
3,107 virtual CPUs
6.7TB RAM
98TB disk

^†Numbers taken from https://openstack-browser.toolforge.org/ on 2024-09-26

OpenStack deployment with Wikimedia focused customizations

7 of 18

Cloud VPS key features

Wikimedia Developer accounts (authentication)
OpenStack Horizon (management dashboard)
🆕 OpenStack Cinder (attachable block storage)
🆕 Ceph Object Gateway (S3 & Swift compatible storage services)
🆕 OpenStack Trove (database as a service)
🆕 OpenStack Magnum (Kubernetes as a service)
🆕 OpenStack APIs & OpenTofu
OpenStack Designate (DNS)
End-user privacy enhancing web proxy

One of our customizations to the OpenStack platform is using Wikimedia Developer account information for authentication both at the OpenStack API layer and in the virtual machine instances inside of the Cloud VPS hosted projects. This makes our system a little different than most other OpenStack clouds you may have seen, but that difference also makes it much easier for the entire Wikimedia technical contributor community to work together in Cloud VPS projects. Each Cloud VPS project has a list of authorized users chosen from the Developer account datastore. New people can be given access to your project without you needing to deal with the lower level things like setting their Linux shell account name or adding their ssh public keys.

Horizon is OpenStack's management dashboard. Our Horizon deployment includes customizations that let you manage things that are unique to Cloud VPS like configuring the Puppet modules applied to instances and registering web proxies. This is also where project admins can do things like add/remove users, create and manage virtual machines, and edit DNS records.

Cinder manages storage for nearly all of our virtual machines. It hosts both the instance storage as well as mountable volumes that make it easier to separate application storage from operating system storage.

In our deployment, Cinder is backed by Ceph which is the same FOSS storage platform used by CERN, OVH, and DigitalOcean. In addition to the Cinder block storage use case, Ceph lets us offer object storage with interfaces compatible with both Amazon S3 and OpenStack Swift APIs.

OpenStack Trove is a database as a service provider. It lets you create a new database server running MariaDB or PostgreSQL for use in your Cloud VPS project.

OpenStack Magnum is a tool for creating Kubernetes clusters in your project. This can be useful for projects that want to use Kubernetes but are somehow not well suited for our managed Kubernetes cluster in Toolforge. We consider Magnum to be a power user service today which mostly means it may have some rough edges that you need to work around when you use it.

In 2022 we moved our OpenStack API endpoints to a public network space. With the proper authentication credentials you can now interact with the Cloud VPS OpenStack services from anywhere on the Internet. One neat thing this enables is using OpenTofu (the FOSS fork of Hashicorp's Terraform) to manage instances, Magnum clusters, DNS, and more from your laptop. Maybe even more interestingly you can set up a gitops workflow running on the Wikimedia GitLab service to apply tofu changes to your project.

Designate is the DNS management service for OpenStack. Each Cloud VPS project is given the ability to manage DNS records in an internal to Cloud VPS use $PROJECT.eqiad1.wikimedia.cloud zone as well as a public $PROJECT.wmcloud.org zone. Records in these zones can be managed with Horizon, the OpenStack APIs, or even OpenTofu.

Many Cloud VPS projects run webservices. To conserve public IP addresses and to enhance end-user privacy, Cloud VPS offers a reverse proxy service that strips the end-user's IP from the information passed to the upstream web server inside a Cloud VPS project. The proxy also provides TLS termination and adds Strict-Transport-Security protection for end users. Proxies can be managed though Horizon or OpenTofu.

8 of 18

Toolforge

03

9 of 18

Toolforge

3373 hosted tools
2686 tool maintainers
106 virtual machines
742 virtual CPUs
1.7TB RAM
35TB disk

^†Numbers taken from

https://toolsadmin.wikimedia.org/ and https://openstack-browser.toolforge.org/ on 2024-09-26

Platform as a Service for bots and webservices

10 of 18

Son of Grid Engine removed

Grid Engine

11 of 18

Toolforge key features

Kubernetes
🆕 Build Service (Cloud Native Buildpacks)
🆕 Jobs framework (one-off, continuous, and scheduled jobs)
🆕 envvars (config secrets storage)
Webservice (per-tool subdomain hosting)
ToolsDB (managed databases)
Redis
Elasticsearch
Dumps (NFS access to all dumps.wikimedia.org files)

Now that Grid Engine is gone, Kubernetes is the only distributed workload scheduling system used by Toolforge tools. Each tool is given control of a Kubernetes Namespace where the tool's jobs can run.

We do not allow arbitrary containers to run on the Kubernetes cluster. Instead we have a collection of images maintained by the Toolforge admins for running bots and webservices written in typical programing languages for our community (Python, PHP, NodeJS, Mono, Java, Perl, Ruby, etc).

Toolforge added a build service for creating custom images that include a tool's code and dependencies in 2022. This service is based on the Cloud Native Buildpacks standard. Buildpacks allow you to generate a container image without having to add extra scripts or Dockerfiles to your code base. A custom image can support multiple runtime languages and add additional libraries and support programs that your tool may need.

2022 also saw the introduction of the Toolforge Jobs framework. This service helps simplify managing jobs on the Kubernetes cluster. Using the Job service you can start a job for a one-time use, create a job that will run continuously, or configure a job to start on a timer. It is very roughly the equivalent of the jsub and crontab services that existed for Grid Engine. The container used can be one of the centrally maintained images or with an image from the build service.

Managing configuration secrets was historically something each tool needed to figure out for itself. A typical solution was reading a non-public file at runtime. In 2023 we introduced an envvars service for managing environment variables which are exposed to your tool's webservice or job service jobs. Your tool can take advantage of this service by implementing the twelve-factor application pattern of storing config in the runtime environment.

Webservice is the Toolforge command for starting an HTTP service exposed to the Internet. It allows using either a centrally maintained image or an image from the build service as the runtime environment. The URL to your tool's webservice will be https://$TOOL.toolforge.org once it has started. A privacy preserving reverse proxy much like the one used for *.wmcloud.org URLs in Cloud VPS will handle TLS termination and basic rate limiting for you.

ToolsDB is a managed MariaDB service for Toolforge tools. Every tool is given a username and password for authenticating to ToolsDB. These credentials will allow you to create new schemas prefixed with your username to avoid name collision issues. Schemas with names ending in _p are readable by any tool. This allows you to create tools which provide interesting data to other tools. One example of a public schema is s53734__toolviews_p which publishes data on the number of successful HTTP requests each tool has handled.

A shared Redis instance is available for minor caching needs. Redis is not natively multi-tenant so we have made some tweaks to the configuration to try and make is it bit harder to snoop on data from other tools. We also ask that you use a random prefix for your tool's storage keys.

A shared Elasticsearch instance is available for full text search needs. Tools must request their own username and password to be able to write to this service.

Toolforge tools can access files from dumps.wikimedia.org via a dedicated NFS share rather than downloading and managing these often large files themselves.

12 of 18

Toolforge highlights

Multiple replicas for continuous jobs and webservices
Internal services
Health checks for jobs and webservices
One-click GitLab git hosting with self-service CI
One-click Phabricator project creation
Toolhub integration
Unwanted tools can be deleted
"Thin" bastions

Both continuous jobs and webservices allow you to specify that more than one copy of the application should be running. This can be used to "scale up" idempotent processes to handle more requests.

---

Internal services are what we call continuous jobs that expose a TCP/IP port to other applications running in the Toolforge Kubernetes cluster. Today this functionality can be used to do things like running a backend API service for webservice frontend or to run a local Redis instance just for your tool. We are working towards a webservice simply being an internal service that is exposed to the toolforge.org reverse proxy.

---

Both continuous jobs and webservices now support a health check option that can be used to help Kubernetes decide if your service needs to be restarted.

For a continuous job a check is written as a shell command to be run every second during job startup and every 10 seconds once the job is established. Startup is the period between when the job first starts running and when the check first passes. If the check does not pass in the first 120 seconds then the job will be force killed and restarted. After the check passes once the check enters a "liveness" phase where a forced restart of the job will be triggered by 3 consecutive check failures.

Checks for a webservice work similarly, but the check takes the form of a path to call on your application. /healthz is a widely used convention for this URL, but you can specify anything you want. If no explicit health check path is provided a simple TCP socket check will be done instead.

---

Toolforge has offered a way to setup git hosting for your tool for quite some time from the toolsadmin.wikimedia.org control panel. In late 2022 all of these git repos were migrated to gitlab.wikimedia.org which provides a more complete code forge environment including self-service CI. An advantage of hosting your tool's source code at github.wikimedia.org is that it will be much easier to collaborate with other Wikimedians. It will also become possible for local admins to help others gain access to your source code if for some reason you abandon your tool.

---

Toolsadmin also lets you create a Phabricator project for your tool since late 2020. Having a dedicated Phabricator project for your tool can make is easier for community members to report bugs and to see what known issues your tool currently has.

---

Toolsadmin lets you make toolinfo.json records for your tool. These records are automatically added to toolhub.wikimedia.org. For those of you who haven't heard of Toolhub, it is a community maintained catalog of tools for the Wikimedia movement that launched in October 2021.

---

Not all tools are intended to live forever. Some are experiments. Some are auditions for other projects. Some finish the work that they were designed to do. Some are just not interesting to their creators anymore. For years folks wondered how they could actually delete a tool when they were done with it. It turns out that this is a rather complex problem involving a number of different Toolforge backend services. In October 2022 we were finally able to provide a solution. A tool maintainer can now use the toolsadmin console to "disable" their tool. This stops all of the running processes for the tool and starts a countdown clock to archive and delete the tool's $HOME directory after 40 days. You can change your mind at any point during this period and stop the processes. The https://disabled-tools.toolforge.org/ tool can be used to see which tools are currently pending deletion.

---

Thin bastions may be another strange thing to highlight as it is more of a feature removal than a feature addition. When we were still using Grid Engine the Toolforge bastions each had a full copy of the runtime applications and libraries available on the job grid. Now that we have removed Grid Engine we have also removed many applications and libraries from the bastions. The main benefit we hope to get from this is reduced contention for resources on the bastions themselves. When the bastions were able to run any code meant for the job grid folks would occasionally run their bots directly on the bastions. This would often lead to login.toolforge.org becoming overwhelmed by bot traffic and prevent others from doing necessary tasks from the bastions like starting their jobs or checking error logs.

13 of 18

Developer accounts & Wikitech

04

14 of 18

Developer accounts & Wikitech

Wikimedia IDM has replaced Wikitech as the primary interface for managing Developer accounts
{{In progress}} Wikitech is becoming a SUL wiki

Wikitech became the tool that the Wikimedia movement used to create and manage Developer accounts early in the history of the WMCS project. Ryan Lane wrote some cool software that let Wikitech connect with an LDAP directory and set things up so that LDAP directory could be used with OpenStack. As the use and desired features for Developer accounts has grown, using Wikitech as a management console for these accounts has become less desirable. The toolsadmin console for Toolforge introduced an alternate management interface for many aspects of Developer accounts, but also did so in a way that was heavily Toolforge branded and optimized for Toolforge workflows.

Since 2022 SREs at the Wikimedia Foundation have been working on a new Identity Management Service (IDM) to take over Developer account management from Wikitech. As the IDM has gained features we have been slowing turning related systems off on Wikitech. This has progressed to the point where all of the functionality for creating, managing, and blocking Developer accounts now lies with the IDM.

Making these changes has also unblocked a long imagined change for Wikitech. Folks from the WMCS and SRE teams are now working to clean up accounts and make config changes that will convert Wikitech to a "normal" SUL wiki like enwiki, mediawiki.org, and commons. We hope this change makes it easier for Wikimedians to contribute to the technical documentation on Wikitech. We also hope it reduces confusion about which account credentials to use when logging in there.

15 of 18

Communication and support

05

16 of 18

Communication and support

Mailing Lists

IRC on libera.chat

Tech blog

https://techblog.wikimedia.org/

Documentation

Bugs & Feature Requests

https://phabricator.wikimedia.org/tag/cloud-services/

17 of 18

Questions & answers

05

18 of 18

Credits

Wikimedia Foundation logo - horizontal.svg By Wikimedia Foundation, CC BY-SA 3.0
Cloud Services products logo.svg By BDavis (WMF) - Own work, includes elements from Wikimedia labs logo.svg, CC BY-SA 4.0
Wikimedia Cloud Services logo.svg By Brandon Harris and Isarra - Vectorisation of Wikimedia labs small logo.png, CC BY-SA 3.0
Toolforge logo.svg By BDavis (WMF) - Own work, includes elements from Wikimedia Community Logo.svg and Forge Noun project 1044767.svg, CC BY-SA 4.0
Quarry-logo.svg By Husky - Own work, CC BY-SA 4.0
PAWS.svg By Barquill0 - PHID-FILE-46r7an25hhlmrkxoault/PAWS.svg, CC BY-SA 3.0
OpenStack-Logo-Horizontal.png By ™/®0penStack Foundation

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International license.