The WikiTeq Calendar
A Maintenance Schedule for MediaWiki
Introduction
2
Welcome to MediaWiki Maintenance
Dive into the essentials of keeping your MediaWiki installation robust, secure, and efficient.
Scope of Presentation
We'll explore the importance of maintenance, tackle common challenges, and guide you through a comprehensive maintenance schedule.
What to Expect
Gain insights into maintenance tasks, their cadence, and the strategic approach to a well-maintained MediaWiki environment.
Importance of Maintenance
3
Ensuring System Health
Regular maintenance is pivotal for the ongoing health, security, and performance of MediaWiki installations.
Risk Mitigation
Neglecting maintenance can expose systems to security vulnerabilities, data loss, and degraded performance.
Maintenance as a Necessity
Proactive maintenance is not optional; it's a critical investment in the stability and reliability of your MediaWiki environment.
Debating the Necessity
4
Myth vs. Reality
Tackling common misconceptions that maintenance is optional or only reactive.
Prevention Through Maintenance
Examples where regular maintenance averted potential issues, enhancing performance and security.
Consequences of Neglect
Illustrating the real-world impact of neglected maintenance, including system failures and security breaches.
Challenges in Maintenance
Time Constraints
Complexity and Expertise
Evolving Software Landscape
Administrators often juggle multiple tasks, making dedicated time for maintenance challenging.
The technical complexity of MediaWiki and its extensions can require specialized knowledge.
Keeping up with the rapid pace of updates and security patches presents a continuous challenge.
Administrators often juggle multiple tasks, making dedicated time for maintenance challenging.
The technical complexity of MediaWiki and its extensions can require specialized knowledge.
Keeping up with the rapid pace of updates and security patches presents a continuous challenge.
Administrators often juggle multiple tasks, making dedicated time for maintenance challenging.
The technical complexity of MediaWiki and its extensions can require specialized knowledge.
Keeping up with the rapid pace of updates and security patches presents a continuous challenge.
Benefits of a Maintenance Schedule
6
Simplification of Processes
A structured schedule breaks down complex tasks into manageable, routine activities.
Risk Reduction
Regularly scheduled maintenance tasks minimize the risk of security breaches and system downtime.
Continuity and Reliability
Ensures the MediaWiki environment remains stable, up-to-date, and reliable for users.
Visual Maintenance Timeline
Semiannually
Quarterly
Monthly & Bimestrially
7
LTS
Visual Maintenance Timeline
8
Monthly and Bimestrial Tasks
Server package checks
Updates to extensions and skins
Log reviews
Visual Maintenance Timeline
9
Quarterly Tasks
Site analysis
Security updates on core
Check backups, caches & monitors
Visual Maintenance Timeline
10
Semiannual Tasks
Site performance and security checks
Disk space cleanup & general wiki maintenance
Test restore from backup
Visual Maintenance Timeline
11
Updates to extensions, skins, composer dependencies
Update server packages
Check server logs
Site analysis
Check backups
Check caches
Security updates on core
Check monitors
Check site performance
Check site security
Disk space cleanup
Test restore from backup
General wiki maintenance
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
…
Walkthrough of Maintenance Tasks
Proactive • Holistic • Continuous
12
Walkthrough of Maintenance Tasks (monthly)
Updates to extensions and skins
13
Performance Impact
Evaluate extensions for their impact on site performance and user experience.
Security and Updates
Ensure extensions are free from known vulnerabilities, maintaining the overall security posture of your MediaWiki installation.
Best Practice:
Systematic Approach:
Stay informed about new releases and updates for extensions and skins and deploy updates systematically.
This also includes updating composer dependencies.
Walkthrough of Maintenance Tasks (monthly)
Updates to extensions and skins (Docker)
14
Updating Docker Images
For Docker-based wikis, update the latest docker image label in the configuration file.
Handling External Extensions and Skins
Add extensions and skins not included in the docker image as submodules, ensuring correct branch alignment.
General Rule:
Adhere to a smart Docker image.
Walkthrough of Maintenance Tasks (bimestrially)
Update server packages
15
Scheduled Reboot
If the system kernel has been upgraded or there are services that need to be restarted, schedule the system reboot, considering the wiki audience timezone.
Keep an eye on compatibility
There can be packages that require a specific version (e.g. elasticsearch) and should be excluded from a routine upgrade.
You may want to upgrade only a few packages at a time if any seem likely to cause breakage.
Check updates availability
Verify server software is up-to-date with the latest security patches and updates.
Benefits:
Walkthrough of Maintenance Tasks (bimestrially)
Check server logs
16
Monitoring System Health
Examine system logs for error messages and failed services to ensure server stability.
Docker Logs Analysis
For Docker instances, scrutinize logs within the stack and container logs for critical error messages.
Optimization Opportunities
Logs can highlight areas for performance optimization, leading to improved user experience and system efficiency.
Be Proactive
Utilize proactive problem-solving and error-handling techniques, leveraging server logs to address issues promptly and prevent escalation.
Mail System Audits
Review mail logs to detect email delivery failures and potential service outages.
Walkthrough of Maintenance Tasks (quarterly)
Comprehensive site analysis
17
Extension Functionality
Verify all extensions, especially their test pages and features, are functioning correctly.
Media Handling
Ensure file uploads and thumbnail generation operate smoothly.
Auxiliary Systems
Test the reliability of email sending and search indexing capabilities.
VisualEditor and Citoid
Confirm the operational status of VisualEditor and, if applicable, the Citoid service for reference management.
System Health Checks
Review maintenance automation reports, cron scripts, job queue, backups, monitoring, and critical updates for extensions and skins.
Walkthrough of Maintenance Tasks (quarterly)
Checking backups
18
Check policy compliance
Ensure that snapshots are taken in accordance with the schedule and rotated following the established policy.
Prove basic recoverability
Retrieve from storage and restore the latest backup in a temporary directory. Compare directory structure with the original.
Check backup storage space
Monitoring storage space allows administrators to address potential issues before they lead to backup failures.
Benefits:
Walkthrough of Maintenance Tasks (quarterly)
Checking cache systems
19
Redis Verification
Confirm the operation of Redis and its configuration within the MediaWiki settings for optimal caching.
Docker Cache Checks
In Docker environments, use commands to ensure Redis services are active and correctly logging read/write actions.
Varnish Cache Confirmation
Validate the use of Varnish by requesting pages and reviewing HTTP headers to check for proper cache delivery and update mechanisms.
Benefits:
Walkthrough of Maintenance Tasks (quarterly)
Security updates on core (“Minor”)
20
Core
Update core MediaWiki to its latest minor version (dot release).
Submodules
Often when there is a MediaWiki core security release, extensions announce security issues at the same time.
Not bundled extensions
Check if any of the extensions with security announcements are used, and if so, update them too.
Docker-based deployments
We follow the same principles, but use a specific routine for updating the Docker image and pulling it into the wiki configuration.
Benefits of keeping to LTS releases:
Walkthrough of Maintenance Tasks (quarterly)
Check monitors
21
Uptime Kuma
Uptime Kuma is an easy-to-use self-hosted tool to monitor uptime for HTTPs / TCP / Ping / DNS Record / Docker Containers / etc.
Walkthrough of Maintenance Tasks (quarterly)
Check monitors
22
Monit
A versatile utility for monitoring and managing services and resources, filesystems and networks on Unix-based systems.
Walkthrough of Maintenance Tasks (quarterly)
Check monitors
23
Benefits:
Monit
A versatile utility for monitoring and managing services and resources, filesystems and networks on Unix-based systems.
Uptime Kuma
Uptime Kuma is an easy-to-use self-hosted tool to monitor uptime for HTTPs / TCP / Ping / DNS Record / Docker Containers / etc.
Walkthrough of Maintenance Tasks (semiannually)
Check Site Performance
24
CPU Load Analysis
Monitor Load Average values for CPU usage to identify potential overloads or bottlenecks.
Memory Utilization
Review RAM and SWAP usage to detect if memory is being excessively used or if there is a need for additional resources.
Job Queue Monitoring
Inspect job logs for errors, particularly OOM, which could indicate insufficient memory allocation.
Docker Health Checks
In Docker environments, check container uptimes and health indicators to ensure stability and sufficient resource allocation.
Benefits:
Walkthrough of Maintenance Tasks (semiannually)
Security Checks
25
SSH Key Audit
Review and update authorized_keys to ensure only current and authorized personnel have access.
Network Services Check
Inspect network services to confirm only expected ports are open to the internet.
Postfix Configuration
Ensure proper configuration of postfix to permit only local network communications.
Security Scans Verification
Confirm the regular operation of security scans by checking for recent activity logs.
Benefits:
Walkthrough of Maintenance Tasks (semiannually)
Disk Space Cleanup
26
Disk Space Review
Regularly check for available disk space to manage storage resources effectively.
Managing Large Files
Identify and safely remove large, obsolete files, especially old backups, to optimize disk usage.
Docker Image Maintenance
Set up automated processes for pruning old Docker images to reclaim space.
Benefits:
Walkthrough of Maintenance Tasks (semiannually)
Test restore from backup and update�staging
27
Backup Restoration Preconditions
Confirm the presence of a staging environment with a matching MediaWiki version.
Initial Checks
Verify no uncommitted changes are present on staging and check for adequate storage space before restoration.
Database and Images Restoration
Ensure accurate restoration by syncing the database and images from production, and verify the integrity post-restore.
Restoration Process
Utilize Restic to restore the latest backup and copy production images to the staging environment for a cohesive test.
Benefits:
Walkthrough of Maintenance Tasks (semiannually)
General MediaWiki Maintenance
Operating System Environment
MediaWiki Stack
Wiki Content
28
Maintenance Scripts Execution
Run necessary maintenance scripts periodically and review their outputs for any anomalies or issues.
Reviewing Recent Changes
Utilize both automated tools and manual oversight to efficiently monitor changes across the wiki. Identify and revert vandalism.
Regular Monitoring
Keep an eye on MediaWiki releases and updates for minor version changes to ensure timely updates.
System Updates
Verify server software is up-to-date with the latest security patches and updates.
The Easiest Maintenance Plan
Leave it to WikiTeq
A partner that cares for your wiki as much as you do.
Reliable
Professional
The simplest
29
Gain peace of mind knowing your wiki's maintenance is in the hands of professionals.
Conclusion and Takeaways
30
Proactive Approach
Highlighted the need for a proactive maintenance strategy to prevent potential issues.
Work Smarter, Not Harder
Have the experts handle it for you!
Maintenance as a Priority
Stressed the importance of regular maintenance for the stability and security of MediaWiki.
Questions?
Necessity • Timeline • Learning • Benefits
31
Thank You!
Ike Hecht + WikiTeq
32