Wikitech-l November 2024

35 participants
40 discussions

Maintenance scripts are moving to Kubernetes
by Reuven Lazarus 06 May '25

06 May '25

Hi all, With MediaWiki at the WMF moving to Kubernetes, it's now time to start running manual maintenance scripts there. Any time you would previously SSH to a mwmaint host and run mwscript, follow these steps instead. The old way will continue working for a little while, but it will be going away. What's familiar: Starting a maintenance script looks like this: rzl@deploy2002:~$ mwscript-k8s --comment="T341553" -- Version.php --wiki=enwiki Any options for the mwscript-k8s tool, as described below, go before the --. After the --, the first argument is the script name; everything else is passed to the script. This is the same as you're used to passing to mwscript. What's different: - Run mwscript-k8s on a deployment host, not the maintenance host. Either deployment host will work; your job will automatically run in whichever data center is active, so you no longer need to change hosts when there’s a switchover. - You don't need a tmux. By default the tool launches your maintenance script and exits immediately, without waiting for your job to finish. If you log out of the deployment host, your job keeps running on the Kubernetes cluster. - Kubernetes saves the maintenance script's output for seven days after completion. By default, mwscript-k8s prints a kubectl command that you (or anyone else) can paste and run to monitor the output or save it to a file. - As a convenience, you can pass -f (--follow) to mwscript-k8s to immediately begin tailing the script output. If you like, you can do this inside a tmux and keep the same workflow as before. Either way, you can safely disconnect and your script will continue running on Kubernetes. rzl@deploy2002:~$ mwscript-k8s -f -- Version.php --wiki=testwiki [...] MediaWiki version: 1.43.0-wmf.24 LTS (built: 22:35, 23 September 2024) - For scripts that take input on stdin, you can pass --attach to mwscript-k8s, either interactively or in a pipeline. rzl@deploy2002:~$ mwscript-k8s --attach -- shell.php --wiki=testwiki [...] Psy Shell v0.12.3 (PHP 7.4.33 — cli) by Justin Hileman > $wmgRealm = "production" > rzl@deploy2002:~$ cat example_url.txt | mwscript-k8s --attach -- purgeList.php [...] Purging 1 urls Done! - Your maintenance script runs in a Docker container which will not outlive it, so it can't save persistent files to disk. Ensure your script logs its important output to stdout, or persists it in a database or other remote storage. - The --comment flag sets an optional (but encouraged) descriptive label, such as a task number. - Using standard kubectl commands[1][2], you can check the status, and view the output, of your running jobs or anyone else's. (Example: `kube_env mw-script codfw; kubectl get pod -l username=rzl`) [1]: https://wikitech.wikimedia.org/wiki/Kubernetes/Kubectl [2]: https://kubernetes.io/docs/reference/kubectl/quick-reference/ What's not supported yet: - Maintenance scripts launched automatically on a timer. We're working on migrating them -- for now, this is for one-off scripts launched by hand. - If your job is interrupted (e.g. by hardware problems), Kubernetes can automatically move it to another machine and restart it, babysitting it until it completes. But we only want to do that if your job is safe to restart. So by default, if your job is interrupted, it will stay stopped until you restart it yourself. Soon, we'll add an option to declare "this is idempotent, please restart it as needed" and that design is recommended for new scripts. - No support yet for mwscriptwikiset, foreachwiki, foreachwikiindblist, etc, but we'll add similar functionality as flags to mwscript_k8s. Your feedback: Let me know by email or IRC, or on Phab (T341553 <https://phabricator.wikimedia.org/T341553>). If mwscript-k8s doesn't work for you, for now you can fall back to using the mwmaint hosts as before -- but they will be going away. Please report any problems sooner rather than later, so that we can ensure the new system meets your needs before that happens. Thanks, Reuven, for Service Ops SRE

7 19

Database dumps
by Byrial Jensen 17 Apr '25

17 Apr '25

Until some weeks ago http://dumps.wikimedia.org/backup-index.html used to show 4 dumps in progress at the same time. That meant that new database dumps normally was available within about 3 weeks for all databases except for enwiki and maybe dewiki where the dump process due to size took longer time. However the 4 dumps processes at one time become 3 some weeks ago. And after massive failures at June 4, only one dump has been in progress at the same time. So at the current speed it will take several months to come thru all dumps. Is it possible to speed up the process again using several dump processes at the same time? Thank you, Byrial

3 2

User-Agent:
by Domas Mituzas 17 Apr '25

17 Apr '25

Hi! from now on specific per-bot/per-software/per-client User-Agent header is mandatory for contacting Wikimedia sites. Domas

19 61

API issue
by ovskmendov＠proton.me 17 Apr '25

17 Apr '25

Whenever I access the API, I get "Wikimedia Error: Our servers are currently under maintenance or experiencing a technical problem." Sometimes if I wait a bit, I'll get a few good requests, but it will shortly go back to the previous message. My code was working just fine a few days ago and I didn't change anything. Sent with [Proton Mail](https://proton.me/) secure email.

7 12

Acquiring list of templates including external links
by Takashi OTA 17 Apr '25

17 Apr '25

Hoi, This is an inquiry from my friend in academia, researching about Wikipedia. He would like to know whether there's a way to acquire a list of templates including external links. Here are some examples including external links. https://ja.wikipedia.org/wiki/Template:JOI/doc https://ja.wikipedia.org/wiki/Template:Twitter/doc Such links are stored in externallinks.sql.gz, in an expanded form. When you want to check increase/decrease of linked domains in chronological order through edit history, you have to check pages-meta-history1.xml etc. In a such case, traditional links and links by templates are mixed, Therefore, the latter ones (links by templates) should be expanded to traditional link forms. Sorry if what I am saying does not make sense. Thanks in advance, --Takashi Ota [[U:Takot]]

13 24

2025 Wikimedia Hackathon - registration is now open
by Deb Tankersley 01 Apr '25

01 Apr '25

*Registration for the 2025 Wikimedia Hackathon is now open! * *(with apologies for cross-posting)* The annual Wikimedia Hackathon <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2025> is organized by the Wikimedia Foundation <https://wikimediafoundation.org/> in collaboration with Wikimedia User Group Turkey (WMTR) <https://meta.wikimedia.org/wiki/Wikimedia_Community_User_Group_Turkey> and will be taking place in Istanbul, Turkey, from May 2nd through May 4th, 2025 at the Renaissance Polat Istanbul Hotel <https://www.marriott.com/en-us/hotels/istrn-renaissance-polat-istanbul-hote…> . The Hackathon is an on-site, in-person event. Those people who cannot attend the Hackathon in-person will be able to follow some of the program content online, but should not register using the link provided below. More information will be provided closer to the event on how to participate remotely. Registration will be open until mid April 2025, or until we reach event capacity. The Hackathon event is free for all* to attend *and each person that attends the Hackathon is required to register. - Catering (lunch, dinner, snacks, and drinks) will be provided during the event. - Covid-related equipment will also be available upon request (e.g. self test kits and masks). - We’ll have discounted room codes for those that want to stay at the hotel venue, to be announced in the coming weeks. - Check if your passport is up to date and if you will need a visa (or an e-visa) to travel to Turkey, citizens of most countries can apply for an e-visa online <https://www.mfa.gov.tr/visa-information-for-foreigners.en.mfa>. Participants are expected to book their own travel arrangements and accommodation. However, the Wikimedia Foundation is offering to take care of travel and accommodation for a limited number of scholars and you will be able to apply for a scholarship during the registration process. Scholarship applicants - please note: - *Scholarship application and Wikimedia Foundation staff sponsorship will only be available to request until December 10th, 2024. * - Priority will be given to scholars who can mentor or support in areas of high demand and staff with ambitious or impactful projects. - Those that have requested a scholarship or sponsorship will be notified by mid to late December 2024 about the status of their application. - All scholarships provided will adhere to the WMF travel policy <https://foundation.wikimedia.org/wiki/Policy:Scholarship_travel_policy> guidelines. *Scholarship to this event will cover:* - Roundtrip airfare between your city of residence and Istanbul, Turkey - arriving on May 1, 2025 and departing on May 5, 2025. - Private accommodation at the conference hotel for up to 4 nights - Medical Insurance Coverage for any scholars traveling outside of their home country. (Scholars local to Turkey will need to use their regular insurance coverage.). - Airport transfers between the Istanbul Airport (IST) and the conference hotel. - All meals will be provided for the duration of the event (lunch, drinks, snacks, and dinner will be provided on event days; breakfast will be provided by the hotel for guests staying there). In addition, dinner will be provided on Thursday before the Hackathon officially kicks off. - Fees associated with obtaining a visa to travel to Turkey (visa application fees, and travel to in-person travel visa appointment (if needed)). - Citizens of most countries can apply for an e-visa online <https://www.mfa.gov.tr/visa-information-for-foreigners.en.mfa>. *Scholarship to this event will not cover:* - Per diem costs on travel days. - Ground transportation costs between your residence and home airport. - Insurance for your personal possessions. The Hackathon registration form runs on Pretix (a free, open-sourced third-party platform that helps organize and manage events) which may subject your data to additional terms <https://pretix.eu/about/en/privacy>. For more information on WMF’s privacy and data-handling for the Hackathon, please read the privacy statement <https://foundation.wikimedia.org/wiki/Legal:Wikimedia_Hackathon_Privacy_Sta…> . <https://pretix.eu/wikimedia/hackathon2025> If you encounter any issues while registering, or have questions about the registration process, feel free to contact the coordination team at hackathon(a)wikimedia.org. Cheers, Deb -- deb tankersley (she/her) developer outreach program management Wikimedia Foundation

1 1

EBNF grammar project status?
by Steve Bennett 01 Apr '25

01 Apr '25

What's the status of the project to create a grammar for Wikitext in EBNF? There are two pages: http://meta.wikimedia.org/wiki/Wikitext_Metasyntax http://www.mediawiki.org/wiki/Markup_spec Nothing seems to have happened since January this year. Also the comments on the latter page seem to indicate a lack of clear goal: is this just a fun project, is it to improve the existing parser, or is it to facilititate a new parser? It's obviously a lot of work, so it needs to be of clear benefit. Brion requested the grammar IIRC (and there's a comment to that effect at http://bugzilla.wikimedia.org/show_bug.cgi?id=7 ), so I'm wondering what became of it. Is there still a goal of replacing the parser? Or is there some alternative plan? Steve

26 217

User style to reduce Gerritbot comments on Phabricator
by Bartosz Dziewoński 25 Mar '25

25 Mar '25

Do your eyes glaze over when trying to make sense of Phabricator tasks with a long and troubled history of patches? Do you struggle to find the real people in the sea of automated bot comments? I've made myself a user style to compact the Gerritbot comments on Phabricator. You might like it too: https://userstyles.world/style/13140/wikimedia-phabricator-reduce-gerritbot… (See screenshots and installation instructions there.) -- Bartosz Dziewoński

8 12

Phabricator upstream shutdown
by Brian Wolff 21 Mar '25

21 Mar '25

It sounds like phabricator upstream is going away: https://admin.phacility.com/phame/post/view/11/phacility_is_winding_down_op… Just curious, are we planning to continue using it long term or move to something else? -- Brian

10 12

Content Credentials
by Bernhard Krabina, KM-A 28 Feb '25

28 Feb '25

Hi, are there any plans or initiatives to support Content Credentials in MediaWiki? https://contentcredentials.org/ https://opensource.contentauthenticity.org/docs/getting-started/ Have just learned about this myself, so I don't have any more background information. Best, Bernhard

4 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l November 2024