Hello, in the next coming months, these changes will happen in databases and the infrastructure. And it might affect you if you rely on them in your tools or queries. This list is ordered based on how soon the change will happen.
We understand that updating your tools and systems can be time consuming, hence we are giving an advanced notice. I truly apologize for the inconvenience but many of these changes are needed to keep the site running smoothly. Image table redesign
Around fourteen years after the creation of T28741 https://phabricator.wikimedia.org/T28741, we are implementing the changes described therein. Currently, every current version of an image has a row in the image table and if there are older versions of that file, those rows could be found in the oldimage table. These two tables (image and oldimage) will be dropped in around two months. The replacement will be two main tables: file and filerevision. Every file will have a row in the file table describing the name and the type. Every version of the file (current and old) will have a row in filerevision describing the file-specific information such as its size or the hash of the file, similar to the existing distinction between pages and revisions. Another improvement is that every file and file revision will get a unique auto increment id simplifying many operations and queries. You can check T28741 https://phabricator.wikimedia.org/T28741 for more information. The new tables are already accessible in wikireplicas but the data hasn’t been fully migrated yet.
Term store split out of wikidata’s database
Wikidata’s database has been growing too fast and we need to move the term store (tables starting with wbt_) to a dedicated cluster to allow growth and improve wikidata’s performance by utilizing cache locality. The new section will be called x3 and you will be able to access it in wikireplicas but this also means you won’t be able to join these tables with the rest of wikidata’s database (such as page table) since they will be residing in two physically separate servers that also means most of your queries to wikidata’s database (and term store) will become faster. We are aiming for the switch to happen in three months’ time. You can follow the work in T351820 https://phabricator.wikimedia.org/T351820.
Additionally, wb_type table will be dropped and the mapping will be hard-coded in the code instead. See gerrit:1110810 https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110810 for more details. This helped us simplify a lot of Wikibase code (example https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110720).
Categorylinks normalization
Categorylinks is the next table in the series of links tables being normalized via the linktarget table (parent ticket https://phabricator.wikimedia.org/T300222, RFC https://phabricator.wikimedia.org/T222224). Similar to templatelinks and pagelinks tables, cl_to will be dropped and instead the new field cl_target_id will point to lt_id in the linktarget table. We will also drop the cl_collation field and replace it with cl_collation_id which will point to the collation_id field on the new table we are introducing called collation. We are aiming to get this fully done by the end of the next quarter (end of June 2025) but it depends on how fast the migration script can operate and that’s outside of our control. You can follow the work in T299951 https://phabricator.wikimedia.org/T299951.It’s worth noting that after this migration is done, we will start working on the imagelinks table.
Thank you
Hello, An update regarding term store split. We are slowly moving towards finalizing the split.
Now you can connect to the term store databases via these endpoints: - termstore.wikidatawiki.analytics.db.svc.wikimedia.cloud - termstore.wikidatawiki.web.db.svc.wikimedia.cloud
They currently point to the wikidata's database but after the split, they will only hold term store tables (wbt_* ones). Please migrate your queries to use the term store endpoints otherwise, in a couple of weeks, they will start to drift and eventually error out.
Emphasizing that it'll be no longer possible to join term store tables with other wikidata tables as they will reside in physically separate servers.
You can follow the work in https://phabricator.wikimedia.org/T351820
Thank you and sorry for the inconvenience
Am Do., 30. Jan. 2025 um 14:47 Uhr schrieb Amir Sarabadani < [email protected]>:
Hello, in the next coming months, these changes will happen in databases and the infrastructure. And it might affect you if you rely on them in your tools or queries. This list is ordered based on how soon the change will happen.
We understand that updating your tools and systems can be time consuming, hence we are giving an advanced notice. I truly apologize for the inconvenience but many of these changes are needed to keep the site running smoothly. Image table redesign
Around fourteen years after the creation of T28741 https://phabricator.wikimedia.org/T28741, we are implementing the changes described therein. Currently, every current version of an image has a row in the image table and if there are older versions of that file, those rows could be found in the oldimage table. These two tables (image and oldimage) will be dropped in around two months. The replacement will be two main tables: file and filerevision. Every file will have a row in the file table describing the name and the type. Every version of the file (current and old) will have a row in filerevision describing the file-specific information such as its size or the hash of the file, similar to the existing distinction between pages and revisions. Another improvement is that every file and file revision will get a unique auto increment id simplifying many operations and queries. You can check T28741 https://phabricator.wikimedia.org/T28741 for more information. The new tables are already accessible in wikireplicas but the data hasn’t been fully migrated yet.
Term store split out of wikidata’s database
Wikidata’s database has been growing too fast and we need to move the term store (tables starting with wbt_) to a dedicated cluster to allow growth and improve wikidata’s performance by utilizing cache locality. The new section will be called x3 and you will be able to access it in wikireplicas but this also means you won’t be able to join these tables with the rest of wikidata’s database (such as page table) since they will be residing in two physically separate servers that also means most of your queries to wikidata’s database (and term store) will become faster. We are aiming for the switch to happen in three months’ time. You can follow the work in T351820 https://phabricator.wikimedia.org/T351820.
Additionally, wb_type table will be dropped and the mapping will be hard-coded in the code instead. See gerrit:1110810 https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110810 for more details. This helped us simplify a lot of Wikibase code (example https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110720 ).
Categorylinks normalization
Categorylinks is the next table in the series of links tables being normalized via the linktarget table (parent ticket https://phabricator.wikimedia.org/T300222, RFC https://phabricator.wikimedia.org/T222224). Similar to templatelinks and pagelinks tables, cl_to will be dropped and instead the new field cl_target_id will point to lt_id in the linktarget table. We will also drop the cl_collation field and replace it with cl_collation_id which will point to the collation_id field on the new table we are introducing called collation. We are aiming to get this fully done by the end of the next quarter (end of June 2025) but it depends on how fast the migration script can operate and that’s outside of our control. You can follow the work in T299951 https://phabricator.wikimedia.org/T299951.It’s worth noting that after this migration is done, we will start working on the imagelinks table.
Thank you
*Amir Sarabadani (he/him)* Staff Database Architect Wikimedia Foundation https://wikimediafoundation.org/