2651-2700 of 10000 results (37ms)
2025-02-05 §
18:04 <swfrench-wmf> scaled mw-api-ext and mw-web next releases to 25% of main - T383845 [production]
15:51 <swfrench-wmf> finished deploying conftool 5.0.1-1 - T383324 [production]
11:32 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1155.eqiad.wmnet with reason: Rebuild tables [production]
10:26 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1179.eqiad.wmnet with reason: Maintenance [production]
09:13 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.15 refs T382366 [production]
2025-02-04 §
20:36 <swfrench-wmf> finished running puppet on A:cp-text after merging https://gerrit.wikimedia.org/r/1084247 [production]
20:09 <swfrench-wmf> running puppet on A:cp-text after merging https://gerrit.wikimedia.org/r/1084247 [production]
20:07 <swfrench-wmf> verified behavior of https://gerrit.wikimedia.org/r/1084247 on cp4040 [production]
19:59 <swfrench-wmf> disabled puppet on A:cp-text before merging https://gerrit.wikimedia.org/r/1084247 [production]
18:42 <swfrench-wmf> mw-api-int to ~ 2% of traffic on PHP 8.1 - T383845 [production]
18:05 <swfrench-wmf> scaled mw-api-ext next to 15% of main release - T383845 [production]
18:02 <swfrench-wmf> scaled mw-web next to 15% of main release - T383845 [production]
12:11 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs500[4-5]*} and A:lvs (T373027) [production]
12:10 <vgutierrez@cumin1002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs500[4-5]*} and A:lvs (T373027) [production]
10:44 <Amir1> foreachwiki sql.php /srv/mediawiki/php-1.44.0-wmf.14/sql/mysql/patch-collation.sql (T384592) [production]
09:18 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.15 refs T382366 [production]
08:57 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1169.eqiad.wmnet with reason: Repooling after clone - T383760 [production]
05:04 <mwpresync@deploy2002> Pruned MediaWiki: 1.44.0-wmf.12 (duration: 04m 49s) [production]
05:01 <mwpresync@deploy2002> Finished scap sync-world: testwikis to 1.44.0-wmf.15 refs T382366 (duration: 58m 53s) [production]
04:02 <mwpresync@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.15 refs T382366 [production]
2025-02-03 §
19:40 <swfrench-wmf> ran reprepro include mercurius 1.1.0-1 - T385225 [production]
18:41 <swfrench-wmf> mw-api-int to ~ 1% of traffic on PHP 8.1 in codfw - T383845 [production]
18:28 <swfrench-wmf> mw-api-int to ~ 1% of traffic on PHP 8.1 in eqiad - T383845 [production]
2025-01-30 §
19:12 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.14 refs T382365 [production]
09:16 <hashar@deploy2002> rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.14 refs T382365 [production]
02:45 <swfrench-wmf> scaled down shellbox-video/migration after switch to PHP 8.1 - T377038 [production]
2025-01-29 §
10:16 <root@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2156 gradually with 4 steps - Repooling after rebuild index T384807 [production]
09:31 <root@cumin1002> START - Cookbook sre.mysql.pool db2156 gradually with 4 steps - Repooling after rebuild index T384807 [production]
08:44 <root@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2157 gradually with 4 steps - Repooling after rebuild index T384994 [production]
07:58 <root@cumin1002> START - Cookbook sre.mysql.pool db2157 gradually with 4 steps - Repooling after rebuild index T384994 [production]
07:40 <root@cumin1002> END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) db1230 gradually with 4 steps - Repooling after rebuild index T384994 [production]
07:40 <root@cumin1002> START - Cookbook sre.mysql.pool db1230 gradually with 4 steps - Repooling after rebuild index T384994 [production]
2025-01-28 §
19:20 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.14 refs T382365 [production]
18:18 <swfrench-wmf> starting shellbox-video pilot on PHP 8.1 (3 replicas, codfw only) - T377038 [production]
18:04 <swfrench-wmf> starting shellbox-constraints pilot on PHP 8.1 (1 replica, eqiad only) - T377038 [production]
16:04 <root@cumin1002> END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) db1166 gradually with 4 steps - Repooling after rebuild index T384807 [production]
16:03 <root@cumin1002> START - Cookbook sre.mysql.pool db1166 gradually with 4 steps - Repooling after rebuild index T384807 [production]
13:18 <root@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2190 gradually with 4 steps - Repooling after rebuild index [production]
12:32 <root@cumin1002> START - Cookbook sre.mysql.pool db2190 gradually with 4 steps - Repooling after rebuild index [production]
05:06 <mwpresync@deploy2002> Pruned MediaWiki: 1.44.0-wmf.11 (duration: 06m 25s) [production]
05:03 <mwpresync@deploy2002> Finished scap sync-world: testwikis to 1.44.0-wmf.14 refs T382365 (duration: 59m 38s) [production]
04:03 <mwpresync@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.14 refs T382365 [production]
2025-01-27 §
18:16 <reedy@deploy2002> Synchronized php-1.44.0-wmf.13/extensions/CommonsMetadata/: T384853 T384854 (duration: 10m 45s) [production]
17:04 <marostegui@cumin1002> dbctl commit (dc=all): 'db1157 (re)pooling @ 4%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72511 and previous config saved to /var/cache/conftool/dbconfig/20250127-170448-root.json [production]
10:56 <cmooney@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b[3-4]-magru.mgmt with reason: upgrading JunOS on magru core routers [production]
2025-01-24 §
22:11 <sukhe@puppetserver1001> conftool action : set/pooled=yes; selector: name=cp700[2-4].magru.wmnet,service=(cdn|ats-be) [production]
08:11 <jynus@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1216.eqiad.wmnet with reason: os upgrade [production]
2025-01-23 §
19:33 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.13 refs T382364 [production]
19:15 <brennen> 1.44.0-wmf.13 train (T382364): no current blockers, logs relatively clean, rolling to all wikis. [production]
15:35 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.13 refs T382364 [production]