1351-1400 of 10000 results (29ms)
2024-11-12 §
09:59 <arnaudb@cumin1002> START - Cookbook sre.mysql.pool db2217 gradually with 4 steps - T379491 [production]
08:02 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
08:02 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
05:01 <mwpresync@deploy2002> Pruned MediaWiki: 1.43.0-wmf.28 (duration: 01m 52s) [production]
2024-11-07 §
19:23 <cdanis> T379199 💙cdanis@mwmaint2002.codfw.wmnet ~ 🕝☕ mwscript sql.php --wiki=testcommonswiki /srv/mediawiki/php-1.44.0-wmf.2/extensions/JsonConfig/sql/mysql/tables-generated.sql [production]
18:14 <swfrench-wmf> updated changeprop-jobqueue to 2024-11-05-170900-production - T356241 [production]
09:14 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.2 refs T375661 [production]
08:05 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
08:05 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
2024-11-06 §
16:10 <arnaudb@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2136 gradually with 4 steps - cloned on db2236 [production]
15:55 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:25:00 on cr[3-4]-ulsfo with reason: prevent bgp alerts firing while lvs4010 is rebooted [production]
15:55 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 0:25:00 on cr[3-4]-ulsfo with reason: prevent bgp alerts firing while lvs4010 is rebooted [production]
15:24 <arnaudb@cumin1002> START - Cookbook sre.mysql.pool db2136 gradually with 4 steps - cloned on db2236 [production]
10:41 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.2 refs T375661 [production]
00:44 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es1021 gradually with 4 steps - Maint over [production]
2024-11-05 §
23:59 <ladsgroup@cumin1002> START - Cookbook sre.mysql.pool es1021 gradually with 4 steps - Maint over [production]
14:48 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.2 refs T375661 [production]
12:10 <jnuche@deploy2002> Finished scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 (duration: 07m 43s) [production]
12:02 <jnuche@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 [production]
11:53 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.2 refs T375661 [production]
11:38 <jnuche@deploy2002> Finished scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 (duration: 36m 28s) [production]
11:02 <jnuche@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 [production]
09:22 <jnuche@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 [production]
05:10 <mwpresync@deploy2002> Pruned MediaWiki: 1.43.0-wmf.27 (duration: 10m 37s) [production]
04:03 <mwpresync@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 [production]
2024-11-04 §
20:26 <swfrench-wmf> zero-replica "migration" releases created for all shellbox instances - T375243 [production]
2024-11-02 §
15:13 <reedy@deploy2002> Synchronized wmf-config/: Comment updates (duration: 07m 31s) [production]
2024-11-01 §
14:06 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2190 gradually with 4 steps - Maint over [production]
13:20 <ladsgroup@cumin1002> START - Cookbook sre.mysql.pool db2190 gradually with 4 steps - Maint over [production]
2024-10-31 §
19:15 <dduvall@deploy2002> rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.1 refs T375660 [production]
16:04 <dancy@deploy2002> scap failed: <CalledProcessError> Command '['sudo', '-u', 'mwbuilder', '-n', '--', '/home/dancy/src/venvs/scap/bin/scap', 'mwshell', '--no-local-config', '--directory', '/srv/mediawiki-staging', '--user', 'www-data', '--', 'rm -f /srv/mediawiki-staging/php-1.43.0-wmf.28/cache/l10n/*.tmp.*']' returned non-zero exit status 1. (scap version: 4.118.0) (duration: 00m 01s) [production]
14:37 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db2190.codfw.wmnet with reason: host has hardware issues T378628 [production]
14:37 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db2190.codfw.wmnet with reason: host has hardware issues T378628 [production]
14:35 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1234 (re)pooling @ 4%: post T378267 reclone', diff saved to https://phabricator.wikimedia.org/P70760 and previous config saved to /var/cache/conftool/dbconfig/20241031-143510-arnaudb.json [production]
10:52 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[1232,1234].eqiad.wmnet with reason: hosts in cloning, avoiding alerts [production]
10:52 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db[1232,1234].eqiad.wmnet with reason: hosts in cloning, avoiding alerts [production]
2024-10-30 §
19:40 <swfrench-wmf> all shellbox instances updated to shellbox 2024-10-15-214239 - T375243 [production]
19:17 <dduvall@deploy2002> rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.1 refs T375660 [production]
14:28 <dreamyjazz@deploy2002> Finished scap sync-world: Backport for [[gerrit:1084761|[BlockManager] Don't assume autoblocks have ::getParentBlockId (T378563)]], [[gerrit:1084152|[GlobalBlocking] Enable global autoblocks on all WMF wikis (T377760)]] (duration: 09m 10s) [production]
14:21 <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:1084761|[BlockManager] Don't assume autoblocks have ::getParentBlockId (T378563)]], [[gerrit:1084152|[GlobalBlocking] Enable global autoblocks on all WMF wikis (T377760)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:19 <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1084761|[BlockManager] Don't assume autoblocks have ::getParentBlockId (T378563)]], [[gerrit:1084152|[GlobalBlocking] Enable global autoblocks on all WMF wikis (T377760)]] [production]
09:11 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2219.codfw.wmnet with reason: Maintenance [production]
09:11 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2219.codfw.wmnet with reason: Maintenance [production]
07:29 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2223 (re)pooling @ 4%: post clone repool', diff saved to https://phabricator.wikimedia.org/P70676 and previous config saved to /var/cache/conftool/dbconfig/20241030-072930-arnaudb.json [production]
07:24 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
07:24 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
2024-10-29 §
18:37 <swfrench-wmf> shellbox-syntaxhighlight updated to shellbox 2024-10-15-214239 - T375243 [production]
18:10 <dduvall@deploy2002> rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.1 refs T375660 [production]
14:40 <reedy@deploy2002> Finished scap sync-world: 1.44.0-wmf.1 backports to fix deprecated logspam T375660 T377521 (duration: 07m 21s) [production]
14:32 <reedy@deploy2002> Started scap sync-world: 1.44.0-wmf.1 backports to fix deprecated logspam T375660 T377521 [production]