2551-2600 of 10000 results (86ms)
2023-08-31 §
05:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1201 (re)pooling @ 75%: Repooling after maintenance ', diff saved to https://phabricator.wikimedia.org/P52127 and previous config saved to /var/cache/conftool/dbconfig/20230831-053300-root.json [production]
05:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1131 T345223', diff saved to https://phabricator.wikimedia.org/P52126 and previous config saved to /var/cache/conftool/dbconfig/20230831-053035-root.json [production]
05:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1173 to s6 primary and set section read-write T345223', diff saved to https://phabricator.wikimedia.org/P52125 and previous config saved to /var/cache/conftool/dbconfig/20230831-052852-marostegui.json [production]
05:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s6 eqiad as read-only for maintenance - T345223', diff saved to https://phabricator.wikimedia.org/P52124 and previous config saved to /var/cache/conftool/dbconfig/20230831-052825-marostegui.json [production]
05:28 <marostegui> Starting s6 eqiad failover from db1131 to db1173 - T345223 [production]
05:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1201 (re)pooling @ 50%: Repooling after maintenance ', diff saved to https://phabricator.wikimedia.org/P52123 and previous config saved to /var/cache/conftool/dbconfig/20230831-051755-root.json [production]
05:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1201 (re)pooling @ 25%: Repooling after maintenance ', diff saved to https://phabricator.wikimedia.org/P52122 and previous config saved to /var/cache/conftool/dbconfig/20230831-050250-root.json [production]
04:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1173 with weight 0 T345223', diff saved to https://phabricator.wikimedia.org/P52121 and previous config saved to /var/cache/conftool/dbconfig/20230831-045719-marostegui.json [production]
04:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s6 T345223 [production]
04:56 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s6 T345223 [production]
04:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1201 (re)pooling @ 10%: Repooling after maintenance ', diff saved to https://phabricator.wikimedia.org/P52120 and previous config saved to /var/cache/conftool/dbconfig/20230831-044746-root.json [production]
02:45 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
02:31 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2037.codfw.wmnet with reason: host reimage [production]
02:26 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2037.codfw.wmnet with reason: host reimage [production]
01:50 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host kubernetes2037.codfw.wmnet with OS bullseye [production]
01:50 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host kubernetes2038.codfw.wmnet with OS bullseye [production]
01:50 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host kubernetes2039.codfw.wmnet with OS bullseye [production]
01:44 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host moss-be2003.codfw.wmnet with OS bullseye [production]
01:43 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye [production]
01:43 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host moss-be2003.codfw.wmnet with OS bullseye [production]
01:43 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye [production]
01:42 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['moss-be2003'] [production]
01:37 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['moss-be2003'] [production]
00:54 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye [production]
00:50 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['moss-be2003'] [production]
00:43 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['moss-be2003'] [production]
2023-08-30 §
22:28 <krinkle@deploy1002> Synchronized php-1.41.0-wmf.24/extensions/WikimediaEvents/: 697ab03ae9a5d5ddb6 (duration: 06m 26s) [production]
22:09 <krinkle@deploy1002> Finished scap: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] (duration: 35m 08s) [production]
21:57 <krinkle@deploy1002> krinkle: Continuing with sync [production]
21:55 <krinkle@deploy1002> krinkle: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
21:34 <krinkle@deploy1002> Started scap: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] [production]
21:31 <Krinkle> krinkle@deploy1002: running `sudo /usr/local/sbin/fix-staging-perms` two fix permissions under /srv/patches/1.41.0-wmf.24 where 2 of the 3 patch files are read-only by jnuche:deployment [production]
20:44 <hmonroy@deploy1002> Finished scap: Backport for [[gerrit:952925|Add comment about mirroring of wgMobileUrlTemplate (T344185)]] (duration: 07m 11s) [production]
20:37 <hmonroy@deploy1002> Started scap: Backport for [[gerrit:952925|Add comment about mirroring of wgMobileUrlTemplate (T344185)]] [production]
20:33 <hmonroy@deploy1002> Finished scap: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] (duration: 08m 18s) [production]
20:27 <hmonroy@deploy1002> matmarex and hmonroy: Continuing with sync [production]
20:26 <hmonroy@deploy1002> matmarex and hmonroy: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:24 <hmonroy@deploy1002> Started scap: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] [production]
20:14 <hmonroy@deploy1002> Finished scap: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] (duration: 09m 13s) [production]
20:08 <hmonroy@deploy1002> hmonroy: Continuing with sync [production]
20:06 <hmonroy@deploy1002> hmonroy: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:05 <hmonroy@deploy1002> Started scap: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] [production]
19:19 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply security updates - bking@cumin1001 - T344587 [production]
19:17 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts wdqs1005.eqiad.wmnet [production]
18:41 <cmooney@cumin1001> END (FAIL) - Cookbook sre.network.provision (exit_code=99) for device ssw1-a1-codfw.mgmt.codfw.wmnet [production]
18:41 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:41 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove management record for ssw1-a1-codfw - cmooney@cumin1001" [production]
18:39 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove management record for ssw1-a1-codfw - cmooney@cumin1001" [production]
18:37 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
18:28 <ryankemper@cumin1001> START - Cookbook sre.hosts.decommission for hosts wdqs1005.eqiad.wmnet [production]