|
2023-08-31
§
|
| 05:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1173 to s6 primary and set section read-write T345223', diff saved to https://phabricator.wikimedia.org/P52125 and previous config saved to /var/cache/conftool/dbconfig/20230831-052852-marostegui.json |
[production] |
| 05:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s6 eqiad as read-only for maintenance - T345223', diff saved to https://phabricator.wikimedia.org/P52124 and previous config saved to /var/cache/conftool/dbconfig/20230831-052825-marostegui.json |
[production] |
| 05:28 |
<marostegui> |
Starting s6 eqiad failover from db1131 to db1173 - T345223 |
[production] |
| 05:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 50%: Repooling after maintenance ', diff saved to https://phabricator.wikimedia.org/P52123 and previous config saved to /var/cache/conftool/dbconfig/20230831-051755-root.json |
[production] |
| 05:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 25%: Repooling after maintenance ', diff saved to https://phabricator.wikimedia.org/P52122 and previous config saved to /var/cache/conftool/dbconfig/20230831-050250-root.json |
[production] |
| 04:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1173 with weight 0 T345223', diff saved to https://phabricator.wikimedia.org/P52121 and previous config saved to /var/cache/conftool/dbconfig/20230831-045719-marostegui.json |
[production] |
| 04:57 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s6 T345223 |
[production] |
| 04:56 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s6 T345223 |
[production] |
| 04:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 10%: Repooling after maintenance ', diff saved to https://phabricator.wikimedia.org/P52120 and previous config saved to /var/cache/conftool/dbconfig/20230831-044746-root.json |
[production] |
| 02:45 |
<jhancock@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" |
[production] |
| 02:31 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2037.codfw.wmnet with reason: host reimage |
[production] |
| 02:26 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2037.codfw.wmnet with reason: host reimage |
[production] |
| 01:50 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host kubernetes2037.codfw.wmnet with OS bullseye |
[production] |
| 01:50 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host kubernetes2038.codfw.wmnet with OS bullseye |
[production] |
| 01:50 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host kubernetes2039.codfw.wmnet with OS bullseye |
[production] |
| 01:44 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host moss-be2003.codfw.wmnet with OS bullseye |
[production] |
| 01:43 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye |
[production] |
| 01:43 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host moss-be2003.codfw.wmnet with OS bullseye |
[production] |
| 01:43 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye |
[production] |
| 01:42 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['moss-be2003'] |
[production] |
| 01:37 |
<jhancock@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['moss-be2003'] |
[production] |
| 00:54 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye |
[production] |
| 00:50 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['moss-be2003'] |
[production] |
| 00:43 |
<jhancock@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['moss-be2003'] |
[production] |
|
2023-08-30
§
|
| 22:28 |
<krinkle@deploy1002> |
Synchronized php-1.41.0-wmf.24/extensions/WikimediaEvents/: 697ab03ae9a5d5ddb6 (duration: 06m 26s) |
[production] |
| 22:09 |
<krinkle@deploy1002> |
Finished scap: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] (duration: 35m 08s) |
[production] |
| 21:57 |
<krinkle@deploy1002> |
krinkle: Continuing with sync |
[production] |
| 21:55 |
<krinkle@deploy1002> |
krinkle: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 21:34 |
<krinkle@deploy1002> |
Started scap: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] |
[production] |
| 21:31 |
<Krinkle> |
krinkle@deploy1002: running `sudo /usr/local/sbin/fix-staging-perms` two fix permissions under /srv/patches/1.41.0-wmf.24 where 2 of the 3 patch files are read-only by jnuche:deployment |
[production] |
| 20:44 |
<hmonroy@deploy1002> |
Finished scap: Backport for [[gerrit:952925|Add comment about mirroring of wgMobileUrlTemplate (T344185)]] (duration: 07m 11s) |
[production] |
| 20:37 |
<hmonroy@deploy1002> |
Started scap: Backport for [[gerrit:952925|Add comment about mirroring of wgMobileUrlTemplate (T344185)]] |
[production] |
| 20:33 |
<hmonroy@deploy1002> |
Finished scap: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] (duration: 08m 18s) |
[production] |
| 20:27 |
<hmonroy@deploy1002> |
matmarex and hmonroy: Continuing with sync |
[production] |
| 20:26 |
<hmonroy@deploy1002> |
matmarex and hmonroy: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 20:24 |
<hmonroy@deploy1002> |
Started scap: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] |
[production] |
| 20:14 |
<hmonroy@deploy1002> |
Finished scap: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] (duration: 09m 13s) |
[production] |
| 20:08 |
<hmonroy@deploy1002> |
hmonroy: Continuing with sync |
[production] |
| 20:06 |
<hmonroy@deploy1002> |
hmonroy: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 20:05 |
<hmonroy@deploy1002> |
Started scap: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] |
[production] |
| 19:19 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply security updates - bking@cumin1001 - T344587 |
[production] |
| 19:17 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts wdqs1005.eqiad.wmnet |
[production] |
| 18:41 |
<cmooney@cumin1001> |
END (FAIL) - Cookbook sre.network.provision (exit_code=99) for device ssw1-a1-codfw.mgmt.codfw.wmnet |
[production] |
| 18:41 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:41 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove management record for ssw1-a1-codfw - cmooney@cumin1001" |
[production] |
| 18:39 |
<cmooney@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove management record for ssw1-a1-codfw - cmooney@cumin1001" |
[production] |
| 18:37 |
<cmooney@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:28 |
<ryankemper@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts wdqs1005.eqiad.wmnet |
[production] |
| 18:19 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host doh2001.wikimedia.org with OS bookworm |
[production] |
| 18:15 |
<jhuneidi@deploy1002> |
Synchronized php: group1 wikis to 1.41.0-wmf.24 refs T343726 (duration: 06m 13s) |
[production] |