|
2024-06-25
§
|
| 16:19 |
<eevans@cumin1002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 |
[production] |
| 16:18 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es1035 (re)pooling @ 75%: post T365986 repool', diff saved to https://phabricator.wikimedia.org/P65418 and previous config saved to /var/cache/conftool/dbconfig/20240625-161824-arnaudb.json |
[production] |
| 16:15 |
<claime> |
Extending vg-srv on mw1437 |
[production] |
| 16:10 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@72ad841]: deploy phab1004 for T368392 - followup T364728 (duration: 00m 39s) |
[production] |
| 16:10 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@72ad841]: deploy phab1004 for T368392 - followup T364728 |
[production] |
| 16:09 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@72ad841]: deploy phab2002 for T368392 - followup T364728 (duration: 00m 33s) |
[production] |
| 16:08 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@72ad841]: deploy phab2002 for T368392 - followup T364728 |
[production] |
| 16:05 |
<brennen> |
silencing phabricator hosts prior to deploy |
[production] |
| 16:03 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es1035 (re)pooling @ 50%: post T365986 repool', diff saved to https://phabricator.wikimedia.org/P65417 and previous config saved to /var/cache/conftool/dbconfig/20240625-160318-arnaudb.json |
[production] |
| 15:33 |
<eevans@cumin1002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 |
[production] |
| 15:33 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs[1011-1021].eqiad.wmnet: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es1035 (re)pooling @ 10%: post T365986 repool', diff saved to https://phabricator.wikimedia.org/P65415 and previous config saved to /var/cache/conftool/dbconfig/20240625-153307-arnaudb.json |
[production] |
| 15:31 |
<Dreamy_Jazz> |
Ran `mwscript extensions/CheckUser/maintenance/deleteReadOldRowsInCuChanges.php --wiki=testwiki` for T366781 |
[production] |
| 15:22 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 15:21 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 15:21 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 15:20 |
<claime> |
Deploying statsd to mw-api-ext - T365265 |
[production] |
| 15:19 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 15:18 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es1035 (re)pooling @ 5%: post T365986 repool', diff saved to https://phabricator.wikimedia.org/P65414 and previous config saved to /var/cache/conftool/dbconfig/20240625-151802-arnaudb.json |
[production] |
| 15:06 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@f58dd50]: deploy phab1004 for T368392 (duration: 00m 50s) |
[production] |
| 15:05 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@f58dd50]: deploy phab1004 for T368392 |
[production] |
| 15:05 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@f58dd50]: deploy phab2002 for T368392 (duration: 00m 33s) |
[production] |
| 15:04 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@f58dd50]: deploy phab2002 for T368392 |
[production] |
| 15:03 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:03 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:03 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:02 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:00 |
<topranks> |
rebooting lsw1-e5-eqiad to upgrade JunOS on switch T365986 |
[production] |
| 14:58 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on 7 hosts with reason: JunOS upgrade lsw1-e5-eqiad |
[production] |
| 14:58 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:40:00 on 7 hosts with reason: JunOS upgrade lsw1-e5-eqiad |
[production] |
| 14:57 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on lsw1-e5-eqiad,lsw1-e5-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt with reason: JunOS upgrade lsw1-e5-eqiad |
[production] |
| 14:57 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:40:00 on lsw1-e5-eqiad,lsw1-e5-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt with reason: JunOS upgrade lsw1-e5-eqiad |
[production] |
| 14:56 |
<cdanis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-web: apply |
[production] |
| 14:56 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:45:00 on es1035.eqiad.wmnet with reason: T365986 |
[production] |
| 14:56 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:45:00 on es1035.eqiad.wmnet with reason: T365986 |
[production] |
| 14:55 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'T365986 - depool es1035', diff saved to https://phabricator.wikimedia.org/P65413 and previous config saved to /var/cache/conftool/dbconfig/20240625-145558-arnaudb.json |
[production] |
| 14:55 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:50:00 on lsw1-e5-eqiad.mgmt with reason: prep JunOS upgrade lsw1-e5-eqiad |
[production] |
| 14:55 |
<cdanis@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-web: apply |
[production] |
| 14:55 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:50:00 on lsw1-e5-eqiad.mgmt with reason: prep JunOS upgrade lsw1-e5-eqiad |
[production] |
| 14:50 |
<cdanis@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-web: apply |
[production] |
| 14:49 |
<cdanis@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-web: apply |
[production] |
| 14:45 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:1049538|WikiPageWriter: Do not run AbuseFilter when UltimateAuthority is used (T368275)]] (duration: 11m 45s) |
[production] |
| 14:40 |
<urbanecm@deploy1002> |
urbanecm: Continuing with sync |
[production] |
| 14:40 |
<urbanecm@deploy1002> |
urbanecm: Backport for [[gerrit:1049538|WikiPageWriter: Do not run AbuseFilter when UltimateAuthority is used (T368275)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:36 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 14:36 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dbproxy2005 to codfw - jhancock@cumin2002" |
[production] |
| 14:35 |
<sukhe> |
sudo cumin -b1 -s900 "A:dnsbox" "run-puppet-agent --enable 'rolling out CR 1049165' && systemctl restart ntp.service" |
[production] |
| 14:35 |
<jhancock@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dbproxy2005 to codfw - jhancock@cumin2002" |
[production] |
| 14:33 |
<vgutierrez> |
rolling upgrade of fifo-log-demux on A:cp-eqiad - T364383 |
[production] |
| 14:33 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:1049538|WikiPageWriter: Do not run AbuseFilter when UltimateAuthority is used (T368275)]] |
[production] |