|
2024-06-18
ยง
|
| 08:51 |
<arnaudb@cumin1002> |
END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 7 days, 0:00:00 on db1165.eqiad.wmnet with reason: repl issues |
[production] |
| 08:51 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1165.eqiad.wmnet with reason: repl issues |
[production] |
| 08:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1160 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P65140 and previous config saved to /var/cache/conftool/dbconfig/20240618-085057-root.json |
[production] |
| 08:45 |
<hashar@deploy1002> |
Finished deploy [integration/docroot@7a92240]: doc: Add mwseaql Rust crate (duration: 00m 07s) |
[production] |
| 08:45 |
<hashar@deploy1002> |
Started deploy [integration/docroot@7a92240]: doc: Add mwseaql Rust crate |
[production] |
| 08:43 |
<fabfur> |
cp4037 currently depooled and puppet disabled for T367756 |
[production] |
| 08:41 |
<fabfur@cumin1002> |
conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 08:40 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-eqiad |
[production] |
| 08:34 |
<marostegui> |
dbmaint eqiad s6 deploy schema change on eqiad master T364069 |
[production] |
| 08:29 |
<XioNoX> |
deploy pfw policy update 1718644831 - T367796 |
[production] |
| 07:56 |
<moritzm> |
uploaded python-irc 8.5.3+dfsg-4+wmf1 to apt.wikimedia.org T331702 |
[production] |
| 07:40 |
<marostegui> |
dbmaint codfw s7 deploy schema change on codfw master T364069 |
[production] |
| 07:33 |
<jiji@cumin1002> |
START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-eqiad |
[production] |
| 07:31 |
<kart_> |
Updated cxserver to 2024-06-13-045621-production (T364122, T138401) |
[production] |
| 07:30 |
<kartik@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 07:29 |
<kartik@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 07:28 |
<kartik@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 07:28 |
<kartik@deploy1002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 07:26 |
<kartik@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 07:26 |
<kartik@deploy1002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 07:20 |
<kartik@deploy1002> |
Finished scap: Backport for [[gerrit:1046810|Content Translation: Adjust the Machine translation limit for Telugu WP from 70% to 75% (T367838)]] (duration: 16m 36s) |
[production] |
| 07:15 |
<marostegui> |
dbmaint eqiad s5 deploy schema change on primary master T364069 |
[production] |
| 07:12 |
<marostegui> |
dbmaint codfw s4 deploy schema change T367261 |
[production] |
| 07:12 |
<marostegui> |
dbmaint codfw s4 deploy schema change |
[production] |
| 07:11 |
<kartik@deploy1002> |
kartik: Continuing with sync |
[production] |
| 07:09 |
<kartik@deploy1002> |
kartik: Backport for [[gerrit:1046810|Content Translation: Adjust the Machine translation limit for Telugu WP from 70% to 75% (T367838)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 07:04 |
<kartik@deploy1002> |
Started scap: Backport for [[gerrit:1046810|Content Translation: Adjust the Machine translation limit for Telugu WP from 70% to 75% (T367838)]] |
[production] |
| 06:52 |
<jynus@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1240.eqiad.wmnet with reason: data reload |
[production] |
| 06:52 |
<jynus@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1240.eqiad.wmnet with reason: data reload |
[production] |
| 06:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1191 (T364069)', diff saved to https://phabricator.wikimedia.org/P65139 and previous config saved to /var/cache/conftool/dbconfig/20240618-060100-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:00 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65138 and previous config saved to /var/cache/conftool/dbconfig/20240618-060038-marostegui.json |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2102.codfw.wmnet |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2102.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002" |
[production] |
| 05:53 |
<jynus@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2102.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002" |
[production] |
| 05:50 |
<jynus@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 05:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P65137 and previous config saved to /var/cache/conftool/dbconfig/20240618-054531-marostegui.json |
[production] |
| 05:44 |
<jynus@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts db2102.codfw.wmnet |
[production] |
| 05:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P65136 and previous config saved to /var/cache/conftool/dbconfig/20240618-053024-marostegui.json |
[production] |
| 05:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65135 and previous config saved to /var/cache/conftool/dbconfig/20240618-051517-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
dbmaint codfw s5 deploy schema change on db2213 T364299 |
[production] |
| 04:57 |
<marostegui> |
dbmaint eqiad s2 deploy schema change on db2207 T364299 |
[production] |
| 04:54 |
<marostegui> |
dbmaint eqiad s4 deploy schema change on db1160 T364299 |
[production] |
| 04:51 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Long schema change |
[production] |
| 04:51 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Long schema change |
[production] |
| 04:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1160 T367378', diff saved to https://phabricator.wikimedia.org/P65134 and previous config saved to /var/cache/conftool/dbconfig/20240618-044908-root.json |
[production] |
| 04:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db1238 to s4 primary and set section read-write T367378', diff saved to https://phabricator.wikimedia.org/P65133 and previous config saved to /var/cache/conftool/dbconfig/20240618-044806-marostegui.json |
[production] |
| 04:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - T367378', diff saved to https://phabricator.wikimedia.org/P65132 and previous config saved to /var/cache/conftool/dbconfig/20240618-044747-marostegui.json |
[production] |