|
2024-06-18
ยง
|
| 07:26 |
<kartik@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 07:26 |
<kartik@deploy1002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 07:20 |
<kartik@deploy1002> |
Finished scap: Backport for [[gerrit:1046810|Content Translation: Adjust the Machine translation limit for Telugu WP from 70% to 75% (T367838)]] (duration: 16m 36s) |
[production] |
| 07:15 |
<marostegui> |
dbmaint eqiad s5 deploy schema change on primary master T364069 |
[production] |
| 07:12 |
<marostegui> |
dbmaint codfw s4 deploy schema change T367261 |
[production] |
| 07:12 |
<marostegui> |
dbmaint codfw s4 deploy schema change |
[production] |
| 07:11 |
<kartik@deploy1002> |
kartik: Continuing with sync |
[production] |
| 07:09 |
<kartik@deploy1002> |
kartik: Backport for [[gerrit:1046810|Content Translation: Adjust the Machine translation limit for Telugu WP from 70% to 75% (T367838)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 07:04 |
<kartik@deploy1002> |
Started scap: Backport for [[gerrit:1046810|Content Translation: Adjust the Machine translation limit for Telugu WP from 70% to 75% (T367838)]] |
[production] |
| 06:52 |
<jynus@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1240.eqiad.wmnet with reason: data reload |
[production] |
| 06:52 |
<jynus@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1240.eqiad.wmnet with reason: data reload |
[production] |
| 06:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1191 (T364069)', diff saved to https://phabricator.wikimedia.org/P65139 and previous config saved to /var/cache/conftool/dbconfig/20240618-060100-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:00 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65138 and previous config saved to /var/cache/conftool/dbconfig/20240618-060038-marostegui.json |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2102.codfw.wmnet |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 05:55 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2102.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002" |
[production] |
| 05:53 |
<jynus@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2102.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002" |
[production] |
| 05:50 |
<jynus@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 05:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P65137 and previous config saved to /var/cache/conftool/dbconfig/20240618-054531-marostegui.json |
[production] |
| 05:44 |
<jynus@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts db2102.codfw.wmnet |
[production] |
| 05:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P65136 and previous config saved to /var/cache/conftool/dbconfig/20240618-053024-marostegui.json |
[production] |
| 05:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65135 and previous config saved to /var/cache/conftool/dbconfig/20240618-051517-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
dbmaint codfw s5 deploy schema change on db2213 T364299 |
[production] |
| 04:57 |
<marostegui> |
dbmaint eqiad s2 deploy schema change on db2207 T364299 |
[production] |
| 04:54 |
<marostegui> |
dbmaint eqiad s4 deploy schema change on db1160 T364299 |
[production] |
| 04:51 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Long schema change |
[production] |
| 04:51 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Long schema change |
[production] |
| 04:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1160 T367378', diff saved to https://phabricator.wikimedia.org/P65134 and previous config saved to /var/cache/conftool/dbconfig/20240618-044908-root.json |
[production] |
| 04:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db1238 to s4 primary and set section read-write T367378', diff saved to https://phabricator.wikimedia.org/P65133 and previous config saved to /var/cache/conftool/dbconfig/20240618-044806-marostegui.json |
[production] |
| 04:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - T367378', diff saved to https://phabricator.wikimedia.org/P65132 and previous config saved to /var/cache/conftool/dbconfig/20240618-044747-marostegui.json |
[production] |
| 04:47 |
<marostegui> |
Starting s4 eqiad failover from db1160 to db1238 - T367378 |
[production] |
| 04:21 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T367378 |
[production] |
| 04:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db1238 with weight 0 T367378', diff saved to https://phabricator.wikimedia.org/P65131 and previous config saved to /var/cache/conftool/dbconfig/20240618-042054-marostegui.json |
[production] |
| 04:20 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 33 hosts with reason: Primary switchover s4 T367378 |
[production] |
| 04:02 |
<mwpresync@deploy1002> |
Pruned MediaWiki: 1.43.0-wmf.7 (duration: 02m 50s) |
[production] |
| 04:01 |
<mwpresync@deploy1002> |
Finished scap: testwikis wikis to 1.43.0-wmf.10 refs T361404 (duration: 58m 57s) |
[production] |
| 03:03 |
<mwpresync@deploy1002> |
Started scap: testwikis wikis to 1.43.0-wmf.10 refs T361404 |
[production] |
| 01:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1181 (T364069)', diff saved to https://phabricator.wikimedia.org/P65130 and previous config saved to /var/cache/conftool/dbconfig/20240618-013639-marostegui.json |
[production] |
| 01:36 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:36 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T364069)', diff saved to https://phabricator.wikimedia.org/P65129 and previous config saved to /var/cache/conftool/dbconfig/20240618-013616-marostegui.json |
[production] |
| 01:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P65128 and previous config saved to /var/cache/conftool/dbconfig/20240618-012109-marostegui.json |
[production] |
| 01:10 |
<brett@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp4044.ulsfo.wmnet |
[production] |
| 01:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P65127 and previous config saved to /var/cache/conftool/dbconfig/20240618-010601-marostegui.json |
[production] |
| 00:57 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS bullseye |
[production] |
| 00:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T364069)', diff saved to https://phabricator.wikimedia.org/P65126 and previous config saved to /var/cache/conftool/dbconfig/20240618-005054-marostegui.json |
[production] |
| 00:34 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage |
[production] |
| 00:31 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage |
[production] |