|
2022-11-14
ยง
|
| 09:38 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 50083 |
[production] |
| 09:38 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 50083 |
[production] |
| 09:37 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 32934 |
[production] |
| 09:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39397 and previous config saved to /var/cache/conftool/dbconfig/20221114-093640-root.json |
[production] |
| 09:35 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 32934 |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39396 and previous config saved to /var/cache/conftool/dbconfig/20221114-093118-marostegui.json |
[production] |
| 09:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39395 and previous config saved to /var/cache/conftool/dbconfig/20221114-092135-root.json |
[production] |
| 09:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39394 and previous config saved to /var/cache/conftool/dbconfig/20221114-091934-marostegui.json |
[production] |
| 09:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:19 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39393 and previous config saved to /var/cache/conftool/dbconfig/20221114-091912-marostegui.json |
[production] |
| 09:18 |
<ayounsi@cumin1001> |
END (ERROR) - Cookbook sre.network.peering (exit_code=97) with action 'configure' for AS: 13335 |
[production] |
| 09:18 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 13335 |
[production] |
| 09:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39392 and previous config saved to /var/cache/conftool/dbconfig/20221114-090630-root.json |
[production] |
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39391 and previous config saved to /var/cache/conftool/dbconfig/20221114-090406-marostegui.json |
[production] |
| 08:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39389 and previous config saved to /var/cache/conftool/dbconfig/20221114-085125-root.json |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39388 and previous config saved to /var/cache/conftool/dbconfig/20221114-084859-marostegui.json |
[production] |
| 08:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39387 and previous config saved to /var/cache/conftool/dbconfig/20221114-083620-root.json |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39386 and previous config saved to /var/cache/conftool/dbconfig/20221114-083352-marostegui.json |
[production] |
| 08:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2145 T322620', diff saved to https://phabricator.wikimedia.org/P39385 and previous config saved to /var/cache/conftool/dbconfig/20221114-082458-root.json |
[production] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39384 and previous config saved to /var/cache/conftool/dbconfig/20221114-082205-marostegui.json |
[production] |
| 08:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:21 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39383 and previous config saved to /var/cache/conftool/dbconfig/20221114-082144-marostegui.json |
[production] |
| 08:20 |
<moritzm> |
installing php7.4 security updates (as packaged in Debian) |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P39381 and previous config saved to /var/cache/conftool/dbconfig/20221114-080637-marostegui.json |
[production] |
| 08:02 |
<marostegui@deploy1002> |
Finished scap: Backport for [[gerrit:855741|Revert "ProductionServices.php: Promote pc2014 to pc1 master"]] (duration: 04m 34s) |
[production] |
| 07:57 |
<marostegui@deploy1002> |
marostegui and marostegui: Backport for [[gerrit:855741|Revert "ProductionServices.php: Promote pc2014 to pc1 master"]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 07:57 |
<marostegui@deploy1002> |
Started scap: Backport for [[gerrit:855741|Revert "ProductionServices.php: Promote pc2014 to pc1 master"]] |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P39380 and previous config saved to /var/cache/conftool/dbconfig/20221114-075131-marostegui.json |
[production] |
| 07:50 |
<moritzm> |
draining ganeti1021 for eventual reimage T311687 |
[production] |
| 07:47 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D |
[production] |
| 07:47 |
<marostegui@deploy1002> |
Finished scap: Backport for [[gerrit:856473|ProductionServices.php: Promote pc2014 to pc1 master (T322295)]] (duration: 05m 14s) |
[production] |
| 07:47 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D |
[production] |
| 07:42 |
<marostegui@deploy1002> |
marostegui and marostegui: Backport for [[gerrit:856473|ProductionServices.php: Promote pc2014 to pc1 master (T322295)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 07:42 |
<marostegui@deploy1002> |
Started scap: Backport for [[gerrit:856473|ProductionServices.php: Promote pc2014 to pc1 master (T322295)]] |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39379 and previous config saved to /var/cache/conftool/dbconfig/20221114-073624-marostegui.json |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1121 (T321130)', diff saved to https://phabricator.wikimedia.org/P39378 and previous config saved to /var/cache/conftool/dbconfig/20221114-072203-marostegui.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:21 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:21 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:13 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:08 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance |
[production] |
| 07:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance |
[production] |
| 06:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39377 and previous config saved to /var/cache/conftool/dbconfig/20221114-065620-marostegui.json |
[production] |
| 06:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39376 and previous config saved to /var/cache/conftool/dbconfig/20221114-064113-marostegui.json |
[production] |
| 06:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P39375 and previous config saved to /var/cache/conftool/dbconfig/20221114-062607-marostegui.json |
[production] |
| 06:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T321130)', diff saved to https://phabricator.wikimedia.org/P39374 and previous config saved to /var/cache/conftool/dbconfig/20221114-061100-marostegui.json |
[production] |