|
2021-08-11
§
|
| 08:17 |
<moritzm> |
restart Turnilo on an-tool1007 to pick up c-ares security updates |
[production] |
| 08:02 |
<moritzm> |
rolling restart of AQS to pick up the c-ares security update |
[production] |
| 07:09 |
<moritzm> |
restart etherpad-lite on etherpad1002 to pick up c-ares security updates |
[production] |
| 06:59 |
<_joe_> |
deleting the staging deployment of mwdebug |
[production] |
| 05:55 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2107.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2107.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:22 |
<marostegui> |
Stop replication on db2107 T287454 |
[production] |
| 05:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2107 T287454', diff saved to https://phabricator.wikimedia.org/P16999 and previous config saved to /var/cache/conftool/dbconfig/20210811-051856-marostegui.json |
[production] |
| 05:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db2104 to s2 master and set section read-write T287454', diff saved to https://phabricator.wikimedia.org/P16998 and previous config saved to /var/cache/conftool/dbconfig/20210811-051041-root.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s2 codfw as read-only for maintenance - T287454', diff saved to https://phabricator.wikimedia.org/P16997 and previous config saved to /var/cache/conftool/dbconfig/20210811-050040-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s2 codfw failover from db2107 to db2104 - T287454 |
[production] |
| 04:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db2104 with weight 0 T287454', diff saved to https://phabricator.wikimedia.org/P16996 and previous config saved to /var/cache/conftool/dbconfig/20210811-041625-root.json |
[production] |
| 04:15 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 |
[production] |
| 04:15 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 |
[production] |
| 03:45 |
<razzi@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 03:45 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 01:49 |
<dpifke@deploy1002> |
Finished deploy [performance/navtiming@12d8381]: Revert https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 05s) |
[production] |
| 01:49 |
<dpifke@deploy1002> |
Started deploy [performance/navtiming@12d8381]: Revert https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 |
[production] |
| 01:47 |
<dpifke@deploy1002> |
Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s) |
[production] |
| 01:47 |
<dpifke@deploy1002> |
Started deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 |
[production] |
| 01:38 |
<legoktm@deploy1002> |
Synchronized docroot/noc/conf/index.php: noc: Expose primary datacenter on conf/ (duration: 01m 06s) |
[production] |
| 01:22 |
<bstorm@cumin1001> |
END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) |
[production] |
| 01:22 |
<bstorm@cumin1001> |
Added views for new wiki: jvwikisource T286245 |
[production] |
| 01:00 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 00:38 |
<bstorm@cumin1001> |
END (ERROR) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=97) |
[production] |
| 00:36 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
|
2021-08-10
§
|
| 23:33 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:710344|Enable user links feature for pilot wikis, modern vector (T288274)]] (duration: 01m 08s) |
[production] |
| 23:18 |
<legoktm@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:06 |
<krinkle@deploy1002> |
Synchronized wmf-config/: I13e88c303a, T284418 (duration: 01m 07s) |
[production] |
| 23:02 |
<legoktm@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:58 |
<eileen> |
process-control config revision is 7bdc78073d |
[production] |
| 22:50 |
<krinkle@deploy1002> |
Synchronized wmf-config/: I8052636, I2038702b7e0 (duration: 01m 21s) |
[production] |
| 21:50 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:48 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1053.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:46 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:46 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1052.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:44 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1053.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:44 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1051.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:42 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1052.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:42 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1050.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:40 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1051.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:40 |
<ryankemper> |
[WDQS] `ryankemper@wdqs2005:~$ sudo pool` |
[production] |
| 21:40 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1049.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:40 |
<ryankemper> |
T288501 `ryankemper@wdqs2003:~$ sudo pool` |
[production] |
| 21:38 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1050.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:37 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1048.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:36 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1049.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:35 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1048.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:35 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1047.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:33 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1046.eqiad.wmnet with reason: REIMAGE |
[production] |