|
2021-08-11
§
|
| 09:26 |
<godog> |
upgrade thanos on prometheus* - T288604 |
[production] |
| 09:21 |
<elukey> |
run "sudo find /var/log/airflow -type f -mtime +15 -delete" on an-airflow1001 to free space (root partition almost full) |
[production] |
| 09:19 |
<jgiannelos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:15 |
<jgiannelos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:11 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 09:05 |
<godog> |
upgrade thanos on thanos-fe* - T288604 |
[production] |
| 08:23 |
<kormat@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Minor cleanup of parsercache entries (duration: 01m 17s) |
[production] |
| 08:19 |
<moritzm> |
restart Aphlict to pick up c-ares security updates |
[production] |
| 08:17 |
<moritzm> |
restart Turnilo on an-tool1007 to pick up c-ares security updates |
[production] |
| 08:02 |
<moritzm> |
rolling restart of AQS to pick up the c-ares security update |
[production] |
| 07:09 |
<moritzm> |
restart etherpad-lite on etherpad1002 to pick up c-ares security updates |
[production] |
| 06:59 |
<_joe_> |
deleting the staging deployment of mwdebug |
[production] |
| 05:55 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2107.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2107.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:22 |
<marostegui> |
Stop replication on db2107 T287454 |
[production] |
| 05:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2107 T287454', diff saved to https://phabricator.wikimedia.org/P16999 and previous config saved to /var/cache/conftool/dbconfig/20210811-051856-marostegui.json |
[production] |
| 05:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db2104 to s2 master and set section read-write T287454', diff saved to https://phabricator.wikimedia.org/P16998 and previous config saved to /var/cache/conftool/dbconfig/20210811-051041-root.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s2 codfw as read-only for maintenance - T287454', diff saved to https://phabricator.wikimedia.org/P16997 and previous config saved to /var/cache/conftool/dbconfig/20210811-050040-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s2 codfw failover from db2107 to db2104 - T287454 |
[production] |
| 04:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db2104 with weight 0 T287454', diff saved to https://phabricator.wikimedia.org/P16996 and previous config saved to /var/cache/conftool/dbconfig/20210811-041625-root.json |
[production] |
| 04:15 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 |
[production] |
| 04:15 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 |
[production] |
| 03:45 |
<razzi@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 03:45 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 01:49 |
<dpifke@deploy1002> |
Finished deploy [performance/navtiming@12d8381]: Revert https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 05s) |
[production] |
| 01:49 |
<dpifke@deploy1002> |
Started deploy [performance/navtiming@12d8381]: Revert https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 |
[production] |
| 01:47 |
<dpifke@deploy1002> |
Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s) |
[production] |
| 01:47 |
<dpifke@deploy1002> |
Started deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 |
[production] |
| 01:38 |
<legoktm@deploy1002> |
Synchronized docroot/noc/conf/index.php: noc: Expose primary datacenter on conf/ (duration: 01m 06s) |
[production] |
| 01:22 |
<bstorm@cumin1001> |
END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) |
[production] |
| 01:22 |
<bstorm@cumin1001> |
Added views for new wiki: jvwikisource T286245 |
[production] |
| 01:00 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 00:38 |
<bstorm@cumin1001> |
END (ERROR) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=97) |
[production] |
| 00:36 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
|
2021-08-10
§
|
| 23:33 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:710344|Enable user links feature for pilot wikis, modern vector (T288274)]] (duration: 01m 08s) |
[production] |
| 23:18 |
<legoktm@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:06 |
<krinkle@deploy1002> |
Synchronized wmf-config/: I13e88c303a, T284418 (duration: 01m 07s) |
[production] |
| 23:02 |
<legoktm@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:58 |
<eileen> |
process-control config revision is 7bdc78073d |
[production] |
| 22:50 |
<krinkle@deploy1002> |
Synchronized wmf-config/: I8052636, I2038702b7e0 (duration: 01m 21s) |
[production] |
| 21:50 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:48 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1053.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:46 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:46 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1052.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:44 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1053.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:44 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1051.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:42 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1052.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:42 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1050.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:40 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1051.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:40 |
<ryankemper> |
[WDQS] `ryankemper@wdqs2005:~$ sudo pool` |
[production] |