|
2021-08-11
ยง
|
| 12:32 |
<godog> |
roll-restart prometheus T284213 |
[production] |
| 12:16 |
<moritzm> |
installing c-ares security updates on stretch |
[production] |
| 12:16 |
<btullis@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 |
[production] |
| 12:14 |
<oblivian@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:08 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:33 |
<Lucas_WMDE> |
EU backport+config window done |
[production] |
| 11:32 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711141|Remove $wmgWikibaseClientEntityNamespaces (T257260)]] (duration: 01m 08s) |
[production] |
| 11:31 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:29 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:29 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:711140|Stop setting $wgWBClientSettings['entityNamespaces'] (T257260)]] (duration: 01m 07s) |
[production] |
| 11:25 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711139|Remove $wmgWikibaseRepoEntityNamespaces (T257260)]] (duration: 01m 08s) |
[production] |
| 11:22 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:711138|Stop setting $wgWBRepoSettings['entityNamespaces'] (T257260)]] (duration: 01m 08s) |
[production] |
| 11:17 |
<oblivian@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:17 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 |
[production] |
| 11:17 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.37.0-wmf.18/extensions/SecurePoll/: Backport: [[gerrit:710720|Add ad-hoc logging to tally process (T288366)]] (duration: 01m 09s) |
[production] |
| 11:11 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:06 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711248|Disable Collection sidebar link on English Wikisource (T288021)]] (duration: 01m 14s) |
[production] |
| 10:58 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:42 |
<moritzm> |
rolling restart of Buster-based maps services to pick up c-ares security updates |
[production] |
| 10:37 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:20 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:02 |
<btullis@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 |
[production] |
| 09:50 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.37.0-wmf.18/includes/specials/SpecialWhatLinksHere.php: Backport: [[gerrit:710719|Fix SelectQueryBuilder use in SpecialWhatLinksHere (T288565)]] (duration: 01m 08s) |
[production] |
| 09:49 |
<godog> |
upgrade thanos on cloudmetrics* - T288604 |
[production] |
| 09:26 |
<godog> |
upgrade thanos on prometheus* - T288604 |
[production] |
| 09:21 |
<elukey> |
run "sudo find /var/log/airflow -type f -mtime +15 -delete" on an-airflow1001 to free space (root partition almost full) |
[production] |
| 09:19 |
<jgiannelos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:15 |
<jgiannelos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:11 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 09:05 |
<godog> |
upgrade thanos on thanos-fe* - T288604 |
[production] |
| 08:23 |
<kormat@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Minor cleanup of parsercache entries (duration: 01m 17s) |
[production] |
| 08:19 |
<moritzm> |
restart Aphlict to pick up c-ares security updates |
[production] |
| 08:17 |
<moritzm> |
restart Turnilo on an-tool1007 to pick up c-ares security updates |
[production] |
| 08:02 |
<moritzm> |
rolling restart of AQS to pick up the c-ares security update |
[production] |
| 07:09 |
<moritzm> |
restart etherpad-lite on etherpad1002 to pick up c-ares security updates |
[production] |
| 06:59 |
<_joe_> |
deleting the staging deployment of mwdebug |
[production] |
| 05:55 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2107.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2107.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:22 |
<marostegui> |
Stop replication on db2107 T287454 |
[production] |
| 05:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2107 T287454', diff saved to https://phabricator.wikimedia.org/P16999 and previous config saved to /var/cache/conftool/dbconfig/20210811-051856-marostegui.json |
[production] |
| 05:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db2104 to s2 master and set section read-write T287454', diff saved to https://phabricator.wikimedia.org/P16998 and previous config saved to /var/cache/conftool/dbconfig/20210811-051041-root.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s2 codfw as read-only for maintenance - T287454', diff saved to https://phabricator.wikimedia.org/P16997 and previous config saved to /var/cache/conftool/dbconfig/20210811-050040-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s2 codfw failover from db2107 to db2104 - T287454 |
[production] |
| 04:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db2104 with weight 0 T287454', diff saved to https://phabricator.wikimedia.org/P16996 and previous config saved to /var/cache/conftool/dbconfig/20210811-041625-root.json |
[production] |
| 04:15 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 |
[production] |
| 04:15 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 |
[production] |
| 03:45 |
<razzi@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 03:45 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 01:49 |
<dpifke@deploy1002> |
Finished deploy [performance/navtiming@12d8381]: Revert https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 05s) |
[production] |