3901-3950 of 10000 results (45ms)
2021-08-11 ยง
12:16 <btullis@cumin1001> START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
12:14 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:08 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:41 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:33 <Lucas_WMDE> EU backport+config window done [production]
11:32 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711141|Remove $wmgWikibaseClientEntityNamespaces (T257260)]] (duration: 01m 08s) [production]
11:31 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:29 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:711140|Stop setting $wgWBClientSettings['entityNamespaces'] (T257260)]] (duration: 01m 07s) [production]
11:25 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711139|Remove $wmgWikibaseRepoEntityNamespaces (T257260)]] (duration: 01m 08s) [production]
11:22 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:711138|Stop setting $wgWBRepoSettings['entityNamespaces'] (T257260)]] (duration: 01m 08s) [production]
11:17 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:17 <btullis@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
11:17 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.37.0-wmf.18/extensions/SecurePoll/: Backport: [[gerrit:710720|Add ad-hoc logging to tally process (T288366)]] (duration: 01m 09s) [production]
11:11 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:06 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711248|Disable Collection sidebar link on English Wikisource (T288021)]] (duration: 01m 14s) [production]
10:58 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:42 <moritzm> rolling restart of Buster-based maps services to pick up c-ares security updates [production]
10:37 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:20 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:02 <btullis@cumin1001> START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
09:50 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.37.0-wmf.18/includes/specials/SpecialWhatLinksHere.php: Backport: [[gerrit:710719|Fix SelectQueryBuilder use in SpecialWhatLinksHere (T288565)]] (duration: 01m 08s) [production]
09:49 <godog> upgrade thanos on cloudmetrics* - T288604 [production]
09:26 <godog> upgrade thanos on prometheus* - T288604 [production]
09:21 <elukey> run "sudo find /var/log/airflow -type f -mtime +15 -delete" on an-airflow1001 to free space (root partition almost full) [production]
09:19 <jgiannelos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
09:15 <jgiannelos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
09:11 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
09:05 <godog> upgrade thanos on thanos-fe* - T288604 [production]
08:23 <kormat@deploy1002> Synchronized wmf-config/ProductionServices.php: Minor cleanup of parsercache entries (duration: 01m 17s) [production]
08:19 <moritzm> restart Aphlict to pick up c-ares security updates [production]
08:17 <moritzm> restart Turnilo on an-tool1007 to pick up c-ares security updates [production]
08:02 <moritzm> rolling restart of AQS to pick up the c-ares security update [production]
07:09 <moritzm> restart etherpad-lite on etherpad1002 to pick up c-ares security updates [production]
06:59 <_joe_> deleting the staging deployment of mwdebug [production]
05:55 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2107.codfw.wmnet with reason: REIMAGE [production]
05:53 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db2107.codfw.wmnet with reason: REIMAGE [production]
05:22 <marostegui> Stop replication on db2107 T287454 [production]
05:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2107 T287454', diff saved to https://phabricator.wikimedia.org/P16999 and previous config saved to /var/cache/conftool/dbconfig/20210811-051856-marostegui.json [production]
05:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db2104 to s2 master and set section read-write T287454', diff saved to https://phabricator.wikimedia.org/P16998 and previous config saved to /var/cache/conftool/dbconfig/20210811-051041-root.json [production]
05:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s2 codfw as read-only for maintenance - T287454', diff saved to https://phabricator.wikimedia.org/P16997 and previous config saved to /var/cache/conftool/dbconfig/20210811-050040-marostegui.json [production]
05:00 <marostegui> Starting s2 codfw failover from db2107 to db2104 - T287454 [production]
04:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db2104 with weight 0 T287454', diff saved to https://phabricator.wikimedia.org/P16996 and previous config saved to /var/cache/conftool/dbconfig/20210811-041625-root.json [production]
04:15 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
04:15 <root@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s2 T287454 [production]
03:45 <razzi@cumin1001> END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 [production]
03:45 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 [production]
01:49 <dpifke@deploy1002> Finished deploy [performance/navtiming@12d8381]: Revert https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 05s) [production]
01:49 <dpifke@deploy1002> Started deploy [performance/navtiming@12d8381]: Revert https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 [production]
01:47 <dpifke@deploy1002> Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s) [production]