651-700 of 10000 results (33ms)
2021-08-11 ยง
16:22 <dancy> Results of testing php_fpm_always_restart: php_fpm_always_restart=false: 1m19.942s php_fpm_always_restart=true: 3m12.836s [production]
16:19 <dancy@deploy1002> Synchronized README: Testing scap php-rpm rolling restart (after) (duration: 03m 12s) [production]
16:16 <thcipriani> moment of truth for php-fpm-always-restart in scap [production]
16:10 <btullis@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
16:05 <dancy@deploy1002> Synchronized README: Testing scap php-rpm rolling restart (before) (duration: 01m 19s) [production]
15:37 <hnowlan@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Restarting to pick up Java security updates - hnowlan@cumin1001 [production]
15:12 <moritzm> import openjdk-8 8u302-b08-1+wmf1 to bullseye-wikimedia (bootstrap build, not to be used yet) T287960 [production]
15:02 <sukhe@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts bast4002.wikimedia.org [production]
14:57 <btullis@cumin1001> START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
14:55 <sukhe@cumin1001> START - Cookbook sre.hosts.decommission for hosts bast4002.wikimedia.org [production]
14:55 <sukhe@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts bast4002.wikimedia.org [production]
14:44 <sukhe@cumin1001> START - Cookbook sre.hosts.decommission for hosts bast4002.wikimedia.org [production]
14:44 <sukhe> s/depool/decommission bast4002.wikimedia.org - T288579 [production]
14:43 <sukhe> depool bast4002.wikimedia.org - T288579 [production]
14:23 <moritzm> installing mx2002 T286911 [production]
14:21 <hnowlan> disabled cassandra-metrics-collector on maps* [production]
13:33 <moritzm> installing Java 8/Java 11 security updates on various analytics hosts [production]
13:29 <btullis@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
12:45 <moritzm> imported openjdk-8 8u302-b08-1~deb10u1 to component/jdk8 for buster-wikimedia (forward port of the latest Java 8 security release) [production]
12:32 <godog> roll-restart prometheus T284213 [production]
12:16 <moritzm> installing c-ares security updates on stretch [production]
12:16 <btullis@cumin1001> START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
12:14 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:08 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:41 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:33 <Lucas_WMDE> EU backport+config window done [production]
11:32 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711141|Remove $wmgWikibaseClientEntityNamespaces (T257260)]] (duration: 01m 08s) [production]
11:31 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:29 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:711140|Stop setting $wgWBClientSettings['entityNamespaces'] (T257260)]] (duration: 01m 07s) [production]
11:25 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711139|Remove $wmgWikibaseRepoEntityNamespaces (T257260)]] (duration: 01m 08s) [production]
11:22 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:711138|Stop setting $wgWBRepoSettings['entityNamespaces'] (T257260)]] (duration: 01m 08s) [production]
11:17 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:17 <btullis@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
11:17 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.37.0-wmf.18/extensions/SecurePoll/: Backport: [[gerrit:710720|Add ad-hoc logging to tally process (T288366)]] (duration: 01m 09s) [production]
11:11 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:06 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711248|Disable Collection sidebar link on English Wikisource (T288021)]] (duration: 01m 14s) [production]
10:58 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:42 <moritzm> rolling restart of Buster-based maps services to pick up c-ares security updates [production]
10:37 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:20 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:02 <btullis@cumin1001> START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 [production]
09:50 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.37.0-wmf.18/includes/specials/SpecialWhatLinksHere.php: Backport: [[gerrit:710719|Fix SelectQueryBuilder use in SpecialWhatLinksHere (T288565)]] (duration: 01m 08s) [production]
09:49 <godog> upgrade thanos on cloudmetrics* - T288604 [production]
09:26 <godog> upgrade thanos on prometheus* - T288604 [production]
09:21 <elukey> run "sudo find /var/log/airflow -type f -mtime +15 -delete" on an-airflow1001 to free space (root partition almost full) [production]
09:19 <jgiannelos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
09:15 <jgiannelos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
09:11 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
09:05 <godog> upgrade thanos on thanos-fe* - T288604 [production]