|
2021-08-11
ยง
|
| 19:45 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:39 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:35 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:29 |
<jhuneidi@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.18 refs T281159 (duration: 01m 08s) |
[production] |
| 19:28 |
<jhuneidi@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.18 refs T281159 |
[production] |
| 19:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:10 |
<jhuneidi@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.18 refs T281159 |
[production] |
| 19:01 |
<jgleeson> |
payments-wiki updated from a70aaa7944 to 0a27dbe9b6 |
[production] |
| 18:32 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Restarting to pick up Java security updates - hnowlan@cumin1001 |
[production] |
| 18:24 |
<legoktm@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:23 |
<legoktm@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:23 |
<legoktm@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:22 |
<legoktm@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 18:22 |
<legoktm@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:21 |
<bstorm> |
removed thirdparty/kubeadm-k8s-1-17 in reprepro |
[production] |
| 18:21 |
<legoktm@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 18:20 |
<legoktm@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:19 |
<legoktm@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:04 |
<ebernhardson@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@563f876]: process_sparql_query: increase parallelism to help backfill (duration: 02m 21s) |
[production] |
| 18:02 |
<ebernhardson@deploy1002> |
Started deploy [wikimedia/discovery/analytics@563f876]: process_sparql_query: increase parallelism to help backfill |
[production] |
| 17:37 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 17:36 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 17:35 |
<jforrester@deploy1002> |
Synchronized php-1.37.0-wmf.18/includes/specials/pagers/ContribsPager.php: T288563 Don't explode Special:Contributions on extension-formatted rows (3/3) (duration: 01m 06s) |
[production] |
| 17:34 |
<jforrester@deploy1002> |
Synchronized php-1.37.0-wmf.18/includes/Revision/RevisionFactory.php: T288563 Don't explode Special:Contributions on extension-formatted rows (2/3) (duration: 01m 08s) |
[production] |
| 17:32 |
<jforrester@deploy1002> |
Synchronized php-1.37.0-wmf.18/includes/Revision/RevisionStore.php: T288563 Don't explode Special:Contributions on extension-formatted rows (1/3) (duration: 01m 09s) |
[production] |
| 16:22 |
<dancy> |
Results of testing php_fpm_always_restart: php_fpm_always_restart=false: 1m19.942s php_fpm_always_restart=true: 3m12.836s |
[production] |
| 16:19 |
<dancy@deploy1002> |
Synchronized README: Testing scap php-rpm rolling restart (after) (duration: 03m 12s) |
[production] |
| 16:16 |
<thcipriani> |
moment of truth for php-fpm-always-restart in scap |
[production] |
| 16:10 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 |
[production] |
| 16:05 |
<dancy@deploy1002> |
Synchronized README: Testing scap php-rpm rolling restart (before) (duration: 01m 19s) |
[production] |
| 15:37 |
<hnowlan@cumin1001> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Restarting to pick up Java security updates - hnowlan@cumin1001 |
[production] |
| 15:12 |
<moritzm> |
import openjdk-8 8u302-b08-1+wmf1 to bullseye-wikimedia (bootstrap build, not to be used yet) T287960 |
[production] |
| 15:02 |
<sukhe@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts bast4002.wikimedia.org |
[production] |
| 14:57 |
<btullis@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 |
[production] |
| 14:55 |
<sukhe@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts bast4002.wikimedia.org |
[production] |
| 14:55 |
<sukhe@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts bast4002.wikimedia.org |
[production] |
| 14:44 |
<sukhe@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts bast4002.wikimedia.org |
[production] |
| 14:44 |
<sukhe> |
s/depool/decommission bast4002.wikimedia.org - T288579 |
[production] |
| 14:43 |
<sukhe> |
depool bast4002.wikimedia.org - T288579 |
[production] |
| 14:23 |
<moritzm> |
installing mx2002 T286911 |
[production] |
| 14:21 |
<hnowlan> |
disabled cassandra-metrics-collector on maps* |
[production] |
| 13:33 |
<moritzm> |
installing Java 8/Java 11 security updates on various analytics hosts |
[production] |
| 13:29 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 |
[production] |
| 12:45 |
<moritzm> |
imported openjdk-8 8u302-b08-1~deb10u1 to component/jdk8 for buster-wikimedia (forward port of the latest Java 8 security release) |
[production] |
| 12:32 |
<godog> |
roll-restart prometheus T284213 |
[production] |
| 12:16 |
<moritzm> |
installing c-ares security updates on stretch |
[production] |
| 12:16 |
<btullis@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. - btullis@cumin1001 |
[production] |