2451-2500 of 10000 results (13ms)
2020-06-04 ยง
20:39 <ryankemper> disabled puppet on `cloudelastic100[5,6]` which are two racked nodes that we are now bringing into service. Will re-enable after successful puppet-merge / elasticsearch cluster join [production]
20:38 <ryankemper> disabled puppet on `cloudelastic100[5,6]` which are two racked nodes that we are now bringing into service. Will re-enable after successful puppet-merge / elasticsearch cluster join [production]
19:04 <jforrester@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.35 [production]
15:12 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=druid1004.eqiad.wmnet [production]
15:11 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:10 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
15:08 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:07 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:06 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:37 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
14:36 <moritzm> installing libexif security updates on jessie [production]
14:08 <moritzm> installing clamav security updates on mendelevium (ticket.wikimedia.org) [production]
14:00 <qchris> Stopping puppet on gerrit1002 (gerrit-test) to run tests for Gerrit upgrade [production]
13:41 <moritzm> bounced ferm on ms-be1023 [production]
13:35 <moritzm> installing exim security updates on jessie (stretch/buster already done) [production]
12:18 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
12:14 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
12:02 <moritzm> upgrading mw1276 to PHP 7.2.31 [production]
11:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11396 and previous config saved to /var/cache/conftool/dbconfig/20200604-115933-marostegui.json [production]
11:53 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: ec07467: wgNamespaceRobotPolicies: thwiki: Add 100 NS to noindex (T253574) (duration: 01m 15s) [production]
11:49 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:48 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 338cb90: 1ade16f: Change $wgNamespaceRobotPolicies on Thai wikis (T253578; T253577; T253576; T253575; T253574) (duration: 01m 07s) [production]
11:46 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11395 and previous config saved to /var/cache/conftool/dbconfig/20200604-114149-marostegui.json [production]
11:29 <marostegui> Compress InnoDB on db1091 before pooling it as new slave on s1 - T254462 [production]
11:21 <hashar@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [metawiki] Add `centralauth-rename` to WMF OIT staff - T254372 (duration: 01m 08s) [production]
11:14 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:12 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:04 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
10:59 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
10:53 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
10:53 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=druid1004.eqiad.wmnet [production]
10:46 <marostegui> Deploy schema change on s3 (only testwiki) eqiad - T238966 [production]
10:42 <marostegui> Deploy schema change on s3 (only testwiki) codfw - T238966 [production]
10:41 <jbond42> deployed new version of puppet-merge revert is https://gerrit.wikimedia.org/r/c/operations/puppet/+/602329 [production]
09:57 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
09:56 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:56 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:55 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:55 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:55 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:54 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
09:53 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:51 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:51 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:50 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:50 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:50 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:50 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:50 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]