2101-2150 of 10000 results (67ms)
2019-10-29 ยง
14:45 <robh> setting up ps1-b2-eqiad, librenms will output a couple reboots from it T227538 [production]
14:32 <Krinkle> krinkle@webperf1001.eqiad Restart navtiming, coal and statsv services [production]
14:29 <elukey> upgrade python-kafka on webperf[12]001 - T234808 [production]
14:27 <Krinkle> krinkle@webperf2001 Restart navtiming, coal and statsv services [production]
12:32 <hashar> Restarting Zuul / Jenkins [production]
12:31 <hashar> Stopping Zuul / Jenkins for upgrade [production]
12:29 <akosiaris> delete all production00 volumes on backup1001 [production]
11:48 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
11:37 <Urbanecm> EU SWAT done [production]
11:34 <urbanecm@deploy1001> Synchronized wmf-config/abusefilter.php: SWAT: faeb8f1: Allow AbuseFilter to issue blocks on es.wikinews (T236730) (duration: 00m 53s) [production]
11:28 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: fc9920e: Rename Author talk namespace at thwikisource (T236640) (duration: 00m 56s) [production]
11:19 <gehel@cumin2001> END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) [production]
11:17 <gehel@cumin2001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
10:51 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:51 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:51 <aborrero@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
10:51 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:46 <jakob@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'termbox' for release 'production' . [production]
10:39 <jakob@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'termbox' for release 'production' . [production]
10:33 <jakob@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . [production]
10:29 <moritzm> installing php5 security updates [production]
10:23 <jakob@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . [production]
10:21 <jynus> running import on m1-master, m1 replicas will lag for a whileT236406 [production]
10:20 <oblivian@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . [production]
10:19 <oblivian@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . [production]
10:15 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . [production]
10:07 <XioNoX> disable cr3-esams:et-1/0/0 (flapping) [production]
09:56 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:55 <filippo@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:55 <filippo@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:54 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:52 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:51 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:50 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:50 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:50 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:49 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:48 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:48 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:28 <gehel> plugin upgrade on relforge - T236123 [production]
09:27 <godog> reimage elastic 7 hw with Buster [production]
09:27 <vgutierrez> restart ats-tls on cp5007 disabling TCP SO_LINGER - T236458 [production]
08:43 <jynus> shutting down db1099 T227538 [production]
08:35 <jynus@cumin1001> dbctl commit (dc=all): 'Depool db1099', diff saved to https://phabricator.wikimedia.org/P9492 and previous config saved to /var/cache/conftool/dbconfig/20191029-083547-jynus.json [production]
08:15 <XioNoX> push term allow_vmhost ro cr3-esams loopback4 filter - T236598 [production]
08:06 <vgutierrez> restarting ats-tls on cp5007 with TCP FASTOPEN disabled - T236458 [production]
07:40 <moritzm> installing php7.3 security updates [production]
07:06 <elukey> roll restart java daemons on analytics1042, druid1003 and aqs1004 to pick up new openjdk upgrades [production]
07:01 <_joe_> restart memcached on mc1024-1036, 1 hour apart, via cumin (T235188) [production]
06:26 <_joe_> restart memcached on mc1023 T23518 [production]