5651-5700 of 10000 results (21ms)
2020-10-06 ยง
15:23 <jayme> updated envoyproxy to 1.15.1-2 on mw-canary and restbase-canary [production]
15:04 <addshore> reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/632482 [releng]
14:57 <sukhe> upload dnsdist_1.5.0-1wm1 to apt.wm.o (buster) - T263789 [production]
14:47 <kormat@cumin1001> dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12936 and previous config saved to /var/cache/conftool/dbconfig/20201006-144701-kormat.json [production]
14:45 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:45 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:45 <arturo> icinga downtime every cloud* lab* host for 60 minutes for keystone maintenance [admin]
14:45 <vgutierrez> Bump ECDHE-ECDSA-AES128-SHA pageview replacement to 5% - T262946 [production]
14:45 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:44 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:40 <jayme> updated envoyproxy to 1.15.1-2 on mw2295.codfw.wmnet,restbase2017.codfw.wmnet [production]
14:38 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=restbase,service=restbase-backend,name=restbase2009.codfw.wmnet [production]
14:38 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=restbase,service=restbase-ssl,name=restbase2009.codfw.wmnet [production]
14:38 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2009.codfw.wmnet [production]
14:36 <hnowlan> repooling restbase2009 [production]
14:31 <kormat@cumin1001> dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 75%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12935 and previous config saved to /var/cache/conftool/dbconfig/20201006-143157-kormat.json [production]
14:19 <volker-e@deploy1001> Finished deploy [design/style-guide@e3fda83]: Deploy design/style-guide: (duration: 00m 05s) [production]
14:19 <volker-e@deploy1001> Started deploy [design/style-guide@e3fda83]: Deploy design/style-guide: [production]
14:15 <jayme> installed envoyproxy 1.15.1-2 on mwdebug1001 [production]
14:10 <andrewbogott> adding Sam Reed (reedy) as a user so he can investigate an rsync issue [iiab]
14:08 <marostegui> Reboot db1076 for kernel upgrade T264755 [production]
14:04 <jayme@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
14:03 <marostegui> Power cycle db1076 T264755 [production]
13:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1076 ', diff saved to https://phabricator.wikimedia.org/P12934 and previous config saved to /var/cache/conftool/dbconfig/20201006-135810-marostegui.json [production]
13:41 <kormat@cumin1001> dbctl commit (dc=all): 'db2137:3314 depooling: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12932 and previous config saved to /var/cache/conftool/dbconfig/20201006-134149-kormat.json [production]
13:41 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:41 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:40 <kormat@cumin1001> dbctl commit (dc=all): 'Remove db2119 from dump/vslow, add to all other contributions/logpager/recentchanges*/watchlist temporarily T259831', diff saved to https://phabricator.wikimedia.org/P12931 and previous config saved to /var/cache/conftool/dbconfig/20201006-134020-kormat.json [production]
13:40 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
13:14 <elukey> cleaned up /srv/jupyter/venv and re-created it to allow jupyterhub to start cleanly on stat1007 [analytics]
13:14 <jayme> pushed docker-registry.discovery.wmnet/envoy:1.15.1-2 - T264157 [production]
13:04 <marostegui> Change innodb_change_buffering = inserts on db2075 db2089 db2099 db2111 db2128 T263443 [production]
12:56 <joal> Restart oozie to pick up new spark settings [analytics]
12:55 <godog> swift codfw-prod: bump weight for ms-be2057 - T261633 [production]
12:47 <elukey> force re-creation of the base virtualenv for jupyter on stat1007 after the reimage [analytics]
12:27 <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/helm-linter:0.2.8 for jayme # T264157 [releng]
12:20 <elukey> update HDFS Namenode GC/Heap settings on an-master100[1,2] [production]
12:20 <elukey> update HDFS Namenode GC/Heap settings on an-master100[1,2] [analytics]
12:19 <elukey> increase spark shuffle io retry logic (10 tries every 10s) [analytics]
12:13 <jayme> imported envoyproxy_1.15.1-2 to buster-wikimedia and stretch-wikimedia [production]
12:08 <jbond42> deploy puppetlabs-stdlib 5.2 [production]
11:50 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:42 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
11:39 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:35 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
11:34 <Urbanecm> EU B&C window done [production]
11:34 <Urbanecm> urbanecm@mwmaint2001:~$ mwscript namespaceDupes.php --wiki=arbcom_ruwiki --fix # T264430 # P12930 [production]
11:32 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 07c19f97c79ec20d6b1657e589acfc242dd53b09: arbcom_ruwiki: Set AK as alias for NS_PROJECT (T264430) (duration: 00m 58s) [production]
11:31 <arturo> cleanup local changes in ops/puppet git repo in cloud-puppetmaster-03 [cloudinfra]
11:31 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 7e4e81129b8697c394ec329dd2b3c784e607a4d1: arbcom_ruwiki: Change favicon to File:Arbcom-ru_favicon.svg from commons (T264430) (duration: 00m 58s) [production]