2251-2300 of 10000 results (80ms)
2019-06-06 §
12:44 <jbond42> reimage neodymium [production]
12:23 <_joe_> running puppet, restarting php-fpm on the canaries to pick up the new opcache size [production]
12:11 <ema> cp1075: repool with varnish 5.1.3-1wm10 T224694 [production]
12:10 <elukey> restart mcrouter on mw2235 [production]
12:05 <Lucas_WMDE> EU SWAT done [production]
12:04 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/: SWAT: [[gerrit:514700|Revert "Specify $wgWBRepoSettings['conceptBaseUri']" (duration: 00m 56s) [production]
12:00 <ema> cp1075: upgrade varnish to 5.1.3-1wm10 T224694 [production]
11:55 <lucaswerkmeister-wmde@deploy1001> scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
11:48 <Urbanecm> running mwscript namespaceDupes.php --wiki=thwikisource --fix (T216322) [production]
11:47 <Urbanecm> running mwscript namespaceDupes.php --wiki=thwikibooks --fix for T216322 [production]
11:46 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[:gerrit:514678|Add new namespaces for several Thai projects]] (T216322) (duration: 00m 54s) [production]
11:38 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:514534|Remove unused config variable wgWikibaseEnableSenses]] (duration: 00m 55s) [production]
11:23 <gehel@cumin2001> END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) [production]
11:22 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.34.0-wmf.8/extensions/CirrusSearch/: SWAT: [[gerrit:514566|Fix event validation error for cirrussearch-request event]] (duration: 01m 06s) [production]
10:55 <elukey> restart mcrouter on mw2163 (codfw mcrouter proxy) [production]
10:43 <mobrovac@deploy1001> scap-helm mathoid finished [production]
10:43 <mobrovac@deploy1001> scap-helm mathoid cluster codfw completed [production]
10:43 <mobrovac@deploy1001> scap-helm mathoid cluster eqiad completed [production]
10:43 <mobrovac@deploy1001> scap-helm mathoid upgrade production stable/mathoid -f mathoid-values.yaml [namespace: mathoid, clusters: eqiad,codfw] [production]
10:30 <ema> varnish 5.1.3-1wm10 uploaded to stretch-wikimedia T224694 [production]
10:19 <elukey> rolling restart of mcrouter on mw1* hosts to pick up config change (batch of 5 hosts, depool/run-puppet/pool) [production]
10:12 <elukey> disable puppet on mw1* and mw[2163,2235,2255,2271] as prep step for mcrouter config deploy [production]
10:10 <fsero> rollbacked last deployment of mathoid to revision 16 [production]
09:59 <mobrovac@deploy1001> scap-helm mathoid finished [production]
09:59 <mobrovac@deploy1001> scap-helm mathoid cluster codfw completed [production]
09:59 <mobrovac@deploy1001> scap-helm mathoid cluster eqiad completed [production]
09:59 <mobrovac@deploy1001> scap-helm mathoid upgrade production stable/mathoid -f mathoid-values.yaml [namespace: mathoid, clusters: eqiad,codfw] [production]
09:31 <moritzm> rebooting mwdebug2002 for some tests [production]
09:31 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:30 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:28 <moritzm> updating qemu on ganeti2004 for some tests [production]
09:24 <gehel@cumin2001> START - Cookbook sre.postgresql.postgres-init [production]
08:38 <marostegui> Stop MySQL on db1117:3322 - this will trigger haproxy alerts - T222682 [production]
07:35 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1121 after upgrade T224852 (duration: 00m 53s) [production]
07:20 <marostegui> Stop MySQL on db1121 for upgrade, this will generate lag on labs hosts for s6 - T224852 [production]
07:16 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2046 to s6 master as db2039 will be decommissioned T221533 (duration: 00m 55s) [production]
06:31 <marostegui> Start topology changes on s6 codfw to promote db2046 as master - T221533 [production]
06:23 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1121 for upgrade T224852 (duration: 00m 55s) [production]
06:15 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1091 after getting its BBU replaced (duration: 00m 54s) [production]
06:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 01m 01s) [production]
05:47 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 00m 55s) [production]
05:41 <marostegui> Upgrade MySQL on s6 codfw hosts in preparation for s6 codfw master failover - T221533 [production]
05:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 00m 55s) [production]
05:18 <marostegui> Remove db2042 from tendril and zarcillo T225090 [production]
05:18 <marostegui> Remove db2042 from tendril and zarcillo [production]
05:14 <marostegui> Stop MySQL on db2042 to copy its content to dbprov2001 as a temporary backup - T225090 [production]
05:11 <marostegui> Disable notifications db2042 - T225090 [production]
05:09 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1091 after getting its BBU replaced T225060 (duration: 00m 56s) [production]
2019-06-05 §
22:15 <chaomodus> restarting gerrit on cobalt due to it being down (seems like Java out of heap space) [production]
20:43 <mforns@deploy1001> Finished deploy [analytics/refinery@0660e70]: deploying analytics/refinery up to 0660e70153dec892ae20bee7119a72cc17e8ec87 (duration: 19m 30s) [production]