3551-3600 of 10000 results (71ms)
2019-06-06 ยง
15:28 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:28 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:06 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:05 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:05 <moritzm> rolling reboot of sessionstore hosts in eqiad for kernel security update [production]
15:02 <_joe_> rolling restart of php-fpm on {appservers,api} in eqiad, in groups of 4, staggered by 10 minutes, to pick up the new opcache settings [production]
14:57 <bstorm_> T224850 update views on labsdb1012 [production]
14:43 <moritzm> updating qemu packages on ganeti hosts to deploy support for md_clear/MDS for Ganeti instances [production]
14:43 <elukey> restart mcrouter on mw2255 (codfw proxy) to pick up new config changes [production]
14:22 <dcausse@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: fix logspam (duration: 00m 48s) [production]
14:18 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.restart-wdqs (exit_code=0) [production]
13:54 <dcausse@deploy1001> Synchronized wmf-config/CirrusSearch-production.php: fix logspam (duration: 00m 47s) [production]
13:44 <moritzm> rolling reboot of sessionstore hosts in codfw for kernel security update [production]
13:44 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:44 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:36 <gehel@cumin1001> START - Cookbook sre.wdqs.restart-wdqs [production]
13:35 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.8 [production]
13:35 <gehel@cumin1001> END (FAIL) - Cookbook sre.wdqs.restart-wdqs (exit_code=99) [production]
13:35 <gehel@cumin1001> START - Cookbook sre.wdqs.restart-wdqs [production]
13:34 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.restart-wdqs (exit_code=0) [production]
13:33 <gehel@cumin1001> START - Cookbook sre.wdqs.restart-wdqs [production]
13:32 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.restart-wdqs (exit_code=0) [production]
13:31 <gehel@cumin1001> START - Cookbook sre.wdqs.restart-wdqs [production]
12:44 <jbond42> reimage neodymium [production]
12:23 <_joe_> running puppet, restarting php-fpm on the canaries to pick up the new opcache size [production]
12:11 <ema> cp1075: repool with varnish 5.1.3-1wm10 T224694 [production]
12:10 <elukey> restart mcrouter on mw2235 [production]
12:05 <Lucas_WMDE> EU SWAT done [production]
12:04 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/: SWAT: [[gerrit:514700|Revert "Specify $wgWBRepoSettings['conceptBaseUri']" (duration: 00m 56s) [production]
12:00 <ema> cp1075: upgrade varnish to 5.1.3-1wm10 T224694 [production]
11:55 <lucaswerkmeister-wmde@deploy1001> scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
11:48 <Urbanecm> running mwscript namespaceDupes.php --wiki=thwikisource --fix (T216322) [production]
11:47 <Urbanecm> running mwscript namespaceDupes.php --wiki=thwikibooks --fix for T216322 [production]
11:46 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[:gerrit:514678|Add new namespaces for several Thai projects]] (T216322) (duration: 00m 54s) [production]
11:38 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:514534|Remove unused config variable wgWikibaseEnableSenses]] (duration: 00m 55s) [production]
11:23 <gehel@cumin2001> END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) [production]
11:22 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.34.0-wmf.8/extensions/CirrusSearch/: SWAT: [[gerrit:514566|Fix event validation error for cirrussearch-request event]] (duration: 01m 06s) [production]
10:55 <elukey> restart mcrouter on mw2163 (codfw mcrouter proxy) [production]
10:43 <mobrovac@deploy1001> scap-helm mathoid finished [production]
10:43 <mobrovac@deploy1001> scap-helm mathoid cluster codfw completed [production]
10:43 <mobrovac@deploy1001> scap-helm mathoid cluster eqiad completed [production]
10:43 <mobrovac@deploy1001> scap-helm mathoid upgrade production stable/mathoid -f mathoid-values.yaml [namespace: mathoid, clusters: eqiad,codfw] [production]
10:30 <ema> varnish 5.1.3-1wm10 uploaded to stretch-wikimedia T224694 [production]
10:19 <elukey> rolling restart of mcrouter on mw1* hosts to pick up config change (batch of 5 hosts, depool/run-puppet/pool) [production]
10:12 <elukey> disable puppet on mw1* and mw[2163,2235,2255,2271] as prep step for mcrouter config deploy [production]
10:10 <fsero> rollbacked last deployment of mathoid to revision 16 [production]
09:59 <mobrovac@deploy1001> scap-helm mathoid finished [production]
09:59 <mobrovac@deploy1001> scap-helm mathoid cluster codfw completed [production]
09:59 <mobrovac@deploy1001> scap-helm mathoid cluster eqiad completed [production]
09:59 <mobrovac@deploy1001> scap-helm mathoid upgrade production stable/mathoid -f mathoid-values.yaml [namespace: mathoid, clusters: eqiad,codfw] [production]