|
2019-06-06
§
|
| 13:31 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.restart-wdqs |
[production] |
| 12:44 |
<jbond42> |
reimage neodymium |
[production] |
| 12:23 |
<_joe_> |
running puppet, restarting php-fpm on the canaries to pick up the new opcache size |
[production] |
| 12:11 |
<ema> |
cp1075: repool with varnish 5.1.3-1wm10 T224694 |
[production] |
| 12:10 |
<elukey> |
restart mcrouter on mw2235 |
[production] |
| 12:05 |
<Lucas_WMDE> |
EU SWAT done |
[production] |
| 12:04 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/: SWAT: [[gerrit:514700|Revert "Specify $wgWBRepoSettings['conceptBaseUri']" (duration: 00m 56s) |
[production] |
| 12:00 |
<ema> |
cp1075: upgrade varnish to 5.1.3-1wm10 T224694 |
[production] |
| 11:55 |
<lucaswerkmeister-wmde@deploy1001> |
scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 11:48 |
<Urbanecm> |
running mwscript namespaceDupes.php --wiki=thwikisource --fix (T216322) |
[production] |
| 11:47 |
<Urbanecm> |
running mwscript namespaceDupes.php --wiki=thwikibooks --fix for T216322 |
[production] |
| 11:46 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[:gerrit:514678|Add new namespaces for several Thai projects]] (T216322) (duration: 00m 54s) |
[production] |
| 11:38 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:514534|Remove unused config variable wgWikibaseEnableSenses]] (duration: 00m 55s) |
[production] |
| 11:23 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) |
[production] |
| 11:22 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized php-1.34.0-wmf.8/extensions/CirrusSearch/: SWAT: [[gerrit:514566|Fix event validation error for cirrussearch-request event]] (duration: 01m 06s) |
[production] |
| 10:55 |
<elukey> |
restart mcrouter on mw2163 (codfw mcrouter proxy) |
[production] |
| 10:43 |
<mobrovac@deploy1001> |
scap-helm mathoid finished |
[production] |
| 10:43 |
<mobrovac@deploy1001> |
scap-helm mathoid cluster codfw completed |
[production] |
| 10:43 |
<mobrovac@deploy1001> |
scap-helm mathoid cluster eqiad completed |
[production] |
| 10:43 |
<mobrovac@deploy1001> |
scap-helm mathoid upgrade production stable/mathoid -f mathoid-values.yaml [namespace: mathoid, clusters: eqiad,codfw] |
[production] |
| 10:30 |
<ema> |
varnish 5.1.3-1wm10 uploaded to stretch-wikimedia T224694 |
[production] |
| 10:19 |
<elukey> |
rolling restart of mcrouter on mw1* hosts to pick up config change (batch of 5 hosts, depool/run-puppet/pool) |
[production] |
| 10:12 |
<elukey> |
disable puppet on mw1* and mw[2163,2235,2255,2271] as prep step for mcrouter config deploy |
[production] |
| 10:10 |
<fsero> |
rollbacked last deployment of mathoid to revision 16 |
[production] |
| 09:59 |
<mobrovac@deploy1001> |
scap-helm mathoid finished |
[production] |
| 09:59 |
<mobrovac@deploy1001> |
scap-helm mathoid cluster codfw completed |
[production] |
| 09:59 |
<mobrovac@deploy1001> |
scap-helm mathoid cluster eqiad completed |
[production] |
| 09:59 |
<mobrovac@deploy1001> |
scap-helm mathoid upgrade production stable/mathoid -f mathoid-values.yaml [namespace: mathoid, clusters: eqiad,codfw] |
[production] |
| 09:31 |
<moritzm> |
rebooting mwdebug2002 for some tests |
[production] |
| 09:31 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:30 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:28 |
<moritzm> |
updating qemu on ganeti2004 for some tests |
[production] |
| 09:24 |
<gehel@cumin2001> |
START - Cookbook sre.postgresql.postgres-init |
[production] |
| 08:38 |
<marostegui> |
Stop MySQL on db1117:3322 - this will trigger haproxy alerts - T222682 |
[production] |
| 07:35 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1121 after upgrade T224852 (duration: 00m 53s) |
[production] |
| 07:20 |
<marostegui> |
Stop MySQL on db1121 for upgrade, this will generate lag on labs hosts for s6 - T224852 |
[production] |
| 07:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Promote db2046 to s6 master as db2039 will be decommissioned T221533 (duration: 00m 55s) |
[production] |
| 06:31 |
<marostegui> |
Start topology changes on s6 codfw to promote db2046 as master - T221533 |
[production] |
| 06:23 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1121 for upgrade T224852 (duration: 00m 55s) |
[production] |
| 06:15 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1091 after getting its BBU replaced (duration: 00m 54s) |
[production] |
| 06:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 01m 01s) |
[production] |
| 05:47 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 00m 55s) |
[production] |
| 05:41 |
<marostegui> |
Upgrade MySQL on s6 codfw hosts in preparation for s6 codfw master failover - T221533 |
[production] |
| 05:32 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 00m 55s) |
[production] |
| 05:18 |
<marostegui> |
Remove db2042 from tendril and zarcillo T225090 |
[production] |
| 05:18 |
<marostegui> |
Remove db2042 from tendril and zarcillo |
[production] |
| 05:14 |
<marostegui> |
Stop MySQL on db2042 to copy its content to dbprov2001 as a temporary backup - T225090 |
[production] |
| 05:11 |
<marostegui> |
Disable notifications db2042 - T225090 |
[production] |
| 05:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1091 after getting its BBU replaced T225060 (duration: 00m 56s) |
[production] |