|
2019-08-07
§
|
| 18:15 |
<jijiki> |
Restart hhvm and php-fpm on canary mw hosts |
[production] |
| 17:54 |
<shdubsh> |
install2002 add fstab entry for /srv mount - T229997 |
[production] |
| 17:46 |
<shdubsh> |
install2002 stop nginx and squid for resync /srv to spare disk and restore mount - T229997 |
[production] |
| 17:42 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Retry - Revert "Switch high-traffic jobs to eventgate." (duration: 00m 58s) |
[production] |
| 16:40 |
<mobrovac@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: JobQueue: Revert switching high-traffic jobs to eventgate (duration: 00m 55s) |
[production] |
| 16:34 |
<mobrovac@deploy1001> |
scap failed: average error rate on 6/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 16:00 |
<thcipriani> |
restarting jenkins for update |
[production] |
| 15:58 |
<jijiki> |
restart npre on stat1004 |
[production] |
| 15:08 |
<_joe_> |
freeing APCu on mw1270, which has degraded performance |
[production] |
| 14:24 |
<marostegui> |
Reboot dbproxy2003 for kernel upgrades |
[production] |
| 14:16 |
<jbond42> |
puppet *now* re-enabled |
[production] |
| 14:16 |
<jbond42> |
puppet not re-enabled |
[production] |
| 14:01 |
<jbond42> |
disable puppet fleet wide for puppetdb restart |
[production] |
| 13:57 |
<marostegui> |
Remove labsdb1004 and labsdb1005 from zarcillo database (instance table), as those hosts were decommissioned months ago |
[production] |
| 13:55 |
<marostegui> |
Remove labsdb1004 and labsdb1005 from zarcillo database, as those hosts were decommissioned months ago |
[production] |
| 13:48 |
<marostegui> |
Apply grants for dbproxy1003 on m3 - T202367 |
[production] |
| 13:22 |
<elukey> |
roll restart aqs on aqs100[4-9] to pick up new Druid backend settings |
[production] |
| 11:48 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:37 |
<kart_> |
Updated cxserver to 2019-08-06-100812-production (T227571) |
[production] |
| 11:33 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:527087|Switch property terms migration to WRITE_NEW on client wikis (T225053)]] (duration: 00m 56s) |
[production] |
| 11:29 |
<@> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:26 |
<pmiazga@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:528458|Enable AMC on all wikipedias (T228916)]] (duration: 00m 55s) |
[production] |
| 11:26 |
<@> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:22 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:09 |
<marostegui> |
Restart gerrit |
[production] |
| 10:11 |
<moritzm> |
deleting poolcounter1001, poolcounter1003, poolcounter2001, poolcounter2002 in Ganeti (T224572) |
[production] |
| 10:03 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:03 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 09:14 |
<marostegui> |
Drop math table from s6 - T196055 |
[production] |
| 08:49 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Provision db2131 into x1 T228969 (duration: 00m 55s) |
[production] |
| 08:48 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Provision db2131 into x1 T228969 (duration: 00m 56s) |
[production] |
| 08:37 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 08:37 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 08:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db2130 into s1 - T228969', diff saved to https://phabricator.wikimedia.org/P8877 and previous config saved to /var/cache/conftool/dbconfig/20190807-080059-marostegui.json |
[production] |
| 07:36 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:36 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1100 after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P8876 and previous config saved to /var/cache/conftool/dbconfig/20190807-073349-marostegui.json |
[production] |
| 07:32 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:31 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:28 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Provision db2130 into s1 T228969 (duration: 00m 56s) |
[production] |
| 07:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Provision db2130 into s1 T228969 (duration: 00m 55s) |
[production] |
| 05:57 |
<marostegui> |
Stop MySQL on db1071 - T229381 |
[production] |
| 05:55 |
<marostegui> |
Remove db1071 from tendril and zarcillo - T229381 |
[production] |
| 05:51 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db1071 from config T229381 (duration: 00m 55s) |
[production] |
| 05:50 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db1071 from config T229381 (duration: 00m 57s) |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1100 after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P8875 and previous config saved to /var/cache/conftool/dbconfig/20190807-053903-marostegui.json |
[production] |
| 00:48 |
<mutante> |
restarting gerrit to apply config change 528276 to exclude some projects from github replication |
[production] |
| 00:21 |
<mutante> |
gerrit2001 - restarting gerrit to apply 528276 |
[production] |