|
2019-08-22
§
|
| 12:39 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:38 |
<_joe_> |
disabled slowlog on mw1348, repooling after reload |
[production] |
| 12:37 |
<jijiki> |
Pooling mv1347 not mw1247 |
[production] |
| 12:35 |
<jijiki> |
Pooling mw1247 |
[production] |
| 12:16 |
<moritzm> |
upgrading mariadb (packaged Debian version) on matomo1001 |
[production] |
| 12:15 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 12:13 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:07 |
<jijiki> |
Depooling mw1347 and mw1348 |
[production] |
| 10:55 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 33.3% - T219150 (duration: 01m 01s) |
[production] |
| 09:09 |
<ema> |
rolling ats-backend-restart to enable @debug system call family |
[production] |
| 09:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:00 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:17 |
<moritzm> |
restarting oozie on an-coord1001 |
[production] |
| 07:54 |
<tarrow@deploy1001> |
Synchronized php-1.34.0-wmf.19/extensions/Wikibase/repo/: Backport for UBN [[gerrit:531527|Hack to avoid trying to termbox render page before save (T230937)]] (duration: 00m 56s) |
[production] |
| 07:46 |
<marostegui> |
Deploy grants on labsdb1009-labsdb1012 to allow connections for haproxy from dbproxy1019 - T202367 |
[production] |
| 06:52 |
<moritzm> |
installing mariadb-10.1 updates from Stretch 9.9 point release (unrelated to wmf-mariadb, mostly client-side clients/libraries as shipped in Debian) |
[production] |
| 06:37 |
<moritzm> |
installing python-pip updates from Stretch 9.9 point release |
[production] |
| 05:41 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2066 from config T230885 (duration: 00m 54s) |
[production] |
| 05:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2066 from config T230885 (duration: 00m 54s) |
[production] |
| 05:14 |
<marostegui> |
Remove db2059 from tendril and zarcillo - T230884 |
[production] |
| 05:08 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2059 from config T230884 (duration: 00m 55s) |
[production] |
| 05:06 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2059 from config T230884 (duration: 00m 59s) |
[production] |
| 00:37 |
<XioNoX> |
run /usr/local/sbin/restart-php7.2-fpm on mwdebug1001/2 |
[production] |
| 00:27 |
<XioNoX> |
push L3 ECMP to eqiad routers - T230955 |
[production] |
| 00:23 |
<XioNoX> |
push L3 ECMP to esams routers - T230955 |
[production] |
| 00:22 |
<XioNoX> |
push L3 ECMP to eqsin routers - T230955 |
[production] |
| 00:21 |
<twentyafterfour> |
phabricator update completed without incident |
[production] |
| 00:19 |
<XioNoX> |
push L3 ECMP to codfw routers - T230955 |
[production] |
| 00:15 |
<twentyafterfour> |
Starting phabricator upgrade from tag release/2019-08-14/1 to release/2019-08-22/1 |
[production] |
|
2019-08-21
§
|
| 21:44 |
<eileen> |
civicrm revision changed from d7370a9d0b to ab2a9b264b, config revision is 58cd6b7ae6 |
[production] |
| 20:51 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@fc270fd]: bulk_daemon: Retune popularity_score bulk sizing (duration: 03m 49s) |
[production] |
| 20:48 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@fc270fd]: bulk_daemon: Retune popularity_score bulk sizing |
[production] |
| 20:17 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@67103e9]: bulk_daemon: Correct super() call (duration: 04m 19s) |
[production] |
| 20:13 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@67103e9]: bulk_daemon: Correct super() call |
[production] |
| 20:02 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@556c4d0]: bulk_daemon: Track timeouts, log indices used, increase thread counts (duration: 04m 42s) |
[production] |
| 20:00 |
<XioNoX> |
test l3 ECMP in ulsfo |
[production] |
| 19:57 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@556c4d0]: bulk_daemon: Track timeouts, log indices used, increase thread counts |
[production] |
| 19:54 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@adff5ad]: bulk_daemon: Track timeouts, log indices used, increase thread counts (duration: 02m 34s) |
[production] |
| 19:52 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@adff5ad]: bulk_daemon: Track timeouts, log indices used, increase thread counts |
[production] |
| 19:34 |
<XioNoX> |
repool codfw and eqsin - T226422 |
[production] |
| 19:31 |
<XioNoX> |
Rollback: Varnish: redirect eqsin/ulsfo text to eqiad - T226422 |
[production] |
| 19:29 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw |
[production] |
| 19:29 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=swift,name=codfw |
[production] |
| 19:26 |
<XioNoX> |
rollback: increase OSPF cost on cr2-codfw links - T226422 |
[production] |
| 19:25 |
<XioNoX> |
rollback deactivate transit links on cr2-codfw - T226422 |
[production] |
| 19:24 |
<XioNoX> |
rollback: move VRRP master from cr2-codfw to cr1-codfw - T226422 |
[production] |
| 19:16 |
<XioNoX> |
restart both REs on cr2-codfw - T226422 |
[production] |
| 19:14 |
<XioNoX> |
failover master RE to RE0 on cr2-codfw - T226422 |
[production] |
| 18:37 |
<XioNoX> |
shutdown re0:cr2-codfw (backup) - T226422 |
[production] |
| 18:32 |
<XioNoX> |
failover master RE to RE1 on cr2-codfw - T226422 |
[production] |