|
2019-08-21
§
|
| 21:44 |
<eileen> |
civicrm revision changed from d7370a9d0b to ab2a9b264b, config revision is 58cd6b7ae6 |
[production] |
| 20:51 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@fc270fd]: bulk_daemon: Retune popularity_score bulk sizing (duration: 03m 49s) |
[production] |
| 20:48 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@fc270fd]: bulk_daemon: Retune popularity_score bulk sizing |
[production] |
| 20:17 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@67103e9]: bulk_daemon: Correct super() call (duration: 04m 19s) |
[production] |
| 20:13 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@67103e9]: bulk_daemon: Correct super() call |
[production] |
| 20:02 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@556c4d0]: bulk_daemon: Track timeouts, log indices used, increase thread counts (duration: 04m 42s) |
[production] |
| 20:00 |
<XioNoX> |
test l3 ECMP in ulsfo |
[production] |
| 19:57 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@556c4d0]: bulk_daemon: Track timeouts, log indices used, increase thread counts |
[production] |
| 19:54 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@adff5ad]: bulk_daemon: Track timeouts, log indices used, increase thread counts (duration: 02m 34s) |
[production] |
| 19:52 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@adff5ad]: bulk_daemon: Track timeouts, log indices used, increase thread counts |
[production] |
| 19:34 |
<XioNoX> |
repool codfw and eqsin - T226422 |
[production] |
| 19:31 |
<XioNoX> |
Rollback: Varnish: redirect eqsin/ulsfo text to eqiad - T226422 |
[production] |
| 19:29 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw |
[production] |
| 19:29 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=swift,name=codfw |
[production] |
| 19:26 |
<XioNoX> |
rollback: increase OSPF cost on cr2-codfw links - T226422 |
[production] |
| 19:25 |
<XioNoX> |
rollback deactivate transit links on cr2-codfw - T226422 |
[production] |
| 19:24 |
<XioNoX> |
rollback: move VRRP master from cr2-codfw to cr1-codfw - T226422 |
[production] |
| 19:16 |
<XioNoX> |
restart both REs on cr2-codfw - T226422 |
[production] |
| 19:14 |
<XioNoX> |
failover master RE to RE0 on cr2-codfw - T226422 |
[production] |
| 18:37 |
<XioNoX> |
shutdown re0:cr2-codfw (backup) - T226422 |
[production] |
| 18:32 |
<XioNoX> |
failover master RE to RE1 on cr2-codfw - T226422 |
[production] |
| 18:19 |
<XioNoX> |
shutdown re1:cr2-codfw (backup) - T226422 |
[production] |
| 18:18 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.19/includes/specialpage/RedirectSpecialPage.php: T230932 RedirectSpecialArticle: Fix PHP notice about undefined index (duration: 00m 54s) |
[production] |
| 18:17 |
<XioNoX> |
move VRRP master from cr2-codfw to cr1-codfw - T226422 |
[production] |
| 18:15 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=false; selector: dnsdisc=swift,name=codfw |
[production] |
| 18:15 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw |
[production] |
| 18:15 |
<tarrow@deploy1001> |
Synchronized php-1.34.0-wmf.19/extensions/Wikibase/client/: SWAT: [[gerrit:531528|Use the backwards-compatible HTML ID for the wikidata item link (T66315)]] (duration: 00m 58s) |
[production] |
| 18:14 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=false; selector: dnsdisc=swift,name=codfw |
[production] |
| 18:14 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw |
[production] |
| 18:12 |
<XioNoX> |
deactivate transit links on cr2-codfw - T226422 |
[production] |
| 18:04 |
<XioNoX> |
increase OSPF cost on cr2-codfw links - T226422 |
[production] |
| 17:56 |
<XioNoX> |
rollback: apply BGP graceful shutdown to cr1-codfw transits - T226422 |
[production] |
| 17:55 |
<XioNoX> |
Rollback: increase OSPF cost on ulsfo-codfw link - T226422 |
[production] |
| 17:53 |
<XioNoX> |
rollback: disable BGP from cr1-codfw to lvs2001/2/3 - T226422 |
[production] |
| 17:43 |
<XioNoX> |
restart both REs on cr1-codfw - T226422 |
[production] |
| 17:33 |
<XioNoX> |
failover master RE to RE0 on cr1-codfw - T226422 |
[production] |
| 17:33 |
<cmjohnson1> |
cloudvirt1015 down for a new motherboard |
[production] |
| 17:25 |
<XioNoX> |
shutdown RE0 on cr1-codfw - T226422 |
[production] |
| 17:17 |
<bstorm_> |
reboot cloudvirt1024 to try and reset raid T230289 |
[production] |
| 17:17 |
<XioNoX> |
failover master RE to RE1 on cr1-codfw - T226422 |
[production] |
| 17:08 |
<XioNoX> |
disable BGP from cr1-codfw to lvs2001/2/3 - T226422 |
[production] |
| 17:02 |
<cmjohnson1> |
rebooting cloudvirt1024 |
[production] |
| 17:00 |
<tarrow> |
continuing the SWAT window to backport train blocker fixes |
[production] |
| 16:56 |
<XioNoX> |
Varnish: redirect eqsin/ulsfo text to eqiad - T226422 |
[production] |
| 16:51 |
<XioNoX> |
increase OSPF cost on ulsfo-codfw link - T226422 |
[production] |
| 16:46 |
<XioNoX> |
apply BGP graceful shutdown to cr1-codfw transits - T226422 |
[production] |
| 16:37 |
<XioNoX> |
depool eqsin and codfw - T226422 |
[production] |