|
2019-07-02
ยง
|
| 18:57 |
<mholloway-shell@deploy1001> |
Started deploy [recommendation-api/deploy@a29da76]: Update recommendation-api to 4f50c71 |
[production] |
| 18:07 |
<XioNoX> |
setup tunnel between eqord and eqiad - T226158 |
[production] |
| 17:49 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@9ca9b0f]: Update mobileapps to 941e14f (T219998 T217352 T219909) (duration: 05m 49s) |
[production] |
| 17:43 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@9ca9b0f]: Update mobileapps to 941e14f (T219998 T217352 T219909) |
[production] |
| 16:59 |
<hashar> |
CI is back, I had to restart Zuul :-\ T227111 |
[production] |
| 16:55 |
<hashar> |
Starting Jenkins and Zuul T227111 |
[production] |
| 16:53 |
<bblack@cumin1001> |
conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org |
[production] |
| 16:52 |
<hashar> |
Stopping Jenkins and Zuul T227111 |
[production] |
| 16:32 |
<bblack> |
testing failure scenarios on dns2002, possible false-alarm alerts (depooled from LVS recdns) |
[production] |
| 16:31 |
<bblack@cumin1001> |
conftool action : set/pooled=no; selector: name=dns2002.wikimedia.org |
[production] |
| 16:31 |
<bblack> |
depool dns2002 from recdns server for testing |
[production] |
| 16:30 |
<hashar> |
CI code-review +2 changes are not quite processed for some unknown reason T227111 |
[production] |
| 16:19 |
<XioNoX> |
add term allow-anycast-dns in filter labs-in4 |
[production] |
| 15:55 |
<ema> |
depool cp2026 and reimage as upload_ats T226637 |
[production] |
| 15:47 |
<ema> |
pool cp2025 w/ ATS backend T226637 |
[production] |
| 15:43 |
<XioNoX> |
"Equinix will be expanding the DA IX subnet from a /24 to a /23." (cf. email) |
[production] |
| 15:34 |
<XioNoX> |
Add BGP to AS15830 in AMS-IX |
[production] |
| 15:26 |
<XioNoX> |
add centrallog1001 to routers ACLs - T226813 |
[production] |
| 15:20 |
<Krinkle> |
Set repo back from active to read-only https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet/cdh (T226474)) |
[production] |
| 14:58 |
<jijiki> |
Run restart-php-fpm in all-mw-codfw - T223391 |
[production] |
| 14:49 |
<ema> |
depool cp2025 and reimage as upload_ats T226637 |
[production] |
| 14:47 |
<XioNoX> |
add anycast BGP statement to eqsin |
[production] |
| 14:25 |
<jbond42> |
restart apache2 on phab1003 |
[production] |
| 14:22 |
<XioNoX> |
add DNS anycast BGP statement to cr3-ulsfo |
[production] |
| 14:18 |
<ema> |
pool cp2024 w/ ATS backend T226637 |
[production] |
| 14:13 |
<otto@deploy1001> |
Finished deploy [eventstreams/deploy@de1d356]: Limit concurrent number of connections per X-Client-IP - T226808 (duration: 06m 17s) |
[production] |
| 14:07 |
<otto@deploy1001> |
Started deploy [eventstreams/deploy@de1d356]: Limit concurrent number of connections per X-Client-IP - T226808 |
[production] |
| 14:02 |
<bblack> |
deploying anycast_healthchecker changes to the recdnses (puppet disabled on all, testing dns4002 first) - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/397723/ |
[production] |
| 13:31 |
<marostegui> |
Upgrade db2086 |
[production] |
| 13:27 |
<ema> |
depool cp2024 and reimage as upload_ats T226637 |
[production] |
| 13:26 |
<XioNoX> |
test fix policy ASXXX_in (missing `then next policy`) |
[production] |
| 13:23 |
<marostegui> |
Upgrade db2085 |
[production] |
| 13:13 |
<XioNoX> |
push RPKI classification to eqiad - T220669 |
[production] |
| 13:09 |
<XioNoX> |
push RPKI classification to eqsin - T220669 |
[production] |
| 13:06 |
<ema> |
pool cp2022 w/ ATS backend T226637 |
[production] |
| 12:51 |
<XioNoX> |
push RPKI classification to AMS - T220669 |
[production] |
| 12:47 |
<marostegui> |
Upgrade db2082 - T227062 |
[production] |
| 12:30 |
<jijiki> |
Power cycle ms-be1021 - T227076 |
[production] |
| 11:51 |
<ema> |
depool cp2022 and reimage as upload_ats T226637 |
[production] |
| 11:40 |
<Urbanecm> |
EU SWAT really done |
[production] |
| 11:37 |
<Urbanecm> |
Ran mwscript resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 86.49.134.37 for T225555 |
[production] |
| 11:37 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[:gerrit:520214|Add new throttle rule for cswiki workshop]] (T225555) (duration: 00m 49s) |
[production] |
| 11:33 |
<Urbanecm> |
Reopen EU SWAT for last-time throttle rule |
[production] |
| 11:33 |
<moritzm> |
re-enabled meta monitoring for icinga2001 |
[production] |
| 11:26 |
<moritzm> |
rebooting icinga2001 for kernel security update |
[production] |
| 11:26 |
<jijiki> |
Run restart-php-fpm in all-mw-eqiad - T223391 |
[production] |
| 11:25 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 11:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:23 |
<moritzm> |
temporarily disabled meta monitoring for icinga2001 |
[production] |
| 11:16 |
<dcausse> |
EU Swat done |
[production] |