801-850 of 10000 results (62ms)
2019-07-02 ยง
18:07 <XioNoX> setup tunnel between eqord and eqiad - T226158 [production]
17:49 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@9ca9b0f]: Update mobileapps to 941e14f (T219998 T217352 T219909) (duration: 05m 49s) [production]
17:43 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@9ca9b0f]: Update mobileapps to 941e14f (T219998 T217352 T219909) [production]
16:59 <hashar> CI is back, I had to restart Zuul :-\ T227111 [production]
16:55 <hashar> Starting Jenkins and Zuul T227111 [production]
16:53 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org [production]
16:52 <hashar> Stopping Jenkins and Zuul T227111 [production]
16:32 <bblack> testing failure scenarios on dns2002, possible false-alarm alerts (depooled from LVS recdns) [production]
16:31 <bblack@cumin1001> conftool action : set/pooled=no; selector: name=dns2002.wikimedia.org [production]
16:31 <bblack> depool dns2002 from recdns server for testing [production]
16:30 <hashar> CI code-review +2 changes are not quite processed for some unknown reason T227111 [production]
16:19 <XioNoX> add term allow-anycast-dns in filter labs-in4 [production]
15:55 <ema> depool cp2026 and reimage as upload_ats T226637 [production]
15:47 <ema> pool cp2025 w/ ATS backend T226637 [production]
15:43 <XioNoX> "Equinix will be expanding the DA IX subnet from a /24 to a /23." (cf. email) [production]
15:34 <XioNoX> Add BGP to AS15830 in AMS-IX [production]
15:26 <XioNoX> add centrallog1001 to routers ACLs - T226813 [production]
15:20 <Krinkle> Set repo back from active to read-only https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet/cdh (T226474)) [production]
14:58 <jijiki> Run restart-php-fpm in all-mw-codfw - T223391 [production]
14:49 <ema> depool cp2025 and reimage as upload_ats T226637 [production]
14:47 <XioNoX> add anycast BGP statement to eqsin [production]
14:25 <jbond42> restart apache2 on phab1003 [production]
14:22 <XioNoX> add DNS anycast BGP statement to cr3-ulsfo [production]
14:18 <ema> pool cp2024 w/ ATS backend T226637 [production]
14:13 <otto@deploy1001> Finished deploy [eventstreams/deploy@de1d356]: Limit concurrent number of connections per X-Client-IP - T226808 (duration: 06m 17s) [production]
14:07 <otto@deploy1001> Started deploy [eventstreams/deploy@de1d356]: Limit concurrent number of connections per X-Client-IP - T226808 [production]
14:02 <bblack> deploying anycast_healthchecker changes to the recdnses (puppet disabled on all, testing dns4002 first) - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/397723/ [production]
13:31 <marostegui> Upgrade db2086 [production]
13:27 <ema> depool cp2024 and reimage as upload_ats T226637 [production]
13:26 <XioNoX> test fix policy ASXXX_in (missing `then next policy`) [production]
13:23 <marostegui> Upgrade db2085 [production]
13:13 <XioNoX> push RPKI classification to eqiad - T220669 [production]
13:09 <XioNoX> push RPKI classification to eqsin - T220669 [production]
13:06 <ema> pool cp2022 w/ ATS backend T226637 [production]
12:51 <XioNoX> push RPKI classification to AMS - T220669 [production]
12:47 <marostegui> Upgrade db2082 - T227062 [production]
12:30 <jijiki> Power cycle ms-be1021 - T227076 [production]
11:51 <ema> depool cp2022 and reimage as upload_ats T226637 [production]
11:40 <Urbanecm> EU SWAT really done [production]
11:37 <Urbanecm> Ran mwscript resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 86.49.134.37 for T225555 [production]
11:37 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: SWAT: [[:gerrit:520214|Add new throttle rule for cswiki workshop]] (T225555) (duration: 00m 49s) [production]
11:33 <Urbanecm> Reopen EU SWAT for last-time throttle rule [production]
11:33 <moritzm> re-enabled meta monitoring for icinga2001 [production]
11:26 <moritzm> rebooting icinga2001 for kernel security update [production]
11:26 <jijiki> Run restart-php-fpm in all-mw-eqiad - T223391 [production]
11:25 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
11:25 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
11:23 <moritzm> temporarily disabled meta monitoring for icinga2001 [production]
11:16 <dcausse> EU Swat done [production]
11:15 <dcausse@deploy1001> Synchronized php-1.34.0-wmf.11/extensions/CirrusSearch/includes/Updater.php: T226592: Ignore broken redirects when updating incoming link counts (duration: 00m 49s) [production]