2101-2150 of 10000 results (65ms)
2019-07-03 §
04:34 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
04:34 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:24 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
04:24 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:23 <vgutierrez> rebooting primary lvs servers for MDS security updates [production]
00:14 <eileen> process-control config revision is 8e215d07f2 (renable jobs) [production]
00:08 <eileen> civicrm revision is 8a4451f390, config revision is ec8c43ee86 Redis [production]
00:05 <eileen> process-control config revision is ec8c43ee86 (Redis turned on) [production]
2019-07-02 §
23:42 <eileen> civicrm revision is 8a4451f390, config revision is c02a038331 (mysql locks enabled) [production]
23:36 <catrope@deploy1001> Synchronized php-1.34.0-wmf.11/extensions/Echo/: T226594 (duration: 00m 51s) [production]
23:34 <catrope@deploy1001> Synchronized php-1.34.0-wmf.11/skins/MonoBook/: T226594 (duration: 00m 50s) [production]
22:35 <eileen> civicrm revision changed from 96985fcc4b to 8a4451f390, config revision is af9e657134 [production]
20:35 <mutante> contint1001 - created new partitions on /dev/sdc and /dev/sdd; created new RAID 1 over /dev/sdc1 and /dev/sdd1 [production]
20:28 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@cc60181]: Weekly WDQS deploy (duration: 14m 43s) [production]
20:20 <mutante> contint1001 - temp installing parted for labeling new disks sdc and sdd for raid for docker images (T207707) [production]
20:13 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@cc60181]: Weekly WDQS deploy [production]
19:37 <krinkle@deploy1001> Finished scap: l10n sync did not work as expected, try full scap to fix missing i18n message for 9963d843622 (duration: 18m 24s) [production]
19:18 <krinkle@deploy1001> Started scap: l10n sync did not work as expected, try full scap to fix missing i18n message for 9963d843622 [production]
19:07 <krinkle@deploy1001> scap sync-l10n completed (1.34.0-wmf.11) (duration: 00m 47s) [production]
19:05 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.11/extensions/AbuseFilter/: 9963d843622b / T227095 (duration: 00m 51s) [production]
19:03 <krinkle@deploy1001> scap sync-l10n completed (1.34.0-wmf.11) (duration: 00m 48s) [production]
19:00 <mholloway-shell@deploy1001> Finished deploy [recommendation-api/deploy@a29da76]: Update recommendation-api to 4f50c71 (duration: 02m 50s) [production]
18:57 <mholloway-shell@deploy1001> Started deploy [recommendation-api/deploy@a29da76]: Update recommendation-api to 4f50c71 [production]
18:07 <XioNoX> setup tunnel between eqord and eqiad - T226158 [production]
17:49 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@9ca9b0f]: Update mobileapps to 941e14f (T219998 T217352 T219909) (duration: 05m 49s) [production]
17:43 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@9ca9b0f]: Update mobileapps to 941e14f (T219998 T217352 T219909) [production]
16:59 <hashar> CI is back, I had to restart Zuul :-\ T227111 [production]
16:55 <hashar> Starting Jenkins and Zuul T227111 [production]
16:53 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org [production]
16:52 <hashar> Stopping Jenkins and Zuul T227111 [production]
16:32 <bblack> testing failure scenarios on dns2002, possible false-alarm alerts (depooled from LVS recdns) [production]
16:31 <bblack@cumin1001> conftool action : set/pooled=no; selector: name=dns2002.wikimedia.org [production]
16:31 <bblack> depool dns2002 from recdns server for testing [production]
16:30 <hashar> CI code-review +2 changes are not quite processed for some unknown reason T227111 [production]
16:19 <XioNoX> add term allow-anycast-dns in filter labs-in4 [production]
15:55 <ema> depool cp2026 and reimage as upload_ats T226637 [production]
15:47 <ema> pool cp2025 w/ ATS backend T226637 [production]
15:43 <XioNoX> "Equinix will be expanding the DA IX subnet from a /24 to a /23." (cf. email) [production]
15:34 <XioNoX> Add BGP to AS15830 in AMS-IX [production]
15:26 <XioNoX> add centrallog1001 to routers ACLs - T226813 [production]
15:20 <Krinkle> Set repo back from active to read-only https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet/cdh (T226474)) [production]
14:58 <jijiki> Run restart-php-fpm in all-mw-codfw - T223391 [production]
14:49 <ema> depool cp2025 and reimage as upload_ats T226637 [production]
14:47 <XioNoX> add anycast BGP statement to eqsin [production]
14:25 <jbond42> restart apache2 on phab1003 [production]
14:22 <XioNoX> add DNS anycast BGP statement to cr3-ulsfo [production]
14:18 <ema> pool cp2024 w/ ATS backend T226637 [production]
14:13 <otto@deploy1001> Finished deploy [eventstreams/deploy@de1d356]: Limit concurrent number of connections per X-Client-IP - T226808 (duration: 06m 17s) [production]
14:07 <otto@deploy1001> Started deploy [eventstreams/deploy@de1d356]: Limit concurrent number of connections per X-Client-IP - T226808 [production]
14:02 <bblack> deploying anycast_healthchecker changes to the recdnses (puppet disabled on all, testing dns4002 first) - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/397723/ [production]