3901-3950 of 10000 results (67ms)
2019-07-25 §
07:02 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
06:42 <elukey> restart kafka* on kafka-jumbo1001 to pick up new openjdk-8 version [production]
06:37 <elukey> restart cassandra instances on aqs1004 to pick up new openjdk-8 version [production]
06:34 <elukey> add term eventgate to analytics-in4 on cr1/cr2-eqiad - T228882 [production]
05:31 <twentyafterfour> set phabricator to read-write mode [production]
05:30 <marostegui> Failover m3 from db1072 to db1128 - T228243 [production]
05:30 <twentyafterfour> phabricator set to read-only mode [production]
04:51 <marostegui> Start pre-failover steps on m3 T228243 [production]
02:02 <XioNoX> remove peer AS63541 from cr1-eqsin [production]
2019-07-24 §
23:46 <nuria@deploy1001> Finished deploy [analytics/refinery@7d93398]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues). Try 2 (duration: 13m 34s) [production]
23:43 <catrope@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/Flow: Fix JS error when saving Flow board descriptions (T228818) (duration: 01m 01s) [production]
23:42 <catrope@deploy1001> Synchronized php-1.34.0-wmf.14/extensions/Flow: Fix JS error when saving Flow board descriptions (T228818) (duration: 01m 03s) [production]
23:39 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable homepage for 50% of new users on arwiki (T228120) (duration: 00m 58s) [production]
23:32 <nuria@deploy1001> Started deploy [analytics/refinery@7d93398]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues). Try 2 [production]
23:30 <nuria@deploy1001> Finished deploy [analytics/refinery@834db0a]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues) (duration: 18m 10s) [production]
23:22 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable GrowthExperiments homepage on arwiki (T228120) (duration: 00m 55s) [production]
23:13 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Correct typo in arwiki help panel config (T228820) (duration: 00m 57s) [production]
23:12 <nuria@deploy1001> Started deploy [analytics/refinery@834db0a]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues) [production]
22:41 <thcipriani@> helmfile [EQIAD] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
22:36 <thcipriani@> helmfile [CODFW] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
22:28 <thcipriani@> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
21:22 <mutante> <+icinga-wm> RECOVERY - Device not healthy -SMART- on restbase-dev1006 is OK: All metrics within thresholds. (T224260) [production]
21:18 <cscott@deploy1001> Finished deploy [parsoid/deploy@abd05ab]: Updating Parsoid to df1af404 (T227216, T226523, T226451) (duration: 18m 35s) [production]
21:16 <nuria@deploy1001> Finished deploy [analytics/refinery@58e64c1]: deploying refinery 0.0.95 (duration: 03m 54s) [production]
21:12 <nuria@deploy1001> Started deploy [analytics/refinery@58e64c1]: deploying refinery 0.0.95 [production]
21:03 <ppchelko@deploy1001> Finished deploy [restbase/deploy@7911f65]: Store PCS endpoints T222384 (duration: 18m 18s) [production]
21:00 <cscott@deploy1001> Started deploy [parsoid/deploy@abd05ab]: Updating Parsoid to df1af404 (T227216, T226523, T226451) [production]
20:45 <ppchelko@deploy1001> Started deploy [restbase/deploy@7911f65]: Store PCS endpoints T222384 [production]
20:39 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@2e2ce6c]: Update mobileapps to 1751a2e (duration: 04m 20s) [production]
20:38 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@bf28187]: Rerender PCS endpoints T222384 (duration: 01m 34s) [production]
20:36 <ppchelko@deploy1001> Started deploy [changeprop/deploy@bf28187]: Rerender PCS endpoints T222384 [production]
20:35 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@2e2ce6c]: Update mobileapps to 1751a2e [production]
20:12 <jeh> redirecting dumps.wikimedia.org back to labstore1007.wikimedia.org T224228 [production]
19:43 <ejegg> updated fundraising CiviCRM from 875ab97742 to 121feb5d53 [production]
19:08 <legoktm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable SecureLinkFixer on group0 wikis - T200751 (duration: 00m 55s) [production]
18:33 <cmjohnson1> moving cloudvirt107 to 10G rack T228691 [production]
18:19 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.15/includes/cache/localisation/LocalisationCache.php: 31d99eb381bc (duration: 00m 54s) [production]
18:15 <ejegg> updated payments-wiki from a28ad541ed to 70b432d309 [production]
18:13 <urandom> creating new restbase keyspaces -- T228804 [production]
18:12 <Krinkle> krinkle@deploy1001: extensions/CheckUser is dirty in php-1.34.0-wmf.15 [production]
17:14 <XioNoX> rollback failover master VIP of ae2.1202 inet6 away from cr1-eqiad - T226782 [production]
17:10 <XioNoX> Add mr1-codfw<->cr1/2-codfw vlan/link config on asw-a-codfw - T228112 [production]
16:44 <jijiki> Rolling puppet-enable and apache reload of jobrunners in codfw [production]
16:12 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org [production]
16:12 <bblack> re-pooling recdns on dns1001 via confctl - T226782 [production]
16:11 <bblack> lvs1014 - restore puppet and resolv.conf contents, restart pybal [production]
16:10 <bblack> dns1001 - restart recursor and re-enable puppet - T226782 [production]
16:07 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.15/includes/export/XmlDumpWriter.php: T228720 make XmlDumpwriter more resilient to blob store corruption (duration: 00m 55s) [production]
16:06 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.14/includes/export/XmlDumpWriter.php: T228720 make XmlDumpwriter more resilient to blob store corruption (duration: 00m 55s) [production]
15:59 <bblack> dns1001 - puppet disable, stop recursor service to kill anycast advert - T226782 [production]