4451-4500 of 10000 results (66ms)
2019-05-24 §
11:25 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:25 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:23 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:23 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:23 <jbond@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
11:23 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:23 <jbond42> rebooting prometheous1004 [production]
10:56 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:56 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:56 <jbond42> rebooting prometheous2003 [production]
10:25 <jbond42> rebooting prometheous2004 [production]
10:25 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:25 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:09 <mobrovac> decommission restbase1010-b - T223976 [production]
07:33 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:33 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:32 <moritzm> rebooting labweb* for kernel security update [production]
07:05 <mobrovac> restbase-dev1006 force-stop the cassandra instances, fsync exception during decomm - T224260 [production]
06:47 <moritzm> bounced ferm on mw2286, wasn't correctly started after reboot [production]
06:45 <mobrovac> restbase-dev1006 decommission cass-b - T224260 [production]
06:43 <_joe_> disable notifications in icinga for restbase-dev1006 T224260 [production]
06:40 <mobrovac> restbase-dev1006 decommission cass-a - T224260 [production]
06:39 <mobrovac> restbase-dev1006 stop restbase - T224260 [production]
06:38 <mobrovac> restbase-dev1006 puppet disabled - T224260 [production]
06:26 <mobrovac@deploy1001> Finished deploy [restbase/deploy@b153f5d] (dev-cluster): Remove Parsoid fallback and rate-limit stashing (duration: 05m 41s) [production]
06:20 <mobrovac@deploy1001> Started deploy [restbase/deploy@b153f5d] (dev-cluster): Remove Parsoid fallback and rate-limit stashing [production]
06:20 <mobrovac@deploy1001> Finished deploy [restbase/deploy@b153f5d]: Remove Parsoid fallback and rate-limit stashing - T215956 T224055 (duration: 21m 30s) [production]
06:17 <marostegui> Stop MySQL on db2078:m1 to clone db2062 - T220170 [production]
06:08 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to new hosts T220170 (duration: 00m 48s) [production]
05:58 <mobrovac@deploy1001> Started deploy [restbase/deploy@b153f5d]: Remove Parsoid fallback and rate-limit stashing - T215956 T224055 [production]
05:35 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2062 from config T220170 (duration: 00m 48s) [production]
05:34 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2062 from config T220170 (duration: 00m 49s) [production]
05:30 <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1011 [production]
00:31 <XioNoX> remove lvs1001-5 bgp sessions from cr1/2-eqiad - T224223 [production]
00:27 <XioNoX> remove term protect-old-lvs-servers from cr1/2-eqiad - T224223 [production]
00:20 <urandom> decommissioning restbase1010-a -- T223976 [production]
00:04 <ebernhardson@deploy1001> Finished scap: php-1.34.0-wmf.6/extensions/CirrusSearch/includes/ T223738 Consider searching out of limits an error (duration: 21m 32s) [production]
2019-05-23 §
23:43 <ebernhardson@deploy1001> Started scap: php-1.34.0-wmf.6/extensions/CirrusSearch/includes/ T223738 Consider searching out of limits an error [production]
23:08 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup VII–X, InitialiseSettings (duration: 00m 48s) [production]
23:06 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Invariant config cleanup VII–X, CommonSettings (duration: 00m 47s) [production]
23:00 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup VI, InitialiseSettings (duration: 00m 47s) [production]
22:59 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Invariant config cleanup VI, CommonSettings (duration: 00m 48s) [production]
22:57 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup V, InitialiseSettings (duration: 00m 47s) [production]
22:56 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Invariant config cleanup V, CommonSettings (duration: 00m 47s) [production]
22:53 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup IV, InitialiseSettings (duration: 00m 47s) [production]
22:51 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Invariant config cleanup IV, CommonSettings (duration: 00m 48s) [production]
22:50 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup III, InitialiseSettings (duration: 00m 47s) [production]
22:47 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Invariant config cleanup III, CommonSettings (duration: 00m 48s) [production]
22:44 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup II, InitialiseSettings (duration: 00m 48s) [production]
22:43 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Invariant config cleanup II, CommonSettings (duration: 00m 48s) [production]