|
2019-05-24
§
|
| 11:23 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:23 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 11:23 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:23 |
<jbond42> |
rebooting prometheous1004 |
[production] |
| 10:56 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:56 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:56 |
<jbond42> |
rebooting prometheous2003 |
[production] |
| 10:25 |
<jbond42> |
rebooting prometheous2004 |
[production] |
| 10:25 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:25 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:09 |
<mobrovac> |
decommission restbase1010-b - T223976 |
[production] |
| 07:33 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:33 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:32 |
<moritzm> |
rebooting labweb* for kernel security update |
[production] |
| 07:05 |
<mobrovac> |
restbase-dev1006 force-stop the cassandra instances, fsync exception during decomm - T224260 |
[production] |
| 06:47 |
<moritzm> |
bounced ferm on mw2286, wasn't correctly started after reboot |
[production] |
| 06:45 |
<mobrovac> |
restbase-dev1006 decommission cass-b - T224260 |
[production] |
| 06:43 |
<_joe_> |
disable notifications in icinga for restbase-dev1006 T224260 |
[production] |
| 06:40 |
<mobrovac> |
restbase-dev1006 decommission cass-a - T224260 |
[production] |
| 06:39 |
<mobrovac> |
restbase-dev1006 stop restbase - T224260 |
[production] |
| 06:38 |
<mobrovac> |
restbase-dev1006 puppet disabled - T224260 |
[production] |
| 06:26 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@b153f5d] (dev-cluster): Remove Parsoid fallback and rate-limit stashing (duration: 05m 41s) |
[production] |
| 06:20 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@b153f5d] (dev-cluster): Remove Parsoid fallback and rate-limit stashing |
[production] |
| 06:20 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@b153f5d]: Remove Parsoid fallback and rate-limit stashing - T215956 T224055 (duration: 21m 30s) |
[production] |
| 06:17 |
<marostegui> |
Stop MySQL on db2078:m1 to clone db2062 - T220170 |
[production] |
| 06:08 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to new hosts T220170 (duration: 00m 48s) |
[production] |
| 05:58 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@b153f5d]: Remove Parsoid fallback and rate-limit stashing - T215956 T224055 |
[production] |
| 05:35 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2062 from config T220170 (duration: 00m 48s) |
[production] |
| 05:34 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2062 from config T220170 (duration: 00m 49s) |
[production] |
| 05:30 |
<marostegui> |
Reload haproxy on dbproxy1010 to repool labsdb1011 |
[production] |
| 00:31 |
<XioNoX> |
remove lvs1001-5 bgp sessions from cr1/2-eqiad - T224223 |
[production] |
| 00:27 |
<XioNoX> |
remove term protect-old-lvs-servers from cr1/2-eqiad - T224223 |
[production] |
| 00:20 |
<urandom> |
decommissioning restbase1010-a -- T223976 |
[production] |
| 00:04 |
<ebernhardson@deploy1001> |
Finished scap: php-1.34.0-wmf.6/extensions/CirrusSearch/includes/ T223738 Consider searching out of limits an error (duration: 21m 32s) |
[production] |
|
2019-05-23
§
|
| 23:43 |
<ebernhardson@deploy1001> |
Started scap: php-1.34.0-wmf.6/extensions/CirrusSearch/includes/ T223738 Consider searching out of limits an error |
[production] |
| 23:08 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup VII–X, InitialiseSettings (duration: 00m 48s) |
[production] |
| 23:06 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Invariant config cleanup VII–X, CommonSettings (duration: 00m 47s) |
[production] |
| 23:00 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup VI, InitialiseSettings (duration: 00m 47s) |
[production] |
| 22:59 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Invariant config cleanup VI, CommonSettings (duration: 00m 48s) |
[production] |
| 22:57 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup V, InitialiseSettings (duration: 00m 47s) |
[production] |
| 22:56 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Invariant config cleanup V, CommonSettings (duration: 00m 47s) |
[production] |
| 22:53 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup IV, InitialiseSettings (duration: 00m 47s) |
[production] |
| 22:51 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Invariant config cleanup IV, CommonSettings (duration: 00m 48s) |
[production] |
| 22:50 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup III, InitialiseSettings (duration: 00m 47s) |
[production] |
| 22:47 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Invariant config cleanup III, CommonSettings (duration: 00m 48s) |
[production] |
| 22:44 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup II, InitialiseSettings (duration: 00m 48s) |
[production] |
| 22:43 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Invariant config cleanup II, CommonSettings (duration: 00m 48s) |
[production] |
| 22:39 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Invariant config cleanup I, InitialiseSettings (duration: 00m 47s) |
[production] |
| 22:37 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Invariant config cleanup I, CommonSettings (duration: 00m 48s) |
[production] |
| 22:32 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Stop setting wmgUseClusterSquid, never varied, no longer used (duration: 00m 48s) |
[production] |