|
2017-08-09
ยง
|
| 21:01 |
<godog> |
add 100G to carbon lv on graphite1003 and graphite2002 |
[production] |
| 20:58 |
<ppchelko@tin> |
Started deploy [restbase/deploy@f97beeb]: Rollback on canary |
[production] |
| 20:54 |
<ppchelko@tin> |
Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy (duration: 07m 12s) |
[production] |
| 20:51 |
<marostegui> |
Remove m3 replication from dbstore1002 - T156758 |
[production] |
| 20:47 |
<ppchelko@tin> |
Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy |
[production] |
| 20:43 |
<ppchelko@tin> |
Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 (duration: 08m 05s) |
[production] |
| 20:37 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2046 - T170662 (duration: 00m 49s) |
[production] |
| 20:35 |
<ppchelko@tin> |
Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 |
[production] |
| 20:22 |
<twentyafterfour> |
twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Actually sync all group1 wikis (except wikidata) to 1.30.0-wmf.13 refs T170631 |
[production] |
| 18:57 |
<ppchelko@tin> |
Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 2 |
[production] |
| 18:39 |
<marostegui> |
Stop replication on db2045 to fix duplicate keys - T151029 |
[production] |
| 18:36 |
<urandom> |
Restarting Cassandra, restbase2001-b.codfw.wmnet (schema jankiness) |
[production] |
| 18:27 |
<urandom> |
Restarting Cassandra, restbase2001-a.codfw.wmnet (schema jankiness) |
[production] |
| 18:10 |
<ebernhardson> |
restart es on elastic1018 to test interleaved numa |
[production] |
| 17:50 |
<marostegui> |
Add db2076 to tendril - T170662 |
[production] |
| 17:41 |
<marostegui> |
Compress innodb on s6 on db2076 - T170662 |
[production] |
| 17:31 |
<legoktm> |
changed mediawiki/vendor to fast forward only |
[production] |
| 17:24 |
<ppchelko@tin> |
Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 01m 27s) |
[production] |
| 17:23 |
<ppchelko@tin> |
Started deploy [restbase/deploy@f97beeb]: Rollback on canary |
[production] |
| 16:35 |
<ppchelko@tin> |
Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation (duration: 04m 46s) |
[production] |
| 16:30 |
<ppchelko@tin> |
Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation |
[production] |
| 16:29 |
<urandom> |
T172384: Upgrading Cassandra in RESTBase dev to 3.11.0-wmf2 (patched to disable use of FastThreadLocal) |
[production] |
| 16:15 |
<marostegui> |
Stop mysql on db2046 to copy its content to db2076 - T170662 |
[production] |
| 16:14 |
<elukey> |
rolling restart of eventstream on scb hosts to deploy https://gerrit.wikimedia.org/r/370793 |
[production] |
| 16:10 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2046 - T170662 (duration: 00m 50s) |
[production] |
| 16:08 |
<gehel> |
config update on logstash, a few logs might be lost during restart - T172713 |
[production] |
| 15:19 |
<XioNoX> |
removing old pfw related config from cr1-codfw - T171970 |
[production] |
| 14:08 |
<mutante> |
purging req.urls with "^/resources" from varnish cluster, to fix redirect with cached 404 |
[production] |
| 14:04 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Add db2075 to s5 - T170662 (duration: 00m 50s) |
[production] |
| 14:03 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Add db2075 to s5 - T170662 (duration: 00m 51s) |
[production] |
| 13:55 |
<marostegui> |
Reboot db2076 for maintenance - T170662 |
[production] |
| 13:21 |
<marostegui> |
Stop replication on db2075 for maintenance - T170662 |
[production] |
| 13:18 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats remove confirmed user group (T101983) (duration: 00m 51s) |
[production] |
| 13:10 |
<gehel@tin> |
Finished deploy [wdqs/wdqs@6620e0f]: (no justification provided) (duration: 01m 34s) |
[production] |
| 13:09 |
<gehel@tin> |
Started deploy [wdqs/wdqs@6620e0f]: (no justification provided) |
[production] |
| 12:41 |
<gehel> |
restarting updater on wdqs1001 (real fix coming up soon |
[production] |
| 10:20 |
<jynus> |
stopping dbstore2002's mysqls and cloning them to dbstore2001 |
[production] |
| 08:39 |
<jynus> |
disable puppet on dbstore2001, about to be reimaged |
[production] |
| 07:47 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1065, db1064, db1070 (duration: 01m 04s) |
[production] |
| 07:12 |
<jynus> |
stopping replication in sync between db1069 and db1065, db1044, db1064, db1070 |
[production] |
| 07:07 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1065, db1064, db1070 (duration: 00m 47s) |
[production] |
| 06:26 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 51s) |
[production] |
| 06:11 |
<jynus> |
stopping db1069 (s7) and db1079 replication in sync |
[production] |
| 06:03 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1085, depool db1079 (duration: 00m 51s) |
[production] |
| 05:42 |
<jynus> |
stopping db1069 (s6) and db1085 replication in sync |
[production] |
| 05:32 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1060, depool db1085 (duration: 00m 50s) |
[production] |
| 04:56 |
<jynus> |
stopping db1069 (s2) and db1060 replication in sync |
[production] |
| 04:54 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 00m 58s) |
[production] |
| 04:48 |
<jynus> |
restarting pdfrender on scb1001, unresponsive |
[production] |
| 04:09 |
<jynus> |
powercycling mw2256 |
[production] |