1601-1650 of 10000 results (43ms)
2017-08-10 §
03:23 <bblack> cp1008, restbase-dev100[456] have puppet disabled, manual "rm /etc/ssh/userkeys/dzahn" [production]
03:20 <bblack> batched cumin puppet agent run on all hosts (not forced) [production]
02:28 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.11) (duration: 09m 12s) [production]
2017-08-09 §
22:06 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: Get baaack you broken CodeMirror! (duration: 00m 51s) [production]
21:20 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: Deploy CodeMirror https://gerrit.wikimedia.org/r/#/c/370943/ (duration: 00m 50s) [production]
21:11 <maxsem@tin> Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370904/ (duration: 00m 51s) [production]
21:05 <ppchelko@tin> Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 07m 30s) [production]
21:01 <godog> add 100G to carbon lv on graphite1003 and graphite2002 [production]
20:58 <ppchelko@tin> Started deploy [restbase/deploy@f97beeb]: Rollback on canary [production]
20:54 <ppchelko@tin> Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy (duration: 07m 12s) [production]
20:51 <marostegui> Remove m3 replication from dbstore1002 - T156758 [production]
20:47 <ppchelko@tin> Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy [production]
20:43 <ppchelko@tin> Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 (duration: 08m 05s) [production]
20:37 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2046 - T170662 (duration: 00m 49s) [production]
20:35 <ppchelko@tin> Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 [production]
20:22 <twentyafterfour> twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Actually sync all group1 wikis (except wikidata) to 1.30.0-wmf.13 refs T170631 [production]
18:57 <ppchelko@tin> Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 2 [production]
18:39 <marostegui> Stop replication on db2045 to fix duplicate keys - T151029 [production]
18:36 <urandom> Restarting Cassandra, restbase2001-b.codfw.wmnet (schema jankiness) [production]
18:27 <urandom> Restarting Cassandra, restbase2001-a.codfw.wmnet (schema jankiness) [production]
18:10 <ebernhardson> restart es on elastic1018 to test interleaved numa [production]
17:50 <marostegui> Add db2076 to tendril - T170662 [production]
17:41 <marostegui> Compress innodb on s6 on db2076 - T170662 [production]
17:31 <legoktm> changed mediawiki/vendor to fast forward only [production]
17:24 <ppchelko@tin> Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 01m 27s) [production]
17:23 <ppchelko@tin> Started deploy [restbase/deploy@f97beeb]: Rollback on canary [production]
16:35 <ppchelko@tin> Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation (duration: 04m 46s) [production]
16:30 <ppchelko@tin> Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation [production]
16:29 <urandom> T172384: Upgrading Cassandra in RESTBase dev to 3.11.0-wmf2 (patched to disable use of FastThreadLocal) [production]
16:15 <marostegui> Stop mysql on db2046 to copy its content to db2076 - T170662 [production]
16:14 <elukey> rolling restart of eventstream on scb hosts to deploy https://gerrit.wikimedia.org/r/370793 [production]
16:10 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2046 - T170662 (duration: 00m 50s) [production]
16:08 <gehel> config update on logstash, a few logs might be lost during restart - T172713 [production]
15:19 <XioNoX> removing old pfw related config from cr1-codfw - T171970 [production]
14:08 <mutante> purging req.urls with "^/resources" from varnish cluster, to fix redirect with cached 404 [production]
14:04 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db2075 to s5 - T170662 (duration: 00m 50s) [production]
14:03 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db2075 to s5 - T170662 (duration: 00m 51s) [production]
13:55 <marostegui> Reboot db2076 for maintenance - T170662 [production]
13:21 <marostegui> Stop replication on db2075 for maintenance - T170662 [production]
13:18 <ladsgroup@tin> Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats remove confirmed user group (T101983) (duration: 00m 51s) [production]
13:10 <gehel@tin> Finished deploy [wdqs/wdqs@6620e0f]: (no justification provided) (duration: 01m 34s) [production]
13:09 <gehel@tin> Started deploy [wdqs/wdqs@6620e0f]: (no justification provided) [production]
12:41 <gehel> restarting updater on wdqs1001 (real fix coming up soon [production]
10:20 <jynus> stopping dbstore2002's mysqls and cloning them to dbstore2001 [production]
08:39 <jynus> disable puppet on dbstore2001, about to be reimaged [production]
07:47 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1065, db1064, db1070 (duration: 01m 04s) [production]
07:12 <jynus> stopping replication in sync between db1069 and db1065, db1044, db1064, db1070 [production]
07:07 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool db1065, db1064, db1070 (duration: 00m 47s) [production]
06:26 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 51s) [production]
06:11 <jynus> stopping db1069 (s7) and db1079 replication in sync [production]