551-600 of 10000 results (56ms)
2019-05-19 §
09:59 <ebernhardson> eqiad psi elasticsearch high disk watermark to 89% to allow unallocated shard to initialize [production]
09:56 <ebernhardson> eqiad psi elasticsearch low disk watermark to 79% to allow unallocated shard to initialize [production]
08:13 <jijiki> varnish-backend-restart on cp1087 [production]
06:56 <mobrovac> bootstrap restbase1024-b - T219404 [production]
05:09 <marostegui> varnish-backend-restart on cp1081 [production]
2019-05-18 §
23:53 <bblack> rebooting lvs1015 for interface changes [production]
22:44 <bblack> imaging lvs1013-lvs1015 [production]
21:01 <bblack> depooling eqiad public front edge in authdns [production]
19:18 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.5/extensions/Collection/templates/CollectionSuggestTemplate.php: T223742 / 89bd434a21a745ec (duration: 00m 49s) [production]
19:16 <mobrovac> bootstrap restbase1024-a - T219404 [production]
18:50 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: T222146 / 9385b2dd66 (duration: 00m 50s) [production]
16:53 <mobrovac> bootstrap restbase1023-c - T219404 [production]
15:57 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.5/extensions/TimedMediaHandler/includes/handlers/WebMHandler/WebMHandler.php: T223445 / a9df59c59d7a30 (duration: 00m 51s) [production]
14:59 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: whitespace is srs (duration: 00m 49s) [production]
14:56 <reedy@deploy1001> Synchronized wmf-config/flaggedrevs.php: Copy in default config (duration: 01m 04s) [production]
13:51 <urandom> bootstrapping restbase1023-b - T219404 [production]
05:41 <mobrovac> bootstrap rb1023-a - T219404 [production]
02:37 <urandom> bootstrapping restbase1022-c - T219404 [production]
2019-05-17 §
23:55 <urandom> bootstrapping restbase1022-b - T219404 [production]
23:11 <foks> removing one file for legal compliance [production]
15:20 <hashar@deploy1001> Synchronized php-1.34.0-wmf.5/includes/api/ApiUpload.php: Revert "Always validate uploads over api" - T223448 (T222994 T223446) (duration: 01m 00s) [production]
15:18 <hashar> Deploying hotfix https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/510924/ . Should restore upload of large files on commons and other wikis #T223448 (poke T22994 T223446 ) [production]
14:51 <mobrovac> bootstrap restbase1022-a - T219404 [production]
14:43 <fsero> reenabling puppet puppet on mcrouter hosts for T221346, checks in place is there any alert for cert expiration and mcrouter this is the source :) [production]
14:17 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1098 & db1131 after maintenance (duration: 00m 49s) [production]
14:09 <fsero> second round of setting up cert check, disablign puppet on mcrouter hosts T221346 [production]
12:58 <mobrovac> bootstrap restbase1021-c - T219404 [production]
10:59 <mobrovac> bootstrap restbase1021-b - T219404 [production]
09:27 <godog> swift remove ms-be101[345] from rings - T220590 [production]
09:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1083 (duration: 00m 48s) [production]
08:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1083 (duration: 00m 49s) [production]
08:43 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1083 (duration: 00m 49s) [production]
08:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1083 (duration: 00m 49s) [production]
08:24 <fsero> reenabling puppet after reverting T221346 [production]
08:19 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1083 (duration: 00m 59s) [production]
07:57 <fsero> disabling puppet on mcrouter hosts for T221346 [production]
07:11 <marostegui> Compress s7 on labsdb1012 T222978 [production]
06:36 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2111 and db2113 into s5 T222772 (duration: 00m 49s) [production]
06:35 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2111 and db2113 into s5 T222772 (duration: 00m 50s) [production]
05:19 <marostegui> Stop MySQL on db1083 to clone db1134 [production]
05:17 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 00m 50s) [production]
05:00 <mobrovac> bootstrap 1021-a - T219404 [production]
2019-05-16 §
21:02 <Jeff_Green> authdns-update to switch payments.wikimedia.org back to eqiad cluster [production]
19:24 <onimisionipe> pooling elastic2038 - shards are properly balanced across nodes [production]
18:31 <onimisionipe> depooling elastic2038 to investigate more [production]
17:26 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:26 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:26 <jbond42> reboot ores1007-1009 [production]
17:15 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:15 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]