4951-5000 of 10000 results (69ms)
2019-05-20 §
10:18 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:18 <fsero> puppet reenabled certs renewed - T221346 [production]
10:08 <fsero> rolling over certs into mcrouter proxies codfw - T221346 [production]
10:03 <fsero> rolling over certs into mcrouter proxies eqiad - T221346 [production]
09:42 <marostegui> Remove db2036 from tendril and zarcillo - T223885 [production]
09:39 <marostegui> Stop MySQL on db2036 T223885 [production]
09:38 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2036, going to be decommissioned T223885 (duration: 00m 49s) [production]
09:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2036, going to be decommissioned T223885 (duration: 00m 49s) [production]
09:35 <fsero> rolling over new certs to all mcrouter hosts except proxys - T221346 [production]
09:26 <fsero> continue to rolling over new certs - T221346 [production]
09:01 <fsero> disabling puppet on mcrouter hosts for regenerating certs - T221346 [production]
08:49 <moritzm> installing atftpd security updates [production]
08:43 <mobrovac> bootstrap restbase1025-c - T219404 [production]
08:38 <moritzm> installing samba security updates [production]
08:36 <moritzm> installing ghostscript security updates on jessie [production]
08:25 <moritzm> installing cups-filter security updates on jessie (prerequisite for ghostscript security update) [production]
07:48 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1126 and db1134 (duration: 00m 48s) [production]
07:26 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2046 (duration: 00m 50s) [production]
06:25 <elukey> rebuild and upload memkeys 20181031-1 to stretch-wikimedia [production]
06:21 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1126 and db1134 (duration: 00m 49s) [production]
06:20 <elukey> upgrade memkeys to version 20181031-1 on all the mc* hosts (was deployied only on a few of them) - T208376 [production]
06:11 <mobrovac> bootstrap restbase1025-b - T219404 [production]
06:00 <elukey> powercycle analytics1071 - soft lockups error messages in the dmesg [production]
05:51 <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1010 [production]
05:42 <marostegui> Reload haproxy on dbproxy1010 and dbproxy1011 to repool labsdb1009 and restore original weights [production]
05:39 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1126 into s8, db1134 into s1 T222682 (duration: 00m 49s) [production]
05:38 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db1126 into s8, db1134 into s1 T222682 (duration: 00m 49s) [production]
05:12 <marostegui> Stop MySQL on db2046 [production]
05:11 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2046 (duration: 00m 50s) [production]
05:07 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2038 (duration: 00m 49s) [production]
05:01 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2038 (duration: 00m 55s) [production]
02:42 <cdanis> cdanis@cp1075.eqiad.wmnet ~ % sudo -i varnish-backend-restart [production]
2019-05-19 §
20:16 <ariel@deploy1001> Finished deploy [dumps/dumps@4febe0c]: for abstract dumps, skip any processing of pages not in main namespace (duration: 00m 03s) [production]
20:16 <ariel@deploy1001> Started deploy [dumps/dumps@4febe0c]: for abstract dumps, skip any processing of pages not in main namespace [production]
17:51 <mobrovac> bootstrap restbase1025-a - T219404 [production]
13:26 <ebernhardson@deploy1001> Synchronized wmf-config/ProductionServices.php: T223734: Depool cloudelastic100[12] (duration: 00m 49s) [production]
12:37 <reedy@deploy1001> Synchronized wmf-config/interwiki-labs.php: update (duration: 00m 57s) [production]
10:32 <reedy@deploy1001> Synchronized wikiversions-labs.json: T223770 (duration: 00m 48s) [production]
10:31 <reedy@deploy1001> Synchronized dblists/all-labs.dblist: T223770 (duration: 00m 51s) [production]
10:12 <mobrovac> bootstrap restbase1024-c - T219404 [production]
09:59 <ebernhardson> eqiad psi elasticsearch high disk watermark to 89% to allow unallocated shard to initialize [production]
09:56 <ebernhardson> eqiad psi elasticsearch low disk watermark to 79% to allow unallocated shard to initialize [production]
08:13 <jijiki> varnish-backend-restart on cp1087 [production]
06:56 <mobrovac> bootstrap restbase1024-b - T219404 [production]
05:09 <marostegui> varnish-backend-restart on cp1081 [production]
2019-05-18 §
23:53 <bblack> rebooting lvs1015 for interface changes [production]
22:44 <bblack> imaging lvs1013-lvs1015 [production]
21:01 <bblack> depooling eqiad public front edge in authdns [production]
19:18 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.5/extensions/Collection/templates/CollectionSuggestTemplate.php: T223742 / 89bd434a21a745ec (duration: 00m 49s) [production]
19:16 <mobrovac> bootstrap restbase1024-a - T219404 [production]