7201-7250 of 10000 results (105ms)
2020-01-20 §
09:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10219 and previous config saved to /var/cache/conftool/dbconfig/20200120-090336-marostegui.json [production]
09:01 <moritzm> installing Java security updates on an-conf* [production]
08:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10218 and previous config saved to /var/cache/conftool/dbconfig/20200120-085537-marostegui.json [production]
08:51 <marostegui> Upgrade db1139:3311 db1139:3316 [production]
08:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10217 and previous config saved to /var/cache/conftool/dbconfig/20200120-084908-marostegui.json [production]
08:44 <marostegui> Upgrade db1094 [production]
08:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1094', diff saved to https://phabricator.wikimedia.org/P10216 and previous config saved to /var/cache/conftool/dbconfig/20200120-084408-marostegui.json [production]
08:10 <marostegui> Compare data on db2085:3318 - T243148 [production]
08:07 <ema> powercycle cp3061 T238305 [production]
07:15 <marostegui> Remove partitions from revision on db2084:3314 T239453 [production]
07:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2084 T239453', diff saved to https://phabricator.wikimedia.org/P10215 and previous config saved to /var/cache/conftool/dbconfig/20200120-071513-marostegui.json [production]
07:10 <marostegui> Stop MySQL on es2020 to clone es2021 - T243052 [production]
06:09 <marostegui> Stop replication on db1107 [production]
06:08 <marostegui> Compress db1121 - T232446 [production]
06:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121, pool db1084 into vslow T232446', diff saved to https://phabricator.wikimedia.org/P10214 and previous config saved to /var/cache/conftool/dbconfig/20200120-060759-marostegui.json [production]
2020-01-19 §
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2085:3311, db2085:3318 T243148', diff saved to https://phabricator.wikimedia.org/P10210 and previous config saved to /var/cache/conftool/dbconfig/20200119-120236-marostegui.json [production]
11:20 <elukey> restart-php-fpm on mw2181 to rule out temporary php-related issues in codfw [production]
00:46 <cdanis> T238305 cp3053.mgmt /admin1-> racadm serveraction hardreset [production]
2020-01-18 §
11:35 <volans|off> upgraded spicerack to 0.0.29 on cumin hosts [production]
09:00 <dcausse> repool wdqs1007 (T242453) [production]
07:05 <marostegui> Remove partitions from enwiki.revision on db2085 T239453 [production]
04:15 <cdanis> cp3065.mgmt: /admin1-> racadm serveraction hardreset T238305 [production]
2020-01-17 §
21:56 <urandom> bootstrapping restbase2023-c — T243000 [production]
21:17 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
20:40 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
20:07 <urandom> bootstrapping restbase2023-b — T243000 [production]
20:01 <bblack> reset bgp peerings with gfiber on cr2-eqiad [production]
19:14 <mutante> gerrit - switching operations/debs/hhvm to READONLY mode and adding ARCHIVED to description (T237038) [production]
18:19 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
18:18 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
17:14 <urandom> bootstrapping restbase2023-a — T243000 [production]
16:33 <marostegui> Stop replication on db1107 [production]
16:25 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@938d253]: Move weekly elasticsearch transfer to airflow (duration: 00m 21s) [production]
16:25 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@938d253]: Move weekly elasticsearch transfer to airflow [production]
14:31 <urandom> bootstrapping restbase2022-c — T243000 [production]
14:09 <awight@deploy1001> Synchronized php-1.35.0-wmf.15/extensions/Cite: UBN backport: [[gerrit:565562|Fix for nested #tag:references and empty name (T242437)]] (duration: 00m 57s) [production]
14:03 <awight> beginning Friday deployment for UBN, T242437 [production]
13:38 <moritzm> masking squid3 on old URL downloaders T224551 [production]
13:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:35 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:35 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:35 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
12:55 <effie> Updgrade netmon* to to php 7.2.26 and restart - T241222 [production]
11:48 <moritzm> upgrading PHP 7.2 on netmon* (also apache restart for SSL update) [production]
11:13 <elukey> restart nginx on analitycs tool hosts to pick up openssl updates [production]
11:05 <moritzm> restarting apache on matomo1001 to pick up SSL updates [production]
11:04 <XioNoX> Running homer to remove decom cloud vlans in eqiad/codfw - T240670 [production]
11:01 <XioNoX> delete vlan cloud-instances1-b-eqiad from asw2-b-eqiad - T240670 [production]