4851-4900 of 10000 results (67ms)
2020-01-20 §
09:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1129', diff saved to https://phabricator.wikimedia.org/P10224 and previous config saved to /var/cache/conftool/dbconfig/20200120-093603-marostegui.json [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1129', diff saved to https://phabricator.wikimedia.org/P10223 and previous config saved to /var/cache/conftool/dbconfig/20200120-092642-marostegui.json [production]
09:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1129', diff saved to https://phabricator.wikimedia.org/P10222 and previous config saved to /var/cache/conftool/dbconfig/20200120-091929-marostegui.json [production]
09:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1094', diff saved to https://phabricator.wikimedia.org/P10221 and previous config saved to /var/cache/conftool/dbconfig/20200120-090850-marostegui.json [production]
09:06 <marostegui> Upgrade db1129 [production]
09:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1129', diff saved to https://phabricator.wikimedia.org/P10220 and previous config saved to /var/cache/conftool/dbconfig/20200120-090617-marostegui.json [production]
09:05 <moritzm> restarting CAS to pick up Java security updates [production]
09:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10219 and previous config saved to /var/cache/conftool/dbconfig/20200120-090336-marostegui.json [production]
09:01 <moritzm> installing Java security updates on an-conf* [production]
08:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10218 and previous config saved to /var/cache/conftool/dbconfig/20200120-085537-marostegui.json [production]
08:51 <marostegui> Upgrade db1139:3311 db1139:3316 [production]
08:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10217 and previous config saved to /var/cache/conftool/dbconfig/20200120-084908-marostegui.json [production]
08:44 <marostegui> Upgrade db1094 [production]
08:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1094', diff saved to https://phabricator.wikimedia.org/P10216 and previous config saved to /var/cache/conftool/dbconfig/20200120-084408-marostegui.json [production]
08:10 <marostegui> Compare data on db2085:3318 - T243148 [production]
08:07 <ema> powercycle cp3061 T238305 [production]
07:15 <marostegui> Remove partitions from revision on db2084:3314 T239453 [production]
07:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2084 T239453', diff saved to https://phabricator.wikimedia.org/P10215 and previous config saved to /var/cache/conftool/dbconfig/20200120-071513-marostegui.json [production]
07:10 <marostegui> Stop MySQL on es2020 to clone es2021 - T243052 [production]
06:09 <marostegui> Stop replication on db1107 [production]
06:08 <marostegui> Compress db1121 - T232446 [production]
06:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121, pool db1084 into vslow T232446', diff saved to https://phabricator.wikimedia.org/P10214 and previous config saved to /var/cache/conftool/dbconfig/20200120-060759-marostegui.json [production]
2020-01-19 §
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2085:3311, db2085:3318 T243148', diff saved to https://phabricator.wikimedia.org/P10210 and previous config saved to /var/cache/conftool/dbconfig/20200119-120236-marostegui.json [production]
11:20 <elukey> restart-php-fpm on mw2181 to rule out temporary php-related issues in codfw [production]
00:46 <cdanis> T238305 cp3053.mgmt /admin1-> racadm serveraction hardreset [production]
2020-01-18 §
11:35 <volans|off> upgraded spicerack to 0.0.29 on cumin hosts [production]
09:00 <dcausse> repool wdqs1007 (T242453) [production]
07:05 <marostegui> Remove partitions from enwiki.revision on db2085 T239453 [production]
04:15 <cdanis> cp3065.mgmt: /admin1-> racadm serveraction hardreset T238305 [production]
2020-01-17 §
21:56 <urandom> bootstrapping restbase2023-c — T243000 [production]
21:17 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
20:40 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
20:07 <urandom> bootstrapping restbase2023-b — T243000 [production]
20:01 <bblack> reset bgp peerings with gfiber on cr2-eqiad [production]
19:14 <mutante> gerrit - switching operations/debs/hhvm to READONLY mode and adding ARCHIVED to description (T237038) [production]
18:19 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
18:18 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
17:14 <urandom> bootstrapping restbase2023-a — T243000 [production]
16:33 <marostegui> Stop replication on db1107 [production]
16:25 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@938d253]: Move weekly elasticsearch transfer to airflow (duration: 00m 21s) [production]
16:25 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@938d253]: Move weekly elasticsearch transfer to airflow [production]
14:31 <urandom> bootstrapping restbase2022-c — T243000 [production]
14:09 <awight@deploy1001> Synchronized php-1.35.0-wmf.15/extensions/Cite: UBN backport: [[gerrit:565562|Fix for nested #tag:references and empty name (T242437)]] (duration: 00m 57s) [production]
14:03 <awight> beginning Friday deployment for UBN, T242437 [production]
13:38 <moritzm> masking squid3 on old URL downloaders T224551 [production]
13:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:35 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:35 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]