1001-1050 of 10000 results (21ms)
2020-11-16 §
14:06 <marostegui> Restart pc1007's mysql T266483 [production]
14:06 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool pc1007 and place pc1010 instead of it T266483 (duration: 01m 00s) [production]
13:23 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) [production]
13:00 <kormat> running schema change against s1 in codfw T259831 [production]
12:59 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:59 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:43 <moritzm> installing tcpdump security updates [production]
12:35 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:35 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:25 <hnowlan@cumin1001> START - Cookbook sre.cassandra.roll-restart [production]
12:25 <hnowlan> roll-restarting restbase-codfw [production]
12:24 <hnowlan@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
12:10 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
12:10 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:49 <hnowlan> roll restarting sessionstore for java updates [production]
11:49 <hnowlan@cumin1001> START - Cookbook sre.cassandra.roll-restart [production]
11:13 <moritzm> installing poppler security updates [production]
10:46 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:46 <klausman@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:45 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:45 <dcaro@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:44 <dcaro@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
10:44 <dcaro@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:31 <gehel@cumin2001> END (FAIL) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=99) [production]
09:31 <gehel@cumin2001> START - Cookbook sre.elasticsearch.force-shard-allocation [production]
08:39 <godog> centrallog1001 move invalid config /etc/logrotate.d/logrotate-debug to /etc [production]
08:35 <moritzm> installing codemirror-js security updates [production]
08:32 <XioNoX> asw-c-codfw> request system power-off member 7 - T267865 [production]
08:24 <joal@deploy1001> Finished deploy [analytics/refinery@3df51cb] (thin): Analytics special train for webrequest table update THIN [analytics/refinery@3df51cb] (duration: 00m 07s) [production]
08:23 <joal@deploy1001> Started deploy [analytics/refinery@3df51cb] (thin): Analytics special train for webrequest table update THIN [analytics/refinery@3df51cb] [production]
08:23 <joal@deploy1001> Finished deploy [analytics/refinery@3df51cb]: Analytics special train for webrequest table update [analytics/refinery@3df51cb] (duration: 10m 09s) [production]
08:13 <joal@deploy1001> Started deploy [analytics/refinery@3df51cb]: Analytics special train for webrequest table update [analytics/refinery@3df51cb] [production]
08:08 <XioNoX> asw-c-codfw> request system power-off member 7 - T267865 [production]
06:35 <marostegui> Stop replication on s3 codfw master (db2105) for MCR schema change deployment T238966 [production]
06:14 <marostegui> Stop MySQL on es1018, es1015, es1019 to clone es1032, es1033, es1034 - T261717 [production]
06:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1018, es1015, es1019 - T261717', diff saved to https://phabricator.wikimedia.org/P13262 and previous config saved to /var/cache/conftool/dbconfig/20201116-060624-marostegui.json [production]
06:02 <marostegui> Restart mysql on db1115 (tendril/dbtree) due to memory usage [production]
00:55 <shdubsh> re-applied mask to kafka and kafka-mirror-main-eqiad_to_main-codfw@0 on kafka-main2003 and disabled puppet to prevent restart - T267865 [production]
00:19 <elukey> run 'systemctl mask kafka' and 'systemctl mask kafka-mirror-main-eqiad_to_main-codfw@0' on kafka-main2003 (for the brief moment when it was up) to avoid purged issues - T267865 [production]
00:09 <elukey> sudo cumin 'cp2028* or cp2036* or cp2039* or cp4022* or cp4025* or cp4028* or cp4031*' 'systemctl restart purged' -b 3 - T267865 [production]
2020-11-15 §
22:10 <cdanis> restart some purgeds in ulsfo as well T267865 T267867 [production]
22:03 <cdanis> T267867 T267865 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕔🍺 sudo cumin -b2 -s10 'A:cp and A:codfw' 'systemctl restart purged' [production]
14:00 <cdanis> powercycling ms-be1022 via mgmt [production]
11:21 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:21 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:12 <vgutierrez> depooling lvs2007, lvs2010 taking over text traffic on codfw - T267865 [production]
10:00 <elukey> cumin 'cp2042* or cp2036* or cp2039*' 'systemctl restart purged' -b 1 [production]
09:57 <elukey> restart purged on cp4028 (consumer stuck due to kafka-main2003 down) [production]
09:55 <elukey> restart purged on cp4025 (consumer stuck due to kafka-main2003 down) [production]
09:53 <elukey> restart purged on cp4031 (consumer stuck due to kafka-main2003 down) [production]