51-100 of 10000 results (55ms)
2020-03-31 ยง
14:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1091 after schema change', diff saved to https://phabricator.wikimedia.org/P10834 and previous config saved to /var/cache/conftool/dbconfig/20200331-141459-marostegui.json [production]
14:10 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:05 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw125[4-8].eqiad.wmnet [production]
13:31 <vgutierrez> Enable TLS Session tickets in eqsin - T245616 [production]
13:05 <XioNoX> update nat on pfw3-codfw - T248906 [production]
13:03 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:03 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:57 <jbond42> updating icu on presto-analytics-canary and hadoop-worker-canary [analytics]
12:49 <_joe_> switching all appserver canaries to envoy [production]
12:46 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:45 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:45 <marostegui> Deploy schema change on db1091 [production]
12:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1091 for schema change', diff saved to https://phabricator.wikimedia.org/P10833 and previous config saved to /var/cache/conftool/dbconfig/20200331-124452-marostegui.json [production]
12:35 <arturo> [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] (T248881) [admin]
12:34 <_joe_> transitioning mw1261 to envoy [production]
12:34 <arturo> [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2001-dev (T248881) [admin]
12:25 <arturo> [codfw1dev] installing neutron-openvswitch-agent on cloudnet200[2,3]-dev (T248881) [admin]
12:23 <vgutierrez> rolling upgrade of ATS to version 8.0.6-1wm5 - T248938 [production]
11:45 <arturo> [codfw1dev] rebooting cloudvirt2003-dev to pick up latest kernel update. Otherwise modprobe is confused trying to load modules and openvswitch won't start (T248881) [admin]
11:30 <Lucas_WMDE> EU SWAT done [production]
11:30 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:584874|Disable TwoColConflict talk page workflow (T230231)]], take II (duration: 00m 57s) [production]
11:29 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:584874|Disable TwoColConflict talk page workflow (T230231)]] (duration: 00m 58s) [production]
11:11 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:584574|Enable ContentTranslation in Lithuanian Wikipedia as a default tool (T248179)]], take II (duration: 00m 59s) [production]
11:10 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:584574|Enable ContentTranslation in Lithuanian Wikipedia as a default tool (T248179)]] (duration: 01m 00s) [production]
10:46 <_joe_> disabled puppet on canary appservers, potentially dangerous change ahead [production]
10:40 <arturo> [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2003-dev (T248881) [admin]
10:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1084 after schema change', diff saved to https://phabricator.wikimedia.org/P10831 and previous config saved to /var/cache/conftool/dbconfig/20200331-101953-marostegui.json [production]
10:09 <arturo> [codfw1dev] reboot cloudnet2003-dev into linux 4.9 (was using 4.14 from a testing operation in 2020-03-10) [admin]
10:03 <XioNoX> add BGP to AS41327 in AMS-IX [production]
09:49 <XioNoX> push homer diffs to mr1-eqsin [production]
09:36 <XioNoX> push homer diffs to mr1-eqiad [production]
09:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:10 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
09:09 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
09:05 <vgutierrez> upload trafficserver 8.0.5-1wm6 to apt.wm.o (buster) - T248938 [production]
09:00 <vgutierrez> depool & decommission cp2011 - T248950 [production]
08:52 <marostegui> Stop MySQL on db2093 for reimage to buster [production]
08:44 <vgutierrez> pool cp2035 - T248816 [production]
08:31 <mutante> signed puppet cert for planet1002.eqiad.wmnet [production]
08:29 <marostegui> Depool db1084 for schema change [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1084 for schema change', diff saved to https://phabricator.wikimedia.org/P10829 and previous config saved to /var/cache/conftool/dbconfig/20200331-082904-marostegui.json [production]
08:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1081 after schema change', diff saved to https://phabricator.wikimedia.org/P10828 and previous config saved to /var/cache/conftool/dbconfig/20200331-082711-marostegui.json [production]
08:17 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
08:08 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
08:01 <XioNoX> delete unused ROA for ARIN v4 prefixes - T235886 [production]
07:49 <vgutierrez@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
07:49 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:17 <vgutierrez> pool cp2034 - T248816 [production]