2751-2800 of 10000 results (111ms)
2023-11-09 ยง
20:55 <otto@deploy2002> helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply [production]
20:55 <otto@deploy2002> helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply [production]
20:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2146 T350916', diff saved to https://phabricator.wikimedia.org/P53248 and previous config saved to /var/cache/conftool/dbconfig/20231109-205445-root.json [production]
20:54 <brouberol@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1005.eqiad.wmnet with OS bullseye [production]
20:54 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1110.eqiad.wmnet with OS bullseye [production]
20:54 <otto@deploy2002> helmfile [staging] START helmfile.d/services/eventstreams-internal: apply [production]
20:52 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P53247 and previous config saved to /var/cache/conftool/dbconfig/20231109-205238-arnaudb.json [production]
20:45 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.3.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
20:45 <cmooney@cumin1001> START - Cookbook sre.dns.wipe-cache 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.3.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
20:41 <topranks> change anycast gw type to single-IP on ssw1-aX-codfw for sandbox1-a-codfw vlan (T350579) [production]
20:37 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P53246 and previous config saved to /var/cache/conftool/dbconfig/20231109-203731-arnaudb.json [production]
20:35 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1110.eqiad.wmnet with reason: host reimage [production]
20:32 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1110.eqiad.wmnet with reason: host reimage [production]
20:32 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1108.eqiad.wmnet with OS bullseye [production]
20:32 <brouberol@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage [production]
20:29 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:29 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update DNS entries for ssw1-aX-codfw xlink IPs. - cmooney@cumin1001" [production]
20:28 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update DNS entries for ssw1-aX-codfw xlink IPs. - cmooney@cumin1001" [production]
20:28 <brouberol@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage [production]
20:25 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
20:22 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T348183)', diff saved to https://phabricator.wikimedia.org/P53245 and previous config saved to /var/cache/conftool/dbconfig/20231109-202225-arnaudb.json [production]
20:18 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2117 (T348183)', diff saved to https://phabricator.wikimedia.org/P53244 and previous config saved to /var/cache/conftool/dbconfig/20231109-201830-arnaudb.json [production]
20:18 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
20:18 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
20:17 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp1110.eqiad.wmnet with OS bullseye [production]
20:17 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1110.eqiad.wmnet with OS bullseye [production]
20:16 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
20:16 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
20:15 <topranks> resetting asw-a-codfw et-2/0/52 to shift traffic away from ssw1-a8-codfw (T347191) [production]
20:14 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
20:14 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
20:14 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1231 (T348183)', diff saved to https://phabricator.wikimedia.org/P53243 and previous config saved to /var/cache/conftool/dbconfig/20231109-201409-arnaudb.json [production]
20:13 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1108.eqiad.wmnet with reason: host reimage [production]
20:13 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp1110.eqiad.wmnet with OS bullseye [production]
20:12 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1110.eqiad.wmnet with OS bullseye [production]
20:12 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw-a-codfw,ssw1-a8-codfw,ssw1-a8-codfw.mgmt with reason: Adjust vlans trunked to asw-a-codfw from ssw1-a8-codfw T347191 [production]
20:12 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on asw-a-codfw,ssw1-a8-codfw,ssw1-a8-codfw.mgmt with reason: Adjust vlans trunked to asw-a-codfw from ssw1-a8-codfw T347191 [production]
20:10 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1108.eqiad.wmnet with reason: host reimage [production]
20:08 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1107.eqiad.wmnet with OS bullseye [production]
20:06 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp1110.eqiad.wmnet with OS bullseye [production]
20:06 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1110.eqiad.wmnet with OS bullseye [production]
19:59 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp1110.eqiad.wmnet with OS bullseye [production]
19:59 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P53242 and previous config saved to /var/cache/conftool/dbconfig/20231109-195903-arnaudb.json [production]
19:55 <volans@cumin1001> START - Cookbook sre.hosts.reimage for host cp1108.eqiad.wmnet with OS bullseye [production]
19:54 <fnegri@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1060.eqiad.wmnet with OS bookworm [production]
19:52 <fnegri@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1057.eqiad.wmnet with OS bookworm [production]
19:51 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs2012.codfw.wmnet: Applying JVM security upgrade - eevans@cumin1001 [production]
19:50 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1107.eqiad.wmnet with reason: host reimage [production]
19:50 <fnegri@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1026.eqiad.wmnet with OS bookworm [production]
19:50 <fnegri@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1059.eqiad.wmnet with OS bookworm [production]