4751-4800 of 10000 results (87ms)
2023-03-01 §
09:57 <root@cumin1001> START - Cookbook sre.hosts.reimage for host ml-serve1001.eqiad.wmnet with OS bullseye [production]
09:51 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 8309 [production]
09:47 <root@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host ml-serve-ctrl1002.eqiad.wmnet with OS bullseye [production]
09:41 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 8309 [production]
09:39 <oblivian@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=appservers-ro,name=eqiad [production]
09:38 <moritzm> installing tiff security updates [production]
09:31 <oblivian@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=appservers-ro [production]
09:31 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve-ctrl1002.eqiad.wmnet with reason: host reimage [production]
09:30 <jnuche@deploy2002> Synchronized php: group1 wikis to 1.40.0-wmf.25 refs T325588 (duration: 07m 48s) [production]
09:26 <root@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve-ctrl1002.eqiad.wmnet with reason: host reimage [production]
09:23 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.25 refs T325588 [production]
09:15 <root@cumin1001> START - Cookbook sre.ganeti.reimage for host ml-serve-ctrl1002.eqiad.wmnet with OS bullseye [production]
09:15 <root@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host ml-serve-ctrl1001.eqiad.wmnet with OS bullseye [production]
08:58 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve-ctrl1001.eqiad.wmnet with reason: host reimage [production]
08:56 <root@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve-ctrl1001.eqiad.wmnet with reason: host reimage [production]
08:51 <moritzm> upgrade mw/eqiad to PHP 1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2 T330270 [production]
08:45 <root@cumin1001> START - Cookbook sre.ganeti.reimage for host ml-serve-ctrl1001.eqiad.wmnet with OS bullseye [production]
08:42 <root@cumin1001> START - Cookbook sre.k8s.upgrade-cluster Upgrade K8s version: Upgrade to k8s 1.23 [production]
08:41 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host ml-etcd1003.eqiad.wmnet with OS bullseye [production]
08:41 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host ml-etcd1002.eqiad.wmnet with OS bullseye [production]
08:40 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host ml-etcd1001.eqiad.wmnet with OS bullseye [production]
08:37 <root@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Emil Chetty out of all services on: 918 hosts [production]
08:36 <root@cumin2002> START - Cookbook sre.idm.logout Logging Emil Chetty out of all services on: 918 hosts [production]
08:35 <root@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Emil Chetty out of all services on: 1110 hosts [production]
08:34 <root@cumin2002> START - Cookbook sre.idm.logout Logging Emil Chetty out of all services on: 1110 hosts [production]
08:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-etcd1001.eqiad.wmnet with reason: host reimage [production]
08:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-etcd1003.eqiad.wmnet with reason: host reimage [production]
08:26 <jynus> stopping db2184 for testing mariadb 10.6 recovery workflow T319383 [production]
08:24 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-etcd1002.eqiad.wmnet with reason: host reimage [production]
08:21 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-etcd1001.eqiad.wmnet with reason: host reimage [production]
08:21 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-etcd1003.eqiad.wmnet with reason: host reimage [production]
08:21 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-etcd1002.eqiad.wmnet with reason: host reimage [production]
08:15 <jynus@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2184.codfw.wmnet with reason: 10.6 recovery [production]
08:14 <jynus@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2184.codfw.wmnet with reason: 10.6 recovery [production]
08:11 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host ml-etcd1001.eqiad.wmnet with OS bullseye [production]
08:11 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host ml-etcd1002.eqiad.wmnet with OS bullseye [production]
08:11 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host ml-etcd1003.eqiad.wmnet with OS bullseye [production]
08:10 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 13 hosts with reason: T330758 [production]
08:10 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 13 hosts with reason: T330758 [production]
06:14 <marostegui> Stop MySQL on db2094 T330828 [production]
05:37 <marostegui> Stop mysql on codfw sanitarium host db2095 (s2, s7, s6, s4) to clone db2187 T326596 [production]
05:37 <eileen> civicrm upgraded from ffc16d2d to fe2c06f6 [production]
00:25 <ejegg> civicrm rolled back from d199694e to ffc16d2d [production]
00:06 <zabe@deploy2002> Finished scap: T198673 (duration: 07m 25s) [production]
2023-02-28 §
23:58 <zabe@deploy2002> Started scap: T198673 [production]
23:45 <ejegg> civicrm upgraded from ffc16d2d to d199694e [production]
23:43 <zabe@deploy2002> Synchronized wmf-config/InitialiseSettings.php: T213295 (duration: 06m 56s) [production]
23:24 <mutante> miscweb2002 rm -rf /srv/org/wikimedia/design/blog/ - this has moved to /srv/org/wikimedia/design-blog but was not deleted in codfw - bringing both to the same state before switching design.wikimedia.org over T330090 [production]
23:20 <zabe@deploy2002> Finished scap: Backport for [[gerrit:893066|Drop custom testcommonswiki groups (T213295)]] (duration: 07m 57s) [production]
23:14 <zabe@deploy2002> zabe: Backport for [[gerrit:893066|Drop custom testcommonswiki groups (T213295)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]