3001-3050 of 10000 results (78ms)
2022-10-04 ยง
11:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading [production]
11:05 <jayme> published calico 3.23.3 debian packages in bullseye component/calico323 as well as corresponding docker images - T307943 [production]
11:04 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye [production]
10:58 <aborrero@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1006.eqiad.wmnet with OS bullseye [production]
10:58 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye [production]
10:56 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host sessionstore2001.codfw.wmnet with OS buster [production]
10:55 <aborrero@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1006.eqiad.wmnet with OS bullseye [production]
10:54 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye [production]
10:54 <hnowlan@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore2001.codfw.wmnet with OS buster [production]
10:53 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudnet1005.eqiad.wmnet with OS bullseye [production]
10:44 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 135158 [production]
10:43 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 135158 [production]
10:43 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 9119 [production]
10:42 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 9119 [production]
10:41 <moritzm> installing expat security updates [production]
09:59 <jmm@cumin2002> END (FAIL) - Cookbook sre.maps.roll-restart (exit_code=1) rolling restart_daemons on A:maps-codfw [production]
09:47 <btullis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply [production]
09:46 <btullis@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply [production]
09:46 <btullis@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply [production]
09:46 <btullis@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply [production]
09:45 <btullis@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply [production]
09:44 <btullis@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply [production]
09:44 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply [production]
09:43 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply [production]
09:42 <jayme> deployed istio-ingressgateway with additional envoy native metrics to wikikube codfw and eqiad [production]
09:40 <hnowlan@cumin2002> START - Cookbook sre.hosts.reimage for host sessionstore2001.codfw.wmnet with OS buster [production]
09:37 <jmm@cumin2002> START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-codfw [production]
09:36 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore2001.codfw.wmnet with reason: Prep for reimage [production]
09:36 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on sessionstore2001.codfw.wmnet with reason: Prep for reimage [production]
09:36 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 20 hosts [production]
09:35 <cgoubert@cumin1001> START - Cookbook sre.hosts.remove-downtime for 20 hosts [production]
09:35 <marostegui@cumin1001> dbctl commit (dc=all): 'db2178 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35338 and previous config saved to /var/cache/conftool/dbconfig/20221004-093530-root.json [production]
09:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db2178 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35337 and previous config saved to /var/cache/conftool/dbconfig/20221004-092025-root.json [production]
09:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading [production]
09:08 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading [production]
09:05 <marostegui@cumin1001> dbctl commit (dc=all): 'db2178 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35336 and previous config saved to /var/cache/conftool/dbconfig/20221004-090520-root.json [production]
08:56 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 20 hosts with reason: php7.2 removal [production]
08:55 <cgoubert@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 20 hosts with reason: php7.2 removal [production]
08:52 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading [production]
08:52 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading [production]
08:50 <marostegui@cumin1001> dbctl commit (dc=all): 'db2178 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35335 and previous config saved to /var/cache/conftool/dbconfig/20221004-085015-root.json [production]
08:35 <marostegui@cumin1001> dbctl commit (dc=all): 'db2178 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35334 and previous config saved to /var/cache/conftool/dbconfig/20221004-083511-root.json [production]
08:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db2178 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35333 and previous config saved to /var/cache/conftool/dbconfig/20221004-082005-root.json [production]
08:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: Upgrading [production]
08:16 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: Upgrading [production]
08:05 <marostegui@cumin1001> dbctl commit (dc=all): 'db2178 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35332 and previous config saved to /var/cache/conftool/dbconfig/20221004-080500-root.json [production]
08:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2181', diff saved to https://phabricator.wikimedia.org/P35331 and previous config saved to /var/cache/conftool/dbconfig/20221004-080338-root.json [production]
07:52 <moritzm> installing libdatetime-timezone-perl updates (catching up with latest timezone changes) [production]
07:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db2178 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35330 and previous config saved to /var/cache/conftool/dbconfig/20221004-074955-root.json [production]
07:36 <elukey@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: sync [production]