5551-5600 of 10000 results (84ms)
2022-12-07 ยง
15:40 <sukhe@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin2002" [production]
15:39 <jiji@deploy1002> jiji and jiji: Backport for [[gerrit:865121|ProductionServices: Use redis_misc servers for LockManager (4/6) (T267581)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
15:37 <jiji@deploy1002> Started scap: Backport for [[gerrit:865121|ProductionServices: Use redis_misc servers for LockManager (4/6) (T267581)]] [production]
15:36 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5003.wikimedia.org with OS buster [production]
15:36 <sukhe@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin2002" [production]
15:34 <sukhe@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin2002" [production]
15:24 <btullis@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. [production]
15:16 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5005.eqsin.wmnet with reason: host reimage [production]
15:13 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5005.eqsin.wmnet with reason: host reimage [production]
15:11 <btullis@cumin1001> START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. [production]
15:06 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns5003.wikimedia.org with reason: host reimage [production]
15:03 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns5003.wikimedia.org with reason: host reimage [production]
14:56 <krinkle@deploy1002> Finished deploy [performance/navtiming@6caa033]: (no justification provided) (duration: 00m 07s) [production]
14:56 <krinkle@deploy1002> Started deploy [performance/navtiming@6caa033]: (no justification provided) [production]
14:44 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host lvs5005.eqsin.wmnet with OS buster [production]
14:38 <XioNoX> draining Arelion eqiad-codfw circuit for optic replacement [production]
14:32 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host dns5003.wikimedia.org with OS buster [production]
14:28 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dns5002.wikimedia.org [production]
14:28 <sukhe@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:28 <sukhe@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
14:27 <moritzm> restarting ntpd [production]
14:26 <sukhe@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
14:24 <sukhe@cumin2002> START - Cookbook sre.dns.netbox [production]
14:20 <sukhe@cumin2002> START - Cookbook sre.hosts.decommission for hosts dns5002.wikimedia.org [production]
14:17 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dns5002.wikimedia.org with reason: downtimed, to be depooled [production]
14:16 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dns5002.wikimedia.org with reason: downtimed, to be depooled [production]
14:03 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on an-tool1005.eqiad.wmnet with reason: redeploying an-tool1005 as bullseye [production]
14:02 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on an-tool1005.eqiad.wmnet with reason: redeploying an-tool1005 as bullseye [production]
13:42 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 (duration: 07m 45s) [production]
13:34 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 [production]
13:25 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Created cloudcumin instances - volans@cumin1001" [production]
13:22 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Created cloudcumin instances - volans@cumin1001" [production]
13:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1206', diff saved to https://phabricator.wikimedia.org/P42443 and previous config saved to /var/cache/conftool/dbconfig/20221207-131858-marostegui.json [production]
13:09 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudcumin1001.eqiad.wmnet with reason: First installation [production]
13:09 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on cloudcumin1001.eqiad.wmnet with reason: First installation [production]
12:37 <moritzm> upgrading mwmaint servers to PHP 7.4.33 [production]
12:33 <moritzm> upgrading deployment servers to PHP 7.4.33 [production]
12:32 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudcumin2001.codfw.wmnet with reason: First installation [production]
12:32 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on cloudcumin2001.codfw.wmnet with reason: First installation [production]
12:30 <moritzm> upgrading cloudweb to PHP 7.4.33 [production]
12:28 <volans@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 4:00:00 on cloudcumin2001.codfw.wmnet with reason: First installation [production]
12:28 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on cloudcumin2001.codfw.wmnet with reason: First installation [production]
12:19 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply [production]
12:18 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply [production]
12:17 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply [production]
12:16 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply [production]
12:16 <oblivian@deploy1002> helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply [production]
12:15 <oblivian@deploy1002> helmfile [staging] START helmfile.d/services/shellbox-timeline: apply [production]
12:15 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
12:15 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]