1951-2000 of 10000 results (60ms)
2022-05-27 ยง
13:51 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:48 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop: sync [production]
13:48 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop: sync [production]
13:35 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:29 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:20 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
13:17 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
13:16 <jnuche@deploy1002> install-world aborted: (duration: 00m 25s) [production]
13:14 <jelto> run authdns-update for gitlab.wikimedia.org https://gerrit.wikimedia.org/r/c/operations/dns/+/800719 - T307142 [production]
13:08 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:04 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:04 <cmooney@cumin1001> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
13:04 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
12:52 <jnuche@deploy1002> install-world aborted: (duration: 03m 22s) [production]
12:43 <jelto> run authdns-update for gitlab-replica https://gerrit.wikimedia.org/r/c/operations/dns/+/800709 - T307142 [production]
12:36 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:35 <jnuche@deploy1002> install-world aborted: (duration: 01m 32s) [production]
12:27 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
12:25 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:23 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
12:17 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:14 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
12:09 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1070.eqiad.wmnet with OS bullseye [production]
11:55 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1069.eqiad.wmnet with OS bullseye [production]
11:55 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1070.eqiad.wmnet with reason: host reimage [production]
11:52 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1070.eqiad.wmnet with reason: host reimage [production]
11:43 <jnuche@deploy1002> install-world aborted: (duration: 00m 02s) [production]
11:38 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be1070.eqiad.wmnet with OS bullseye [production]
11:18 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: sync [production]
11:12 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: sync [production]
11:03 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1069.eqiad.wmnet with reason: host reimage [production]
11:01 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1071.eqiad.wmnet with OS bullseye [production]
11:00 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1069.eqiad.wmnet with reason: host reimage [production]
10:46 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1071.eqiad.wmnet with reason: host reimage [production]
10:45 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be1069.eqiad.wmnet with OS bullseye [production]
10:43 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1071.eqiad.wmnet with reason: host reimage [production]
10:38 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1068.eqiad.wmnet with OS bullseye [production]
10:24 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1068.eqiad.wmnet with reason: host reimage [production]
10:20 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1068.eqiad.wmnet with reason: host reimage [production]
10:12 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be1071.eqiad.wmnet with OS bullseye [production]
10:05 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be1068.eqiad.wmnet with OS bullseye [production]
09:48 <jelto> run authdns-update for gitlab-new https://gerrit.wikimedia.org/r/c/operations/dns/+/799334 - T307142 [production]
09:26 <kevinbazira@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
09:26 <kevinbazira@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
09:24 <Amir1> killed hewiki's refresh link suggestions job (T299021) [production]
09:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1127 (T298560)', diff saved to https://phabricator.wikimedia.org/P28630 and previous config saved to /var/cache/conftool/dbconfig/20220527-092233-ladsgroup.json [production]
09:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
09:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
08:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1175 (T298560)', diff saved to https://phabricator.wikimedia.org/P28629 and previous config saved to /var/cache/conftool/dbconfig/20220527-085819-ladsgroup.json [production]
08:58 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]