4451-4500 of 10000 results (96ms)
2022-10-14 ยง
14:22 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
14:21 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
14:19 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
14:17 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
14:09 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
14:09 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
14:07 <wm-bot2> Increased quotas by 1 instances - cookbook ran by dcaro@vulcanus [cloudinfra]
14:06 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
13:59 <jclark@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:57 <jclark@cumin1001> START - Cookbook sre.dns.netbox [production]
13:57 <jclark@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
13:55 <jclark@cumin1001> START - Cookbook sre.dns.netbox [production]
13:48 <James_F> Zuul: [css-sanitizer] Re-enable PHP 8.1 jobs for T311451 [releng]
12:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db1202 - Degraded RAID (T320786)', diff saved to https://phabricator.wikimedia.org/P35487 and previous config saved to /var/cache/conftool/dbconfig/20221014-120155-ladsgroup.json [production]
10:22 <godog> upgrade grafana to 8.5.14 [production]
10:15 <dcausse> Deployed patch for T320785 [production]
08:47 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts elastic[2028-2030].codfw.wmnet [production]
08:47 <ryankemper@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:46 <ryankemper@cumin2002> START - Cookbook sre.dns.netbox [production]
08:31 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts elastic[2028-2030].codfw.wmnet [production]
08:29 <moritzm> installing git security updates on buster [production]
08:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1008.eqiad.wmnet [production]
08:15 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:15 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts elastic[2025-2027].codfw.wmnet [production]
08:15 <ryankemper@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:14 <ryankemper@cumin2002> START - Cookbook sre.dns.netbox [production]
08:12 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:07 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti1008.eqiad.wmnet [production]
08:05 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1007.eqiad.wmnet [production]
08:05 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:02 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
07:57 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti1007.eqiad.wmnet [production]
07:57 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1006.eqiad.wmnet [production]
07:56 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:54 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
07:54 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (0cc020e) (T311466) - cookbook ran by taavi@runko [tools]
07:53 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (0cc020e) - cookbook ran by taavi@runko [toolsbeta]
07:45 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti1006.eqiad.wmnet [production]
07:43 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1005.eqiad.wmnet [production]
07:43 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:41 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
07:37 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts elastic[2025-2027].codfw.wmnet [production]
07:36 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti1005.eqiad.wmnet [production]
07:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1008.eqiad.wmnet with reason: Remove from cluster for eventual decom [production]
07:16 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1008.eqiad.wmnet with reason: Remove from cluster for eventual decom [production]
06:46 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 7843 [production]
06:45 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 7843 [production]
06:37 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Not working well [production]
06:37 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Not working well [production]
03:42 <oblivian@cumin1001> dbctl commit (dc=all): 'depool db1143, lagging', diff saved to https://phabricator.wikimedia.org/P35485 and previous config saved to /var/cache/conftool/dbconfig/20221014-034223-oblivian.json [production]