301-350 of 10000 results (69ms)
2022-10-14 ยง
13:57 <jclark@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
13:55 <jclark@cumin1001> START - Cookbook sre.dns.netbox [production]
12:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db1202 - Degraded RAID (T320786)', diff saved to https://phabricator.wikimedia.org/P35487 and previous config saved to /var/cache/conftool/dbconfig/20221014-120155-ladsgroup.json [production]
10:22 <godog> upgrade grafana to 8.5.14 [production]
10:15 <dcausse> Deployed patch for T320785 [production]
08:47 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts elastic[2028-2030].codfw.wmnet [production]
08:47 <ryankemper@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:46 <ryankemper@cumin2002> START - Cookbook sre.dns.netbox [production]
08:31 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts elastic[2028-2030].codfw.wmnet [production]
08:29 <moritzm> installing git security updates on buster [production]
08:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1008.eqiad.wmnet [production]
08:15 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:15 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts elastic[2025-2027].codfw.wmnet [production]
08:15 <ryankemper@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:14 <ryankemper@cumin2002> START - Cookbook sre.dns.netbox [production]
08:12 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:07 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti1008.eqiad.wmnet [production]
08:05 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1007.eqiad.wmnet [production]
08:05 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:02 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
07:57 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti1007.eqiad.wmnet [production]
07:57 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1006.eqiad.wmnet [production]
07:56 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:54 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
07:45 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti1006.eqiad.wmnet [production]
07:43 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1005.eqiad.wmnet [production]
07:43 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:41 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
07:37 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts elastic[2025-2027].codfw.wmnet [production]
07:36 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti1005.eqiad.wmnet [production]
07:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1008.eqiad.wmnet with reason: Remove from cluster for eventual decom [production]
07:16 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1008.eqiad.wmnet with reason: Remove from cluster for eventual decom [production]
06:46 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 7843 [production]
06:45 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 7843 [production]
06:37 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Not working well [production]
06:37 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Not working well [production]
03:42 <oblivian@cumin1001> dbctl commit (dc=all): 'depool db1143, lagging', diff saved to https://phabricator.wikimedia.org/P35485 and previous config saved to /var/cache/conftool/dbconfig/20221014-034223-oblivian.json [production]
02:24 <tstarling@deploy1002> Synchronized wmf-config: clean up deleted file (duration: 03m 46s) [production]
02:11 <ryankemper> T300943 Decom of elastic20[25-36] complete. Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/842547. This is done [production]
02:11 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts elastic[2034,2036].codfw.wmnet [production]
02:10 <ryankemper@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
02:05 <ryankemper@cumin2002> START - Cookbook sre.dns.netbox [production]
02:01 <ryankemper> T300943 Final batch of decom'ing `elastic20[25-36]` => already decommissioned rows A/B/C; starting final row D (corresponding to `203[4,6]`) [production]
01:59 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts elastic[2034,2036].codfw.wmnet [production]
01:42 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts elastic[2031-2033].codfw.wmnet [production]
01:42 <ryankemper@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
01:40 <ryankemper@cumin2002> START - Cookbook sre.dns.netbox [production]
01:32 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts elastic[2031-2033].codfw.wmnet [production]
01:26 <tstarling@deploy1002> Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 03m 36s) [production]
01:20 <tstarling@deploy1002> Synchronized wmf-config/UcfirstOverrides.php: for T292552, should have no effect at this stage (duration: 03m 46s) [production]