1351-1400 of 10000 results (75ms)
2022-07-13 §
05:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db2162 in s8 T311493', diff saved to https://phabricator.wikimedia.org/P31023 and previous config saved to /var/cache/conftool/dbconfig/20220713-051239-marostegui.json [production]
2022-07-12 §
22:32 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2039.codfw.wmnet with OS bullseye [production]
22:19 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@45ae36d]: subgraph_and_query_metrics: Drop wiki from sparql event partition spec (duration: 02m 04s) [production]
22:17 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@45ae36d]: subgraph_and_query_metrics: Drop wiki from sparql event partition spec [production]
22:15 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2039.codfw.wmnet with reason: host reimage [production]
22:11 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2039.codfw.wmnet with reason: host reimage [production]
21:50 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2039.codfw.wmnet with OS bullseye [production]
20:28 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2038.codfw.wmnet with OS bullseye [production]
20:11 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2038.codfw.wmnet with reason: host reimage [production]
20:07 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2038.codfw.wmnet with reason: host reimage [production]
19:49 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2038.codfw.wmnet with OS bullseye [production]
19:38 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2038.codfw.wmnet with OS bullseye [production]
19:35 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2038.codfw.wmnet with OS bullseye [production]
19:34 <bking@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host elastic2038.codfw.wmnet with OS bullseye [production]
19:31 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2038.codfw.wmnet with OS bullseye [production]
19:31 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2038.codfw.wmnet with OS bullseye [production]
19:31 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2038.codfw.wmnet with OS bullseye [production]
19:30 <bking@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host elastic2038.codfw.wmnet with OS bullseye [production]
19:27 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2038.codfw.wmnet with OS bullseye [production]
19:26 <krinkle@deploy1002> Synchronized wmf-config/InitialiseSettings.php: I3071c009c (2) (duration: 02m 45s) [production]
19:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:20 <krinkle@deploy1002> Synchronized wmf-config/InitialiseSettings.php: I3071c009c (duration: 03m 09s) [production]
19:20 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:20 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic2038.codfw.wmnet with reason: firmware update T312298 [production]
19:19 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on elastic2038.codfw.wmnet with reason: firmware update T312298 [production]
19:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:13 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for elastic1065.eqiad.wmnet [production]
19:13 <bking@cumin1001> START - Cookbook sre.hosts.remove-downtime for elastic1065.eqiad.wmnet [production]
18:54 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:53 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:53 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:26 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:24 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:22 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:18 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2037.codfw.wmnet with OS bullseye [production]
16:59 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2037.codfw.wmnet with reason: host reimage [production]
16:55 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2037.codfw.wmnet with reason: host reimage [production]
16:55 <bblack> codfw dns repooled for front edge traffic [production]
16:50 <herron> ran failed codfw puppet agents [production]
16:47 <mutante> doc1002 - systemctl reset-failed [production]
16:45 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=restbase1026.eqiad.wmnet [production]
16:36 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2037.codfw.wmnet with OS bullseye [production]
16:19 <mutante> rebooting mwdebug2001 via ganeti2022 [production]
16:15 <cwhite> repair networking on people2002 [production]
16:11 <cwhite> repair networking on puppetdb2002 [production]
16:10 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=restbase1026.eqiad.wmnet [production]
16:05 <mutante> parse200[1-3] - restarted ferm [production]