1951-2000 of 10000 results (71ms)
2022-06-29 ยง
17:31 <sukhe> running puppet agent on centrallog2002 to finalize T310574 [production]
17:21 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]
17:21 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]
17:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T298560)', diff saved to https://phabricator.wikimedia.org/P30627 and previous config saved to /var/cache/conftool/dbconfig/20220629-172127-ladsgroup.json [production]
17:19 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host dumpsdata1007.eqiad.wmnet with OS bullseye [production]
17:18 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1007.eqiad.wmnet with OS bullseye [production]
17:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P30626 and previous config saved to /var/cache/conftool/dbconfig/20220629-170622-ladsgroup.json [production]
17:04 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host dumpsdata1007.eqiad.wmnet with OS bullseye [production]
17:00 <dduvall@deploy1002> helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply [production]
17:00 <dduvall@deploy1002> helmfile [eqiad] START helmfile.d/services/blubberoid: apply [production]
16:59 <dduvall@deploy1002> helmfile [codfw] DONE helmfile.d/services/blubberoid: apply [production]
16:59 <dduvall@deploy1002> helmfile [codfw] START helmfile.d/services/blubberoid: apply [production]
16:58 <dduvall@deploy1002> helmfile [staging] DONE helmfile.d/services/blubberoid: apply [production]
16:58 <dduvall@deploy1002> helmfile [staging] START helmfile.d/services/blubberoid: apply [production]
16:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P30625 and previous config saved to /var/cache/conftool/dbconfig/20220629-165117-ladsgroup.json [production]
16:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T298560)', diff saved to https://phabricator.wikimedia.org/P30624 and previous config saved to /var/cache/conftool/dbconfig/20220629-163612-ladsgroup.json [production]
16:22 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster restart to pickup swift-s3 plugin - bking@cumin1001 - T309648 [production]
15:55 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
15:54 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
15:54 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
15:53 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
15:51 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:808941|Increase weights on the language selector statement boosts (T307869)]] (expected to be a no-op) (duration: 03m 21s) [production]
15:25 <sukhe> upload anycast-healthchecker 0.8.2-1wm1 to apt.wm.o (bullseye) - T310574 [production]
14:49 <dancy@deploy1002> rebuilt and synchronized wikiversions files: Debugging [production]
14:48 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1032.eqiad.wmnet with OS buster [production]
14:40 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1033.eqiad.wmnet with OS buster [production]
14:36 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1029.eqiad.wmnet with OS buster [production]
14:30 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1034.eqiad.wmnet with OS buster [production]
14:25 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1030.eqiad.wmnet with OS buster [production]
14:24 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1028.eqiad.wmnet with OS buster [production]
14:14 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1034.eqiad.wmnet with reason: host reimage [production]
14:13 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1026.eqiad.wmnet with OS buster [production]
14:12 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host stat1010.eqiad.wmnet with OS bullseye [production]
14:11 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1033.eqiad.wmnet with reason: host reimage [production]
14:09 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1032.eqiad.wmnet with reason: host reimage [production]
14:06 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1029.eqiad.wmnet with reason: host reimage [production]
14:06 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1034.eqiad.wmnet with reason: host reimage [production]
14:06 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1033.eqiad.wmnet with reason: host reimage [production]
14:06 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1032.eqiad.wmnet with reason: host reimage [production]
14:04 <btullis@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host stat1010.eqiad.wmnet with OS bullseye [production]
14:04 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1028.eqiad.wmnet with reason: host reimage [production]
14:03 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephosd1030.eqiad.wmnet with reason: host reimage [production]
14:02 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1030.eqiad.wmnet with reason: host reimage [production]
14:01 <sukhe> sudo cumin -b 1 -s 5 'A:wikidough' 'run-puppet-agent -q' [production]
14:01 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1029.eqiad.wmnet with reason: host reimage [production]
14:00 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1028.eqiad.wmnet with reason: host reimage [production]
14:00 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1031.eqiad.wmnet with OS buster [production]
13:58 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1027.eqiad.wmnet with OS buster [production]
13:55 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcephosd1034.eqiad.wmnet with OS buster [production]
13:55 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcephosd1033.eqiad.wmnet with OS buster [production]