1-50 of 10000 results (92ms)
2024-07-03 ยง
23:47 <tzatziki> removing 11 files for legal compliance [production]
23:23 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1196 (T364069)', diff saved to https://phabricator.wikimedia.org/P65776 and previous config saved to /var/cache/conftool/dbconfig/20240703-232302-marostegui.json [production]
23:22 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
23:22 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
23:22 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
23:22 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
23:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1195 (T364069)', diff saved to https://phabricator.wikimedia.org/P65775 and previous config saved to /var/cache/conftool/dbconfig/20240703-232221-marostegui.json [production]
23:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T352010)', diff saved to https://phabricator.wikimedia.org/P65774 and previous config saved to /var/cache/conftool/dbconfig/20240703-232154-ladsgroup.json [production]
23:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P65773 and previous config saved to /var/cache/conftool/dbconfig/20240703-230713-marostegui.json [production]
23:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P65772 and previous config saved to /var/cache/conftool/dbconfig/20240703-230646-ladsgroup.json [production]
22:52 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P65771 and previous config saved to /var/cache/conftool/dbconfig/20240703-225206-marostegui.json [production]
22:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P65770 and previous config saved to /var/cache/conftool/dbconfig/20240703-225139-ladsgroup.json [production]
22:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1195 (T364069)', diff saved to https://phabricator.wikimedia.org/P65769 and previous config saved to /var/cache/conftool/dbconfig/20240703-223659-marostegui.json [production]
22:36 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T352010)', diff saved to https://phabricator.wikimedia.org/P65768 and previous config saved to /var/cache/conftool/dbconfig/20240703-223632-ladsgroup.json [production]
22:36 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host parsoidtest1001.eqiad.wmnet with OS bullseye [production]
21:56 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host parsoidtest1001.eqiad.wmnet with OS bullseye [production]
21:40 <ryankemper@cumin2002> START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [production]
21:40 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster [production]
21:35 <ryankemper@cumin2002> START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [production]
20:13 <cjming> end of UTC late backport window [production]
20:11 <cjming@deploy1002> Finished scap: Backport for [[gerrit:1051756|Remove QuickSurvey for Automoderator patroller workstream survey (T362969)]] (duration: 08m 22s) [production]
20:10 <cmooney@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host sretest2002.codfw.wmnet with OS bookworm [production]
20:06 <cjming@deploy1002> kgraessle, cjming: Continuing with sync [production]
20:05 <cjming@deploy1002> kgraessle, cjming: Backport for [[gerrit:1051756|Remove QuickSurvey for Automoderator patroller workstream survey (T362969)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:05 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply [production]
20:04 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow: apply [production]
20:04 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply [production]
20:03 <cjming@deploy1002> Started scap sync-world: Backport for [[gerrit:1051756|Remove QuickSurvey for Automoderator patroller workstream survey (T362969)]] [production]
19:56 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:55 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
19:54 <cmooney@cumin1002> START - Cookbook sre.hosts.reimage for host sretest2002.codfw.wmnet with OS bookworm [production]
19:49 <cmooney@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host sretest2002.codfw.wmnet with OS bookworm [production]
19:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1195 (T364069)', diff saved to https://phabricator.wikimedia.org/P65766 and previous config saved to /var/cache/conftool/dbconfig/20240703-194055-marostegui.json [production]
19:40 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance [production]
19:40 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance [production]
19:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T364069)', diff saved to https://phabricator.wikimedia.org/P65765 and previous config saved to /var/cache/conftool/dbconfig/20240703-194033-marostegui.json [production]
19:30 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow: apply [production]
19:30 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply [production]
19:25 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P65761 and previous config saved to /var/cache/conftool/dbconfig/20240703-192526-marostegui.json [production]
19:25 <cmooney@cumin1002> START - Cookbook sre.hosts.reimage for host sretest2002.codfw.wmnet with OS bookworm [production]
19:24 <cmooney@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2002.codfw.wmnet with OS bookworm [production]
19:19 <cmooney@cumin1002> START - Cookbook sre.hosts.reimage for host sretest2002.codfw.wmnet with OS bookworm [production]
19:16 <cmooney@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host sretest2002.codfw.wmnet with OS bookworm [production]
19:12 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@d773cac]: (no justification provided) (duration: 00m 33s) [production]
19:11 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@d773cac]: (no justification provided) [production]
19:10 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P65760 and previous config saved to /var/cache/conftool/dbconfig/20240703-191019-marostegui.json [production]
19:08 <SandraEbele_> deploying airflow dags [production]
18:55 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T364069)', diff saved to https://phabricator.wikimedia.org/P65759 and previous config saved to /var/cache/conftool/dbconfig/20240703-185511-marostegui.json [production]
18:54 <cmooney@cumin1002> START - Cookbook sre.hosts.reimage for host sretest2002.codfw.wmnet with OS bookworm [production]
18:36 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]