2251-2300 of 10000 results (87ms)
2024-01-24 ยง
16:04 <klausman@deploy2002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
16:04 <klausman@deploy2002> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
16:03 <klausman@deploy2002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
16:03 <klausman@deploy2002> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
15:58 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host phab2002.codfw.wmnet [production]
15:57 <hashar@deploy2002> Synchronized php-1.42.0-wmf.15/extensions/Echo/includes/Formatters/EchoRevertedPresentationModel.php: Fix EchoRevertedPresentationModel using null as string - T355751 (duration: 09m 06s) [production]
15:55 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P55539 and previous config saved to /var/cache/conftool/dbconfig/20240124-155519-marostegui.json [production]
15:50 <vgutierrez> disable puppet on cp3066 - T354424 [production]
15:48 <sukhe> sudo cumin -b1 -s120 'A:dns-rec' "enable-puppet 'merging CR 980929' && run-puppet-agent" [production]
15:47 <hashar@deploy2002> Synchronized php-1.42.0-wmf.15/extensions/CentralAuth/tests/phpunit/CentralAuthIdLookupTest.php: Fix CentralIdLookup tests (duration: 11m 18s) [production]
15:45 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2446.codfw.wmnet with OS bullseye [production]
15:42 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2430.codfw.wmnet with OS bullseye [production]
15:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2188 (T354336)', diff saved to https://phabricator.wikimedia.org/P55538 and previous config saved to /var/cache/conftool/dbconfig/20240124-154013-marostegui.json [production]
15:39 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2427.codfw.wmnet with OS bullseye [production]
15:38 <sukhe> sudo cumin 'A:dns-rec' "disable-puppet 'merging CR 980929'" [production]
15:38 <klausman@deploy2002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:38 <klausman@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
15:38 <klausman@deploy2002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
15:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2188 (T354336)', diff saved to https://phabricator.wikimedia.org/P55537 and previous config saved to /var/cache/conftool/dbconfig/20240124-153752-marostegui.json [production]
15:37 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2188.codfw.wmnet with reason: Maintenance [production]
15:37 <klausman@deploy2002> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
15:37 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2188.codfw.wmnet with reason: Maintenance [production]
15:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2176 (T354336)', diff saved to https://phabricator.wikimedia.org/P55536 and previous config saved to /var/cache/conftool/dbconfig/20240124-153730-marostegui.json [production]
15:37 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host phab2002.codfw.wmnet [production]
15:37 <klausman@deploy2002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
15:36 <klausman@deploy2002> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
15:32 <moritzm> imported jenkins 2.426.3 for buster/bullseye T355503 [production]
15:25 <aqu@deploy2002> Finished deploy [airflow-dags/analytics@da2e61c]: Regular analytics weekly train [airflow-dags/analytics@da2e61c7] (duration: 00m 42s) [production]
15:25 <aqu@deploy2002> Started deploy [airflow-dags/analytics@da2e61c]: Regular analytics weekly train [airflow-dags/analytics@da2e61c7] [production]
15:25 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2446.codfw.wmnet with reason: host reimage [production]
15:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P55534 and previous config saved to /var/cache/conftool/dbconfig/20240124-152224-marostegui.json [production]
15:22 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2430.codfw.wmnet with reason: host reimage [production]
15:21 <aqu> Refinery weekly deployment train - end (scap, then deployed onto hdfs) (test cluster deploy still broken T354703) [production]
15:19 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2427.codfw.wmnet with reason: host reimage [production]
15:17 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2430.codfw.wmnet with reason: host reimage [production]
15:16 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2446.codfw.wmnet with reason: host reimage [production]
15:16 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2427.codfw.wmnet with reason: host reimage [production]
15:12 <aqu@deploy2002> Finished deploy [analytics/refinery@13f7a06] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13f7a06c] (duration: 03m 28s) [production]
15:11 <moritzm> uploading pymsql 1.0.2-2~wmf11u1 to apt.wikimedia.org T355531 [production]
15:09 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host mc2055.codfw.wmnet [production]
15:08 <aqu@deploy2002> Started deploy [analytics/refinery@13f7a06] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13f7a06c] [production]
15:08 <aqu@deploy2002> Finished deploy [analytics/refinery@13f7a06] (thin): Regular analytics weekly train THIN [analytics/refinery@13f7a06c] (duration: 00m 05s) [production]
15:08 <aqu@deploy2002> Started deploy [analytics/refinery@13f7a06] (thin): Regular analytics weekly train THIN [analytics/refinery@13f7a06c] [production]
15:07 <aqu@deploy2002> Finished deploy [analytics/refinery@13f7a06]: Regular analytics weekly train [analytics/refinery@13f7a06c] (duration: 10m 12s) [production]
15:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P55533 and previous config saved to /var/cache/conftool/dbconfig/20240124-150718-marostegui.json [production]
15:04 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host mc2055.codfw.wmnet [production]
14:59 <hnowlan@cumin2002> START - Cookbook sre.hosts.reimage for host mw2446.codfw.wmnet with OS bullseye [production]
14:59 <hnowlan@cumin2002> START - Cookbook sre.hosts.reimage for host mw2430.codfw.wmnet with OS bullseye [production]
14:59 <hnowlan@cumin2002> START - Cookbook sre.hosts.reimage for host mw2427.codfw.wmnet with OS bullseye [production]
14:57 <aqu@deploy2002> Started deploy [analytics/refinery@13f7a06]: Regular analytics weekly train [analytics/refinery@13f7a06c] [production]