1651-1700 of 10000 results (49ms)
2022-06-30 ยง
13:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:23 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2155.codfw.wmnet with reason: host reimage [production]
13:23 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
13:23 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
13:22 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s) [production]
13:22 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
13:19 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2155.codfw.wmnet with reason: host reimage [production]
13:19 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host db2155.codfw.wmnet with OS bullseye [production]
13:16 <moritzm> installing libsndfile security updates [production]
13:16 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:15 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:15 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:13 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: fb399065b123db536ae244a0c0fada61eb906a6e: Structured task: enable free text for "other" rejection reason (T304099) (duration: 03m 46s) [production]
13:09 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host stat1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
13:09 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:05 <moritzm> upgrade mwdebug* servers to 2:76+wmf1~buster2 T311386 [production]
13:00 <moritzm> uploaded php-defaults 76+wmf1+buster2 for component/php74 (drops a Breaks: on php72-common) T311386 [production]
12:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2083.codfw.wmnet [production]
12:44 <marostegui@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:40 <marostegui@cumin1001> START - Cookbook sre.dns.netbox [production]
12:36 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db2083.codfw.wmnet [production]
12:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 100%: Maint done', diff saved to https://phabricator.wikimedia.org/P30660 and previous config saved to /var/cache/conftool/dbconfig/20220630-122931-ladsgroup.json [production]
12:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 75%: Maint done', diff saved to https://phabricator.wikimedia.org/P30659 and previous config saved to /var/cache/conftool/dbconfig/20220630-121427-ladsgroup.json [production]
11:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 25%: Maint done', diff saved to https://phabricator.wikimedia.org/P30658 and previous config saved to /var/cache/conftool/dbconfig/20220630-115923-ladsgroup.json [production]
11:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 10%: Maint done', diff saved to https://phabricator.wikimedia.org/P30657 and previous config saved to /var/cache/conftool/dbconfig/20220630-114419-ladsgroup.json [production]
09:47 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS buster [production]
09:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db2083 from dbctl', diff saved to https://phabricator.wikimedia.org/P30655 and previous config saved to /var/cache/conftool/dbconfig/20220630-094239-marostegui.json [production]
09:36 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS buster [production]
09:35 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on sretest1001.eqiad.wmnet with reason: Testing [production]
09:35 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 0:20:00 on sretest1001.eqiad.wmnet with reason: Testing [production]
08:57 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS buster [production]
08:56 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage [production]
08:56 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage [production]
08:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove weight from x1 master - not neeed anymore', diff saved to https://phabricator.wikimedia.org/P30654 and previous config saved to /var/cache/conftool/dbconfig/20220630-084621-marostegui.json [production]
08:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1103 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P30653 and previous config saved to /var/cache/conftool/dbconfig/20220630-084550-root.json [production]
08:44 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage [production]
08:42 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS buster [production]
08:42 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage [production]
08:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 100%: After on-site maintenance', diff saved to https://phabricator.wikimedia.org/P30652 and previous config saved to /var/cache/conftool/dbconfig/20220630-084148-root.json [production]
08:33 <elukey@deploy1002> Finished deploy [ores/deploy@dfaec93]: Update ores submodule to its latest commit and scap canary settings (duration: 14m 48s) [production]
08:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1103 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P30651 and previous config saved to /var/cache/conftool/dbconfig/20220630-083046-root.json [production]
08:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage [production]
08:28 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS buster [production]
08:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 75%: After on-site maintenance', diff saved to https://phabricator.wikimedia.org/P30650 and previous config saved to /var/cache/conftool/dbconfig/20220630-082644-root.json [production]
08:26 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage [production]