801-850 of 10000 results (102ms)
2024-07-04 ยง
22:03 <mvolz@deploy1002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
22:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1234 (T364069)', diff saved to https://phabricator.wikimedia.org/P65840 and previous config saved to /var/cache/conftool/dbconfig/20240704-220227-marostegui.json [production]
22:02 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance [production]
22:02 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance [production]
22:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1232 (T364069)', diff saved to https://phabricator.wikimedia.org/P65839 and previous config saved to /var/cache/conftool/dbconfig/20240704-220205-marostegui.json [production]
22:01 <mvolz@deploy1002> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
22:00 <mvolz@deploy1002> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
21:59 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
21:59 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
21:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P65838 and previous config saved to /var/cache/conftool/dbconfig/20240704-214658-marostegui.json [production]
21:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P65837 and previous config saved to /var/cache/conftool/dbconfig/20240704-213151-marostegui.json [production]
21:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1232 (T364069)', diff saved to https://phabricator.wikimedia.org/P65836 and previous config saved to /var/cache/conftool/dbconfig/20240704-211644-marostegui.json [production]
20:17 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:1050671|[July 4th] Reduce list of exclusions for dark mode (1.43.0-wmf.12)]], [[gerrit:1050424|Remove modifications of wgCheckUserLogAdditionalRights (T346022)]], [[gerrit:1052158|Add editcontentmodel to interface-admin for French Wikipedia (T369113)]] (duration: 12m 14s) [production]
20:12 <jdrewniak@deploy1002> jdlrobson, nmw03, jdrewniak, dreamyjazz: Continuing with sync [production]
20:08 <jdrewniak@deploy1002> jdlrobson, nmw03, jdrewniak, dreamyjazz: Backport for [[gerrit:1050671|[July 4th] Reduce list of exclusions for dark mode (1.43.0-wmf.12)]], [[gerrit:1050424|Remove modifications of wgCheckUserLogAdditionalRights (T346022)]], [[gerrit:1052158|Add editcontentmodel to interface-admin for French Wikipedia (T369113)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:05 <jdrewniak@deploy1002> Started scap sync-world: Backport for [[gerrit:1050671|[July 4th] Reduce list of exclusions for dark mode (1.43.0-wmf.12)]], [[gerrit:1050424|Remove modifications of wgCheckUserLogAdditionalRights (T346022)]], [[gerrit:1052158|Add editcontentmodel to interface-admin for French Wikipedia (T369113)]] [production]
19:57 <fabfur@cumin1002> END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_eqiad [production]
19:55 <fabfur@cumin1002> END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad [production]
18:23 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1232 (T364069)', diff saved to https://phabricator.wikimedia.org/P65835 and previous config saved to /var/cache/conftool/dbconfig/20240704-182308-marostegui.json [production]
18:23 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance [production]
18:23 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance [production]
18:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T364069)', diff saved to https://phabricator.wikimedia.org/P65834 and previous config saved to /var/cache/conftool/dbconfig/20240704-182257-marostegui.json [production]
18:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P65833 and previous config saved to /var/cache/conftool/dbconfig/20240704-180749-marostegui.json [production]
17:52 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P65832 and previous config saved to /var/cache/conftool/dbconfig/20240704-175242-marostegui.json [production]
17:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T364069)', diff saved to https://phabricator.wikimedia.org/P65831 and previous config saved to /var/cache/conftool/dbconfig/20240704-173735-marostegui.json [production]
17:10 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1078.eqiad.wmnet [production]
16:20 <btullis@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:19 <btullis@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
16:15 <btullis@cumin1002> START - Cookbook sre.hosts.reboot-single for host an-worker1078.eqiad.wmnet [production]
16:14 <btullis@cumin1002> END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster [production]
16:14 <btullis@cumin1002> START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [production]
16:06 <kevinbazira@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
15:49 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
15:49 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
15:02 <elukey@deploy1002> helmfile [staging] DONE helmfile.d/services/thumbor: sync [production]
15:02 <elukey@deploy1002> helmfile [staging] START helmfile.d/services/thumbor: sync [production]
14:33 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1219 (T364069)', diff saved to https://phabricator.wikimedia.org/P65830 and previous config saved to /var/cache/conftool/dbconfig/20240704-143350-marostegui.json [production]
14:33 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
14:33 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
14:33 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218 (T364069)', diff saved to https://phabricator.wikimedia.org/P65829 and previous config saved to /var/cache/conftool/dbconfig/20240704-143327-marostegui.json [production]
14:18 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P65827 and previous config saved to /var/cache/conftool/dbconfig/20240704-141820-marostegui.json [production]
14:03 <Lucas_WMDE> UTC afternoon backport+config window done [production]
14:03 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P65826 and previous config saved to /var/cache/conftool/dbconfig/20240704-140313-marostegui.json [production]
14:01 <claime> Enabling and running puppet on P:trafficserver::backend to merge 1050293 - T367949 [production]
14:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P65825 and previous config saved to /var/cache/conftool/dbconfig/20240704-140145-root.json [production]
13:57 <claime> Enabling puppet on cp4037.ulsfo.wmnet to test 1050293 - T367949 [production]
13:53 <claime> disabling puppet on P:trafficserver::backend to merge 1049507 - T367949 [production]
13:48 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218 (T364069)', diff saved to https://phabricator.wikimedia.org/P65824 and previous config saved to /var/cache/conftool/dbconfig/20240704-134806-marostegui.json [production]
13:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db1185 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P65823 and previous config saved to /var/cache/conftool/dbconfig/20240704-134656-root.json [production]
13:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P65822 and previous config saved to /var/cache/conftool/dbconfig/20240704-134639-root.json [production]