501-550 of 10000 results (103ms)
2024-07-05 §
05:12 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235 (T364069)', diff saved to https://phabricator.wikimedia.org/P65852 and previous config saved to /var/cache/conftool/dbconfig/20240705-051202-marostegui.json [production]
05:00 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2136', diff saved to https://phabricator.wikimedia.org/P65851 and previous config saved to /var/cache/conftool/dbconfig/20240705-050028-root.json [production]
04:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P65850 and previous config saved to /var/cache/conftool/dbconfig/20240705-045655-marostegui.json [production]
04:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2123 (T367856)', diff saved to https://phabricator.wikimedia.org/P65849 and previous config saved to /var/cache/conftool/dbconfig/20240705-045145-marostegui.json [production]
04:51 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
04:51 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
04:49 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1246 (T367856)', diff saved to https://phabricator.wikimedia.org/P65848 and previous config saved to /var/cache/conftool/dbconfig/20240705-044912-marostegui.json [production]
04:49 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1246.eqiad.wmnet with reason: Maintenance [production]
04:49 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1246.eqiad.wmnet with reason: Maintenance [production]
04:41 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P65847 and previous config saved to /var/cache/conftool/dbconfig/20240705-044148-marostegui.json [production]
04:26 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1235 (T364069)', diff saved to https://phabricator.wikimedia.org/P65846 and previous config saved to /var/cache/conftool/dbconfig/20240705-042641-marostegui.json [production]
01:32 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1235 (T364069)', diff saved to https://phabricator.wikimedia.org/P65845 and previous config saved to /var/cache/conftool/dbconfig/20240705-013250-marostegui.json [production]
01:32 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance [production]
01:32 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance [production]
01:32 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1234 (T364069)', diff saved to https://phabricator.wikimedia.org/P65844 and previous config saved to /var/cache/conftool/dbconfig/20240705-013229-marostegui.json [production]
01:17 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P65843 and previous config saved to /var/cache/conftool/dbconfig/20240705-011721-marostegui.json [production]
01:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P65842 and previous config saved to /var/cache/conftool/dbconfig/20240705-010214-marostegui.json [production]
00:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1234 (T364069)', diff saved to https://phabricator.wikimedia.org/P65841 and previous config saved to /var/cache/conftool/dbconfig/20240705-004707-marostegui.json [production]
2024-07-04 §
22:04 <mvolz@deploy1002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
22:03 <mvolz@deploy1002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
22:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1234 (T364069)', diff saved to https://phabricator.wikimedia.org/P65840 and previous config saved to /var/cache/conftool/dbconfig/20240704-220227-marostegui.json [production]
22:02 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance [production]
22:02 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance [production]
22:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1232 (T364069)', diff saved to https://phabricator.wikimedia.org/P65839 and previous config saved to /var/cache/conftool/dbconfig/20240704-220205-marostegui.json [production]
22:01 <mvolz@deploy1002> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
22:00 <mvolz@deploy1002> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
21:59 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
21:59 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
21:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P65838 and previous config saved to /var/cache/conftool/dbconfig/20240704-214658-marostegui.json [production]
21:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P65837 and previous config saved to /var/cache/conftool/dbconfig/20240704-213151-marostegui.json [production]
21:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1232 (T364069)', diff saved to https://phabricator.wikimedia.org/P65836 and previous config saved to /var/cache/conftool/dbconfig/20240704-211644-marostegui.json [production]
20:17 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:1050671|[July 4th] Reduce list of exclusions for dark mode (1.43.0-wmf.12)]], [[gerrit:1050424|Remove modifications of wgCheckUserLogAdditionalRights (T346022)]], [[gerrit:1052158|Add editcontentmodel to interface-admin for French Wikipedia (T369113)]] (duration: 12m 14s) [production]
20:12 <jdrewniak@deploy1002> jdlrobson, nmw03, jdrewniak, dreamyjazz: Continuing with sync [production]
20:08 <jdrewniak@deploy1002> jdlrobson, nmw03, jdrewniak, dreamyjazz: Backport for [[gerrit:1050671|[July 4th] Reduce list of exclusions for dark mode (1.43.0-wmf.12)]], [[gerrit:1050424|Remove modifications of wgCheckUserLogAdditionalRights (T346022)]], [[gerrit:1052158|Add editcontentmodel to interface-admin for French Wikipedia (T369113)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:05 <jdrewniak@deploy1002> Started scap sync-world: Backport for [[gerrit:1050671|[July 4th] Reduce list of exclusions for dark mode (1.43.0-wmf.12)]], [[gerrit:1050424|Remove modifications of wgCheckUserLogAdditionalRights (T346022)]], [[gerrit:1052158|Add editcontentmodel to interface-admin for French Wikipedia (T369113)]] [production]
19:57 <fabfur@cumin1002> END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_eqiad [production]
19:55 <fabfur@cumin1002> END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad [production]
18:23 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1232 (T364069)', diff saved to https://phabricator.wikimedia.org/P65835 and previous config saved to /var/cache/conftool/dbconfig/20240704-182308-marostegui.json [production]
18:23 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance [production]
18:23 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance [production]
18:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T364069)', diff saved to https://phabricator.wikimedia.org/P65834 and previous config saved to /var/cache/conftool/dbconfig/20240704-182257-marostegui.json [production]
18:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P65833 and previous config saved to /var/cache/conftool/dbconfig/20240704-180749-marostegui.json [production]
17:52 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P65832 and previous config saved to /var/cache/conftool/dbconfig/20240704-175242-marostegui.json [production]
17:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T364069)', diff saved to https://phabricator.wikimedia.org/P65831 and previous config saved to /var/cache/conftool/dbconfig/20240704-173735-marostegui.json [production]
17:10 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1078.eqiad.wmnet [production]
16:20 <btullis@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:19 <btullis@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
16:15 <btullis@cumin1002> START - Cookbook sre.hosts.reboot-single for host an-worker1078.eqiad.wmnet [production]
16:14 <btullis@cumin1002> END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster [production]
16:14 <btullis@cumin1002> START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [production]