351-400 of 10000 results (75ms)
2024-07-03 ยง
17:35 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-misc: apply [production]
17:35 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-misc: apply [production]
17:35 <cdanis@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-misc: apply [production]
17:35 <cdanis@deploy1002> helmfile [codfw] START helmfile.d/services/mw-misc: apply [production]
17:35 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply [production]
17:34 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply [production]
17:34 <cdanis@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply [production]
17:34 <cdanis@deploy1002> helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply [production]
17:34 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [production]
17:33 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [production]
17:33 <cdanis@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply [production]
17:31 <cdanis@deploy1002> helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply [production]
17:30 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
17:29 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
17:28 <cdanis@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
17:28 <cdanis@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
17:22 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
17:20 <marostegui@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P65756 and previous config saved to /var/cache/conftool/dbconfig/20240703-172055-root.json [production]
17:19 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
17:19 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
17:17 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
17:17 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
17:15 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
17:11 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
17:10 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
17:10 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
17:09 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
17:08 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
17:07 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
17:05 <marostegui@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P65755 and previous config saved to /var/cache/conftool/dbconfig/20240703-170549-root.json [production]
16:50 <marostegui@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P65754 and previous config saved to /var/cache/conftool/dbconfig/20240703-165044-root.json [production]
16:47 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-presto1004.eqiad.wmnet with reason: Cold booting to investigate RAM issue [production]
16:46 <btullis@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-presto1004.eqiad.wmnet with reason: Cold booting to investigate RAM issue [production]
16:44 <jhathaway> adding inbound email servers mx-in{1001,2001} to our MX record [production]
16:35 <marostegui@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P65752 and previous config saved to /var/cache/conftool/dbconfig/20240703-163538-root.json [production]
16:20 <marostegui@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P65751 and previous config saved to /var/cache/conftool/dbconfig/20240703-162032-root.json [production]
16:05 <marostegui@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 1%: Repooling', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20240703-160521-root.json [production]
16:04 <kevinbazira@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
15:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1186 (T364069)', diff saved to https://phabricator.wikimedia.org/P65750 and previous config saved to /var/cache/conftool/dbconfig/20240703-154716-marostegui.json [production]
15:47 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]
15:46 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]
15:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T364069)', diff saved to https://phabricator.wikimedia.org/P65749 and previous config saved to /var/cache/conftool/dbconfig/20240703-154643-marostegui.json [production]
15:41 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1191 (re)pooling @ 100%: post T365994 repool', diff saved to https://phabricator.wikimedia.org/P65748 and previous config saved to /var/cache/conftool/dbconfig/20240703-154142-arnaudb.json [production]
15:41 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1196 (re)pooling @ 100%: post T365994 repool', diff saved to https://phabricator.wikimedia.org/P65747 and previous config saved to /var/cache/conftool/dbconfig/20240703-154121-arnaudb.json [production]
15:41 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1197 (re)pooling @ 100%: post T365994 repool', diff saved to https://phabricator.wikimedia.org/P65746 and previous config saved to /var/cache/conftool/dbconfig/20240703-154109-arnaudb.json [production]
15:32 <kevinbazira@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
15:31 <sukhe> restart haproxy on dns1005 [production]
15:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P65744 and previous config saved to /var/cache/conftool/dbconfig/20240703-153136-marostegui.json [production]
15:26 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1191 (re)pooling @ 75%: post T365994 repool', diff saved to https://phabricator.wikimedia.org/P65743 and previous config saved to /var/cache/conftool/dbconfig/20240703-152636-arnaudb.json [production]
15:26 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1196 (re)pooling @ 75%: post T365994 repool', diff saved to https://phabricator.wikimedia.org/P65742 and previous config saved to /var/cache/conftool/dbconfig/20240703-152616-arnaudb.json [production]