501-550 of 10000 results (99ms)
2024-04-18 §
00:06 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
00:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60832 and previous config saved to /var/cache/conftool/dbconfig/20240418-000616-ladsgroup.json [production]
2024-04-17 §
23:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60831 and previous config saved to /var/cache/conftool/dbconfig/20240417-235105-ladsgroup.json [production]
23:48 <amastilovic@deploy1002> Finished deploy [airflow-dags/analytics@c9d6969]: (no justification provided) (duration: 00m 37s) [production]
23:47 <amastilovic@deploy1002> Started deploy [airflow-dags/analytics@c9d6969]: (no justification provided) [production]
23:37 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
23:37 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
23:37 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P60830 and previous config saved to /var/cache/conftool/dbconfig/20240417-233731-ladsgroup.json [production]
23:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60829 and previous config saved to /var/cache/conftool/dbconfig/20240417-233557-ladsgroup.json [production]
23:22 <sukhe> sukhe@cp1114:~$ sudo -i haproxy-restart [production]
23:22 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P60828 and previous config saved to /var/cache/conftool/dbconfig/20240417-232221-ladsgroup.json [production]
23:20 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60827 and previous config saved to /var/cache/conftool/dbconfig/20240417-232050-ladsgroup.json [production]
23:14 <mutante> rsyncing jenkins data from contint2002 to contint1002, pre-sync in preparation for migration next week - /srv/jenkins (291G) and much smaller zuul and jenkins data dirs T334517 [production]
23:07 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P60826 and previous config saved to /var/cache/conftool/dbconfig/20240417-230714-ladsgroup.json [production]
22:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P60825 and previous config saved to /var/cache/conftool/dbconfig/20240417-225206-ladsgroup.json [production]
22:42 <zabe@deploy1002> Finished scap: Backport for [[gerrit:1020910|Revert "REST: Deprecate using "post" as the parameter source" (T362817)]] (duration: 17m 14s) [production]
22:29 <zabe@deploy1002> jforrester and zabe: Continuing with sync [production]
22:27 <zabe@deploy1002> jforrester and zabe: Backport for [[gerrit:1020910|Revert "REST: Deprecate using "post" as the parameter source" (T362817)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:24 <zabe@deploy1002> Started scap: Backport for [[gerrit:1020910|Revert "REST: Deprecate using "post" as the parameter source" (T362817)]] [production]
22:11 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 19 hosts with reason: T362508 [production]
22:10 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 19 hosts with reason: T362508 [production]
21:50 <mutante> deploying scap config change (gerrit:1020321) - [cumin2002:~] $ sudo cumin -b 4 -s 40 'C:scap AND mw*' 'run-puppet-agent' T359643 [production]
21:09 <mutante> DNS - created ae.wikimedia.org for United Arab Emirates User Group wiki - T362529 [production]
21:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60824 and previous config saved to /var/cache/conftool/dbconfig/20240417-210256-marostegui.json [production]
20:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60823 and previous config saved to /var/cache/conftool/dbconfig/20240417-204748-marostegui.json [production]
20:44 <eevans@deploy1002> helmfile [staging] DONE helmfile.d/services/echostore: apply [production]
20:44 <eevans@deploy1002> helmfile [staging] START helmfile.d/services/echostore: apply [production]
20:44 <cjming> end of UTC late backport window [production]
20:44 <eevans@deploy1002> helmfile [staging] DONE helmfile.d/services/sessionstore: apply [production]
20:43 <eevans@deploy1002> helmfile [staging] START helmfile.d/services/sessionstore: apply [production]
20:43 <cjming@deploy1002> Finished scap: Backport for [[gerrit:1020736|Upstream tablet infobox styles (T3603861)]], [[gerrit:1020737|Upstream tablet infobox styles (T3603861)]] (duration: 17m 30s) [production]
20:32 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60822 and previous config saved to /var/cache/conftool/dbconfig/20240417-203241-marostegui.json [production]
20:30 <cjming@deploy1002> cjming and jdlrobson: Continuing with sync [production]
20:29 <cjming@deploy1002> cjming and jdlrobson: Backport for [[gerrit:1020736|Upstream tablet infobox styles (T3603861)]], [[gerrit:1020737|Upstream tablet infobox styles (T3603861)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:26 <cjming@deploy1002> Started scap: Backport for [[gerrit:1020736|Upstream tablet infobox styles (T3603861)]], [[gerrit:1020737|Upstream tablet infobox styles (T3603861)]] [production]
20:24 <cjming@deploy1002> Finished scap: Backport for [[gerrit:1020854|Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726)]], [[gerrit:1020883|Enable night mode in AMC for all projects (T361555)]] (duration: 18m 13s) [production]
20:17 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60821 and previous config saved to /var/cache/conftool/dbconfig/20240417-201733-marostegui.json [production]
20:11 <cjming@deploy1002> cjming and jdlrobson: Continuing with sync [production]
20:09 <cjming@deploy1002> cjming and jdlrobson: Backport for [[gerrit:1020854|Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726)]], [[gerrit:1020883|Enable night mode in AMC for all projects (T361555)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:06 <cjming@deploy1002> Started scap: Backport for [[gerrit:1020854|Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726)]], [[gerrit:1020883|Enable night mode in AMC for all projects (T361555)]] [production]
19:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60820 and previous config saved to /var/cache/conftool/dbconfig/20240417-195628-marostegui.json [production]
19:56 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance [production]
19:56 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance [production]
19:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60819 and previous config saved to /var/cache/conftool/dbconfig/20240417-195605-marostegui.json [production]
19:46 <eileen> civicrm upgraded from fdd12ed1 to 28adb4da [production]
19:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60818 and previous config saved to /var/cache/conftool/dbconfig/20240417-194058-marostegui.json [production]
19:25 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60817 and previous config saved to /var/cache/conftool/dbconfig/20240417-192551-marostegui.json [production]
19:10 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60816 and previous config saved to /var/cache/conftool/dbconfig/20240417-191043-marostegui.json [production]
18:56 <ejegg> payments-wiki upgraded from 72e3bf19 to fb0367a4 [production]
18:49 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60815 and previous config saved to /var/cache/conftool/dbconfig/20240417-184931-marostegui.json [production]