2701-2750 of 10000 results (67ms)
2022-09-12 ยง
07:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Set es2024 with weight 0 T317508', diff saved to https://phabricator.wikimedia.org/P34456 and previous config saved to /var/cache/conftool/dbconfig/20220912-075739-root.json [production]
07:56 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T317508 [production]
07:56 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T317508 [production]
07:47 <hashar> Upgraded Jenkins instances from 2.346.1 to 2.346.3 # T317418 [production]
07:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2021 T317507', diff saved to https://phabricator.wikimedia.org/P34455 and previous config saved to /var/cache/conftool/dbconfig/20220912-074258-root.json [production]
07:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote es2020 to es4 primary and set section read-write T317507', diff saved to https://phabricator.wikimedia.org/P34454 and previous config saved to /var/cache/conftool/dbconfig/20220912-074100-root.json [production]
07:39 <marostegui> Starting es4 codfw failover from es2021 to es2020 - T317507 [production]
07:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Set es2020 with weight 0 T317507', diff saved to https://phabricator.wikimedia.org/P34453 and previous config saved to /var/cache/conftool/dbconfig/20220912-073408-root.json [production]
07:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es4 T317507 [production]
07:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T317507 [production]
07:31 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
07:31 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
07:31 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2174 (T312863)', diff saved to https://phabricator.wikimedia.org/P34452 and previous config saved to /var/cache/conftool/dbconfig/20220912-072931-ladsgroup.json [production]
07:29 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance [production]
07:29 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance [production]
07:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2173 (T312863)', diff saved to https://phabricator.wikimedia.org/P34450 and previous config saved to /var/cache/conftool/dbconfig/20220912-072909-ladsgroup.json [production]
07:27 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:18 <marostegui@cumin1001> dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34449 and previous config saved to /var/cache/conftool/dbconfig/20220912-071829-root.json [production]
07:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3317 (T314041)', diff saved to https://phabricator.wikimedia.org/P34448 and previous config saved to /var/cache/conftool/dbconfig/20220912-071700-ladsgroup.json [production]
07:16 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
07:16 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
07:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P34447 and previous config saved to /var/cache/conftool/dbconfig/20220912-071403-ladsgroup.json [production]
07:11 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:831374|Stop writing to the old templatelinks fields everywhere (T312865)]] (duration: 06m 57s) [production]
07:07 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:06 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:04 <ladsgroup@deploy1002> ladsgroup and ladsgroup: Backport for [[gerrit:831374|Stop writing to the old templatelinks fields everywhere (T312865)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
07:04 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:831374|Stop writing to the old templatelinks fields everywhere (T312865)]] [production]
07:03 <marostegui@cumin1001> dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34446 and previous config saved to /var/cache/conftool/dbconfig/20220912-070324-root.json [production]
06:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P34445 and previous config saved to /var/cache/conftool/dbconfig/20220912-065856-ladsgroup.json [production]
06:50 <marostegui@cumin1001> dbctl commit (dc=all): 'es2020 (re)pooling @ 100%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34444 and previous config saved to /var/cache/conftool/dbconfig/20220912-065028-root.json [production]
06:48 <marostegui@cumin1001> dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34443 and previous config saved to /var/cache/conftool/dbconfig/20220912-064819-root.json [production]
06:47 <moritzm> installing 5.10.136 updates on buster systems running 5.10 [production]
06:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2173 (T312863)', diff saved to https://phabricator.wikimedia.org/P34442 and previous config saved to /var/cache/conftool/dbconfig/20220912-064350-ladsgroup.json [production]
06:35 <marostegui@cumin1001> dbctl commit (dc=all): 'es2020 (re)pooling @ 75%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34441 and previous config saved to /var/cache/conftool/dbconfig/20220912-063523-root.json [production]
06:33 <marostegui@cumin1001> dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34440 and previous config saved to /var/cache/conftool/dbconfig/20220912-063314-root.json [production]
06:20 <marostegui@cumin1001> dbctl commit (dc=all): 'es2020 (re)pooling @ 50%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34439 and previous config saved to /var/cache/conftool/dbconfig/20220912-062018-root.json [production]
06:18 <marostegui@cumin1001> dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34438 and previous config saved to /var/cache/conftool/dbconfig/20220912-061810-root.json [production]
06:05 <marostegui@cumin1001> dbctl commit (dc=all): 'es2020 (re)pooling @ 25%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34437 and previous config saved to /var/cache/conftool/dbconfig/20220912-060513-root.json [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34436 and previous config saved to /var/cache/conftool/dbconfig/20220912-060305-root.json [production]
05:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2024 for upgrade', diff saved to https://phabricator.wikimedia.org/P34435 and previous config saved to /var/cache/conftool/dbconfig/20220912-055101-root.json [production]
05:50 <marostegui@cumin1001> dbctl commit (dc=all): 'es2020 (re)pooling @ 10%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34434 and previous config saved to /var/cache/conftool/dbconfig/20220912-055008-root.json [production]
05:35 <marostegui@cumin1001> dbctl commit (dc=all): 'es2020 (re)pooling @ 5%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34433 and previous config saved to /var/cache/conftool/dbconfig/20220912-053504-root.json [production]
05:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es4 T317507 [production]
05:22 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T317507 [production]
05:21 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]