351-400 of 10000 results (44ms)
2022-09-06 ยง
07:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P33857 and previous config saved to /var/cache/conftool/dbconfig/20220906-075455-ladsgroup.json [production]
07:52 <XioNoX> depool ulsfo for routers upgrade - T295690 [production]
07:51 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 1%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P33856 and previous config saved to /var/cache/conftool/dbconfig/20220906-075113-root.json [production]
07:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 25%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33855 and previous config saved to /var/cache/conftool/dbconfig/20220906-074504-root.json [production]
07:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 25%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33854 and previous config saved to /var/cache/conftool/dbconfig/20220906-074448-root.json [production]
07:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T314041)', diff saved to https://phabricator.wikimedia.org/P33853 and previous config saved to /var/cache/conftool/dbconfig/20220906-073948-ladsgroup.json [production]
07:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1130 T316342', diff saved to https://phabricator.wikimedia.org/P33851 and previous config saved to /var/cache/conftool/dbconfig/20220906-073434-root.json [production]
07:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 10%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33850 and previous config saved to /var/cache/conftool/dbconfig/20220906-072959-root.json [production]
07:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 10%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33849 and previous config saved to /var/cache/conftool/dbconfig/20220906-072943-root.json [production]
07:26 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0) [production]
07:25 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 5%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33848 and previous config saved to /var/cache/conftool/dbconfig/20220906-071455-root.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 5%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33847 and previous config saved to /var/cache/conftool/dbconfig/20220906-071438-root.json [production]
07:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:11 <oblivian@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:823679|Move 1 of 6 users to php 7.4 (T271736)]] (duration: 04m 06s) [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 4%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33846 and previous config saved to /var/cache/conftool/dbconfig/20220906-065950-root.json [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 4%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33845 and previous config saved to /var/cache/conftool/dbconfig/20220906-065934-root.json [production]
06:53 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 3%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33844 and previous config saved to /var/cache/conftool/dbconfig/20220906-064445-root.json [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 3%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33843 and previous config saved to /var/cache/conftool/dbconfig/20220906-064429-root.json [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1189 T316342', diff saved to https://phabricator.wikimedia.org/P33841 and previous config saved to /var/cache/conftool/dbconfig/20220906-064021-root.json [production]
06:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1188 T316342', diff saved to https://phabricator.wikimedia.org/P33839 and previous config saved to /var/cache/conftool/dbconfig/20220906-063322-root.json [production]
06:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 2%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33838 and previous config saved to /var/cache/conftool/dbconfig/20220906-062940-root.json [production]
06:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 2%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33837 and previous config saved to /var/cache/conftool/dbconfig/20220906-062924-root.json [production]
06:15 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
06:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 1%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33836 and previous config saved to /var/cache/conftool/dbconfig/20220906-061434-root.json [production]
06:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 1%: Repooling again', diff saved to https://phabricator.wikimedia.org/P33835 and previous config saved to /var/cache/conftool/dbconfig/20220906-061419-root.json [production]
06:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1134 (T312863)', diff saved to https://phabricator.wikimedia.org/P33833 and previous config saved to /var/cache/conftool/dbconfig/20220906-061150-ladsgroup.json [production]
06:11 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
06:11 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
06:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
06:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
06:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Give some weight to current x1 eqiad master', diff saved to https://phabricator.wikimedia.org/P33832 and previous config saved to /var/cache/conftool/dbconfig/20220906-060833-root.json [production]
06:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1103 T316745', diff saved to https://phabricator.wikimedia.org/P33831 and previous config saved to /var/cache/conftool/dbconfig/20220906-060815-root.json [production]
06:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1120 to x1 primary T316745', diff saved to https://phabricator.wikimedia.org/P33830 and previous config saved to /var/cache/conftool/dbconfig/20220906-060602-root.json [production]
06:05 <marostegui> Starting x1 eqiad failover from db1103 to db1120 - T316745 [production]
06:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db1118 T316623', diff saved to https://phabricator.wikimedia.org/P33829 and previous config saved to /var/cache/conftool/dbconfig/20220906-060418-ladsgroup.json [production]
06:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Promote db1163 to s1 primary and set section read-write T316623', diff saved to https://phabricator.wikimedia.org/P33828 and previous config saved to /var/cache/conftool/dbconfig/20220906-060055-ladsgroup.json [production]
06:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T316623', diff saved to https://phabricator.wikimedia.org/P33827 and previous config saved to /var/cache/conftool/dbconfig/20220906-060032-ladsgroup.json [production]
06:00 <Amir1> Starting s1 eqiad failover from db1118 to db1163 - T316623 [production]
05:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1107 to dbctl depooled T316870', diff saved to https://phabricator.wikimedia.org/P33826 and previous config saved to /var/cache/conftool/dbconfig/20220906-053238-marostegui.json [production]
05:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3316 (T314041)', diff saved to https://phabricator.wikimedia.org/P33825 and previous config saved to /var/cache/conftool/dbconfig/20220906-052609-ladsgroup.json [production]
05:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
05:25 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
05:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T314041)', diff saved to https://phabricator.wikimedia.org/P33824 and previous config saved to /var/cache/conftool/dbconfig/20220906-052547-ladsgroup.json [production]
05:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1120 with weight 0 T316745', diff saved to https://phabricator.wikimedia.org/P33823 and previous config saved to /var/cache/conftool/dbconfig/20220906-051304-root.json [production]
05:12 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 10 hosts with reason: Primary switchover x1 T316745 [production]
05:12 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 10 hosts with reason: Primary switchover x1 T316745 [production]