|
2024-09-11
§
|
| 08:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P68907 and previous config saved to /var/cache/conftool/dbconfig/20240911-082324-ladsgroup.json |
[production] |
| 08:22 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: post fix', diff saved to https://phabricator.wikimedia.org/P68906 and previous config saved to /var/cache/conftool/dbconfig/20240911-082200-arnaudb.json |
[production] |
| 08:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P68905 and previous config saved to /var/cache/conftool/dbconfig/20240911-080817-ladsgroup.json |
[production] |
| 08:06 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: post fix', diff saved to https://phabricator.wikimedia.org/P68904 and previous config saved to /var/cache/conftool/dbconfig/20240911-080654-arnaudb.json |
[production] |
| 08:03 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2137 (re)pooling @ 100%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68903 and previous config saved to /var/cache/conftool/dbconfig/20240911-080319-arnaudb.json |
[production] |
| 07:53 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1235 (T371742)', diff saved to https://phabricator.wikimedia.org/P68899 and previous config saved to /var/cache/conftool/dbconfig/20240911-075310-ladsgroup.json |
[production] |
| 07:53 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[2003,2008].codfw.wmnet with reason: Hardware refresh |
[production] |
| 07:52 |
<jayme@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[2003,2008].codfw.wmnet with reason: Hardware refresh |
[production] |
| 07:51 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: post fix', diff saved to https://phabricator.wikimedia.org/P68898 and previous config saved to /var/cache/conftool/dbconfig/20240911-075149-arnaudb.json |
[production] |
| 07:49 |
<jayme> |
evacuating leadership for all partitions assigned to broker id 2003 on kafka-main-codfw - T363210 |
[production] |
| 07:48 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2137 (re)pooling @ 75%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68897 and previous config saved to /var/cache/conftool/dbconfig/20240911-074813-arnaudb.json |
[production] |
| 07:36 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: post fix', diff saved to https://phabricator.wikimedia.org/P68896 and previous config saved to /var/cache/conftool/dbconfig/20240911-073643-arnaudb.json |
[production] |
| 07:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'prod issue kmwiki.pagelinks', diff saved to https://phabricator.wikimedia.org/P68895 and previous config saved to /var/cache/conftool/dbconfig/20240911-073420-arnaudb.json |
[production] |
| 07:33 |
<sgimeno@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1062416|EventStreamConfig and stream registration for homepage modules analytics (T370907)]] (duration: 13m 56s) |
[production] |
| 07:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2137 (re)pooling @ 50%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68894 and previous config saved to /var/cache/conftool/dbconfig/20240911-073307-arnaudb.json |
[production] |
| 07:29 |
<sgimeno@deploy1003> |
sgimeno: Continuing with sync |
[production] |
| 07:26 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2179 T374512', diff saved to https://phabricator.wikimedia.org/P68893 and previous config saved to /var/cache/conftool/dbconfig/20240911-072612-arnaudb.json |
[production] |
| 07:24 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2179 T374512', diff saved to https://phabricator.wikimedia.org/P68892 and previous config saved to /var/cache/conftool/dbconfig/20240911-072458-arnaudb.json |
[production] |
| 07:24 |
<sgimeno@deploy1003> |
sgimeno: Backport for [[gerrit:1062416|EventStreamConfig and stream registration for homepage modules analytics (T370907)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 07:22 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Promote db2140 to s4 primary T374512', diff saved to https://phabricator.wikimedia.org/P68891 and previous config saved to /var/cache/conftool/dbconfig/20240911-072210-arnaudb.json |
[production] |
| 07:21 |
<arnaudb> |
Starting s4 codfw failover from db2179 to db2140 - T374512 |
[production] |
| 07:19 |
<sgimeno@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1062416|EventStreamConfig and stream registration for homepage modules analytics (T370907)]] |
[production] |
| 07:18 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2137 (re)pooling @ 25%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68890 and previous config saved to /var/cache/conftool/dbconfig/20240911-071802-arnaudb.json |
[production] |
| 07:13 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Remove db2140 from API/vslow/dump T374512', diff saved to https://phabricator.wikimedia.org/P68889 and previous config saved to /var/cache/conftool/dbconfig/20240911-071335-arnaudb.json |
[production] |
| 07:12 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T374512 |
[production] |
| 07:12 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Set db2140 with weight 0 T374512', diff saved to https://phabricator.wikimedia.org/P68888 and previous config saved to /var/cache/conftool/dbconfig/20240911-071205-arnaudb.json |
[production] |
| 07:11 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 33 hosts with reason: Primary switchover s4 T374512 |
[production] |
| 07:02 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2137 (re)pooling @ 10%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68887 and previous config saved to /var/cache/conftool/dbconfig/20240911-070254-arnaudb.json |
[production] |
| 06:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P68884 and previous config saved to /var/cache/conftool/dbconfig/20240911-063458-ladsgroup.json |
[production] |
| 06:19 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P68883 and previous config saved to /var/cache/conftool/dbconfig/20240911-061951-ladsgroup.json |
[production] |
| 06:04 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1234 (T371742)', diff saved to https://phabricator.wikimedia.org/P68882 and previous config saved to /var/cache/conftool/dbconfig/20240911-060444-ladsgroup.json |
[production] |
| 05:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1234 (T371742)', diff saved to https://phabricator.wikimedia.org/P68881 and previous config saved to /var/cache/conftool/dbconfig/20240911-050506-ladsgroup.json |
[production] |
| 05:04 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:04 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:04 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1232 (T371742)', diff saved to https://phabricator.wikimedia.org/P68880 and previous config saved to /var/cache/conftool/dbconfig/20240911-050444-ladsgroup.json |
[production] |
| 04:49 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P68879 and previous config saved to /var/cache/conftool/dbconfig/20240911-044936-ladsgroup.json |
[production] |
| 04:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P68878 and previous config saved to /var/cache/conftool/dbconfig/20240911-043429-ladsgroup.json |
[production] |
| 04:19 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1232 (T371742)', diff saved to https://phabricator.wikimedia.org/P68877 and previous config saved to /var/cache/conftool/dbconfig/20240911-041922-ladsgroup.json |
[production] |
| 03:16 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1232 (T371742)', diff saved to https://phabricator.wikimedia.org/P68876 and previous config saved to /var/cache/conftool/dbconfig/20240911-031643-ladsgroup.json |
[production] |
| 03:16 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:16 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:16 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1219 (T371742)', diff saved to https://phabricator.wikimedia.org/P68875 and previous config saved to /var/cache/conftool/dbconfig/20240911-031621-ladsgroup.json |
[production] |
| 03:01 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P68874 and previous config saved to /var/cache/conftool/dbconfig/20240911-030112-ladsgroup.json |
[production] |
| 02:46 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P68873 and previous config saved to /var/cache/conftool/dbconfig/20240911-024605-ladsgroup.json |
[production] |
| 02:30 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1219 (T371742)', diff saved to https://phabricator.wikimedia.org/P68872 and previous config saved to /var/cache/conftool/dbconfig/20240911-023058-ladsgroup.json |
[production] |
| 01:33 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1219 (T371742)', diff saved to https://phabricator.wikimedia.org/P68871 and previous config saved to /var/cache/conftool/dbconfig/20240911-013327-ladsgroup.json |
[production] |
| 01:33 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:33 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:33 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1218 (T371742)', diff saved to https://phabricator.wikimedia.org/P68870 and previous config saved to /var/cache/conftool/dbconfig/20240911-013305-ladsgroup.json |
[production] |
| 01:17 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P68869 and previous config saved to /var/cache/conftool/dbconfig/20240911-011758-ladsgroup.json |
[production] |