|
2023-08-22
ยง
|
| 07:11 |
<sgimeno@deploy1002> |
sgimeno: Backport for [[gerrit:950168|GrowthExperiments: turn off AddLink in aswiki (T344319)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P50805 and previous config saved to /var/cache/conftool/dbconfig/20230822-071053-ladsgroup.json |
[production] |
| 07:09 |
<sgimeno@deploy1002> |
Started scap: Backport for [[gerrit:950168|GrowthExperiments: turn off AddLink in aswiki (T344319)]] |
[production] |
| 07:05 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1115.eqiad.wmnet with OS bullseye |
[production] |
| 07:04 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 07:04 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 07:03 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1116.eqiad.wmnet with reason: host reimage |
[production] |
| 07:02 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 45356 |
[production] |
| 07:02 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 45356 |
[production] |
| 07:01 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org |
[production] |
| 07:00 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1116.eqiad.wmnet with reason: host reimage |
[production] |
| 06:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1200 (T343718)', diff saved to https://phabricator.wikimedia.org/P50804 and previous config saved to /var/cache/conftool/dbconfig/20230822-065819-ladsgroup.json |
[production] |
| 06:57 |
<moritzm> |
installing intel-microcode security updates on buster hosts |
[production] |
| 06:57 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 06:57 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 06:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223 (T344589)', diff saved to https://phabricator.wikimedia.org/P50803 and previous config saved to /var/cache/conftool/dbconfig/20230822-065547-ladsgroup.json |
[production] |
| 06:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depool db2103 T344666', diff saved to https://phabricator.wikimedia.org/P50802 and previous config saved to /var/cache/conftool/dbconfig/20230822-065518-ladsgroup.json |
[production] |
| 06:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2157 (T343718)', diff saved to https://phabricator.wikimedia.org/P50801 and previous config saved to /var/cache/conftool/dbconfig/20230822-065440-ladsgroup.json |
[production] |
| 06:54 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 06:54 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 06:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T343718)', diff saved to https://phabricator.wikimedia.org/P50800 and previous config saved to /var/cache/conftool/dbconfig/20230822-065430-ladsgroup.json |
[production] |
| 06:53 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org |
[production] |
| 06:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Promote db2112 to s1 primary T344666', diff saved to https://phabricator.wikimedia.org/P50799 and previous config saved to /var/cache/conftool/dbconfig/20230822-065316-ladsgroup.json |
[production] |
| 06:52 |
<Amir1> |
Starting s1 codfw failover from db2103 to db2112 - T344666 |
[production] |
| 06:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1223 (T344589)', diff saved to https://phabricator.wikimedia.org/P50798 and previous config saved to /var/cache/conftool/dbconfig/20230822-064828-ladsgroup.json |
[production] |
| 06:48 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:48 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1212 (T344589)', diff saved to https://phabricator.wikimedia.org/P50797 and previous config saved to /var/cache/conftool/dbconfig/20230822-064804-ladsgroup.json |
[production] |
| 06:47 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:951366|Enable URL shortener in sidebar in jawiki and zhwiki (T267921)]] (duration: 10m 06s) |
[production] |
| 06:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1200 (T343718)', diff saved to https://phabricator.wikimedia.org/P50796 and previous config saved to /var/cache/conftool/dbconfig/20230822-064716-ladsgroup.json |
[production] |
| 06:47 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:47 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T343718)', diff saved to https://phabricator.wikimedia.org/P50795 and previous config saved to /var/cache/conftool/dbconfig/20230822-064706-ladsgroup.json |
[production] |
| 06:46 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1116.eqiad.wmnet with OS bullseye |
[production] |
| 06:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T344589)', diff saved to https://phabricator.wikimedia.org/P50794 and previous config saved to /var/cache/conftool/dbconfig/20230822-064106-ladsgroup.json |
[production] |
| 06:40 |
<ladsgroup@deploy1002> |
ladsgroup: Continuing with sync |
[production] |
| 06:40 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1115.eqiad.wmnet with reason: host reimage |
[production] |
| 06:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P50793 and previous config saved to /var/cache/conftool/dbconfig/20230822-063923-ladsgroup.json |
[production] |
| 06:39 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:951366|Enable URL shortener in sidebar in jawiki and zhwiki (T267921)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 06:37 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:951366|Enable URL shortener in sidebar in jawiki and zhwiki (T267921)]] |
[production] |
| 06:36 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1115.eqiad.wmnet with reason: host reimage |
[production] |
| 06:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P50792 and previous config saved to /var/cache/conftool/dbconfig/20230822-063258-ladsgroup.json |
[production] |
| 06:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P50791 and previous config saved to /var/cache/conftool/dbconfig/20230822-063200-ladsgroup.json |
[production] |
| 06:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Set db2112 with weight 0 T344666', diff saved to https://phabricator.wikimedia.org/P50790 and previous config saved to /var/cache/conftool/dbconfig/20230822-062854-ladsgroup.json |
[production] |
| 06:27 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 35 hosts with reason: Primary switchover s1 T344666 |
[production] |
| 06:26 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 35 hosts with reason: Primary switchover s1 T344666 |
[production] |
| 06:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P50789 and previous config saved to /var/cache/conftool/dbconfig/20230822-062600-ladsgroup.json |
[production] |
| 06:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P50788 and previous config saved to /var/cache/conftool/dbconfig/20230822-062417-ladsgroup.json |
[production] |
| 06:21 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1115.eqiad.wmnet with OS bullseye |
[production] |
| 06:21 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1114.eqiad.wmnet with OS bullseye |
[production] |