|
2024-08-26
ยง
|
| 22:59 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217 (T370903)', diff saved to https://phabricator.wikimedia.org/P67851 and previous config saved to /var/cache/conftool/dbconfig/20240826-225933-ladsgroup.json |
[production] |
| 22:51 |
<sukhe@cumin1002> |
START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox |
[production] |
| 22:44 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P67850 and previous config saved to /var/cache/conftool/dbconfig/20240826-224426-ladsgroup.json |
[production] |
| 22:36 |
<swfrench-wmf> |
running homer 'cr*codfw*' commit 'T372878' (remove old BGP session config for kubernetes2018, kubernetes2025) |
[production] |
| 22:29 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs-main.discovery.wmnet wdqs-scholarly.discovery.wmnet on all recursors |
[production] |
| 22:29 |
<bking@cumin2002> |
START - Cookbook sre.dns.wipe-cache wdqs-main.discovery.wmnet wdqs-scholarly.discovery.wmnet on all recursors |
[production] |
| 22:29 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P67849 and previous config saved to /var/cache/conftool/dbconfig/20240826-222919-ladsgroup.json |
[production] |
| 22:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1238 (T371742)', diff saved to https://phabricator.wikimedia.org/P67848 and previous config saved to /var/cache/conftool/dbconfig/20240826-222351-ladsgroup.json |
[production] |
| 22:23 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1238.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:23 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1238.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221 (T371742)', diff saved to https://phabricator.wikimedia.org/P67847 and previous config saved to /var/cache/conftool/dbconfig/20240826-222328-ladsgroup.json |
[production] |
| 22:17 |
<zabe@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1049500|Removing 'spamblacklistlog' right from usergroups (T367683)]] (duration: 06m 58s) |
[production] |
| 22:14 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2042.codfw.wmnet |
[production] |
| 22:14 |
<swfrench@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet |
[production] |
| 22:14 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217 (T370903)', diff saved to https://phabricator.wikimedia.org/P67846 and previous config saved to /var/cache/conftool/dbconfig/20240826-221411-ladsgroup.json |
[production] |
| 22:13 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2217 (T370903)', diff saved to https://phabricator.wikimedia.org/P67845 and previous config saved to /var/cache/conftool/dbconfig/20240826-221302-ladsgroup.json |
[production] |
| 22:13 |
<zabe@deploy1003> |
superpes, zabe: Continuing with sync |
[production] |
| 22:12 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2217.codfw.wmnet with reason: Maintenance |
[production] |
| 22:12 |
<swfrench-wmf> |
ran homer 'lsw1-a8-codfw*' commit 'T372878' |
[production] |
| 22:12 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2217.codfw.wmnet with reason: Maintenance |
[production] |
| 22:12 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2197.codfw.wmnet with reason: Maintenance |
[production] |
| 22:12 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2197.codfw.wmnet with reason: Maintenance |
[production] |
| 22:12 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2193 (T370903)', diff saved to https://phabricator.wikimedia.org/P67844 and previous config saved to /var/cache/conftool/dbconfig/20240826-221245-ladsgroup.json |
[production] |
| 22:12 |
<zabe@deploy1003> |
superpes, zabe: Backport for [[gerrit:1049500|Removing 'spamblacklistlog' right from usergroups (T367683)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 22:10 |
<zabe@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1049500|Removing 'spamblacklistlog' right from usergroups (T367683)]] |
[production] |
| 22:09 |
<zabe@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1051757|[sysop_plwiki] Change the logo/icon and the favicon (T368712)]], [[gerrit:1052063|[arbcom_itwiki] Enable importing from itwiki (T369264)]] (duration: 07m 13s) |
[production] |
| 22:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P67843 and previous config saved to /var/cache/conftool/dbconfig/20240826-220821-ladsgroup.json |
[production] |
| 22:05 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2042.codfw.wmnet with OS bullseye |
[production] |
| 22:05 |
<zabe@deploy1003> |
superpes, zabe: Continuing with sync |
[production] |
| 22:04 |
<zabe@deploy1003> |
superpes, zabe: Backport for [[gerrit:1051757|[sysop_plwiki] Change the logo/icon and the favicon (T368712)]], [[gerrit:1052063|[arbcom_itwiki] Enable importing from itwiki (T369264)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 22:02 |
<zabe@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1051757|[sysop_plwiki] Change the logo/icon and the favicon (T368712)]], [[gerrit:1052063|[arbcom_itwiki] Enable importing from itwiki (T369264)]] |
[production] |
| 22:01 |
<inflatador> |
bking@dns1004.wikimedia.org `sudo -i authdns-update` T364364 |
[production] |
| 21:57 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P67842 and previous config saved to /var/cache/conftool/dbconfig/20240826-215738-ladsgroup.json |
[production] |
| 21:53 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P67841 and previous config saved to /var/cache/conftool/dbconfig/20240826-215314-ladsgroup.json |
[production] |
| 21:45 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2042.codfw.wmnet with reason: host reimage |
[production] |
| 21:42 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P67840 and previous config saved to /var/cache/conftool/dbconfig/20240826-214230-ladsgroup.json |
[production] |
| 21:41 |
<swfrench@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2042.codfw.wmnet with reason: host reimage |
[production] |
| 21:38 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221 (T371742)', diff saved to https://phabricator.wikimedia.org/P67839 and previous config saved to /var/cache/conftool/dbconfig/20240826-213807-ladsgroup.json |
[production] |
| 21:31 |
<catrope@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1066881|Revert "Activates the "compact" Parsoid indicator on all wikivoyage wikis"]] (duration: 31m 21s) |
[production] |
| 21:27 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2193 (T370903)', diff saved to https://phabricator.wikimedia.org/P67838 and previous config saved to /var/cache/conftool/dbconfig/20240826-212723-ladsgroup.json |
[production] |
| 21:27 |
<catrope@deploy1003> |
catrope, trainbranchbot: Continuing with sync |
[production] |
| 21:25 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2193 (T370903)', diff saved to https://phabricator.wikimedia.org/P67837 and previous config saved to /var/cache/conftool/dbconfig/20240826-212513-ladsgroup.json |
[production] |
| 21:25 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2193.codfw.wmnet with reason: Maintenance |
[production] |
| 21:25 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2193.codfw.wmnet with reason: Maintenance |
[production] |
| 21:25 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T370903)', diff saved to https://phabricator.wikimedia.org/P67836 and previous config saved to /var/cache/conftool/dbconfig/20240826-212458-ladsgroup.json |
[production] |
| 21:24 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host <spicerack.netbox.NetboxServer object at 0x7fe3cb9c1700> |
[production] |
| 21:24 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2042 |
[production] |
| 21:23 |
<swfrench@cumin2002> |
START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2042 |
[production] |
| 21:23 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2042.codfw.wmnet 20.0.192.10.in-addr.arpa 0.2.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors |
[production] |
| 21:23 |
<swfrench@cumin2002> |
START - Cookbook sre.dns.wipe-cache wikikube-worker2042.codfw.wmnet 20.0.192.10.in-addr.arpa 0.2.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors |
[production] |