|
2025-01-23
ยง
|
| 13:49 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_eqiad |
[production] |
| 13:49 |
<vgutierrez@cumin1002> |
END (PASS) - Cookbook sre.cdn.roll-restart-tcp-mss-clamper (exit_code=0) rolling restart_daemons on A:cp-text_esams |
[production] |
| 13:46 |
<jynus@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1239.eqiad.wmnet with reason: reimage |
[production] |
| 13:42 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1005 to wikikube-worker1146 |
[production] |
| 13:42 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1146 |
[production] |
| 13:41 |
<kamila@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:41 |
<godog> |
bounce mtail on centrallog2002 - high system cpu usage and perf top reports native_queued_spin_lock_slowpath |
[production] |
| 13:41 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.rename from parse1006 to wikikube-worker1147 |
[production] |
| 13:38 |
<kamila@cumin1002> |
START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1146 |
[production] |
| 13:38 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 13:38 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1005 to wikikube-worker1146 - kamila@cumin1002" |
[production] |
| 13:37 |
<kamila@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1005 to wikikube-worker1146 - kamila@cumin1002" |
[production] |
| 13:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2166 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72256 and previous config saved to /var/cache/conftool/dbconfig/20250123-133311-root.json |
[production] |
| 13:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1165 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72255 and previous config saved to /var/cache/conftool/dbconfig/20250123-133304-root.json |
[production] |
| 13:31 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_esams |
[production] |
| 13:31 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1113799|file: Add caller to write queries (T384481)]] (duration: 09m 43s) |
[production] |
| 13:30 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1004 to wikikube-worker1145 |
[production] |
| 13:30 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1145 |
[production] |
| 13:29 |
<cmooney@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow1002.eqiad.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd |
[production] |
| 13:28 |
<kamila@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:28 |
<kamila@cumin1002> |
START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1145 |
[production] |
| 13:28 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 13:28 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1004 to wikikube-worker1145 - kamila@cumin1002" |
[production] |
| 13:28 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.rename from parse1005 to wikikube-worker1146 |
[production] |
| 13:28 |
<kamila@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1004 to wikikube-worker1145 - kamila@cumin1002" |
[production] |
| 13:25 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1003 to wikikube-worker1144 |
[production] |
| 13:24 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1144 |
[production] |
| 13:24 |
<ladsgroup@deploy2002> |
ladsgroup: Continuing with sync |
[production] |
| 13:24 |
<ladsgroup@deploy2002> |
ladsgroup: Backport for [[gerrit:1113799|file: Add caller to write queries (T384481)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:23 |
<kamila@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:23 |
<kamila@cumin1002> |
START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1144 |
[production] |
| 13:23 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 13:23 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1003 to wikikube-worker1144 - kamila@cumin1002" |
[production] |
| 13:23 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.rename from parse1004 to wikikube-worker1145 |
[production] |
| 13:23 |
<kamila@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1003 to wikikube-worker1144 - kamila@cumin1002" |
[production] |
| 13:21 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1113799|file: Add caller to write queries (T384481)]] |
[production] |
| 13:20 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1002 to wikikube-worker1143 |
[production] |
| 13:20 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1143 |
[production] |
| 13:19 |
<kamila@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:19 |
<kamila@cumin1002> |
START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1143 |
[production] |
| 13:19 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 13:18 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1002 to wikikube-worker1143 - kamila@cumin1002" |
[production] |
| 13:18 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.rename from parse1003 to wikikube-worker1144 |
[production] |
| 13:18 |
<kamila@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1002 to wikikube-worker1143 - kamila@cumin1002" |
[production] |
| 13:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2166 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72253 and previous config saved to /var/cache/conftool/dbconfig/20250123-131805-root.json |
[production] |
| 13:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1165 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72252 and previous config saved to /var/cache/conftool/dbconfig/20250123-131758-root.json |
[production] |
| 13:17 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1162.eqiad.wmnet with reason: Index rebuild |
[production] |
| 13:16 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1001 to wikikube-worker1142 |
[production] |
| 13:16 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1142 |
[production] |
| 13:15 |
<kamila@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |