|
2022-12-16
§
|
| 08:43 |
<jayme@deploy1002> |
helmfile [staging] DONE helmfile.d/services/echostore: apply |
[production] |
| 08:42 |
<jayme@deploy1002> |
helmfile [staging] START helmfile.d/services/echostore: apply |
[production] |
| 08:41 |
<jayme@deploy1002> |
helmfile [staging] DONE helmfile.d/services/echostore: apply |
[production] |
| 08:41 |
<jayme@deploy1002> |
helmfile [staging] START helmfile.d/services/echostore: apply |
[production] |
| 08:35 |
<moritzm> |
power down ganeti5003 manually (mgmt/IPMI broken) for pending decom T322048 |
[production] |
| 08:25 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2002.wikimedia.org |
[production] |
| 08:18 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host bast2002.wikimedia.org |
[production] |
| 08:16 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org |
[production] |
| 08:10 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org |
[production] |
| 00:51 |
<mutante> |
puppetmasters - merged gerrit:868481 to "revert" gerrit:866644,ran puppet and 'systemctl reset-failed' via cumin on 10 masters, resolved monitoring alerts |
[production] |
| 00:22 |
<mutante> |
aphlict1001 - systemctl start man-db - T325246 |
[production] |
| 00:21 |
<mutante> |
aphlict1001 - systemctl start logrotate - T325246 |
[production] |
| 00:21 |
<mutante> |
aphlict1001 - :/var/log/aphlict# truncate aphlict.log --size 100M - T325246 |
[production] |
|
2022-12-15
§
|
| 23:10 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netmon1002.wikimedia.org |
[production] |
| 23:10 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 23:10 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netmon1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" |
[production] |
| 23:09 |
<denisse@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netmon1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" |
[production] |
| 23:05 |
<denisse@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 23:00 |
<denisse@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts netmon1002.wikimedia.org |
[production] |
| 22:49 |
<TheresNoTime> |
`[samtar@mwmaint1002 imports]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Coffeeandcrumbs /home/samtar/imports` T325330 |
[production] |
| 22:48 |
<denisse@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts netmon2001.wikimedia.org |
[production] |
| 22:48 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:47 |
<denisse@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:42 |
<denisse@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts netmon2001.wikimedia.org |
[production] |
| 22:11 |
<denisse@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts netmon2001.wikimedia.org |
[production] |
| 22:11 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:10 |
<denisse@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:02 |
<denisse@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts netmon2001.wikimedia.org |
[production] |
| 21:59 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) |
[production] |
| 21:59 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-reload |
[production] |
| 21:52 |
<TheresNoTime> |
done UTC late backport and config training |
[production] |
| 21:47 |
<samtar@deploy1002> |
Finished scap: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] (duration: 24m 38s) |
[production] |
| 21:42 |
<TheresNoTime> |
logging to note that this deploy is unusually slow, P42715 |
[production] |
| 21:35 |
<samtar@deploy1002> |
samtar and tgr: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 21:25 |
<bking@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) |
[production] |
| 21:24 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-reload |
[production] |
| 21:23 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) |
[production] |
| 21:23 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-reload |
[production] |
| 21:22 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] |
[production] |
| 21:17 |
<bking@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) |
[production] |
| 21:16 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-reload |
[production] |
| 21:15 |
<cwhite@cumin2002> |
conftool action : set/pooled=no; selector: name=logstash2024.codfw.wmnet,service=kibana7 |
[production] |
| 21:11 |
<samtar@deploy1002> |
Finished scap: Backport for [[gerrit:863434|Update reference to CommandLineInc (T184782)]] (duration: 08m 18s) |
[production] |
| 21:04 |
<samtar@deploy1002> |
samtar and zabe: Backport for [[gerrit:863434|Update reference to CommandLineInc (T184782)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 21:03 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:863434|Update reference to CommandLineInc (T184782)]] |
[production] |
| 20:22 |
<bking@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) |
[production] |
| 20:21 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-reload |
[production] |
| 20:20 |
<bking@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) |
[production] |
| 20:19 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-reload |
[production] |
| 20:02 |
<denisse@cumin1001> |
END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-eqiad cluster: Reboot kafka nodes |
[production] |