3851-3900 of 10000 results (94ms)
2022-12-16 §
09:13 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:13 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti5003.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
09:11 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti5003.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:53 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:45 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/sessionstore: apply [production]
08:45 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/sessionstore: apply [production]
08:45 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti5003.eqsin.wmnet [production]
08:43 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/echostore: apply [production]
08:42 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/echostore: apply [production]
08:41 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/echostore: apply [production]
08:41 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/echostore: apply [production]
08:35 <moritzm> power down ganeti5003 manually (mgmt/IPMI broken) for pending decom T322048 [production]
08:25 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2002.wikimedia.org [production]
08:18 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host bast2002.wikimedia.org [production]
08:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org [production]
08:10 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org [production]
00:51 <mutante> puppetmasters - merged gerrit:868481 to "revert" gerrit:866644,ran puppet and 'systemctl reset-failed' via cumin on 10 masters, resolved monitoring alerts [production]
00:22 <mutante> aphlict1001 - systemctl start man-db - T325246 [production]
00:21 <mutante> aphlict1001 - systemctl start logrotate - T325246 [production]
00:21 <mutante> aphlict1001 - :/var/log/aphlict# truncate aphlict.log --size 100M - T325246 [production]
2022-12-15 §
23:10 <denisse@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netmon1002.wikimedia.org [production]
23:10 <denisse@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:10 <denisse@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netmon1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" [production]
23:09 <denisse@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netmon1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" [production]
23:05 <denisse@cumin1001> START - Cookbook sre.dns.netbox [production]
23:00 <denisse@cumin1001> START - Cookbook sre.hosts.decommission for hosts netmon1002.wikimedia.org [production]
22:49 <TheresNoTime> `[samtar@mwmaint1002 imports]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Coffeeandcrumbs /home/samtar/imports` T325330 [production]
22:48 <denisse@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts netmon2001.wikimedia.org [production]
22:48 <denisse@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:47 <denisse@cumin1001> START - Cookbook sre.dns.netbox [production]
22:42 <denisse@cumin1001> START - Cookbook sre.hosts.decommission for hosts netmon2001.wikimedia.org [production]
22:11 <denisse@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts netmon2001.wikimedia.org [production]
22:11 <denisse@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:10 <denisse@cumin1001> START - Cookbook sre.dns.netbox [production]
22:02 <denisse@cumin1001> START - Cookbook sre.hosts.decommission for hosts netmon2001.wikimedia.org [production]
21:59 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
21:59 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
21:52 <TheresNoTime> done UTC late backport and config training [production]
21:47 <samtar@deploy1002> Finished scap: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] (duration: 24m 38s) [production]
21:42 <TheresNoTime> logging to note that this deploy is unusually slow, P42715 [production]
21:35 <samtar@deploy1002> samtar and tgr: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:25 <bking@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
21:24 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
21:23 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
21:23 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
21:22 <samtar@deploy1002> Started scap: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] [production]
21:17 <bking@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
21:16 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
21:15 <cwhite@cumin2002> conftool action : set/pooled=no; selector: name=logstash2024.codfw.wmnet,service=kibana7 [production]
21:11 <samtar@deploy1002> Finished scap: Backport for [[gerrit:863434|Update reference to CommandLineInc (T184782)]] (duration: 08m 18s) [production]