5751-5800 of 10000 results (87ms)
2022-12-16 §
08:10 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org [production]
00:51 <mutante> puppetmasters - merged gerrit:868481 to "revert" gerrit:866644,ran puppet and 'systemctl reset-failed' via cumin on 10 masters, resolved monitoring alerts [production]
00:22 <mutante> aphlict1001 - systemctl start man-db - T325246 [production]
00:21 <mutante> aphlict1001 - systemctl start logrotate - T325246 [production]
00:21 <mutante> aphlict1001 - :/var/log/aphlict# truncate aphlict.log --size 100M - T325246 [production]
2022-12-15 §
23:10 <denisse@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netmon1002.wikimedia.org [production]
23:10 <denisse@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:10 <denisse@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netmon1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" [production]
23:09 <denisse@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netmon1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - denisse@cumin1001" [production]
23:05 <denisse@cumin1001> START - Cookbook sre.dns.netbox [production]
23:00 <denisse@cumin1001> START - Cookbook sre.hosts.decommission for hosts netmon1002.wikimedia.org [production]
22:49 <TheresNoTime> `[samtar@mwmaint1002 imports]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Coffeeandcrumbs /home/samtar/imports` T325330 [production]
22:48 <denisse@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts netmon2001.wikimedia.org [production]
22:48 <denisse@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:47 <denisse@cumin1001> START - Cookbook sre.dns.netbox [production]
22:42 <denisse@cumin1001> START - Cookbook sre.hosts.decommission for hosts netmon2001.wikimedia.org [production]
22:11 <denisse@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts netmon2001.wikimedia.org [production]
22:11 <denisse@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:10 <denisse@cumin1001> START - Cookbook sre.dns.netbox [production]
22:02 <denisse@cumin1001> START - Cookbook sre.hosts.decommission for hosts netmon2001.wikimedia.org [production]
21:59 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
21:59 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
21:52 <TheresNoTime> done UTC late backport and config training [production]
21:47 <samtar@deploy1002> Finished scap: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] (duration: 24m 38s) [production]
21:42 <TheresNoTime> logging to note that this deploy is unusually slow, P42715 [production]
21:35 <samtar@deploy1002> samtar and tgr: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:25 <bking@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
21:24 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
21:23 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
21:23 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
21:22 <samtar@deploy1002> Started scap: Backport for [[gerrit:868364|NewImpact: Use "View all edits" in footer (T325216)]] [production]
21:17 <bking@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
21:16 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
21:15 <cwhite@cumin2002> conftool action : set/pooled=no; selector: name=logstash2024.codfw.wmnet,service=kibana7 [production]
21:11 <samtar@deploy1002> Finished scap: Backport for [[gerrit:863434|Update reference to CommandLineInc (T184782)]] (duration: 08m 18s) [production]
21:04 <samtar@deploy1002> samtar and zabe: Backport for [[gerrit:863434|Update reference to CommandLineInc (T184782)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
21:03 <samtar@deploy1002> Started scap: Backport for [[gerrit:863434|Update reference to CommandLineInc (T184782)]] [production]
20:22 <bking@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
20:21 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
20:20 <bking@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
20:19 <bking@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
20:02 <denisse@cumin1001> END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-eqiad cluster: Reboot kafka nodes [production]
19:35 <mutante> short downtime for misc websites, iegreview, racktables, transparency.wm, annual.wm, design.wm, sitemaps.wm, research.wm, bienvenida.wm, wikiworkshop.org [production]
19:08 <xcollazo@deploy1002> Finished deploy [airflow-dags/platform_eng@d23127b]: Deploying Spark3 upgrade of image_suggestions job to the platform_eng Airflow instance. (duration: 00m 10s) [production]
19:07 <xcollazo@deploy1002> Started deploy [airflow-dags/platform_eng@d23127b]: Deploying Spark3 upgrade of image_suggestions job to the platform_eng Airflow instance. [production]
18:50 <robh> starting msw2-ulsfo swap for rack .23, mgmt will flap with no expected user impact [production]
18:48 <denisse@cumin1001> START - Cookbook sre.kafka.reboot-workers for Kafka logging-eqiad cluster: Reboot kafka nodes [production]
18:37 <denisse@cumin1001> END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-codfw cluster: Reboot kafka nodes [production]
18:08 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1010.eqiad.wmnet with OS bullseye [production]
18:08 <btullis@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1001" [production]