601-650 of 10000 results (83ms)
2023-10-20 §
15:36 <ebernhardson@deploy2002> Finished deploy [airflow-dags/search@fd88cfa]: Update kafka hosts mjolnir communicates with (duration: 00m 27s) [production]
15:36 <ebernhardson@deploy2002> Started deploy [airflow-dags/search@fd88cfa]: Update kafka hosts mjolnir communicates with [production]
15:26 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 12 hosts with reason: changing bgp config on esams switches [production]
15:25 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 12 hosts with reason: changing bgp config on esams switches [production]
15:18 <topranks> Disabling BGP from asw1-b13-drmrs to cr1-drmrs to move BGP peers to new group T349125 [production]
15:16 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
15:15 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
15:13 <topranks> Disabling BGP from asw1-b13-drmrs to cr2-drmrs to move BGP peers to new group T349125 [production]
15:09 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 12 hosts with reason: changing bgp config on drmrs switches [production]
15:09 <topranks> Disabling BGP from asw1-b12-drmrs to cr2-drmrs to move BGP peers to new group T349125 [production]
15:08 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 12 hosts with reason: changing bgp config on drmrs switches [production]
14:57 <topranks> Disabling BGP from asw1-b12-drmrs to cr1-drmrs to move BGP peers to new group T349125 [production]
14:49 <ejegg> payments-wiki upgraded from 87cda414 to 7575f0e6 [production]
14:33 <topranks> Disabling BGP from ssw1-f1-eqiad to cr2-eqiad to move BGP peers to new group T349125 [production]
14:08 <ejegg|away> fundraising civicrm upgraded from f11ad380 to e57425a9 [production]
13:19 <topranks> Disabling BGP from ssw1-e1-eqiad to cr1-eqiad to move BGP peers to new group T349125 [production]
12:11 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-be1003.eqiad.wmnet with OS bullseye [production]
11:55 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-be1003.eqiad.wmnet with reason: host reimage [production]
11:52 <jclark@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-be1003.eqiad.wmnet with reason: host reimage [production]
11:42 <jynus> refactoring tables @ db1164[bbackups] T349360 [production]
11:37 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host moss-be1003.eqiad.wmnet with OS bullseye [production]
11:36 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host moss-be1003.eqiad.wmnet with OS bullseye [production]
10:46 <kevinbazira@deploy2002> helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
10:46 <kevinbazira@deploy2002> helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
10:39 <kevinbazira@deploy2002> helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
10:19 <godog> powercycle titan1001 [production]
10:13 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:13 <ayounsi@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add A/PTR for lsw1-f8/ssw links - ayounsi@cumin1001" [production]
10:12 <ayounsi@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add A/PTR for lsw1-f8/ssw links - ayounsi@cumin1001" [production]
10:04 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
09:58 <kevinbazira@deploy2002> helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
08:45 <brouberol@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts kafka-jumbo1006.eqiad.wmnet [production]
08:43 <brouberol@cumin1001> START - Cookbook sre.hosts.decommission for hosts kafka-jumbo1006.eqiad.wmnet [production]
07:43 <isaranto@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
07:29 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:29 <ayounsi@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add A/PTR for lsw1-f8/ssw links - ayounsi@cumin1001" [production]
07:26 <ayounsi@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add A/PTR for lsw1-f8/ssw links - ayounsi@cumin1001" [production]
07:25 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on etherpad1003.eqiad.wmnet with reason: Reboot to use new CPU and memory config [production]
07:24 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on etherpad1003.eqiad.wmnet with reason: Reboot to use new CPU and memory config [production]
07:22 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
07:21 <jelto> increase etherpad1003 CPU and memory (1CPU,1GB -> 2CPU,2GB) - T348386 [production]
06:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1119 from dbctl T349272', diff saved to https://phabricator.wikimedia.org/P53021 and previous config saved to /var/cache/conftool/dbconfig/20231020-061822-marostegui.json [production]
03:15 <tstarling@deploy2002> Synchronized wmf-config/InitialiseSettings.php: Enable source maps everywhere T47514 (duration: 06m 26s) [production]
03:03 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host moss-be1003.eqiad.wmnet with OS bullseye [production]
2023-10-19 §
22:37 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with reboot policy FORCED [production]
22:36 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with reboot policy FORCED [production]
21:32 <hmonroy@deploy2002> Finished scap: Backport for [[gerrit:967188|PhonosButton: use text() instead of append() (T349312)]] (duration: 06m 48s) [production]
21:28 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host moss-be1003.eqiad.wmnet with OS bullseye [production]
21:27 <hmonroy@deploy2002> hmonroy: Continuing with sync [production]
21:27 <hmonroy@deploy2002> hmonroy: Backport for [[gerrit:967188|PhonosButton: use text() instead of append() (T349312)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]