1101-1150 of 10000 results (96ms)
2024-07-01 ยง
20:23 <cjming@deploy1002> jdlrobson, cjming: Backport for [[gerrit:1050084|[July 1st] Mobile: Enable dark mode for all tier 1 wikis (logged in) (T367151)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2145 (T364069)', diff saved to https://phabricator.wikimedia.org/P65600 and previous config saved to /var/cache/conftool/dbconfig/20240701-201949-marostegui.json [production]
20:03 <cjming@deploy1002> Started scap sync-world: Backport for [[gerrit:1050084|[July 1st] Mobile: Enable dark mode for all tier 1 wikis (logged in) (T367151)]] [production]
19:33 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
19:19 <dancy@deploy1002> Installation of scap version "4.91.0" completed for 233 hosts [production]
19:19 <dancy@deploy1002> Installing scap version "4.91.0" for 233 hosts [production]
19:16 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1041.eqiad.wmnet with reason: host reimage [production]
19:15 <dancy@deploy1002> Installing scap version "4.91.0" for 234 hosts [production]
19:14 <dancy@deploy1002> Installing scap version "4.91.0" for 234 hosts [production]
19:13 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1041.eqiad.wmnet with reason: host reimage [production]
18:57 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd1041.eqiad.wmnet with OS bullseye [production]
18:56 <jclark@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1041 [production]
18:56 <jclark@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1041 [production]
17:49 <jclark@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1041 [production]
17:49 <jclark@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1041 [production]
17:49 <jclark@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:49 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt for dbproxy1028,9 - jclark@cumin1002" [production]
17:48 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt for dbproxy1028,9 - jclark@cumin1002" [production]
17:45 <jclark@cumin1002> START - Cookbook sre.dns.netbox [production]
17:44 <jclark@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1041 [production]
17:44 <jclark@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1041 [production]
17:42 <jclark@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1041 [production]
17:42 <jclark@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1041 [production]
17:41 <jclark@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1041 [production]
17:41 <jclark@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1041 [production]
17:37 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1041.eqiad.wmnet with OS bullseye [production]
17:36 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr2-codfw,ssw1-a[1,8]-codfw.mgmt with reason: reboot ssw1-d8-codfw [production]
17:35 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on cr2-codfw,ssw1-a[1,8]-codfw.mgmt with reason: reboot ssw1-d8-codfw [production]
17:27 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1039.eqiad.wmnet with OS bullseye [production]
17:27 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
17:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2145 (T364069)', diff saved to https://phabricator.wikimedia.org/P65599 and previous config saved to /var/cache/conftool/dbconfig/20240701-171609-marostegui.json [production]
17:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2145.codfw.wmnet with reason: Maintenance [production]
17:15 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2145.codfw.wmnet with reason: Maintenance [production]
17:08 <klausman@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
17:08 <klausman@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
17:05 <klausman@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
17:04 <klausman@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
16:51 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
16:51 <swfrench@deploy1002> helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply [production]
16:51 <swfrench@deploy1002> helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply [production]
16:38 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1039.eqiad.wmnet with reason: host reimage [production]
16:35 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1039.eqiad.wmnet with reason: host reimage [production]
16:34 <dancy@deploy1002> Installing scap version "4.90.0" for 234 hosts [production]
16:34 <dancy@deploy1002> Installing scap version "4.90.0" for 234 hosts [production]
16:33 <dancy@deploy1002> Installing scap version "4.90.0" for 234 hosts [production]
16:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1182 (T367856)', diff saved to https://phabricator.wikimedia.org/P65598 and previous config saved to /var/cache/conftool/dbconfig/20240701-163010-marostegui.json [production]
16:30 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
16:29 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
16:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T367856)', diff saved to https://phabricator.wikimedia.org/P65597 and previous config saved to /var/cache/conftool/dbconfig/20240701-162948-marostegui.json [production]
16:22 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd1039.eqiad.wmnet with OS bullseye [production]