951-1000 of 10000 results (77ms)
2023-08-02 §
06:12 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
06:11 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
06:10 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
05:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1130 (re)pooling @ 5%: Repooling after replacing its memory', diff saved to https://phabricator.wikimedia.org/P49946 and previous config saved to /var/cache/conftool/dbconfig/20230802-055916-root.json [production]
05:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1130 (re)pooling @ 3%: Repooling after replacing its memory', diff saved to https://phabricator.wikimedia.org/P49945 and previous config saved to /var/cache/conftool/dbconfig/20230802-054411-root.json [production]
05:33 <oblivian@deploy1002> Synchronized wmf-config/InitialiseSettings.php: enabling emergency captcha on enwiki - T343294 (take 2) (duration: 06m 40s) [production]
05:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1130 (re)pooling @ 1%: Repooling after replacing its memory', diff saved to https://phabricator.wikimedia.org/P49944 and previous config saved to /var/cache/conftool/dbconfig/20230802-052906-root.json [production]
05:23 <marostegui> Stop mariadb on es2025 for onsite maintenance dbmaint codfw T343254 [production]
05:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2025 T343254', diff saved to https://phabricator.wikimedia.org/P49943 and previous config saved to /var/cache/conftool/dbconfig/20230802-052021-root.json [production]
05:11 <oblivian@deploy1002> Synchronized wmf-config/InitialiseSettings.php: enabling emergency captcha on enwiki - T343294 (duration: 06m 36s) [production]
04:49 <_joe_> running scap pull on mwmaint1002 to pick up the noc.w.o changes [production]
01:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['pc2015'] [production]
01:24 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['pc2015'] [production]
01:20 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2015.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:14 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host pc2016.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:04 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host pc2016.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:02 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
01:02 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: setup switch port and DNS for pc2016 - pt1979@cumin2002" [production]
01:02 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: setup switch port and DNS for pc2016 - pt1979@cumin2002" [production]
00:59 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
00:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2005-dev.codfw.wmnet with OS bullseye [production]
00:58 <pt1979@cumin2002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
00:52 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host pc2015.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:41 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
00:41 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: setup switch interfaces and DNS for pc201[5-6] - pt1979@cumin2002" [production]
00:40 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: setup switch interfaces and DNS for pc201[5-6] - pt1979@cumin2002" [production]
00:38 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
00:37 <pt1979@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
00:37 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
00:36 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
00:30 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2004-dev.codfw.wmnet with OS bullseye [production]
00:30 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
00:29 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
00:29 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2006-dev.codfw.wmnet with OS bullseye [production]
00:29 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
00:25 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
00:20 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2005-dev.codfw.wmnet with reason: host reimage [production]
00:17 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2005-dev.codfw.wmnet with reason: host reimage [production]
00:14 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2004-dev.codfw.wmnet with reason: host reimage [production]
00:11 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2004-dev.codfw.wmnet with reason: host reimage [production]
00:09 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2006-dev.codfw.wmnet with reason: host reimage [production]
00:06 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2006-dev.codfw.wmnet with reason: host reimage [production]
2023-08-01 §
23:13 <eileen> config revision changed from 8b3a46c3 to f5e6425b - updated process controll (added segmentation_aging job - rollback if it doesn't work) [production]
22:53 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudvirt2006-dev.codfw.wmnet with OS bullseye [production]
22:40 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudvirt2005-dev.codfw.wmnet with OS bullseye [production]
22:29 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudvirt2004-dev.codfw.wmnet with OS bullseye [production]
22:25 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudvirt2006-dev'] [production]
22:23 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudvirt2005-dev'] [production]
22:23 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudvirt2004-dev'] [production]
22:19 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt2006-dev'] [production]