2301-2350 of 10000 results (41ms)
2023-06-30 §
10:22 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
10:20 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
10:15 <jbond@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest1003'] [production]
10:15 <jbond@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['sretest1003'] [production]
10:14 <jbond@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest1003'] [production]
10:13 <jbond@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['sretest1003'] [production]
10:11 <jbond@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest1003'] [production]
08:00 <jayme> rolled back envoyproxy package in buster-wikimedia component/envoy-future to 1.18.3-1 - T300324 [production]
07:52 <jayme> removed docker-registry.discovery.wmnet/envoy-future:1.26.1-1 - T300324 [production]
06:41 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on urldownloader[2001-2002].wikimedia.org with reason: pending decom [production]
06:41 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on urldownloader[2001-2002].wikimedia.org with reason: pending decom [production]
06:40 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on urldownloader[1001-1002].wikimedia.org with reason: Setup in progress [production]
06:40 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on urldownloader[1001-1002].wikimedia.org with reason: Setup in progress [production]
2023-06-29 §
21:28 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [production]
21:25 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [production]
21:22 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
21:18 <samtar@deploy1002> Finished scap: Backport for [[gerrit:934391|IS: Phonos, reorder and enable for mediawikiwiki (T336763)]] (duration: 08m 26s) [production]
21:11 <samtar@deploy1002> samtar: Backport for [[gerrit:934391|IS: Phonos, reorder and enable for mediawikiwiki (T336763)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
21:10 <samtar@deploy1002> Started scap: Backport for [[gerrit:934391|IS: Phonos, reorder and enable for mediawikiwiki (T336763)]] [production]
20:13 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS bullseye [production]
20:01 <mutante> contint* servers: restarted apache after deploying gerrit:932435 [production]
19:50 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2021.codfw.wmnet with OS bullseye [production]
19:49 <wm-bot2> Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye [admin]
19:49 <wm-bot2> Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye [admin]
19:49 <wm-bot2> Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:49 <wm-bot2> Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:49 <wm-bot2> Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye [admin]
19:49 <wm-bot2> Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye [admin]
19:48 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
19:48 <wm-bot2> Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
19:48 <wm-bot2> Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:48 <wm-bot2> Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:48 <wm-bot2> Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:48 <wm-bot2> Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
19:47 <wm-bot2> Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
19:34 <wm-bot2> Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye [admin]
19:34 <wm-bot2> Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye [admin]
19:34 <wm-bot2> Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:34 <wm-bot2> Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:34 <wm-bot2> Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye [admin]
19:33 <wm-bot2> Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye [admin]
19:33 <wm-bot2> Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye [admin]
19:33 <wm-bot2> Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye [admin]
19:33 <wm-bot2> Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye [admin]
19:33 <wm-bot2> Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:33 <wm-bot2> Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:33 <wm-bot2> Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
19:30 <urbanecm@deploy1002> helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply [production]
19:30 <wm-bot2> Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
19:30 <urbanecm@deploy1002> helmfile [codfw] START helmfile.d/services/linkrecommendation: apply [production]