6951-7000 of 10000 results (26ms)
2023-01-19 §
06:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db2118 T327372', diff saved to https://phabricator.wikimedia.org/P43190 and previous config saved to /var/cache/conftool/dbconfig/20230119-060449-ladsgroup.json [production]
06:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Promote db2121 to s7 primary T327372', diff saved to https://phabricator.wikimedia.org/P43189 and previous config saved to /var/cache/conftool/dbconfig/20230119-060316-ladsgroup.json [production]
06:02 <Amir1> Starting s7 codfw failover from db2118 to db2121 - T327372 [production]
05:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set db2121 with weight 0 T327372', diff saved to https://phabricator.wikimedia.org/P43188 and previous config saved to /var/cache/conftool/dbconfig/20230119-054243-ladsgroup.json [production]
05:42 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s7 T327372 [production]
05:41 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 30 hosts with reason: Primary switchover s7 T327372 [production]
2023-01-18 §
23:47 <zabe> run populateCulComment.php on all group0 and group1 wikis # T327290 [production]
23:42 <cstone> civicrm upgraded from 164270b0 to f6093fb2 [production]
22:35 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: raise heap memory to 12G - bking@cumin1001 - T323646 [production]
22:32 <wm-bot2> Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 347cb75e-215e-4b85-ae14-4ce1934c70c7, use this to unset) - cookbook ran by andrew@bullseye [admin]
22:03 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: raise heap memory to 12G - bking@cumin1001 - T323646 [production]
22:01 <wm-bot2> Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 9bf6212f-7fdb-4869-8190-b07387b2bc7e, use this to unset) - cookbook ran by andrew@bullseye [admin]
21:50 <kindrobot> close UTC late backport window [production]
21:50 <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:881462|[config]: Undeploy GDI Safety Survey Wave 4 (T327296)]] (duration: 10m 45s) [production]
21:41 <kindrobot@deploy1002> essexigyan and kindrobot: Backport for [[gerrit:881462|[config]: Undeploy GDI Safety Survey Wave 4 (T327296)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:40 <wm-bot2> Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 11aec8ea-f443-41ae-b79a-5e1e3aa94546, use this to unset) - cookbook ran by andrew@bullseye [admin]
21:39 <kindrobot@deploy1002> Started scap: Backport for [[gerrit:881462|[config]: Undeploy GDI Safety Survey Wave 4 (T327296)]] [production]
21:36 <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]] (duration: 13m 01s) [production]
21:33 <dancy> Rebooting deployment-jobrunner04 for T327329 [releng]
21:25 <kindrobot@deploy1002> kindrobot and jdlrobson: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:23 <kindrobot@deploy1002> Started scap: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]] [production]
21:08 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1037.eqiad.wmnet with OS bullseye [production]
21:05 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1036.eqiad.wmnet with OS bullseye [production]
21:03 <kindrobot> start UTC late backport window [production]
20:54 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1037.eqiad.wmnet with reason: host reimage [production]
20:51 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1036.eqiad.wmnet with reason: host reimage [production]
20:49 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1037.eqiad.wmnet with reason: host reimage [production]
20:48 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1036.eqiad.wmnet with reason: host reimage [production]
20:36 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1037.eqiad.wmnet with OS bullseye [production]
20:35 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1036.eqiad.wmnet with OS bullseye [production]
20:34 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: installation failed due to read-only database [production]
20:34 <aokoth@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: installation failed due to read-only database [production]
20:20 <wm-bot2> Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye [admin]
20:20 <wm-bot2> Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye [admin]
20:20 <wm-bot2> Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
20:20 <wm-bot2> Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
20:19 <wm-bot2> Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye [admin]
20:19 <wm-bot2> Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye [admin]
20:19 <wm-bot2> Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
20:19 <wm-bot2> Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
20:19 <wm-bot2> Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
20:18 <wm-bot2> Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
20:18 <wm-bot2> Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
20:18 <wm-bot2> Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
19:54 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1037.eqiad.wmnet with OS buster [production]
19:54 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:52 <bblack> db1129 and lvs1017: removed misconfigured IP address in wrong vlan from eno1 and /e/n/i [production]
19:51 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:47 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1036.eqiad.wmnet with OS buster [production]
19:47 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]