901-950 of 10000 results (34ms)
2023-06-05 §
07:23 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
07:23 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
07:23 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
07:21 <taavi@deploy1002> Finished scap: Backport for [[gerrit:926497|[SearchVue] Enable on Norwegian, Hungarian, Catalan, Dutch, and Ukrainian (T336870)]] (duration: 18m 27s) [production]
07:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1001.eqiad.wmnet [production]
07:12 <taavi@deploy1002> mlitn and taavi: Backport for [[gerrit:926497|[SearchVue] Enable on Norwegian, Hungarian, Catalan, Dutch, and Ukrainian (T336870)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
07:09 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host krb1001.eqiad.wmnet [production]
07:02 <taavi@deploy1002> Started scap: Backport for [[gerrit:926497|[SearchVue] Enable on Norwegian, Hungarian, Catalan, Dutch, and Ukrainian (T336870)]] [production]
06:20 <_joe_> killing a pod with consistently high haproxy queue for thumbor in codfw [production]
06:16 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 60427 [production]
06:15 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 60427 [production]
2023-06-04 §
22:05 <wm-bot> <lucaswerkmeister> deployed 08962e4902 (update past transgressive item ID after merge; only affects czech-verb-perfective) [tools.lexeme-forms]
18:28 <legoktm> sudo deleted job 490152 per IRC request from GreenC [tools.botwikiawk]
09:17 <wm-bot> <valeriobozzolan> Update homepage info in https://itwiki.toolforge.org/ [tools.itwiki]
08:57 <wm-bot> <valeriobozzolan> ehm, this SAL works [tools.itwiki]
08:40 <wm-bot> <valeriobozzolan> Testing SAL [tools.itwiki]
2023-06-03 §
13:41 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on an-test-worker1001.eqiad.wmnet with reason: Host under testing/upgrade [production]
13:41 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on an-test-worker1001.eqiad.wmnet with reason: Host under testing/upgrade [production]
13:28 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs2012.codfw.wmnet [production]
13:28 <bking@cumin1001> START - Cookbook sre.hosts.remove-downtime for wdqs2012.codfw.wmnet [production]
2023-06-02 §
21:44 <mutante> based on T336427#8890714 pending a response, everything already copied to gerrit1003, and extra paths being added to Bacule... shell access to gerrit1001 will be removed next week [releng]
20:16 <apergos> rsync in ariel screen session, bwlimit 100000, running on dumpsdata1003, pulling from dumpsdata1002, copying over 'other dumps' [production]
18:42 <bblack> dns*: puppets are all re-enabled, ntp restarts are done, etc [production]
18:16 <Krinkle> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/926593 [releng]
17:48 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:48 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a1-codfw - pt1979@cumin2002" [production]
17:47 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a1-codfw - pt1979@cumin2002" [production]
17:45 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
17:45 <pt1979@cumin2002> START - Cookbook sre.network.provision for device ssw1-a1-codfw.mgmt.codfw.wmnet [production]
17:27 <bblack> dns*: disabling puppet to control rollout of NTP config fixups [production]
16:03 <bblack> dns*: removed faulty authdns[12]001 lines from /etc/hosts via cumin+sed [production]
15:35 <sukhe> restart ntp.service on dns1002 [production]
14:40 <wm-bot2> Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by arturo@nostromo [admin]
13:26 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
13:26 <otto@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
13:25 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
13:25 <otto@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
13:25 <ottomata> deploying flink-operator change to dse-k8s and wikikube to add ingress for health check port - https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/926479 [production]
13:24 <otto@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:24 <otto@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
13:24 <otto@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
13:24 <otto@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
13:22 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:22 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
12:59 <wm-bot2> Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
12:58 <wm-bot2> Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
12:58 <wm-bot2> Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
12:44 <wm-bot2> Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye [admin]
12:44 <wm-bot2> Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]
12:43 <wm-bot2> Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye [admin]