5801-5850 of 10000 results (103ms)
2023-09-26 ยง
14:58 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:58 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding ganeti-test server to codfw - jhancock@cumin2002" [production]
14:57 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1021.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:57 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding ganeti-test server to codfw - jhancock@cumin2002" [production]
14:57 <jclark@cumin1001> START - Cookbook sre.dns.netbox [production]
14:54 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
14:53 <jclark@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:53 <jclark@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wdqs1017-20 - jclark@cumin1001" [production]
14:52 <jclark@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wdqs1017-20 - jclark@cumin1001" [production]
14:50 <jclark@cumin1001> START - Cookbook sre.dns.netbox [production]
14:49 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1021.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:47 <moritzm> installing lldpd security updates [production]
14:46 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2021.codfw.wmnet with OS bullseye [production]
14:45 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P52640 and previous config saved to /var/cache/conftool/dbconfig/20230926-144521-arnaudb.json [production]
14:38 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: sync [production]
14:38 <effie> Rump up traffic to mw-on-k8s to 6.5% - T346422 [production]
14:37 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: sync [production]
14:36 <ejegg> fundraising civicrm upgraded from 9efea665 to 41a4c2cf [production]
14:35 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1021.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:34 <jbond@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "puppetserver2002.codfw.wmnet - jbond@cumin2002" [production]
14:33 <ejegg> disabled recurring donations charge job for civi deploy [production]
14:30 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P52639 and previous config saved to /var/cache/conftool/dbconfig/20230926-143015-arnaudb.json [production]
14:27 <jbond@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "puppetserver2002.codfw.wmnet - jbond@cumin2002" [production]
14:25 <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host puppetserver2002.codfw.wmnet with OS bookworm [production]
14:25 <jbond@cumin2002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jbond@cumin2002" [production]
14:24 <jbond@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jbond@cumin2002" [production]
14:23 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host puppetserver1003.eqiad.wmnet with OS bookworm [production]
14:23 <jbond@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jbond@cumin1001" [production]
14:21 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1021.eqiad.wmnet with OS bullseye [production]
14:21 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs1021.eqiad.wmnet with OS bullseye [production]
14:17 <moritzm> prune obsolete nginx packages from durum hosts after migration to new library scheme T329529 [production]
14:16 <jbond@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jbond@cumin1001" [production]
14:15 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131 (T343198)', diff saved to https://phabricator.wikimedia.org/P52638 and previous config saved to /var/cache/conftool/dbconfig/20230926-141508-arnaudb.json [production]
14:13 <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetserver2002.codfw.wmnet with reason: host reimage [production]
14:10 <jbond@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on puppetserver2002.codfw.wmnet with reason: host reimage [production]
14:08 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs1017.eqiad.wmnet with OS bullseye [production]
14:08 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs1021.eqiad.wmnet with OS bullseye [production]
14:05 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetserver1003.eqiad.wmnet with reason: host reimage [production]
14:02 <Lucas_WMDE> UTC afternoon backport+config window done [production]
14:02 <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:961038|Enable Minerva site notice for wikifunctions wiki (T345463)]] (duration: 09m 51s) [production]
14:02 <jbond@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on puppetserver1003.eqiad.wmnet with reason: host reimage [production]
14:01 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase2021.codfw.wmnet with reason: host reimage [production]
13:57 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on restbase2021.codfw.wmnet with reason: host reimage [production]
13:55 <lucaswerkmeister-wmde@deploy2002> ammarpad and lucaswerkmeister-wmde: Continuing with sync [production]
13:54 <lucaswerkmeister-wmde@deploy2002> ammarpad and lucaswerkmeister-wmde: Backport for [[gerrit:961038|Enable Minerva site notice for wikifunctions wiki (T345463)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
13:52 <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:961038|Enable Minerva site notice for wikifunctions wiki (T345463)]] [production]
13:51 <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:961030|arwikisource: Increase autoconfirm edit count to 10 (T347264)]] (duration: 11m 27s) [production]
13:47 <Lucas_WMDE> lucaswerkmeister-wmde@deploy2002 ammarpad and lucaswerkmeister-wmde: Continuing with sync [originally 13:44 UTC] [production]
13:43 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host restbase2021.codfw.wmnet with OS bullseye [production]
13:43 <lucaswerkmeister-wmde@deploy2002> ammarpad and lucaswerkmeister-wmde: Backport for [[gerrit:961030|arwikisource: Increase autoconfirm edit count to 10 (T347264)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]