4851-4900 of 10000 results (94ms)
2023-05-30 §
07:03 <slyngshede@cumin1001> START - Cookbook sre.ganeti.makevm for new host testvm2006.codfw.wmnet [production]
07:02 <slyngshede@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=2) for new host testvm2006.codfw.wmnet [production]
07:02 <slyngshede@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors [production]
07:02 <slyngshede@cumin1001> START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors [production]
07:02 <slyngshede@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:02 <slyngshede@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" [production]
07:01 <slyngshede@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48629 and previous config saved to /var/cache/conftool/dbconfig/20230530-065932-root.json [production]
06:58 <slyngshede@cumin1001> START - Cookbook sre.dns.netbox [production]
06:58 <slyngshede@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2006.codfw.wmnet - slyngshede@cumin1001" [production]
06:57 <slyngshede@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2006.codfw.wmnet - slyngshede@cumin1001" [production]
06:51 <slyngshede@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors [production]
06:51 <slyngshede@cumin1001> START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors [production]
06:51 <slyngshede@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:51 <slyngshede@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" [production]
06:50 <slyngshede@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" [production]
06:48 <slyngshede@cumin1001> START - Cookbook sre.dns.netbox [production]
06:48 <slyngshede@cumin1001> START - Cookbook sre.ganeti.makevm for new host testvm2006.codfw.wmnet [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48628 and previous config saved to /var/cache/conftool/dbconfig/20230530-064427-root.json [production]
06:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48625 and previous config saved to /var/cache/conftool/dbconfig/20230530-062922-root.json [production]
06:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48624 and previous config saved to /var/cache/conftool/dbconfig/20230530-061417-root.json [production]
05:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48623 and previous config saved to /var/cache/conftool/dbconfig/20230530-055913-root.json [production]
05:43 <ayounsi@cumin1001> END (ERROR) - Cookbook sre.network.peering (exit_code=97) with action 'configure' for AS: 62597 [production]
05:42 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 62597 [production]
05:41 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Nray out of all services on: 1255 hosts [production]
05:40 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Nray out of all services on: 1255 hosts [production]
05:40 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Nray out of all services on: 784 hosts [production]
05:40 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Nray out of all services on: 784 hosts [production]
05:28 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Hxi-ctr out of all services on: 784 hosts [production]
05:27 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Hxi-ctr out of all services on: 784 hosts [production]
05:26 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Hxi-ctr out of all services on: 1255 hosts [production]
05:25 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Hxi-ctr out of all services on: 1255 hosts [production]
05:22 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 62597 [production]
05:17 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 62597 [production]
04:28 <kart_> Updated cxserver to 2023-05-29-112644-production (T337657) [production]
04:28 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
04:27 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
04:24 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
04:24 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
04:21 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
04:20 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
03:54 <mwpresync@deploy1002> Pruned MediaWiki: 1.41.0-wmf.9 (duration: 02m 10s) [production]
03:52 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.41.0-wmf.11 refs T337525 (duration: 49m 54s) [production]
03:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.41.0-wmf.11 refs T337525 [production]
2023-05-29 §
15:19 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: This is being worked on [production]
15:19 <eoghan@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: This is being worked on [production]
14:18 <stevemunene@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on stat1009.eqiad.wmnet with reason: Bringing stat1009 into service [production]
14:18 <stevemunene@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on stat1009.eqiad.wmnet with reason: Bringing stat1009 into service [production]
13:57 <vgutierrez@puppetmaster1001> conftool action : set/weight=10; selector: name=dbproxy.*,dc=eqiad [production]
11:25 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]