|
2023-05-30
ยง
|
| 08:30 |
<tgr@deploy1002> |
tgr: Backport for [[gerrit:923643|Improve logging of invalid image recommendation kinds]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 08:29 |
<slyngshede@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2006.codfw.wmnet |
[production] |
| 08:29 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors |
[production] |
| 08:29 |
<tgr@deploy1002> |
Started scap: Backport for [[gerrit:923643|Improve logging of invalid image recommendation kinds]] |
[production] |
| 08:29 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors |
[production] |
| 08:28 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 08:28 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 08:27 |
<slyngshede@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 08:27 |
<jayme> |
re-enable puppet on P:kubernetes::node for https://gerrit.wikimedia.org/r/c/operations/puppet/+/909687 |
[production] |
| 08:25 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 08:20 |
<jayme> |
disable puppet on P:kubernetes::node (apart from staging-codfw) for https://gerrit.wikimedia.org/r/c/operations/puppet/+/909687 |
[production] |
| 08:15 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors |
[production] |
| 08:15 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors |
[production] |
| 08:15 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 08:15 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 08:14 |
<slyngshede@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 08:12 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 08:12 |
<slyngshede@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host testvm2006.codfw.wmnet |
[production] |
| 08:11 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage |
[production] |
| 08:08 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on puppetdb2003.codfw.wmnet with reason: host reimage |
[production] |
| 08:08 |
<tgr@deploy1002> |
Finished scap: Backport for [[gerrit:924356|Section images: Accept more recommendation types]] (duration: 07m 51s) |
[production] |
| 08:01 |
<tgr@deploy1002> |
tgr: Backport for [[gerrit:924356|Section images: Accept more recommendation types]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 08:00 |
<tgr@deploy1002> |
Started scap: Backport for [[gerrit:924356|Section images: Accept more recommendation types]] |
[production] |
| 07:56 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:924086|Revert "Rename wgPageContentLanguage to wgPageViewLanguage" partially (T337634)]] (duration: 09m 17s) |
[production] |
| 07:49 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host puppetdb2003.codfw.wmnet with OS bookworm |
[production] |
| 07:48 |
<ladsgroup@deploy1002> |
func and ladsgroup: Backport for [[gerrit:924086|Revert "Rename wgPageContentLanguage to wgPageViewLanguage" partially (T337634)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 07:46 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:924086|Revert "Rename wgPageContentLanguage to wgPageViewLanguage" partially (T337634)]] |
[production] |
| 07:45 |
<slyngshede@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2006.codfw.wmnet |
[production] |
| 07:45 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors |
[production] |
| 07:45 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors |
[production] |
| 07:45 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:45 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 07:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48633 and previous config saved to /var/cache/conftool/dbconfig/20230530-074445-root.json |
[production] |
| 07:44 |
<slyngshede@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 07:42 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 07:41 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors |
[production] |
| 07:41 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors |
[production] |
| 07:41 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:41 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 07:40 |
<slyngshede@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 07:38 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 07:38 |
<slyngshede@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host testvm2006.codfw.wmnet |
[production] |
| 07:31 |
<slyngshede@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=2) for new host testvm2006.codfw.wmnet |
[production] |
| 07:31 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors |
[production] |
| 07:31 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors |
[production] |
| 07:31 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:31 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 07:30 |
<moritzm> |
move LDAP permissions for hghani from cn=nda to cn=wmf T322145 |
[production] |
| 07:30 |
<slyngshede@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM testvm2006.codfw.wmnet - slyngshede@cumin1001" |
[production] |
| 07:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48632 and previous config saved to /var/cache/conftool/dbconfig/20230530-072941-root.json |
[production] |