|
2025-02-28
§
|
| 00:33 |
<jclark@cumin1002> |
START - Cookbook sre.network.configure-switch-interfaces for host an-worker1193 |
[production] |
| 00:18 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host an-worker1192.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 00:18 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host an-worker1191.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 00:18 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host an-worker1190.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 00:18 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host an-worker1187.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 00:18 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host an-worker1189.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 00:18 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host an-worker1188.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 00:15 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 00:15 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for an-worker - jclark@cumin1002" |
[production] |
| 00:14 |
<jclark@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for an-worker - jclark@cumin1002" |
[production] |
| 00:10 |
<jclark@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 00:10 |
<jclark@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm |
[production] |
| 00:03 |
<cmooney@cumin1002> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 00:01 |
<cmooney@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
|
2025-02-27
§
|
| 23:50 |
<sbassett@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1123489|Bump wikimedia/parsoid to v0.21.0-a18]], [[gerrit:1123490|Bump wikimedia/parsoid to 0.21.0-a18]] (duration: 08m 58s) |
[production] |
| 23:44 |
<sbassett@deploy2002> |
sbassett, cscott: Continuing with sync |
[production] |
| 23:44 |
<sbassett@deploy2002> |
sbassett, cscott: Backport for [[gerrit:1123489|Bump wikimedia/parsoid to v0.21.0-a18]], [[gerrit:1123490|Bump wikimedia/parsoid to 0.21.0-a18]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 23:41 |
<sbassett@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1123489|Bump wikimedia/parsoid to v0.21.0-a18]], [[gerrit:1123490|Bump wikimedia/parsoid to 0.21.0-a18]] |
[production] |
| 22:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73838 and previous config saved to /var/cache/conftool/dbconfig/20250227-224612-root.json |
[production] |
| 22:46 |
<sbassett> |
Deployed core security patch for T387130 (apologies for previous sync-world log msgs) |
[production] |
| 22:44 |
<sbassett@deploy2002> |
Finished scap sync-world: help (duration: 09m 04s) |
[production] |
| 22:35 |
<sbassett@deploy2002> |
Started scap sync-world: help |
[production] |
| 22:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73837 and previous config saved to /var/cache/conftool/dbconfig/20250227-223107-root.json |
[production] |
| 22:25 |
<dzahn@dns1004> |
END - running authdns-update |
[production] |
| 22:23 |
<dzahn@dns1004> |
START - running authdns-update |
[production] |
| 22:17 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1123466|Reduce logspam]] (duration: 10m 23s) |
[production] |
| 22:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73836 and previous config saved to /var/cache/conftool/dbconfig/20250227-221601-root.json |
[production] |
| 22:11 |
<ladsgroup@deploy2002> |
ladsgroup: Continuing with sync |
[production] |
| 22:09 |
<ladsgroup@deploy2002> |
ladsgroup: Backport for [[gerrit:1123466|Reduce logspam]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 22:07 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1123466|Reduce logspam]] |
[production] |
| 22:06 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1123416|PageCollectionMetadataApi: don't parse pages (T387370)]] (duration: 14m 42s) |
[production] |
| 22:04 |
<vriley@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002" |
[production] |
| 22:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73835 and previous config saved to /var/cache/conftool/dbconfig/20250227-220056-root.json |
[production] |
| 21:59 |
<ladsgroup@deploy2002> |
sbisson, ladsgroup: Continuing with sync |
[production] |
| 21:56 |
<ladsgroup@deploy2002> |
sbisson, ladsgroup: Backport for [[gerrit:1123416|PageCollectionMetadataApi: don't parse pages (T387370)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:51 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1123416|PageCollectionMetadataApi: don't parse pages (T387370)]] |
[production] |
| 21:50 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1093399|Turn on Parsoid fragment support everywhere (T374661 T386233)]] (duration: 23m 25s) |
[production] |
| 21:47 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage |
[production] |
| 21:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73834 and previous config saved to /var/cache/conftool/dbconfig/20250227-214551-root.json |
[production] |
| 21:44 |
<vriley@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage |
[production] |
| 21:43 |
<ladsgroup@deploy2002> |
cscott, ladsgroup: Continuing with sync |
[production] |
| 21:34 |
<ladsgroup@deploy2002> |
cscott, ladsgroup: Backport for [[gerrit:1093399|Turn on Parsoid fragment support everywhere (T374661 T386233)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:27 |
<vriley@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1256.eqiad.wmnet with OS bookworm |
[production] |
| 21:26 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1093399|Turn on Parsoid fragment support everywhere (T374661 T386233)]] |
[production] |
| 21:25 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1123312|Revert "CentralAuth: Enable SUL3 signup on group 0 (attempt 2)" (T384007)]] (duration: 12m 10s) |
[production] |
| 21:24 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1256.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 21:19 |
<ladsgroup@deploy2002> |
tgr, ladsgroup: Continuing with sync |
[production] |
| 21:18 |
<ladsgroup@deploy2002> |
tgr, ladsgroup: Backport for [[gerrit:1123312|Revert "CentralAuth: Enable SUL3 signup on group 0 (attempt 2)" (T384007)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:13 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1123312|Revert "CentralAuth: Enable SUL3 signup on group 0 (attempt 2)" (T384007)]] |
[production] |
| 21:13 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1123463|MediaWikiConfigReader: Lower logging level]], [[gerrit:1123464|MediaWikiConfigReader: Lower logging level]] (duration: 11m 11s) |
[production] |