|
2025-01-27
ยง
|
| 22:45 |
<pt1979@cumin1002> |
START - Cookbook sre.hosts.dhcp for host cloudgw1003.eqiad.wmnet |
[production] |
| 22:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P72552 and previous config saved to /var/cache/conftool/dbconfig/20250127-224455-marostegui.json |
[production] |
| 22:41 |
<mstyles@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 22:41 |
<mstyles@deploy2002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 22:41 |
<mstyles@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 22:41 |
<mstyles@deploy2002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 22:40 |
<mstyles@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 22:40 |
<mstyles@deploy2002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 22:39 |
<mstyles@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 22:39 |
<mstyles@deploy2002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 22:39 |
<mstyles@deploy2002> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 22:38 |
<mstyles@deploy2002> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 22:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228 (T384592)', diff saved to https://phabricator.wikimedia.org/P72550 and previous config saved to /var/cache/conftool/dbconfig/20250127-222947-marostegui.json |
[production] |
| 21:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P72547 and previous config saved to /var/cache/conftool/dbconfig/20250127-215710-marostegui.json |
[production] |
| 21:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P72546 and previous config saved to /var/cache/conftool/dbconfig/20250127-214203-marostegui.json |
[production] |
| 21:33 |
<cjming> |
end of UTC late backport window |
[production] |
| 21:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2223 (T384592)', diff saved to https://phabricator.wikimedia.org/P72544 and previous config saved to /var/cache/conftool/dbconfig/20250127-212656-marostegui.json |
[production] |
| 21:14 |
<cjming@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1114396|testwiki: Enable MetricsPlatform experiment enrollment overrides (T384728)]] (duration: 10m 30s) |
[production] |
| 21:08 |
<cjming@deploy2002> |
phuedx, cjming: Continuing with sync |
[production] |
| 21:08 |
<cjming@deploy2002> |
phuedx, cjming: Backport for [[gerrit:1114396|testwiki: Enable MetricsPlatform experiment enrollment overrides (T384728)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:04 |
<cjming@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1114396|testwiki: Enable MetricsPlatform experiment enrollment overrides (T384728)]] |
[production] |
| 21:04 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2223 (T384592)', diff saved to https://phabricator.wikimedia.org/P72543 and previous config saved to /var/cache/conftool/dbconfig/20250127-210415-marostegui.json |
[production] |
| 21:04 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2223.codfw.wmnet with reason: Maintenance |
[production] |
| 21:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211 (T384592)', diff saved to https://phabricator.wikimedia.org/P72542 and previous config saved to /var/cache/conftool/dbconfig/20250127-210353-marostegui.json |
[production] |
| 21:01 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 20:50 |
<vriley@cumin1002> |
START - Cookbook sre.hosts.provision for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 20:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P72541 and previous config saved to /var/cache/conftool/dbconfig/20250127-204846-marostegui.json |
[production] |
| 20:45 |
<vriley@cumin1002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 20:45 |
<vriley@cumin1002> |
START - Cookbook sre.hosts.provision for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 20:36 |
<papaul> |
power down logging-hd1005 for maintenance |
[production] |
| 20:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P72540 and previous config saved to /var/cache/conftool/dbconfig/20250127-203339-marostegui.json |
[production] |
| 19:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2211 (T384592)', diff saved to https://phabricator.wikimedia.org/P72538 and previous config saved to /var/cache/conftool/dbconfig/20250127-195939-marostegui.json |
[production] |
| 19:59 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2211.codfw.wmnet with reason: Maintenance |
[production] |
| 19:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72537 and previous config saved to /var/cache/conftool/dbconfig/20250127-195736-root.json |
[production] |
| 19:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72536 and previous config saved to /var/cache/conftool/dbconfig/20250127-194231-root.json |
[production] |
| 19:36 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2201.codfw.wmnet with reason: Maintenance |
[production] |
| 19:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192 (T384592)', diff saved to https://phabricator.wikimedia.org/P72535 and previous config saved to /var/cache/conftool/dbconfig/20250127-193625-marostegui.json |
[production] |
| 19:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72534 and previous config saved to /var/cache/conftool/dbconfig/20250127-192725-root.json |
[production] |
| 19:25 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:25 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:24 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P72533 and previous config saved to /var/cache/conftool/dbconfig/20250127-192118-marostegui.json |
[production] |
| 19:19 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:18 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:18 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:17 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:17 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserver2004 to codfw - jhancock@cumin2002" |
[production] |
| 19:17 |
<jhancock@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserver2004 to codfw - jhancock@cumin2002" |
[production] |
| 19:13 |
<jhancock@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:13 |
<ejegg> |
deployed fallback SMTP settings for Fundraising CiviCRM |
[production] |