|
2025-01-27
ยง
|
| 20:36 |
<papaul> |
power down logging-hd1005 for maintenance |
[production] |
| 20:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P72540 and previous config saved to /var/cache/conftool/dbconfig/20250127-203339-marostegui.json |
[production] |
| 19:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2211 (T384592)', diff saved to https://phabricator.wikimedia.org/P72538 and previous config saved to /var/cache/conftool/dbconfig/20250127-195939-marostegui.json |
[production] |
| 19:59 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2211.codfw.wmnet with reason: Maintenance |
[production] |
| 19:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72537 and previous config saved to /var/cache/conftool/dbconfig/20250127-195736-root.json |
[production] |
| 19:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72536 and previous config saved to /var/cache/conftool/dbconfig/20250127-194231-root.json |
[production] |
| 19:36 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2201.codfw.wmnet with reason: Maintenance |
[production] |
| 19:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192 (T384592)', diff saved to https://phabricator.wikimedia.org/P72535 and previous config saved to /var/cache/conftool/dbconfig/20250127-193625-marostegui.json |
[production] |
| 19:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72534 and previous config saved to /var/cache/conftool/dbconfig/20250127-192725-root.json |
[production] |
| 19:25 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:25 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:24 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P72533 and previous config saved to /var/cache/conftool/dbconfig/20250127-192118-marostegui.json |
[production] |
| 19:19 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:18 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:18 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 19:17 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:17 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserver2004 to codfw - jhancock@cumin2002" |
[production] |
| 19:17 |
<jhancock@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserver2004 to codfw - jhancock@cumin2002" |
[production] |
| 19:13 |
<jhancock@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:13 |
<ejegg> |
deployed fallback SMTP settings for Fundraising CiviCRM |
[production] |
| 19:12 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72532 and previous config saved to /var/cache/conftool/dbconfig/20250127-191220-root.json |
[production] |
| 19:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P72531 and previous config saved to /var/cache/conftool/dbconfig/20250127-190611-marostegui.json |
[production] |
| 18:59 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:59 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt pay-lb1002~ - vriley@cumin1002" |
[production] |
| 18:59 |
<vriley@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt pay-lb1002~ - vriley@cumin1002" |
[production] |
| 18:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72530 and previous config saved to /var/cache/conftool/dbconfig/20250127-185715-root.json |
[production] |
| 18:56 |
<vriley@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192 (T384592)', diff saved to https://phabricator.wikimedia.org/P72529 and previous config saved to /var/cache/conftool/dbconfig/20250127-185104-marostegui.json |
[production] |
| 18:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2194', diff saved to https://phabricator.wikimedia.org/P72528 and previous config saved to /var/cache/conftool/dbconfig/20250127-184839-marostegui.json |
[production] |
| 18:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P72527 and previous config saved to /var/cache/conftool/dbconfig/20250127-184642-root.json |
[production] |
| 18:35 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:35 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt pay-lb1001 - vriley@cumin1002" |
[production] |
| 18:35 |
<vriley@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt pay-lb1001 - vriley@cumin1002" |
[production] |
| 18:35 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72526 and previous config saved to /var/cache/conftool/dbconfig/20250127-183519-root.json |
[production] |
| 18:34 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2192 (T384592)', diff saved to https://phabricator.wikimedia.org/P72525 and previous config saved to /var/cache/conftool/dbconfig/20250127-183417-marostegui.json |
[production] |
| 18:34 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2192.codfw.wmnet with reason: Maintenance |
[production] |
| 18:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178 (T384592)', diff saved to https://phabricator.wikimedia.org/P72524 and previous config saved to /var/cache/conftool/dbconfig/20250127-183355-marostegui.json |
[production] |
| 18:31 |
<vriley@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:27 |
<andrew@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudgw1004.eqiad.wmnet with OS bookworm |
[production] |
| 18:20 |
<andrew@cumin1002> |
START - Cookbook sre.hosts.reimage for host cloudgw1004.eqiad.wmnet with OS bookworm |
[production] |
| 18:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72523 and previous config saved to /var/cache/conftool/dbconfig/20250127-182014-root.json |
[production] |
| 18:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P72522 and previous config saved to /var/cache/conftool/dbconfig/20250127-181847-marostegui.json |
[production] |
| 18:16 |
<reedy@deploy2002> |
Synchronized php-1.44.0-wmf.13/extensions/CommonsMetadata/: T384853 T384854 (duration: 10m 45s) |
[production] |
| 18:10 |
<tchin@deploy2002> |
Finished deploy [airflow-dags/analytics@c49f40b]: Deploying airflow for T357684 (duration: 01m 01s) |
[production] |
| 18:10 |
<tchin@deploy2002> |
Started deploy [airflow-dags/analytics@c49f40b]: Deploying airflow for T357684 |
[production] |
| 18:05 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 50%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72521 and previous config saved to /var/cache/conftool/dbconfig/20250127-180509-root.json |
[production] |
| 18:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P72520 and previous config saved to /var/cache/conftool/dbconfig/20250127-180341-marostegui.json |
[production] |
| 17:55 |
<andrew@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudgw1003.eqiad.wmnet with OS bookworm |
[production] |
| 17:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 25%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72518 and previous config saved to /var/cache/conftool/dbconfig/20250127-175004-root.json |
[production] |