|
2023-11-22
§
|
| 07:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 75%: Testing 10.4.32', diff saved to https://phabricator.wikimedia.org/P53691 and previous config saved to /var/cache/conftool/dbconfig/20231122-070742-root.json |
[production] |
| 07:02 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 07:01 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:976337|Revert "ProductionServices.php: Promote pc2014 to pc2 master"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 06:59 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:976337|Revert "ProductionServices.php: Promote pc2014 to pc2 master"]] |
[production] |
| 06:57 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2012.codfw.wmnet with OS bookworm |
[production] |
| 06:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 50%: Testing 10.4.32', diff saved to https://phabricator.wikimedia.org/P53690 and previous config saved to /var/cache/conftool/dbconfig/20231122-065238-root.json |
[production] |
| 06:44 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2012.codfw.wmnet with reason: host reimage |
[production] |
| 06:41 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on pc2012.codfw.wmnet with reason: host reimage |
[production] |
| 06:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 25%: Testing 10.4.32', diff saved to https://phabricator.wikimedia.org/P53689 and previous config saved to /var/cache/conftool/dbconfig/20231122-063733-root.json |
[production] |
| 06:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host pc2012.codfw.wmnet with OS bookworm |
[production] |
| 06:22 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:976383|ProductionServices.php: Promote pc2014 to pc2 master (T351620)]] (duration: 07m 28s) |
[production] |
| 06:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 10%: Testing 10.4.32', diff saved to https://phabricator.wikimedia.org/P53688 and previous config saved to /var/cache/conftool/dbconfig/20231122-062228-root.json |
[production] |
| 06:17 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 06:16 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:976383|ProductionServices.php: Promote pc2014 to pc2 master (T351620)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 06:16 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc[2012,2014].codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Switch |
[production] |
| 06:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on pc[2012,2014].codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Switch |
[production] |
| 06:15 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:976383|ProductionServices.php: Promote pc2014 to pc2 master (T351620)]] |
[production] |
| 04:25 |
<eileen> |
civicrm upgraded from 43d191c8 to 3c5db93b |
[production] |
| 02:01 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1158.eqiad.wmnet with OS bullseye |
[production] |
| 01:27 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['an-worker1158'] |
[production] |
| 01:21 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1158'] |
[production] |
| 00:55 |
<tstarling@deploy2002> |
Synchronized wmf-config/CommonSettings.php: enable LoginNotify seen subnets table g965663 T346989 (duration: 06m 23s) |
[production] |
| 00:41 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1158.eqiad.wmnet with OS bullseye |
[production] |
|
2023-11-21
§
|
| 23:05 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 23:05 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: message - vriley@cumin1001" |
[production] |
| 23:04 |
<vriley@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: message - vriley@cumin1001" |
[production] |
| 23:02 |
<vriley@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 23:02 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1036 |
[production] |
| 23:01 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1038 |
[production] |
| 23:01 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1037 |
[production] |
| 23:00 |
<vriley@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host ganeti1038 |
[production] |
| 23:00 |
<vriley@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host ganeti1037 |
[production] |
| 23:00 |
<vriley@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host ganeti1036 |
[production] |
| 22:59 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:59 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: message - vriley@cumin1001" |
[production] |
| 22:58 |
<vriley@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: message - vriley@cumin1001" |
[production] |
| 22:54 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs1021.eqiad.wmnet |
[production] |
| 22:54 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for aqs1021.eqiad.wmnet |
[production] |
| 22:53 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1021.eqiad.wmnet with OS bullseye |
[production] |
| 22:48 |
<vriley@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:47 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1035 |
[production] |
| 22:46 |
<vriley@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host ganeti1035 |
[production] |
| 22:32 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1021.eqiad.wmnet with reason: host reimage |
[production] |
| 22:30 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2179 (T348183)', diff saved to https://phabricator.wikimedia.org/P53687 and previous config saved to /var/cache/conftool/dbconfig/20231121-223053-arnaudb.json |
[production] |
| 22:29 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1021.eqiad.wmnet with reason: host reimage |
[production] |
| 22:23 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1038.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 22:20 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1037.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 22:18 |
<vriley@cumin1001> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1036.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 22:18 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.reimage for host aqs1021.eqiad.wmnet with OS bullseye |
[production] |
| 22:17 |
<eevans@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host aqs1021.eqiad.wmnet with OS bullseye |
[production] |