|
2024-03-21
ยง
|
| 14:05 |
<klausman@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . |
[production] |
| 14:02 |
<moritzm> |
installing squid security updates |
[production] |
| 13:51 |
<damilare> |
SmashPig upgraded from 47cd65d9 to 5d275ad6 |
[production] |
| 13:50 |
<sukhe> |
upgrading pdns-rec to 4.8.7-1 on dns* and doh* hosts |
[production] |
| 13:16 |
<isaranto@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . |
[production] |
| 13:14 |
<jiji@deploy1002> |
Finished scap: Check new deployment server (deploy1002) post switchover - March 2024 (duration: 35m 20s) |
[production] |
| 13:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1222 (T356166)', diff saved to https://phabricator.wikimedia.org/P58869 and previous config saved to /var/cache/conftool/dbconfig/20240321-130213-marostegui.json |
[production] |
| 13:02 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:02 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1197 (T356166)', diff saved to https://phabricator.wikimedia.org/P58868 and previous config saved to /var/cache/conftool/dbconfig/20240321-130151-marostegui.json |
[production] |
| 12:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P58867 and previous config saved to /var/cache/conftool/dbconfig/20240321-124644-marostegui.json |
[production] |
| 12:39 |
<jiji@deploy1002> |
Started scap: Check new deployment server (deploy1002) post switchover - March 2024 |
[production] |
| 12:37 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage |
[production] |
| 12:34 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage |
[production] |
| 12:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P58866 and previous config saved to /var/cache/conftool/dbconfig/20240321-123135-marostegui.json |
[production] |
| 12:19 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS bullseye |
[production] |
| 12:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1197 (T356166)', diff saved to https://phabricator.wikimedia.org/P58865 and previous config saved to /var/cache/conftool/dbconfig/20240321-121628-marostegui.json |
[production] |
| 12:15 |
<akosiaris@cumin1002> |
END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad |
[production] |
| 12:00 |
<effie> |
disable puppet on deployment servers |
[production] |
| 11:46 |
<akosiaris@cumin1002> |
START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad |
[production] |
| 11:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P58860 and previous config saved to /var/cache/conftool/dbconfig/20240321-112108-marostegui.json |
[production] |
| 11:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P58857 and previous config saved to /var/cache/conftool/dbconfig/20240321-110600-marostegui.json |
[production] |
| 10:58 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1153.eqiad.wmnet with OS bookworm |
[production] |
| 10:55 |
<akosiaris@cumin1002> |
END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad |
[production] |
| 10:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1188 (T356166)', diff saved to https://phabricator.wikimedia.org/P58856 and previous config saved to /var/cache/conftool/dbconfig/20240321-105052-marostegui.json |
[production] |
| 10:43 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1153.eqiad.wmnet with reason: host reimage |
[production] |
| 10:41 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1153.eqiad.wmnet with reason: host reimage |
[production] |
| 10:34 |
<slyngshede@cumin1002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host idp1003.wikimedia.org |
[production] |
| 10:34 |
<slyngshede@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host idp1003.wikimedia.org with OS bookworm |
[production] |
| 10:32 |
<fabfur@cumin1002> |
conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 10:29 |
<akosiaris@cumin1002> |
START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad |
[production] |
| 10:28 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1153.eqiad.wmnet with OS bookworm |
[production] |
| 10:20 |
<slyngshede@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on idp1003.wikimedia.org with reason: host reimage |
[production] |
| 10:17 |
<slyngshede@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on idp1003.wikimedia.org with reason: host reimage |
[production] |
| 10:13 |
<akosiaris@cumin1002> |
END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw |
[production] |
| 10:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1188 (T356166)', diff saved to https://phabricator.wikimedia.org/P58855 and previous config saved to /var/cache/conftool/dbconfig/20240321-101119-marostegui.json |
[production] |
| 10:11 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:11 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T356166)', diff saved to https://phabricator.wikimedia.org/P58854 and previous config saved to /var/cache/conftool/dbconfig/20240321-101056-marostegui.json |
[production] |
| 10:05 |
<slyngshede@cumin1002> |
START - Cookbook sre.hosts.reimage for host idp1003.wikimedia.org with OS bookworm |
[production] |
| 10:05 |
<slyngshede@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM idp1003.wikimedia.org - slyngshede@cumin1002" |
[production] |
| 10:04 |
<slyngshede@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM idp1003.wikimedia.org - slyngshede@cumin1002" |
[production] |
| 10:03 |
<slyngshede@cumin1002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) idp1003.wikimedia.org on all recursors |
[production] |
| 10:03 |
<slyngshede@cumin1002> |
START - Cookbook sre.dns.wipe-cache idp1003.wikimedia.org on all recursors |
[production] |
| 10:03 |
<slyngshede@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:03 |
<slyngshede@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM idp1003.wikimedia.org - slyngshede@cumin1002" |
[production] |
| 10:02 |
<slyngshede@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM idp1003.wikimedia.org - slyngshede@cumin1002" |
[production] |
| 10:01 |
<Emperor> |
update ceph-reef packages to 18.2.2 on apt.wm.org |
[production] |
| 10:00 |
<fabfur@cumin1002> |
conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 10:00 |
<fabfur> |
repooling cp4037 for about ~30m (this is last time I'll notice here, no need for this in the future) (T358109) |
[production] |