|
2023-09-19
ยง
|
| 13:52 |
<elukey> |
clean old puppet certs kafka_logging-eqiad_broker |
[production] |
| 13:52 |
<jhancock@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:52 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 13:52 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Updating DNS record of kuberbetes2026 - jhancock@cumin2002" |
[production] |
| 13:51 |
<stevemunene@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:50 |
<jhancock@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Updating DNS record of kuberbetes2026 - jhancock@cumin2002" |
[production] |
| 13:47 |
<jhancock@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:47 |
<jebe@deploy1002> |
Finished deploy [airflow-dags/analytics@6b9855a]: (no justification provided) (duration: 00m 43s) |
[production] |
| 13:46 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts an-test-client1001.eqiad.wmnet |
[production] |
| 13:46 |
<jebe@deploy1002> |
Started deploy [airflow-dags/analytics@6b9855a]: (no justification provided) |
[production] |
| 13:34 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flerovium.eqiad.wmnet |
[production] |
| 13:33 |
<samtar@deploy1002> |
samtar and aleksandar: Continuing with sync |
[production] |
| 13:28 |
<samtar@deploy1002> |
samtar and aleksandar: Backport for [[gerrit:958421|Add namespace aliases to shwiki (T346588)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 13:27 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host flerovium.eqiad.wmnet |
[production] |
| 13:17 |
<jebe@deploy1002> |
Finished deploy [analytics/refinery@2d9d6d0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2d9d6d0] (duration: 02m 06s) |
[production] |
| 13:15 |
<Emperor> |
ms-be10[44-60] swift package updates T346730 |
[production] |
| 13:15 |
<jebe@deploy1002> |
Started deploy [analytics/refinery@2d9d6d0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2d9d6d0] |
[production] |
| 13:14 |
<jebe@deploy1002> |
Finished deploy [analytics/refinery@2d9d6d0] (thin): Regular analytics weekly train THIN [analytics/refinery@2d9d6d0] (duration: 00m 04s) |
[production] |
| 13:14 |
<jebe@deploy1002> |
Started deploy [analytics/refinery@2d9d6d0] (thin): Regular analytics weekly train THIN [analytics/refinery@2d9d6d0] |
[production] |
| 13:14 |
<jebe@deploy1002> |
Finished deploy [analytics/refinery@2d9d6d0]: Regular analytics weekly train [analytics/refinery@2d9d6d0] (duration: 05m 52s) |
[production] |
| 13:08 |
<jebe@deploy1002> |
Started deploy [analytics/refinery@2d9d6d0]: Regular analytics weekly train [analytics/refinery@2d9d6d0] |
[production] |
| 13:05 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:958421|Add namespace aliases to shwiki (T346588)]] |
[production] |
| 13:05 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-jumbo-eqiad cluster: Roll restart of jvm daemons. |
[production] |
| 12:44 |
<Emperor> |
ms-be20[60-73] swift package updates T346730 |
[production] |
| 12:22 |
<Emperor> |
ms-be20[49-59] swift package updates T346730 |
[production] |
| 12:19 |
<jebe@deploy1002> |
Finished deploy [analytics/refinery@91bb4a0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@91bb4a0] (duration: 02m 03s) |
[production] |
| 12:18 |
<Emperor> |
ms-be2048 swift package updates T346730 |
[production] |
| 12:17 |
<jebe@deploy1002> |
Started deploy [analytics/refinery@91bb4a0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@91bb4a0] |
[production] |
| 12:17 |
<jebe@deploy1002> |
Finished deploy [analytics/refinery@91bb4a0] (thin): Regular analytics weekly train THIN [analytics/refinery@91bb4a0] (duration: 00m 05s) |
[production] |
| 12:17 |
<jebe@deploy1002> |
Started deploy [analytics/refinery@91bb4a0] (thin): Regular analytics weekly train THIN [analytics/refinery@91bb4a0] |
[production] |
| 12:14 |
<Emperor> |
ms-be2047 swift package updates T346730 |
[production] |
| 12:12 |
<Emperor> |
ms-be204{5,6} swift package updates T346730 |
[production] |
| 12:10 |
<jebe@deploy1002> |
Finished deploy [analytics/refinery@91bb4a0]: Regular analytics weekly train [analytics/refinery@91bb4a0] (duration: 06m 53s) |
[production] |
| 12:08 |
<cmooney@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:03 |
<jebe@deploy1002> |
Started deploy [analytics/refinery@91bb4a0]: Regular analytics weekly train [analytics/refinery@91bb4a0] |
[production] |
| 11:51 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox |
[production] |
| 11:51 |
<ayounsi@cumin1001> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox |
[production] |
| 11:50 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary |
[production] |
| 11:48 |
<ayounsi@cumin1001> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary |
[production] |
| 11:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 100%: Repooling after recloning db1128', diff saved to https://phabricator.wikimedia.org/P52530 and previous config saved to /var/cache/conftool/dbconfig/20230919-112156-root.json |
[production] |
| 11:09 |
<Emperor> |
eqiad swift front-end swift package updates T346730 |
[production] |
| 11:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 75%: Repooling after recloning db1128', diff saved to https://phabricator.wikimedia.org/P52529 and previous config saved to /var/cache/conftool/dbconfig/20230919-110651-root.json |
[production] |
| 10:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 50%: Repooling after recloning db1128', diff saved to https://phabricator.wikimedia.org/P52528 and previous config saved to /var/cache/conftool/dbconfig/20230919-105147-root.json |
[production] |
| 10:38 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1148.eqiad.wmnet with OS bullseye |
[production] |
| 10:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 25%: Repooling after recloning db1128', diff saved to https://phabricator.wikimedia.org/P52527 and previous config saved to /var/cache/conftool/dbconfig/20230919-103642-root.json |
[production] |
| 10:34 |
<Emperor> |
codfw swift front-end swift package updates T346730 |
[production] |
| 10:25 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1147.eqiad.wmnet with OS bullseye |
[production] |
| 10:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 10%: Repooling after recloning db1128', diff saved to https://phabricator.wikimedia.org/P52526 and previous config saved to /var/cache/conftool/dbconfig/20230919-102137-root.json |
[production] |
| 10:15 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1148.eqiad.wmnet with reason: host reimage |
[production] |
| 10:11 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1148.eqiad.wmnet with reason: host reimage |
[production] |