7351-7400 of 10000 results (51ms)
2024-08-12 ยง
11:06 <isaranto@deploy1003> helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
11:04 <isaranto@deploy1003> helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
11:03 <isaranto@deploy1003> helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
10:37 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [toolsbeta]
10:31 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [toolsbeta]
10:30 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [tools]
10:24 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [tools]
10:24 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [toolsbeta]
10:19 <vgutierrez> restarting apache on puppetmaster1003 [production]
10:18 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [toolsbeta]
10:01 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [toolsbeta]
09:59 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [toolsbeta]
09:57 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [tools]
09:54 <kamila_> rebooting puppetmaster1001 due to intermittent network failures [production]
09:50 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [tools]
09:50 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [toolsbeta]
09:46 <ayounsi@cumin1002> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 54994 [production]
09:45 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [toolsbeta]
09:43 <ayounsi@cumin1002> START - Cookbook sre.network.peering with action 'email' for AS: 54994 [production]
09:41 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [toolsbeta]
09:40 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [toolsbeta]
09:17 <urbanecm@deploy1003> Finished scap: Backport for [[gerrit:1061148|MenteeOverviewApi: Do not apply undefined/null params (T372164)]] (duration: 19m 54s) [production]
09:14 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [toolsbeta]
09:13 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [toolsbeta]
09:11 <urbanecm@deploy1003> urbanecm: Continuing with sync [production]
09:11 <godog> bounce grafana after https://gerrit.wikimedia.org/r/c/operations/puppet/+/1061955 [production]
09:10 <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1061148|MenteeOverviewApi: Do not apply undefined/null params (T372164)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:09 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [toolsbeta]
09:04 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [toolsbeta]
09:02 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [toolsbeta]
09:01 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [toolsbeta]
08:57 <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1061148|MenteeOverviewApi: Do not apply undefined/null params (T372164)]] [production]
08:52 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.wait_for_rebalance [admin]
08:46 <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=0) [admin]
08:43 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_rack [admin]
08:37 <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) [admin]
08:37 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_rack [admin]
08:37 <wmbot~dcaro@urcuchillay> END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97) (T371878) [admin]
08:37 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_rack (T371878) [admin]
08:37 <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) [admin]
08:33 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_rack [admin]
08:32 <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) (T371878) [admin]
08:27 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_rack (T371878) [admin]
08:26 <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) (T371878) [admin]
08:26 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_rack (T371878) [admin]
08:17 <dcaro@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97) [admin]
08:17 <dcaro@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_rack [admin]
07:39 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: index corruption [production]
07:39 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: index corruption [production]
07:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2189 - s2', diff saved to https://phabricator.wikimedia.org/P67270 and previous config saved to /var/cache/conftool/dbconfig/20240812-073846-arnaudb.json [production]