6701-6750 of 10000 results (121ms)
2023-08-18 §
07:34 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet [production]
06:58 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:57 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
06:56 <jmm@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
06:51 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
05:48 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host bast3007.wikimedia.org [production]
05:48 <jmm@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
05:43 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
05:43 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host bast3007.wikimedia.org [production]
01:42 <taavi@deploy1002> Finished scap: Backport for [[gerrit:949629|Set WRITE_BOTH for OAuth multiple devices to checkuserwiki (T242031)]] (duration: 07m 48s) [production]
01:36 <taavi@deploy1002> taavi: Continuing with sync [production]
01:36 <taavi@deploy1002> taavi: Backport for [[gerrit:949629|Set WRITE_BOTH for OAuth multiple devices to checkuserwiki (T242031)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
01:34 <taavi@deploy1002> Started scap: Backport for [[gerrit:949629|Set WRITE_BOTH for OAuth multiple devices to checkuserwiki (T242031)]] [production]
2023-08-17 §
23:27 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:22 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:54 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
21:41 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
21:40 <bking@deploy1002> Finished deploy [wdqs/wdqs@f1a6177]: deploying WDQS on newly-reimaged Bullseye hosts T343124 (duration: 00m 16s) [production]
21:40 <bking@deploy1002> Started deploy [wdqs/wdqs@f1a6177]: deploying WDQS on newly-reimaged Bullseye hosts T343124 [production]
21:34 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:34 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1011.eqiad.wmnet with OS bullseye [production]
21:21 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
21:21 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:11 <ebernhardson@deploy1002> Finished deploy [airflow-dags/search@1d60a29]: make wikibase ttl imports to hdfs world readable (duration: 00m 11s) [production]
21:11 <ebernhardson@deploy1002> Started deploy [airflow-dags/search@1d60a29]: make wikibase ttl imports to hdfs world readable [production]
21:10 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
21:09 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:03 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
21:03 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
20:58 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1011.eqiad.wmnet with reason: host reimage [production]
20:56 <urandom> Rolling Cassandra restart eqiad/d (RESTBase cluster) — T339298 [production]
20:55 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1011.eqiad.wmnet with reason: host reimage [production]
20:50 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:950011|Revert "Add newline to README for backport training"]] (duration: 10m 43s) [production]
20:49 <urandom> Rolling Cassandra restart eqiad/b (RESTBase cluster) — T339298 [production]
20:48 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
20:46 <sukhe> restart pybal on lvs3008 [production]
20:44 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1010.eqiad.wmnet with OS bullseye [production]
20:44 <thcipriani@deploy1002> thcipriani: Continuing with sync [production]
20:41 <sukhe> restart pybal on lvs3010 [production]
20:41 <thcipriani@deploy1002> thcipriani: Backport for [[gerrit:950011|Revert "Add newline to README for backport training"]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:40 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs1011.eqiad.wmnet with OS bullseye [production]
20:40 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs1010.eqiad.wmnet with OS bullseye [production]
20:40 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:950011|Revert "Add newline to README for backport training"]] [production]
20:34 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:950036|Add newline to README for backport training]] (duration: 13m 29s) [production]
20:30 <urandom> Rolling Cassandra restart eqiad/a (RESTBase cluster) — T339298 [production]
20:28 <thcipriani@deploy1002> thcipriani: Continuing with sync [production]
20:22 <thcipriani@deploy1002> thcipriani: Backport for [[gerrit:950036|Add newline to README for backport training]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:21 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:950036|Add newline to README for backport training]] [production]
20:20 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:20 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add reverses for Lumen transport esams eqiad - cmooney@cumin1001" [production]