8001-8050 of 10000 results (104ms)
2023-08-17 §
08:16 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:16 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ncredir3002.esams.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:15 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ncredir3002.esams.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:14 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
08:14 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
08:13 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:08 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ncredir3002.esams.wmnet [production]
08:07 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
08:07 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
08:01 <apergos> UTC morning backport and config window done [production]
07:59 <ariel@deploy1002> Finished scap: Backport for [[gerrit:949612|zhwiki: Create abusefilter-helper group (T344398)]] (duration: 11m 18s) [production]
07:51 <ariel@deploy1002> stang and ariel: Continuing with sync [production]
07:49 <ariel@deploy1002> stang and ariel: Backport for [[gerrit:949612|zhwiki: Create abusefilter-helper group (T344398)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
07:48 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts install3002.wikimedia.org [production]
07:48 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:48 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: install3002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
07:47 <ariel@deploy1002> Started scap: Backport for [[gerrit:949612|zhwiki: Create abusefilter-helper group (T344398)]] [production]
07:45 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: install3002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
07:44 <taavi@deploy1002> Finished scap: Backport for [[gerrit:949573|Use UserIdentity::LOCAL in PreliminaryCheckService when appropriate (T344403)]], [[gerrit:949574|Use UserIdentity::LOCAL in PreliminaryCheckService when appropriate (T344403)]] (duration: 11m 25s) [production]
07:40 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
07:39 <gehel@cumin1001> conftool action : set/pooled=yes; selector: name=cloudelastic1006.wikimedia.org [production]
07:37 <jelto@deploy1002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
07:36 <taavi@deploy1002> dreamyjazz and taavi: Continuing with sync [production]
07:35 <jelto@deploy1002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
07:35 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts install3002.wikimedia.org [production]
07:34 <taavi@deploy1002> dreamyjazz and taavi: Backport for [[gerrit:949573|Use UserIdentity::LOCAL in PreliminaryCheckService when appropriate (T344403)]], [[gerrit:949574|Use UserIdentity::LOCAL in PreliminaryCheckService when appropriate (T344403)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible vi [production]
07:33 <taavi@deploy1002> Started scap: Backport for [[gerrit:949573|Use UserIdentity::LOCAL in PreliminaryCheckService when appropriate (T344403)]], [[gerrit:949574|Use UserIdentity::LOCAL in PreliminaryCheckService when appropriate (T344403)]] [production]
07:32 <gehel> restarting elasticsearch on cloudelastic1006 (high GC) [production]
07:32 <gehel@cumin1001> conftool action : set/pooled=no; selector: name=cloudelastic1006.wikimedia.org [production]
07:31 <jelto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
07:28 <jelto@deploy1002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
07:27 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM install3003.wikimedia.org [production]
07:23 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM install3003.wikimedia.org [production]
07:12 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on install3002.wikimedia.org with reason: decom in progress [production]
07:12 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on install3002.wikimedia.org with reason: decom in progress [production]
06:59 <_joe_> updated vopsbot on the icinga hosts T344316 [production]
01:59 <fab@deploy1002> Finished deploy [airflow-dags/research@ff0a21b]: (no justification provided) (duration: 00m 22s) [production]
01:58 <eileen> civicrm upgraded from 0f981bc4 to 4f7f1e68 [production]
01:58 <fab@deploy1002> Started deploy [airflow-dags/research@ff0a21b]: (no justification provided) [production]
01:55 <fab@deploy1002> Finished deploy [airflow-dags/research@ff0a21b]: (no justification provided) (duration: 00m 19s) [production]
01:55 <fab@deploy1002> Started deploy [airflow-dags/research@ff0a21b]: (no justification provided) [production]
01:54 <fab@deploy1002> Finished deploy [airflow-dags/research@ff0a21b]: (no justification provided) (duration: 00m 20s) [production]
01:53 <fab@deploy1002> Started deploy [airflow-dags/research@ff0a21b]: (no justification provided) [production]
00:10 <eileen> civicrm upgraded from 5e631101 to 0f981bc4 - add email custom fields [production]
2023-08-16 §
21:52 <bking@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host flink-zk2002.codfw.wmnet [production]
21:52 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host flink-zk2002.codfw.wmnet with OS bookworm [production]
21:49 <ryankemper> T343124 [WDQS] Pooled `wdqs1012` and `wdqs1013` (passing checks after reimage/data transfer) [production]
20:37 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host flink-zk2002.codfw.wmnet with OS bookworm [production]
20:36 <bking@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk2002.codfw.wmnet - bking@cumin1001" [production]
20:35 <bking@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM flink-zk2002.codfw.wmnet - bking@cumin1001" [production]