901-950 of 10000 results (106ms)
2025-09-05 §
07:45 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3007.esams.wmnet [production]
07:42 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh3006.wikimedia.org [production]
07:42 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host doh3006.wikimedia.org with OS bookworm [production]
07:38 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps2012.codfw.wmnet with reason: host reimage [production]
07:33 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps2012.codfw.wmnet with reason: host reimage [production]
07:32 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2206 (T402925)', diff saved to https://phabricator.wikimedia.org/P82603 and previous config saved to /var/cache/conftool/dbconfig/20250905-073225-ladsgroup.json [production]
07:23 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on doh3006.wikimedia.org with reason: host reimage [production]
07:18 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on doh3006.wikimedia.org with reason: host reimage [production]
07:13 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host maps2012.codfw.wmnet with OS bookworm [production]
06:56 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host doh3006.wikimedia.org with OS bookworm [production]
06:51 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM doh3006.wikimedia.org - jmm@cumin2002" [production]
06:51 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM doh3006.wikimedia.org - jmm@cumin2002" [production]
06:50 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) doh3006.wikimedia.org on all recursors [production]
06:50 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache doh3006.wikimedia.org on all recursors [production]
06:50 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:50 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM doh3006.wikimedia.org - jmm@cumin2002" [production]
06:50 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM doh3006.wikimedia.org - jmm@cumin2002" [production]
06:46 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
06:46 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host doh3006.wikimedia.org [production]
06:44 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts install2004.wikimedia.org [production]
06:44 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:44 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: install2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
06:43 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: install2004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
06:39 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
06:38 <kharlan@deploy1003> Sync cancelled. [production]
06:36 <kharlan@deploy1003> kharlan: Backport for [[gerrit:1184968|hCaptcha: Update secure enclave API endpoint]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
06:34 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts install2004.wikimedia.org [production]
06:30 <kharlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1184968|hCaptcha: Update secure enclave API endpoint]] [production]
06:04 <ayounsi@cumin1003> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 139628 [production]
05:54 <ayounsi@cumin1003> START - Cookbook sre.network.peering with action 'configure' for AS: 139628 [production]
04:50 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2206 (T402925)', diff saved to https://phabricator.wikimedia.org/P82602 and previous config saved to /var/cache/conftool/dbconfig/20250905-045020-ladsgroup.json [production]
04:50 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance [production]
03:19 <ryankemper@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749 [production]
02:18 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2199.codfw.wmnet with reason: Maintenance [production]
02:18 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2179 (T402925)', diff saved to https://phabricator.wikimedia.org/P82601 and previous config saved to /var/cache/conftool/dbconfig/20250905-021830-ladsgroup.json [production]
02:03 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P82600 and previous config saved to /var/cache/conftool/dbconfig/20250905-020323-ladsgroup.json [production]
01:55 <ryankemper@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 55 hosts with reason: rolling restart cirrus eqiad [production]
01:52 <ryankemper@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749 [production]
01:48 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P82599 and previous config saved to /var/cache/conftool/dbconfig/20250905-014815-ladsgroup.json [production]
01:33 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2179 (T402925)', diff saved to https://phabricator.wikimedia.org/P82598 and previous config saved to /var/cache/conftool/dbconfig/20250905-013307-ladsgroup.json [production]
00:07 <ryankemper@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749 [production]
2025-09-04 §
23:42 <swfrench-wmf> finished single-replica PHP 8.3 pilot on shellbox-syntaxhighlight - T403284 [production]
23:41 <swfrench@deploy1003> helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
23:41 <swfrench@deploy1003> helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
23:41 <swfrench@deploy1003> helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
23:41 <swfrench@deploy1003> helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
23:41 <swfrench@deploy1003> helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
23:40 <swfrench@deploy1003> helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
23:12 <ryankemper@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749 [production]
22:46 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2179 (T402925)', diff saved to https://phabricator.wikimedia.org/P82597 and previous config saved to /var/cache/conftool/dbconfig/20250904-224604-ladsgroup.json [production]