651-700 of 10000 results (14ms)
2026-11-19 §
08:05 <slyngshede@cumin1004> START - Cookbook sre.hosts.provision for host cp5027.mgmt.eqsin.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
08:03 <slyngshede@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp5027.eqsin.wmnet with reason: provisioning [production]
07:59 <marostegui@cumin1004> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage [production]
07:58 <slyngshede@puppetserver1001> conftool action : set/pooled=no; selector: name=cp5027.eqsin.wmnet [production]
07:58 <slyngshede@puppetserver1001> conftool action : set/pooled=no; selector: name= cp5027.eqsin.wmnet [production]
07:55 <marostegui@cumin1004> START - Cookbook sre.hosts.downtime for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage [production]
07:53 <marostegui@cumin1004> END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db1164.eqiad.wmnet (remove config only) [production]
07:53 <marostegui@cumin1004> START - Cookbook sre.hosts.dhcp for host db1164.eqiad.wmnet (remove config only) [production]
07:51 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P97553 and previous config saved to /var/cache/conftool/dbconfig/20261119-075144-marostegui.json [production]
07:42 <marostegui@cumin1004> END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db1164.eqiad.wmnet (push config only) [production]
07:42 <marostegui@cumin1004> START - Cookbook sre.hosts.dhcp for host db1164.eqiad.wmnet (push config only) [production]
07:41 <marostegui@cumin1004> START - Cookbook sre.hosts.reimage for host db1164.eqiad.wmnet with OS trixie [production]
07:36 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P97552 and previous config saved to /var/cache/conftool/dbconfig/20261119-073642-marostegui.json [production]
07:21 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T445307)', diff saved to https://phabricator.wikimedia.org/P97551 and previous config saved to /var/cache/conftool/dbconfig/20261119-072140-marostegui.json [production]
07:19 <marostegui@cumin1004> dbctl commit (dc=all): 'Depooling db1168 (T445307)', diff saved to https://phabricator.wikimedia.org/P97550 and previous config saved to /var/cache/conftool/dbconfig/20261119-071919-marostegui.json [production]
07:19 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
07:19 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T445307)', diff saved to https://phabricator.wikimedia.org/P97549 and previous config saved to /var/cache/conftool/dbconfig/20261119-071901-marostegui.json [production]
07:17 <ayounsi@cumin1004> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 32934 [production]
07:12 <ayounsi@cumin1004> START - Cookbook sre.network.peering with action 'configure' for AS: 32934 [production]
07:04 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P97548 and previous config saved to /var/cache/conftool/dbconfig/20261119-070359-marostegui.json [production]
06:48 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P97547 and previous config saved to /var/cache/conftool/dbconfig/20261119-064857-marostegui.json [production]
06:33 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T445307)', diff saved to https://phabricator.wikimedia.org/P97546 and previous config saved to /var/cache/conftool/dbconfig/20261119-063355-marostegui.json [production]
06:31 <marostegui@cumin1004> dbctl commit (dc=all): 'Depooling db1165 (T445307)', diff saved to https://phabricator.wikimedia.org/P97545 and previous config saved to /var/cache/conftool/dbconfig/20261119-063134-marostegui.json [production]
06:31 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1165.eqiad.wmnet with reason: Maintenance [production]
06:30 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2229.codfw.wmnet with reason: Maintenance [production]
05:04 <fab@deploy2003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/research-tools: apply [production]
05:03 <fab@deploy2003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/research-tools: apply [production]
04:31 <fab@deploy2003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/research-tools: apply [production]
04:30 <fab@deploy2003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/research-tools: apply [production]
04:29 <fab@deploy2003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/research-tools: apply [production]
04:29 <fab@deploy2003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/research-tools: apply [production]
01:33 <sukhe@cumin1004> END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough [production]
01:13 <sukhe@cumin1004> START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough [production]
00:00 <root@deploy2003> helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [production]
00:00 <root@deploy2003> helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [production]
00:00 <root@deploy2003> helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [production]
00:00 <root@deploy2003> helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [production]
00:00 <root@deploy2003> helmfile [staging] DONE helmfile.d/services/shellbox: sync [production]
00:00 <root@deploy2003> helmfile [staging] START helmfile.d/services/shellbox: sync [production]
2026-11-18 §
22:54 <cooltey@deploy2003> helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [production]
22:53 <cooltey@deploy2003> helmfile [codfw] START helmfile.d/services/wikifeeds: apply [production]
22:53 <cooltey@deploy2003> helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [production]
22:53 <cooltey@deploy2003> helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [production]
22:52 <cooltey@deploy2003> helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [production]
22:52 <cooltey@deploy2003> helmfile [staging] START helmfile.d/services/wikifeeds: apply [production]
22:31 <jforrester@deploy2003> Finished scap sync-world: Backport for [[gerrit:1361533|Abstractwiki: Define virtual-abstractwiki DB pointer (T443295)]] (duration: 13m 19s) [production]
22:25 <jforrester@deploy2003> jforrester: Continuing with deployment [production]
22:25 <jforrester@deploy2003> jforrester: Backport for [[gerrit:1361533|Abstractwiki: Define virtual-abstractwiki DB pointer (T443295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
22:21 <James_F> mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_render_status.sql # T443522 [production]
22:18 <jforrester@deploy2003> Started scap sync-world: Backport for [[gerrit:1361533|Abstractwiki: Define virtual-abstractwiki DB pointer (T443295)]] [production]