|
2025-05-29
§
|
| 16:17 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cirrussearch2112*,cirrussearch2113* for T394543 - bking@cumin2002 |
[production] |
| 16:17 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2007.codfw.wmnet with OS bookworm |
[production] |
| 16:17 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1002" |
[production] |
| 16:17 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.ban Banning hosts: cirrussearch2112*,cirrussearch2113* for T394543 - bking@cumin2002 |
[production] |
| 16:17 |
<cmooney@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1002" |
[production] |
| 16:16 |
<cmooney@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:09 |
<bvibber@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1151787|Lua transform backend for JsonConfig Data: pages (T388434)]], [[gerrit:1151788|Chart-side support for Lua transforms (T388616)]] |
[production] |
| 16:07 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1175 (T395241)', diff saved to https://phabricator.wikimedia.org/P76681 and previous config saved to /var/cache/conftool/dbconfig/20250529-160758-fceratto.json |
[production] |
| 15:58 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1175 (T395241)', diff saved to https://phabricator.wikimedia.org/P76680 and previous config saved to /var/cache/conftool/dbconfig/20250529-155843-fceratto.json |
[production] |
| 15:58 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:58 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T395241)', diff saved to https://phabricator.wikimedia.org/P76679 and previous config saved to /var/cache/conftool/dbconfig/20250529-155817-fceratto.json |
[production] |
| 15:56 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2007.codfw.wmnet with reason: host reimage |
[production] |
| 15:56 |
<bvibber@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1151750|Enable Chart for Phase 4 wikis (all remaining public wikis) (T393788)]] (duration: 10m 20s) |
[production] |
| 15:52 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2007.codfw.wmnet with reason: host reimage |
[production] |
| 15:48 |
<bvibber@deploy1003> |
jforrester, bvibber: Continuing with sync |
[production] |
| 15:47 |
<bvibber@deploy1003> |
jforrester, bvibber: Backport for [[gerrit:1151750|Enable Chart for Phase 4 wikis (all remaining public wikis) (T393788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 15:45 |
<bvibber@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1151750|Enable Chart for Phase 4 wikis (all remaining public wikis) (T393788)]] |
[production] |
| 15:43 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P76678 and previous config saved to /var/cache/conftool/dbconfig/20250529-154309-fceratto.json |
[production] |
| 15:34 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.reimage for host sretest2007.codfw.wmnet with OS bookworm |
[production] |
| 15:32 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 15:28 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P76677 and previous config saved to /var/cache/conftool/dbconfig/20250529-152802-fceratto.json |
[production] |
| 15:27 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed 894d8b3ef5 (three minor improvements) |
[tools.wdactle] |
| 15:25 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 15:12 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T395241)', diff saved to https://phabricator.wikimedia.org/P76676 and previous config saved to /var/cache/conftool/dbconfig/20250529-151255-fceratto.json |
[production] |
| 15:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1166 (T395241)', diff saved to https://phabricator.wikimedia.org/P76675 and previous config saved to /var/cache/conftool/dbconfig/20250529-150332-fceratto.json |
[production] |
| 15:03 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76674 and previous config saved to /var/cache/conftool/dbconfig/20250529-150306-fceratto.json |
[production] |
| 14:55 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:54 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:53 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:52 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:48 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:47 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P76673 and previous config saved to /var/cache/conftool/dbconfig/20250529-144759-fceratto.json |
[production] |
| 14:47 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:36 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed 3ad25bd9e0 (support globe coordinates – all value types now supported 🎉) |
[tools.wdactle] |
| 14:32 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P76672 and previous config saved to /var/cache/conftool/dbconfig/20250529-143251-fceratto.json |
[production] |
| 14:31 |
<dcausse> |
T395546: rebuildind eqiad completion indices for cswikiversity pnbwiktionary avkwiki mlwikiquote ladwiki ptwikiquote cswikiversity olowiki azwikibooks wikimania2018wiki rowikibooks rswikimedia liwiktionary oswiki nnwiktionary trwikisource vowikibooks ndswiktionary bmwikiquote ilowiki pawiktionary nowikibooks zhwikiversity tawikiquote hawiktionary akwikibooks udmwiki xhwikibooks |
[production] |
| 14:17 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76671 and previous config saved to /var/cache/conftool/dbconfig/20250529-141744-fceratto.json |
[production] |
| 14:10 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed 2db51d07e9 (support for most time values) |
[tools.wdactle] |
| 14:08 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76670 and previous config saved to /var/cache/conftool/dbconfig/20250529-140811-fceratto.json |
[production] |
| 14:08 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:39 |
<marostegui> |
Reboot dbproxy[1022,1025,1028-1029].eqiad.wmnet T395241 |
[production] |
| 13:39 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy[1022,1025,1028-1029].eqiad.wmnet with reason: Maintenance |
[production] |
| 13:39 |
<taavi@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1148299|Allow itwiki bureaucrat to remove sysop permission (T394752)]] (duration: 10m 13s) |
[production] |
| 13:32 |
<taavi@deploy1003> |
simmed, taavi: Continuing with sync |
[production] |
| 13:31 |
<taavi@deploy1003> |
simmed, taavi: Backport for [[gerrit:1148299|Allow itwiki bureaucrat to remove sysop permission (T394752)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 13:28 |
<taavi@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1148299|Allow itwiki bureaucrat to remove sysop permission (T394752)]] |
[production] |
| 13:26 |
<taavi@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1151813|Fixes issues with recommendations config in production (T393943)]] (duration: 12m 09s) |
[production] |
| 13:19 |
<taavi@deploy1003> |
jdlrobson, taavi: Continuing with sync |
[production] |
| 13:16 |
<taavi@deploy1003> |
jdlrobson, taavi: Backport for [[gerrit:1151813|Fixes issues with recommendations config in production (T393943)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |