|
2025-11-24
ยง
|
| 23:30 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1029.eqiad.wmnet with OS trixie |
[production] |
| 23:29 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1028.eqiad.wmnet with OS trixie |
[production] |
| 23:29 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudnet2005-dev.codfw.wmnet |
[production] |
| 23:29 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2010-dev.codfw.wmnet |
[production] |
| 23:22 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudcontrol2010-dev.codfw.wmnet |
[production] |
| 23:22 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2003-dev.codfw.wmnet |
[production] |
| 23:13 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudlb2003-dev.codfw.wmnet |
[production] |
| 23:13 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2006-dev.codfw.wmnet |
[production] |
| 23:03 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudcontrol2006-dev.codfw.wmnet |
[production] |
| 23:03 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2002-dev.codfw.wmnet |
[production] |
| 22:59 |
<ryankemper@cumin2002> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reboot (apply updates) - ryankemper@cumin2002 - T390860 |
[production] |
| 22:59 |
<ryankemper@cumin2002> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reboot (apply updates) - ryankemper@cumin2002 - T390860 |
[production] |
| 22:54 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudlb2002-dev.codfw.wmnet |
[production] |
| 22:54 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2005-dev.codfw.wmnet |
[production] |
| 22:46 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudcontrol2005-dev.codfw.wmnet |
[production] |
| 22:46 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet |
[production] |
| 22:45 |
<ryankemper@cumin2002> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reboot (apply updates) - ryankemper@cumin2002 - T390860 |
[production] |
| 22:45 |
<jgleeson> |
payments-wiki upgraded from f1fe8fa6 to 5afaf7c8 |
[production] |
| 22:42 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudbackup1002-dev.eqiad.wmnet |
[production] |
| 22:41 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit2001-dev.codfw.wmnet |
[production] |
| 22:40 |
<dani@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1210722|Revert "Deploy experiment for 2025 Global Readers Survey"]], [[gerrit:1210723|Revert "Pre-deploy 2025 Global Readers Survey"]] (duration: 20m 42s) |
[production] |
| 22:40 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet |
[production] |
| 22:36 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet |
[production] |
| 22:35 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit2001-dev.codfw.wmnet |
[production] |
| 22:34 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1032.eqiad.wmnet with OS trixie |
[production] |
| 22:31 |
<eileen> |
tools upgraded from 773e8d11 to 5649c921 |
[production] |
| 22:28 |
<dani@deploy2002> |
dani: Continuing with sync |
[production] |
| 22:26 |
<dani@deploy2002> |
dani: Backport for [[gerrit:1210722|Revert "Deploy experiment for 2025 Global Readers Survey"]], [[gerrit:1210723|Revert "Pre-deploy 2025 Global Readers Survey"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 22:21 |
<Amir1> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1210719 |
[releng] |
| 22:19 |
<dani@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1210722|Revert "Deploy experiment for 2025 Global Readers Survey"]], [[gerrit:1210723|Revert "Pre-deploy 2025 Global Readers Survey"]] |
[production] |
| 22:12 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2239.codfw.wmnet with reason: Maintenance |
[production] |
| 22:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2227 (T410531)', diff saved to https://phabricator.wikimedia.org/P85559 and previous config saved to /var/cache/conftool/dbconfig/20251124-221207-marostegui.json |
[production] |
| 22:10 |
<bking@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs1029.eqiad.wmnet with OS trixie |
[production] |
| 22:09 |
<bking@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs1028.eqiad.wmnet with OS trixie |
[production] |
| 22:09 |
<btullis> |
failing over the Hadoop nameserver from an-master1003 to an-master1004 for T405943 |
[analytics] |
| 22:01 |
<dani@deploy2002> |
dani, bvibber: Backport for [[gerrit:1208408|Pre-deploy 2025 Global Readers Survey (T410696)]], [[gerrit:1210655|Deploy experiment for 2025 Global Readers Survey (T410696)]], [[gerrit:1210669|Show "no data" message when tooltip does not contain to show (T401990)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 21:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P85558 and previous config saved to /var/cache/conftool/dbconfig/20251124-215659-marostegui.json |
[production] |
| 21:56 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1031.eqiad.wmnet with OS trixie |
[production] |
| 21:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P85557 and previous config saved to /var/cache/conftool/dbconfig/20251124-214151-marostegui.json |
[production] |
| 21:38 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1031.eqiad.wmnet with reason: host reimage |
[production] |
| 21:34 |
<bking@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1031.eqiad.wmnet with reason: host reimage |
[production] |
| 21:33 |
<dani@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1208408|Pre-deploy 2025 Global Readers Survey (T410696)]], [[gerrit:1210655|Deploy experiment for 2025 Global Readers Survey (T410696)]], [[gerrit:1210669|Show "no data" message when tooltip does not contain to show (T401990)]] |
[production] |
| 21:30 |
<arlolra@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1209791|arwiktionary: make Cite button in main VE bar (T410840)]] (duration: 08m 54s) |
[production] |
| 21:27 |
<bd808> |
Unblock 47.193.0.0/17 (T410951) |
[releng] |
| 21:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2227 (T410531)', diff saved to https://phabricator.wikimedia.org/P85556 and previous config saved to /var/cache/conftool/dbconfig/20251124-212643-marostegui.json |
[production] |
| 21:26 |
<arlolra@deploy2002> |
arlolra, hubaishan: Continuing with sync |
[production] |
| 21:26 |
<arlolra@deploy2002> |
arlolra, hubaishan: Backport for [[gerrit:1209791|arwiktionary: make Cite button in main VE bar (T410840)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 21:21 |
<arlolra@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1209791|arwiktionary: make Cite button in main VE bar (T410840)]] |
[production] |
| 21:17 |
<arlolra@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1207276|Deploy Parsoid Read Views to 18 wikis (T410564)]], [[gerrit:1206466|Mark non-wikimedia.org math APIs as deprecated in the sandbox (T409773)]] (duration: 11m 49s) |
[production] |
| 21:16 |
<bking@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs1031.eqiad.wmnet with OS trixie |
[production] |