|
2023-03-08
ยง
|
| 22:12 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host sessionstore1001.eqiad.wmnet |
[production] |
| 22:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P45585 and previous config saved to /var/cache/conftool/dbconfig/20230308-221006-marostegui.json |
[production] |
| 22:09 |
<kindrobot> |
hand off backport window UTC late to tgr for self-service |
[production] |
| 22:07 |
<kindrobot@deploy2002> |
Finished scap: Backport for [[gerrit:895833|Enable new Linter UI for namespace, tag and template for all wikis (T299612)]] (duration: 09m 36s) |
[production] |
| 21:59 |
<kindrobot@deploy2002> |
sbailey and kindrobot: Backport for [[gerrit:895833|Enable new Linter UI for namespace, tag and template for all wikis (T299612)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 21:57 |
<kindrobot@deploy2002> |
Started scap: Backport for [[gerrit:895833|Enable new Linter UI for namespace, tag and template for all wikis (T299612)]] |
[production] |
| 21:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T329260)', diff saved to https://phabricator.wikimedia.org/P45584 and previous config saved to /var/cache/conftool/dbconfig/20230308-215500-marostegui.json |
[production] |
| 21:54 |
<kindrobot@deploy2002> |
Finished scap: Backport for [[gerrit:895297|Switch order of "Add topic" and language dropdown (T267444)]], [[gerrit:895375|Release DiscussionTools on mobile on enwiki (T328942)]], [[gerrit:888804|Enable history page visual diffs everywhere except Wikipedias and Wiktionaries (T314588)]] (duration: 07m 49s) |
[production] |
| 21:48 |
<kindrobot@deploy2002> |
kemayo and kindrobot and esanders: Backport for [[gerrit:895297|Switch order of "Add topic" and language dropdown (T267444)]], [[gerrit:895375|Release DiscussionTools on mobile on enwiki (T328942)]], [[gerrit:888804|Enable history page visual diffs everywhere except Wikipedias and Wiktionaries (T314588)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.cod |
[production] |
| 21:46 |
<kindrobot@deploy2002> |
Started scap: Backport for [[gerrit:895297|Switch order of "Add topic" and language dropdown (T267444)]], [[gerrit:895375|Release DiscussionTools on mobile on enwiki (T328942)]], [[gerrit:888804|Enable history page visual diffs everywhere except Wikipedias and Wiktionaries (T314588)]] |
[production] |
| 21:37 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1001.eqiad.wmnet |
[production] |
| 21:31 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host sessionstore1001.eqiad.wmnet |
[production] |
| 21:30 |
<kindrobot@deploy2002> |
kemayo and kindrobot and esanders: Backport for [[gerrit:888804|Enable history page visual diffs everywhere except Wikipedias and Wiktionaries (T314588)]], [[gerrit:895375|Release DiscussionTools on mobile on enwiki (T328942)]], [[gerrit:895297|Switch order of "Add topic" and language dropdown (T267444)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqi |
[production] |
| 21:29 |
<kindrobot@deploy2002> |
Started scap: Backport for [[gerrit:888804|Enable history page visual diffs everywhere except Wikipedias and Wiktionaries (T314588)]], [[gerrit:895375|Release DiscussionTools on mobile on enwiki (T328942)]], [[gerrit:895297|Switch order of "Add topic" and language dropdown (T267444)]] |
[production] |
| 21:22 |
<ebernhardson@deploy2002> |
Finished deploy [airflow-dags/search@3419b7d]: test deploy after deployment fix (duration: 00m 05s) |
[production] |
| 21:22 |
<ebernhardson@deploy2002> |
Started deploy [airflow-dags/search@3419b7d]: test deploy after deployment fix |
[production] |
| 21:19 |
<kindrobot> |
start UTC-late backport window |
[production] |
| 21:08 |
<hashar@deploy2002> |
Finished deploy [releng/jenkins-deploy@0e465ac] (releasing): (no justification provided) (duration: 01m 01s) |
[production] |
| 21:07 |
<hashar@deploy2002> |
Started deploy [releng/jenkins-deploy@0e465ac] (releasing): (no justification provided) |
[production] |
| 20:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2163 (T329260)', diff saved to https://phabricator.wikimedia.org/P45583 and previous config saved to /var/cache/conftool/dbconfig/20230308-205435-marostegui.json |
[production] |
| 20:54 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance |
[production] |
| 20:54 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance |
[production] |
| 20:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2162 (T329260)', diff saved to https://phabricator.wikimedia.org/P45582 and previous config saved to /var/cache/conftool/dbconfig/20230308-205414-marostegui.json |
[production] |
| 20:51 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host acmechief2001.codfw.wmnet with OS bullseye |
[production] |
| 20:41 |
<mutante> |
deploy2002 - systemctl restart keyholder-proxy.service to fix T331568 - after this SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i /etc/keyholder.d/deploy_jenkins -l deploy-jenkins releases1002.eqiad.wmnet works |
[production] |
| 20:39 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on acmechief2001.codfw.wmnet with reason: host reimage |
[production] |
| 20:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P45581 and previous config saved to /var/cache/conftool/dbconfig/20230308-203907-marostegui.json |
[production] |
| 20:36 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on acmechief2001.codfw.wmnet with reason: host reimage |
[production] |
| 20:24 |
<brett@cumin2002> |
START - Cookbook sre.ganeti.reimage for host acmechief2001.codfw.wmnet with OS bullseye |
[production] |
| 20:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P45580 and previous config saved to /var/cache/conftool/dbconfig/20230308-202401-marostegui.json |
[production] |
| 20:18 |
<urandom> |
power cycle restbase2022 (unresponsive; cannot SSH) |
[production] |
| 20:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2162 (T329260)', diff saved to https://phabricator.wikimedia.org/P45579 and previous config saved to /var/cache/conftool/dbconfig/20230308-200855-marostegui.json |
[production] |
| 20:01 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host acmechief-test1001.eqiad.wmnet with OS bullseye |
[production] |
| 19:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2162 (T329260)', diff saved to https://phabricator.wikimedia.org/P45578 and previous config saved to /var/cache/conftool/dbconfig/20230308-194646-marostegui.json |
[production] |
| 19:46 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2162.codfw.wmnet with reason: Maintenance |
[production] |
| 19:46 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2162.codfw.wmnet with reason: Maintenance |
[production] |
| 19:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2161 (T329260)', diff saved to https://phabricator.wikimedia.org/P45577 and previous config saved to /var/cache/conftool/dbconfig/20230308-194625-marostegui.json |
[production] |
| 19:44 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on acmechief-test1001.eqiad.wmnet with reason: host reimage |
[production] |
| 19:41 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on acmechief-test1001.eqiad.wmnet with reason: host reimage |
[production] |
| 19:31 |
<brett@cumin2002> |
START - Cookbook sre.ganeti.reimage for host acmechief-test1001.eqiad.wmnet with OS bullseye |
[production] |
| 19:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P45576 and previous config saved to /var/cache/conftool/dbconfig/20230308-193118-marostegui.json |
[production] |
| 19:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P45575 and previous config saved to /var/cache/conftool/dbconfig/20230308-191612-marostegui.json |
[production] |
| 19:16 |
<jhuneidi@deploy2002> |
Synchronized php: group1 wikis to 1.40.0-wmf.26 refs T330204 (duration: 06m 16s) |
[production] |
| 19:14 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:14 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add reverse entries for new links from CRs to cloudsw1-b1-codfw. - cmooney@cumin1001" |
[production] |
| 19:13 |
<cmooney@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add reverse entries for new links from CRs to cloudsw1-b1-codfw. - cmooney@cumin1001" |
[production] |
| 19:09 |
<jhuneidi@deploy2002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.26 refs T330204 |
[production] |
| 19:09 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for acmechief-test2001.codfw.wmnet |
[production] |
| 19:09 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for acmechief-test2001.codfw.wmnet |
[production] |
| 19:08 |
<cmooney@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |