|
2021-12-13
ยง
|
| 12:05 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on restbase[2025-2026].codfw.wmnet with reason: New cassandra hosts awaiting syncing |
[production] |
| 12:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P18156 and previous config saved to /var/cache/conftool/dbconfig/20211213-120259-marostegui.json |
[production] |
| 11:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P18155 and previous config saved to /var/cache/conftool/dbconfig/20211213-114754-marostegui.json |
[production] |
| 11:39 |
<majavah> |
deployed patch for T297574 |
[production] |
| 11:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T277354)', diff saved to https://phabricator.wikimedia.org/P18154 and previous config saved to /var/cache/conftool/dbconfig/20211213-113249-marostegui.json |
[production] |
| 11:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1179 (T277354)', diff saved to https://phabricator.wikimedia.org/P18153 and previous config saved to /var/cache/conftool/dbconfig/20211213-112245-marostegui.json |
[production] |
| 11:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1179.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:22 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1179.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:22 |
<urbanecm> |
Run namespaceDupes.php --wiki=$WIKI --fix --add-prefix=BROKEN for wikis in P18152 |
[production] |
| 11:12 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:12 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:53 |
<urbanecm> |
mwscript namespaceDupes.php --wiki={mswiki,sqwiki,bclwiki,idwiki,siwiki,tlwiki,rowiki} --add-prefix=BROKEN --fix |
[production] |
| 10:51 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 7 hosts with reason: Maintenance |
[production] |
| 10:51 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on 7 hosts with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1172', diff saved to https://phabricator.wikimedia.org/P18151 and previous config saved to /var/cache/conftool/dbconfig/20211213-101707-marostegui.json |
[production] |
| 10:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1172 (T277354)', diff saved to https://phabricator.wikimedia.org/P18150 and previous config saved to /var/cache/conftool/dbconfig/20211213-101427-marostegui.json |
[production] |
| 10:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1172.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:14 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1172.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:14 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1126', diff saved to https://phabricator.wikimedia.org/P18149 and previous config saved to /var/cache/conftool/dbconfig/20211213-101013-marostegui.json |
[production] |
| 10:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1126 (T277354)', diff saved to https://phabricator.wikimedia.org/P18148 and previous config saved to /var/cache/conftool/dbconfig/20211213-100238-marostegui.json |
[production] |
| 10:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1126.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1126.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1116.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1116.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1114', diff saved to https://phabricator.wikimedia.org/P18147 and previous config saved to /var/cache/conftool/dbconfig/20211213-100143-marostegui.json |
[production] |
| 09:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2130 and db2074', diff saved to https://phabricator.wikimedia.org/P18146 and previous config saved to /var/cache/conftool/dbconfig/20211213-095949-marostegui.json |
[production] |
| 09:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1114 (T277354)', diff saved to https://phabricator.wikimedia.org/P18145 and previous config saved to /var/cache/conftool/dbconfig/20211213-095851-marostegui.json |
[production] |
| 09:58 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1114.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1114.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1111', diff saved to https://phabricator.wikimedia.org/P18144 and previous config saved to /var/cache/conftool/dbconfig/20211213-095728-marostegui.json |
[production] |
| 09:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1111 (T277354)', diff saved to https://phabricator.wikimedia.org/P18143 and previous config saved to /var/cache/conftool/dbconfig/20211213-094900-marostegui.json |
[production] |
| 09:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1111.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1111.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1104 (T277354)', diff saved to https://phabricator.wikimedia.org/P18142 and previous config saved to /var/cache/conftool/dbconfig/20211213-094853-marostegui.json |
[production] |
| 09:45 |
<urbanecm> |
Staging at mwdebug1001 ended |
[production] |
| 09:43 |
<urbanecm> |
pwnwiki: Create DB tables for GrowthExperiments |
[production] |
| 09:43 |
<dcausse@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 09:42 |
<urbanecm> |
Stagging at mwdebug1001 |
[production] |
| 09:38 |
<Amir1> |
revoking DROP from centralauth grant of wikiadmin (T249683) |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P18141 and previous config saved to /var/cache/conftool/dbconfig/20211213-093348-marostegui.json |
[production] |
| 09:32 |
<dcausse@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 09:31 |
<elukey@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 09:31 |
<elukey@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 09:31 |
<elukey@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 09:31 |
<elukey@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 09:29 |
<dcausse> |
restarting blazegraph on wdqs1012 (jvm stuck for 6h) |
[production] |
| 09:29 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 09:28 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. |
[production] |