|
2022-01-18
ยง
|
| 12:25 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T285149)', diff saved to https://phabricator.wikimedia.org/P18770 and previous config saved to /var/cache/conftool/dbconfig/20220118-122538-marostegui.json |
[production] |
| 12:16 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P18769 and previous config saved to /var/cache/conftool/dbconfig/20220118-121034-marostegui.json |
[production] |
| 11:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P18768 and previous config saved to /var/cache/conftool/dbconfig/20220118-115529-marostegui.json |
[production] |
| 11:46 |
<hashar> |
Rolled back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389 |
[production] |
| 11:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T285149)', diff saved to https://phabricator.wikimedia.org/P18767 and previous config saved to /var/cache/conftool/dbconfig/20220118-114024-marostegui.json |
[production] |
| 11:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1165 (T285149)', diff saved to https://phabricator.wikimedia.org/P18766 and previous config saved to /var/cache/conftool/dbconfig/20220118-113916-marostegui.json |
[production] |
| 11:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:39 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:39 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:39 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:38 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:38 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:28 |
<Amir1> |
mwscript findBadBlobs.php --wiki=dewiki --revisions 5730218 --mark "T299387" |
[production] |
| 11:06 |
<moritzm> |
running gnt-cluster renew-crypto --new-node-certificates for ganeti/eqiad cluster following 2.16 update |
[production] |
| 11:06 |
<mmandere> |
start rolling upgrade to varnish 6.0.9 T298758 |
[production] |
| 10:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1117.eqiad.wmnet with OS bullseye |
[production] |
| 10:46 |
<moritzm> |
gnt-cluster upgrade --to 2.16 for ganeti/eqiad cluster |
[production] |
| 10:31 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1117.eqiad.wmnet with OS bullseye |
[production] |
| 10:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 10:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 10:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 10:02 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 10:01 |
<moritzm> |
running gnt-cluster renew-crypto --new-cluster-certificate --new-rapi-certificate --new-spice-certificate for ganeti/eqiad cluster |
[production] |
| 10:00 |
<marostegui> |
Move pc1014 to pc3 T299046 |
[production] |
| 09:59 |
<marostegui@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Revert: Promote pc1014 to master in pc2 T299046 (duration: 00m 50s) |
[production] |
| 09:50 |
<taavi> |
mwscript extensions/GlobalBlocking/maintenance/FixBlockerUsername.php --wiki metawiki "QuiteUnusual" "MarcGarver" # T298707 |
[production] |
| 09:50 |
<moritzm> |
installing ganeti 2.16.0-1~bpo9+1+wmf1 on ganeti/eqiad servers T296721 |
[production] |
| 09:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 09:45 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 09:45 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 09:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 09:41 |
<taavi@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:752344|Enable temporary global user groups on production (T153815)]] (duration: 00m 51s) |
[production] |
| 09:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 09:35 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 09:35 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 09:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 09:32 |
<ladsgroup@deploy1002> |
Synchronized php-1.38.0-wmf.17/includes: Backport: [[gerrit:754602|page: Use MainObjectStash instead of 'db-replicated' cache (T272512)]] (duration: 00m 56s) |
[production] |
| 09:31 |
<ladsgroup@deploy1002> |
Synchronized php-1.38.0-wmf.17/extensions/Linter/extension.json: Backport: [[gerrit:754145|Disable "inline-media-caption" category (T297443)]] (duration: 00m 51s) |
[production] |
| 09:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 09:06 |
<ladsgroup@deploy1002> |
Synchronized php-1.38.0-wmf.17/includes/watcheditem/WatchedItemStore.php: Backport: [[gerrit:754599|watcheditem: Try getting the cached version in resetNotificationTimestamp]] (duration: 00m 51s) |
[production] |
| 09:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 09:05 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 09:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |