|
2021-09-20
ยง
|
| 22:44 |
<andrewbogott> |
sudo touch /tmp/galera.disabled on cloudcontrol1004, the service seems troubled there |
[admin] |
| 22:36 |
<bstorm> |
created cloudstore-nfs-01 with a floating ip and a 10GB cinder volume T291406 |
[cloudstore] |
| 22:21 |
<bd808@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 22:14 |
<mutante> |
wdqs1004 - depool |
[production] |
| 22:10 |
<mutante> |
wdqs1004 - service wdqs-updater restart |
[production] |
| 22:06 |
<mutante> |
wdqs1004 - HTTP/1.1 503 Service Unavailable - systemctl restart wdqs-blazegraph |
[production] |
| 22:05 |
<foks> |
changing user email for MIskander (WMF)@collabwiki |
[production] |
| 21:57 |
<andrewbogott> |
moving cloudvirt1043 into the 'nfs' aggregate for T291405 |
[admin] |
| 21:41 |
<mutante> |
ms-fe1005 - systemctl start swift_dispersion_stats.service (gerrit:719285) |
[production] |
| 21:30 |
<bd808@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 21:23 |
<wm-bot> |
<lucaswerkmeister> successfully rolled out node12 for function-orchestrator too |
[tools.notwikilambda] |
| 21:19 |
<wm-bot> |
<lucaswerkmeister> successfully rolled out node12 for function-evaluator after changing `npm ci` to `npm install` (ecfa5ab9ab) |
[tools.notwikilambda] |
| 21:00 |
<wm-bot> |
<lucaswerkmeister> tried to deploy d028cb3667 (switch to node12) for function-evaluator but the new pod never came up so I undid the rollout; will consider further steps |
[tools.notwikilambda] |
| 20:50 |
<wm-bot> |
<lucaswerkmeister> set cronjob successfulJobsHistoryLimit to 0 (0aa054b391) |
[tools.notwikilambda] |
| 20:40 |
<wm-bot> |
<lucaswerkmeister> `git -C ~/www/js/function-orchestrator/ config pull.rebase true` ditto |
[tools.notwikilambda] |
| 20:40 |
<wm-bot> |
<lucaswerkmeister> `git -C ~/www/js/function-evaluator/ config pull.rebase true` to silence `git pull` warnings (presumably due to newer git in node12 container) |
[tools.notwikilambda] |
| 19:49 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:47 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:45 |
<legoktm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Disable jQuery Migrate on group1" (T291410) (duration: 00m 56s) |
[production] |
| 17:26 |
<wm-bot> |
<root> Added missing "title" in $HOME/public_html/static/toolinfo.json. |
[tools.gerrit-patch-uploader] |
| 17:24 |
<wm-bot> |
<root> Added missing "title" in $HOME/public_html/toolinfo.json. |
[tools.gerrit-reviewer-bot] |
| 17:23 |
<wm-bot> |
<root> Fixed JSON syntax errors in $HOME/public_html/toolinfo.json. Updated URLs in same file to new toolforge.org standard. |
[tools.tsreports] |
| 17:22 |
<wm-bot> |
<root> Updated $HOME/public_html/toolinfo.json to add missing "title" and update URL to toolforge.org |
[tools.wikilinkbot] |
| 17:20 |
<wm-bot> |
<root> Fixed JSON syntax errors in $HOME/public_html/toolinfo.json. Updated URLs in same file to new toolforge.org standard. |
[tools.apersonbot] |
| 17:13 |
<bd808> |
Freed up disk space on toolhub-demo01 with `docker image prune -a` to remove all unused containers pulled to the instance (mostly old snapshots of the toolhub image). |
[toolhub] |
| 17:05 |
<bd808> |
Updated toolhub-demo.wmcloud.org to 8c8c4fb |
[toolhub] |
| 17:02 |
<legoktm> |
repooled codfw (traffic/caches) 1 week after DC switchover |
[production] |
| 16:41 |
<effie> |
upgrading php on wtp[1025-1029] to 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf2 - T291052 |
[production] |
| 16:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: After migrating wikitech to codfw', diff saved to https://phabricator.wikimedia.org/P17305 and previous config saved to /var/cache/conftool/dbconfig/20210920-144844-root.json |
[production] |
| 14:42 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 14:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: After migrating wikitech to codfw', diff saved to https://phabricator.wikimedia.org/P17304 and previous config saved to /var/cache/conftool/dbconfig/20210920-143340-root.json |
[production] |
| 14:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: After migrating wikitech to codfw', diff saved to https://phabricator.wikimedia.org/P17303 and previous config saved to /var/cache/conftool/dbconfig/20210920-141836-root.json |
[production] |
| 14:11 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 14:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: After migrating wikitech to codfw', diff saved to https://phabricator.wikimedia.org/P17302 and previous config saved to /var/cache/conftool/dbconfig/20210920-140333-root.json |
[production] |
| 13:51 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 13:47 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) |
[production] |
| 13:45 |
<moritzm> |
restarting apache on Logstash ELK5 cluster to pick up GNUTLS update T283165 |
[production] |
| 13:39 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons. - elukey@cumin1001 |
[production] |
| 13:20 |
<elukey@cumin1001> |
START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons. - elukey@cumin1001 |
[production] |
| 13:13 |
<damilare> |
updated payments-wiki from f9cbf95a12 to 23d0ffac66 |
[production] |
| 13:04 |
<arturo> |
bump RAM quota from 20GB to 36GB (T290793) |
[masz] |
| 12:59 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 12:58 |
<marostegui> |
Drop ct_tag_id_log key from db1144:3314 T277416 |
[production] |
| 12:54 |
<moritzm> |
installing gnutls28 updates for stretch with backport for forthcoming Let's encrypt issuance chain update (T283165) |
[production] |
| 12:44 |
<majavah> |
deploying volume-admission to tools, should not affect anything yet T279106 |
[tools] |
| 12:42 |
<marostegui> |
Add ct_tag_id_log key to db1144:3314 T277416 |
[production] |
| 11:50 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:48 |
<hnowlan@cumin1001> |
START - Cookbook sre.postgresql.postgres-init |
[production] |