|
2021-09-20
ยง
|
| 22:06 |
<mutante> |
wdqs1004 - HTTP/1.1 503 Service Unavailable - systemctl restart wdqs-blazegraph |
[production] |
| 22:05 |
<foks> |
changing user email for MIskander (WMF)@collabwiki |
[production] |
| 21:41 |
<mutante> |
ms-fe1005 - systemctl start swift_dispersion_stats.service (gerrit:719285) |
[production] |
| 21:30 |
<bd808@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 19:49 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:47 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:45 |
<legoktm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Disable jQuery Migrate on group1" (T291410) (duration: 00m 56s) |
[production] |
| 17:02 |
<legoktm> |
repooled codfw (traffic/caches) 1 week after DC switchover |
[production] |
| 16:41 |
<effie> |
upgrading php on wtp[1025-1029] to 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf2 - T291052 |
[production] |
| 16:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: After migrating wikitech to codfw', diff saved to https://phabricator.wikimedia.org/P17305 and previous config saved to /var/cache/conftool/dbconfig/20210920-144844-root.json |
[production] |
| 14:42 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 14:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: After migrating wikitech to codfw', diff saved to https://phabricator.wikimedia.org/P17304 and previous config saved to /var/cache/conftool/dbconfig/20210920-143340-root.json |
[production] |
| 14:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: After migrating wikitech to codfw', diff saved to https://phabricator.wikimedia.org/P17303 and previous config saved to /var/cache/conftool/dbconfig/20210920-141836-root.json |
[production] |
| 14:11 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 14:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: After migrating wikitech to codfw', diff saved to https://phabricator.wikimedia.org/P17302 and previous config saved to /var/cache/conftool/dbconfig/20210920-140333-root.json |
[production] |
| 13:51 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 13:47 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) |
[production] |
| 13:45 |
<moritzm> |
restarting apache on Logstash ELK5 cluster to pick up GNUTLS update T283165 |
[production] |
| 13:39 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons. - elukey@cumin1001 |
[production] |
| 13:20 |
<elukey@cumin1001> |
START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons. - elukey@cumin1001 |
[production] |
| 13:13 |
<damilare> |
updated payments-wiki from f9cbf95a12 to 23d0ffac66 |
[production] |
| 12:59 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 12:58 |
<marostegui> |
Drop ct_tag_id_log key from db1144:3314 T277416 |
[production] |
| 12:54 |
<moritzm> |
installing gnutls28 updates for stretch with backport for forthcoming Let's encrypt issuance chain update (T283165) |
[production] |
| 12:42 |
<marostegui> |
Add ct_tag_id_log key to db1144:3314 T277416 |
[production] |
| 11:50 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:48 |
<hnowlan@cumin1001> |
START - Cookbook sre.postgresql.postgres-init |
[production] |
| 11:47 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) |
[production] |
| 11:46 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:31 |
<urbanecm@deploy1002> |
Finished scap: b9031bc: Mentor dashboard: Mentor tools (T280307) (duration: 11m 44s) |
[production] |
| 11:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:20 |
<urbanecm@deploy1002> |
Started scap: b9031bc: Mentor dashboard: Mentor tools (T280307) |
[production] |
| 11:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:18 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:722348|Disable jQuery Migrate on group1 (T280944)]] (duration: 00m 56s) |
[production] |
| 11:16 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: b518d8ba03e85afdf98f2e06bf569b4f2b551b1b: Mentor dashboard: Enable beta mode at testwiki (T281534) (duration: 00m 55s) |
[production] |
| 11:11 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.23/extensions/GrowthExperiments/includes/: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 5) (duration: 00m 56s) |
[production] |
| 11:10 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.23/extensions/GrowthExperiments/ServiceWiring.php: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 4) (duration: 00m 56s) |
[production] |
| 11:09 |
<hnowlan> |
roll restarting restbase service in codfw |
[production] |
| 11:08 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.23/extensions/GrowthExperiments/includes/MentorDashboard/Modules/MentorTools.php: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 2) (duration: 00m 55s) |
[production] |
| 11:07 |
<urbanecm@deploy1002> |
sync-file aborted: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 1) (duration: 00m 00s) |
[production] |
| 11:07 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.23/extensions/GrowthExperiments/includes/MentorDashboard/MentorTools/MentorStatusManager.php: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 1) (duration: 00m 57s) |
[production] |
| 11:05 |
<hnowlan> |
roll restarting restbase service in eqiad for openssl updates |
[production] |
| 10:45 |
<hnowlan> |
roll restarting kartotherian and tilerator on maps2* |
[production] |
| 10:41 |
<hnowlan> |
roll restarting kartotherian and tilerator on maps1* |
[production] |
| 10:36 |
<jynus> |
rolling restart bacula & minio daemons on backup hosts |
[production] |
| 09:59 |
<moritzm> |
restarting apache2 on thorium |
[production] |