|
2020-11-05
§
|
| 06:34 |
<elukey> |
truncate application_1601916545561_129457's taskmanager.log (~600G) on an-worker1113 due to partition 'e' full |
[production] |
| 06:30 |
<elukey> |
truncate application_1601916545561_129457's taskmanager.log (~600G) on an-worker1113 due to partition 'e' full |
[analytics] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1016 (re)pooling @ 25%: After cloning es1029 T261717', diff saved to https://phabricator.wikimedia.org/P13190 and previous config saved to /var/cache/conftool/dbconfig/20201105-062507-root.json |
[production] |
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1013 (re)pooling @ 25%: After cloning es1027 T261717', diff saved to https://phabricator.wikimedia.org/P13189 and previous config saved to /var/cache/conftool/dbconfig/20201105-062454-root.json |
[production] |
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1017 (re)pooling @ 25%: After cloning es1028 T261717', diff saved to https://phabricator.wikimedia.org/P13188 and previous config saved to /var/cache/conftool/dbconfig/20201105-062446-root.json |
[production] |
| 02:05 |
<milimetric> |
deployed refinery pointing to refinery-source v0.0.138 |
[analytics] |
| 01:57 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@6913407] (thin): Regular analytics weekly train THIN [analytics/refinery@6913407] (duration: 00m 08s) |
[production] |
| 01:56 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@6913407] (thin): Regular analytics weekly train THIN [analytics/refinery@6913407] |
[production] |
| 01:56 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@6913407]: Regular analytics weekly train [analytics/refinery@6913407] (duration: 08m 34s) |
[production] |
| 01:47 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@6913407]: Regular analytics weekly train [analytics/refinery@6913407] |
[production] |
|
2020-11-04
§
|
| 22:04 |
<Krinkle> |
restarting cvn-app servers |
[cvn] |
| 21:33 |
<hashar> |
devtools upgraded gerrit da40615f726..9275b308ffd |
[releng] |
| 20:36 |
<Urbanecm> |
Late B&C Morning window completed, deployment host is clear |
[production] |
| 20:36 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: ee0ba541fa55f6707276fdc5bd3f032cb9be3e60: Disable the search in header A/B test (T265333) (duration: 01m 06s) |
[production] |
| 20:33 |
<ejegg> |
updated payments-wiki from 1ad4ba9639 to 388490e86d |
[production] |
| 20:33 |
<hashar> |
devtools: fixed java version on gerrit-prod-1001.devtools.eqiad.wmflabs by setting java::profile hieradata variable at https://horizon.wikimedia.org/project/instances/48c5ff1c-2885-410d-beeb-d5a57a0a91c7/ |
[releng] |
| 20:31 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Migrate NewcomerTask event stream to EventGate on testwiki - T259163 (duration: 01m 07s) |
[production] |
| 20:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 82579bf9d71bd3c9d97da0132ce8d92a8863da5b: Enable wgImagePreconnect on remaining wikis (T123582) (duration: 01m 06s) |
[production] |
| 20:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: d2a57725f8f6fdaa3f40c834e84b43a0260077f2: Enable DiscussionTools as a beta feature on almost all Wikipedias (T266303) (duration: 01m 07s) |
[production] |
| 20:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: fb5c03262c20b5e99b3c2f6e91abb024f12da1f5: Enable wgCheckUserLogLogins at all wikis but loginwiki (T253802) (duration: 01m 08s) |
[production] |
| 19:59 |
<hashar> |
operations/software/gerrit : pushed upstream git tag, synced the master branch # T262241 |
[releng] |
| 19:59 |
<brennen@deploy1001> |
Finished scap: testwikis wikis to 1.36.0-wmf.16 (duration: 62m 44s) |
[production] |
| 18:57 |
<brennen@deploy1001> |
Started scap: testwikis wikis to 1.36.0-wmf.16 |
[production] |
| 18:52 |
<brennen@deploy1001> |
Pruned MediaWiki: 1.36.0-wmf.10 (duration: 27m 38s) |
[production] |
| 18:51 |
<Urbanecm> |
Strip 2FA for Mark83 at SUL (T267257) |
[production] |
| 18:50 |
<dpifke> |
Re-cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/639216 in beta. |
[releng] |
| 18:20 |
<elukey> |
restart memcached on mc1036 to pick up new settings (see https://gerrit.wikimedia.org/r/639099) |
[production] |
| 18:15 |
<hknust> |
holger@mwmaint1002 END - Run updateRestrictions.php |
[production] |
| 18:05 |
<dpifke> |
Cherry-picking https://gerrit.wikimedia.org/r/c/analytics/statsv/+/639223 in beta. |
[releng] |
| 17:50 |
<dpifke> |
Reverting previous cherry-pick for now; requires minor code change first. |
[releng] |
| 17:44 |
<hknust> |
holger@mwmaint1002 START - Run updateRestrictions.php |
[production] |
| 17:43 |
<dpifke> |
Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/639216 in beta. |
[releng] |
| 17:22 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:20 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:15 |
<zpapierski@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@8e8d2d4]: Deploying dc switch (duration: 01m 15s) |
[production] |
| 17:13 |
<zpapierski@deploy1001> |
Started deploy [wikimedia/discovery/analytics@8e8d2d4]: Deploying dc switch |
[production] |
| 17:07 |
<effie> |
Reimage mc1036 for real this time |
[production] |
| 16:57 |
<wm-bot> |
<bd808> Rolled back to 1.18.3 and filed T267239 |
[tools.bridgebot] |
| 16:43 |
<wm-bot> |
<bd808> Restart to upgrade to matterbridge 1.19.0 release |
[tools.bridgebot] |
| 16:40 |
<brennen> |
1.36.0-wmf.16 was branched at f51ccd2ccef8cba0e7d874b6f7cf4b73bcd36636 for T263182 |
[production] |
| 16:12 |
<dpifke> |
Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/639197 in beta, should only affect deployment-webperf11. |
[releng] |
| 16:10 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:10 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:42 |
<dcaro> |
re-creating the toolsbeta-proxy-03, used wrong image on the first try (T267140) |
[toolsbeta] |
| 15:39 |
<effie> |
Reimage mc1036 to buster - T252391 |
[production] |
| 15:25 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Migrate ContentTranslationAbuseFilter event stream to EventGate on all wikis - T259163 (duration: 00m 58s) |
[production] |
| 15:21 |
<dcaro> |
creating new proxy instance toolsbeta-proxy-03 |
[toolsbeta] |
| 15:19 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:19 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:18 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |