|
2019-12-03
ยง
|
| 09:05 |
<godog> |
downtime new logstash hosts in codfw/eqiad until thurs |
[production] |
| 09:02 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 09:02 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 09:00 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 08:48 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 08:45 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 08:45 |
<effie> |
Restart php-fpm on mw[1330-1333].eqiad.wmnet |
[production] |
| 08:45 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) |
[production] |
| 08:45 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 08:35 |
<ema> |
cp3050: set cache.max_open_read_retries=-1 and proxy.config.http.cache.max_open_write_retries=1 (default values) T238494 |
[production] |
| 08:30 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db1062 from config T239188 (duration: 01m 02s) |
[production] |
| 08:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db1062 from config T239188 (duration: 01m 08s) |
[production] |
| 08:20 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 08:19 |
<akosiaris> |
apply calico rules for eventgate-logging-external. T236386 |
[production] |
| 08:18 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 08:14 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@7978f0d]: Deploy design/style-guide: (duration: 00m 06s) |
[production] |
| 08:14 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@7978f0d]: Deploy design/style-guide: |
[production] |
| 07:39 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 06:29 |
<marostegui> |
Deploy schema change on db1112 with replication (this will generate lag on s3 on labs) |
[production] |
| 06:19 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@8e08740]: Deploy design/style-guide: (duration: 00m 08s) |
[production] |
| 06:19 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@8e08740]: Deploy design/style-guide: |
[production] |
| 06:07 |
<marostegui> |
Stop MySQL on db1062 for decommissioning T239188 |
[production] |
| 06:00 |
<marostegui> |
Remove db2065 from tendril and zarcillo T239046 |
[production] |
| 05:53 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 05:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:50 |
<ema> |
cp3050: ats-be restart with proxy.config.http.server_session_sharing.pool=thread T238494 |
[production] |
| 05:47 |
<marostegui> |
Remove ar_comment triggers from s3 db1124:3313 - T234704 |
[production] |
| 05:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1112 for schema change', diff saved to https://phabricator.wikimedia.org/P9798 and previous config saved to /var/cache/conftool/dbconfig/20191203-054528-marostegui.json |
[production] |
| 04:19 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.5/includes/Rest/EntryPoint.php: disable IE6 safety checks for T239666 (duration: 01m 00s) |
[production] |
| 04:15 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.8/includes/Rest/EntryPoint.php: disable IE6 safety checks for T239666 (duration: 01m 01s) |
[production] |
| 03:53 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@d00c6ad]: Fix: Apply language headers to zhwiki mobile-html responses (T239659) (duration: 05m 51s) |
[production] |
| 03:47 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@d00c6ad]: Fix: Apply language headers to zhwiki mobile-html responses (T239659) |
[production] |
| 02:54 |
<mutante> |
mw1269 restarted nginx, php |
[production] |
| 02:48 |
<mutante> |
mw1320, mw1321 restarted php-fpm |
[production] |
| 02:32 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T78711 Display 'twice a month' or 'once a month' on cached reports (duration: 01m 19s) |
[production] |
| 02:25 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Stop setting testwiki => true for wmgUseCentralAuth, already implied by default (duration: 01m 24s) |
[production] |
| 02:23 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T237698 Stop setting wmgUseDPL, unread (duration: 01m 11s) |
[production] |
| 02:21 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T237698 Read wmgUseDynamicPageList not wmgUseDPL (duration: 01m 22s) |
[production] |
| 02:19 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T237698 Set wmgUseDynamicPageList, less cryptic form of wmgUseDPL (duration: 01m 16s) |
[production] |
| 02:16 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Stop setting wgTorLoadNodes, not read for a while (duration: 01m 14s) |
[production] |
| 02:13 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Stop setting wgGEHelpPanelSearchEnabled, no longer used (duration: 01m 08s) |
[production] |
| 02:04 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T239091 Enable Translate extension on sewikimedia, second try (duration: 01m 24s) |
[production] |
| 01:58 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/VisualEditor/: T239209 Sanitize HTML on paste (duration: 01m 33s) |
[production] |
| 01:55 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/VisualEditor/: T239209 Sanitize HTML on paste (duration: 01m 24s) |
[production] |
| 01:42 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 01:42 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 01:36 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2252.codfw.wmnet |
[production] |
| 01:35 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2250.codfw.wmnet |
[production] |
| 01:33 |
<mutante> |
mw2250 - E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. |
[production] |
| 01:33 |
<mutante> |
mw2252 rebooting |
[production] |