|
2021-03-17
§
|
| 07:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 100%: Slowly repool db1111', diff saved to https://phabricator.wikimedia.org/P14915 and previous config saved to /var/cache/conftool/dbconfig/20210317-073024-root.json |
[production] |
| 07:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 75%: Slowly repool db1111', diff saved to https://phabricator.wikimedia.org/P14914 and previous config saved to /var/cache/conftool/dbconfig/20210317-071520-root.json |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 50%: Slowly repool db1111', diff saved to https://phabricator.wikimedia.org/P14913 and previous config saved to /var/cache/conftool/dbconfig/20210317-070017-root.json |
[production] |
| 06:52 |
<marostegui> |
Stop MySQL on db1082 to clone db1161 (lag will appear on s5 on wikireplicas) - T258361 |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082 to clone db1161 T258361', diff saved to https://phabricator.wikimedia.org/P14911 and previous config saved to /var/cache/conftool/dbconfig/20210317-065146-marostegui.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db2150 into s7 T275633', diff saved to https://phabricator.wikimedia.org/P14910 and previous config saved to /var/cache/conftool/dbconfig/20210317-064606-marostegui.json |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 25%: Slowly repool db1111', diff saved to https://phabricator.wikimedia.org/P14909 and previous config saved to /var/cache/conftool/dbconfig/20210317-064513-root.json |
[production] |
| 06:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db2150 to s7, depooled T275633', diff saved to https://phabricator.wikimedia.org/P14908 and previous config saved to /var/cache/conftool/dbconfig/20210317-060358-marostegui.json |
[production] |
| 05:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111 for schema change', diff saved to https://phabricator.wikimedia.org/P14907 and previous config saved to /var/cache/conftool/dbconfig/20210317-054206-marostegui.json |
[production] |
| 02:25 |
<eileen> |
civicrm revision changed from 8c137b94f0 to 99bf1c9210, config revision is ef2767ab91 |
[production] |
| 01:55 |
<eileen> |
civicrm revision changed from 550be50105 to 8c137b94f0, config revision is ef2767ab91 |
[production] |
|
2021-03-16
§
|
| 23:56 |
<krinkle@deploy1002> |
Synchronized php-1.36.0-wmf.35/includes/Revision/: I8619ab9e92b, T277362, T275531 (duration: 00m 58s) |
[production] |
| 23:51 |
<krinkle@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/Scribunto/: I84e8732d8d - tmp logging (duration: 00m 58s) |
[production] |
| 23:47 |
<Krinkle> |
There is an uncommitted dirty diff in /srv/mediawiki-staging/php-1.36.0-wmf.34/extensions/WikimediaMaintenance/createExtensionTables.php |
[production] |
| 23:31 |
<krinkle@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: I1ca4f30c2, T262612 (duration: 00m 57s) |
[production] |
| 23:22 |
<krinkle@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Icd6635cb302cc, T277332 (duration: 00m 58s) |
[production] |
| 23:07 |
<krinkle@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: I8d8c94d95c6 (duration: 00m 59s) |
[production] |
| 23:03 |
<twentyafterfour> |
applied hotfix to phabricator/src/infrastructure/customfield/storage/PhabricatorCustomFieldStorage.php and restarted php-fpm |
[production] |
| 23:02 |
<krinkle@deploy1002> |
Synchronized wmf-config/CommonSettings.php: I4097cbcb1d5 (duration: 00m 59s) |
[production] |
| 22:59 |
<krinkle@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Ie24eb2077 (duration: 00m 58s) |
[production] |
| 20:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2232.codfw.wmnet |
[production] |
| 20:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2231.codfw.wmnet |
[production] |
| 20:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2230.codfw.wmnet |
[production] |
| 20:49 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@e4fd934]: tiny horizon patch to support flavor deprecation (duration: 03m 44s) |
[production] |
| 20:45 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@e4fd934]: tiny horizon patch to support flavor deprecation |
[production] |
| 20:15 |
<XioNoX> |
remove DMZ zone from pfw3-eqiad - T174203 |
[production] |
| 20:00 |
<brennen> |
1.36.0-wmf.35 train status (T274939): blocked at group0 on T277362 |
[production] |
| 19:52 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.34 |
[production] |
| 19:51 |
<XioNoX> |
commit changes to pfw3-eqiad - T274422 |
[production] |
| 19:44 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.35 |
[production] |
| 19:31 |
<dancy@deploy1002> |
Finished scap: testwikis wikis to 1.36.0-wmf.35 (duration: 33m 41s) |
[production] |
| 19:20 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2229.codfw.wmnet |
[production] |
| 19:11 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2229.codfw.wmnet |
[production] |
| 19:10 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2229.codfw.wmnet |
[production] |
| 19:08 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2228.codfw.wmnet |
[production] |
| 19:07 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2228.codfw.wmnet |
[production] |
| 19:06 |
<XioNoX> |
commit changes to pfw3-codfw - T274422 |
[production] |
| 18:58 |
<dancy@deploy1002> |
Started scap: testwikis wikis to 1.36.0-wmf.35 |
[production] |
| 18:55 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2228.codfw.wmnet |
[production] |
| 18:48 |
<mbsantos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:43 |
<mbsantos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:41 |
<mbsantos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 18:03 |
<ppchelko@deploy1002> |
Finished deploy [restbase/deploy@f99ddaa]: Add new wikis T275837 T271983 T273466 T276127 T273460 T276249 (duration: 31m 31s) |
[production] |
| 17:44 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on aqs1011.eqiad.wmnet with reason: New buster hosts, not in use |
[production] |
| 17:44 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 6 days, 0:00:00 on aqs1011.eqiad.wmnet with reason: New buster hosts, not in use |
[production] |
| 17:37 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2227.codfw.wmnet |
[production] |
| 17:32 |
<ppchelko@deploy1002> |
Started deploy [restbase/deploy@f99ddaa]: Add new wikis T275837 T271983 T273466 T276127 T273460 T276249 |
[production] |
| 17:09 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2227.codfw.wmnet |
[production] |
| 17:04 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2226.codfw.wmnet |
[production] |
| 16:47 |
<eevans@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'sessionstore' for release 'production' . |
[production] |