|
2019-05-30
§
|
| 10:18 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:15 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 10:15 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:07 |
<jynus> |
upgrade and restart test-s4 hosts (db1111, db1112) |
[production] |
| 09:42 |
<jynus> |
stop and upgrade db1102 |
[production] |
| 09:32 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:32 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:31 |
<_joe_> |
depooling mw1261 for benchmarking for T224491 |
[production] |
| 09:26 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1089 with low weight (duration: 00m 55s) |
[production] |
| 08:54 |
<jynus> |
stop and restart db1089 for upgrade |
[production] |
| 08:50 |
<onimisionipe> |
maps2001 postgres initialization - T224395 |
[production] |
| 08:44 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1089 for maintenance (duration: 00m 57s) |
[production] |
| 08:32 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2087 for maintenance (duration: 01m 00s) |
[production] |
| 08:10 |
<mobrovac> |
drop old Parsoid tables from cassandra -- T223998 |
[production] |
| 07:40 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@92591a7]: Switch to OpenAPI v3 and drop page/html/title/revision/tid - T218218 T215956 (duration: 19m 28s) |
[production] |
| 07:33 |
<_joe_> |
upgraded service-checker on icinga1001,2 |
[production] |
| 07:21 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@92591a7]: Switch to OpenAPI v3 and drop page/html/title/revision/tid - T218218 T215956 |
[production] |
| 00:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2091 - T224393 (duration: 00m 56s) |
[production] |
| 00:24 |
<mutante> |
re-enabling puppet on phab1001 now that it does not have the phab role anymore (T221389) |
[production] |
| 00:17 |
<mutante> |
rsyncing /srv/repos again. pulling on phab2001 from phab1003 (T221389) |
[production] |
|
2019-05-29
§
|
| 23:37 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove wikibase sameAs A/B test config, part II (duration: 00m 56s) |
[production] |
| 23:36 |
<jforrester@deploy1001> |
sync-file aborted: Remove wikibase sameAs A/B test config, part I (duration: 00m 00s) |
[production] |
| 23:35 |
<jforrester@deploy1001> |
Synchronized wmf-config/Wikibase.php: Remove wikibase sameAs A/B test config, part I (duration: 00m 56s) |
[production] |
| 23:26 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.7/extensions/AbuseFilter/includes/parser/AbuseFilterTokenizer.php: SWAT AbuseFilter: Tokenizer caching back to APC I8c6a4a95e (duration: 00m 54s) |
[production] |
| 23:19 |
<jforrester@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: Replace FR constants with numbers Ia52f644948 (duration: 00m 56s) |
[production] |
| 23:17 |
<jforrester@deploy1001> |
Synchronized multiversion/MWScript.php: Mark refreshMessageBlobs.php as a global script (duration: 00m 56s) |
[production] |
| 23:15 |
<mutante> |
repooled phab2001-vcs , fixes pybal / lvs alerts |
[production] |
| 23:13 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 23:10 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT Enable wgSpecialSearchFormOptions on production Wikidata T55652 (duration: 00m 57s) |
[production] |
| 23:01 |
<mutante> |
phab2001 - same issue with tin.eqiad.wmnet still showing up when first trying to git clone |
[production] |
| 22:52 |
<mutante> |
misweb2001 - a2dismod mpm_event ; systemctl restart apache2 to fix php7.0 dependency issue |
[production] |
| 22:50 |
<mutante> |
miscweb2001 - when first trying to git pull iegreview - still tries to resolve 'tin.eqiad.wmnet' which is long gone. fix is still to manually edit /srv/deployment/iegreview/iegreview-cache/cache/.git/config |
[production] |
| 22:46 |
<jforrester@deploy1001> |
Synchronized wmf-config/CirrusSearch-common.php: Hot-deploy T224634 to fix CirrusSearch for extension registration (duration: 00m 57s) |
[production] |
| 21:50 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 21:47 |
<mutante> |
installing OS on miscweb2001 VM failed at grub install step :( T224323 |
[production] |
| 21:47 |
<mutante> |
sign puppet cert request for phab2001 after reinstall (for some reason it needed me to connect to console and hit enter, reimage script itself was stuck) |
[production] |
| 20:54 |
<mutante> |
creating new ganeti VM miscweb2001.codfw.wmnet with same specs as krypton.eqiad.wmnet (T224323) |
[production] |
| 20:35 |
<arlolra> |
Updated Parsoid to 8546c79 (T219927, T211125) |
[production] |
| 20:35 |
<ejegg> |
updated payments-wiki from 332aaa96e2 to 45b73e7749 |
[production] |
| 20:28 |
<arlolra@deploy1001> |
Finished deploy [parsoid/deploy@6caac43]: Updating Parsoid to 8546c79 (duration: 07m 46s) |
[production] |
| 20:20 |
<arlolra@deploy1001> |
Started deploy [parsoid/deploy@6caac43]: Updating Parsoid to 8546c79 |
[production] |
| 20:10 |
<bblack> |
pool cp3044 (esams cache_upload ats-be) - T222937 |
[production] |
| 19:46 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.7/extensions/Collection/: Replace missing wfCollectionSuggestAction (duration: 00m 57s) |
[production] |
| 19:45 |
<XioNoX> |
enable cr1-codfw:et-0/2/1 - T224511 |
[production] |
| 19:45 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.6/extensions/Collection/: Replace missing wfCollectionSuggestAction (duration: 01m 01s) |
[production] |
| 19:42 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 19:32 |
<mutante> |
phba2001 - reinstalling with stretch - upgrade from jessie (T190568) |
[production] |
| 19:09 |
<XioNoX> |
enable cr1-codfw:et-0/2/0 - T224511 |
[production] |
| 18:37 |
<bblack@cumin1001> |
conftool action : set/pooled=no; selector: name=cp3044.esams.wmnet |
[production] |
| 17:44 |
<XioNoX> |
enable cr1-codfw:et-0/0/1 - T224511 |
[production] |