|
2018-04-23
ยง
|
| 11:50 |
<moritzm> |
reimaging mw1238,mw1239,mw1240 (app servers) to stretch |
[production] |
| 11:46 |
<moritzm> |
reimaging mw1285 (previous attempt had a hardware problem which failed to trigger the reboot via IPMI) ,mw1287,mw1288 (API servers) to stretch |
[production] |
| 11:41 |
<moritzm> |
installing poppler security updates |
[production] |
| 11:25 |
<mobrovac@tin> |
Finished deploy [citoid/deploy@b3c0818]: Add support for restful crossRef API and Wikidata QIDs - T108175 T176411 (duration: 03m 36s) |
[production] |
| 11:22 |
<mobrovac@tin> |
Started deploy [citoid/deploy@b3c0818]: Add support for restful crossRef API and Wikidata QIDs - T108175 T176411 |
[production] |
| 11:17 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@3f3f989]: Add lfnwiki, inhwiki, gorwiki and euwikisource, take #2 - T192678 (duration: 07m 21s) |
[production] |
| 11:10 |
<mobrovac@tin> |
Started deploy [restbase/deploy@3f3f989]: Add lfnwiki, inhwiki, gorwiki and euwikisource, take #2 - T192678 |
[production] |
| 11:09 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@3f3f989]: Add lfnwiki, inhwiki, gorwiki and euwikisource - T192678 (duration: 11m 47s) |
[production] |
| 11:00 |
<gehel> |
restarting wdqs updater on all wdqs notes |
[production] |
| 10:57 |
<mobrovac@tin> |
Started deploy [restbase/deploy@3f3f989]: Add lfnwiki, inhwiki, gorwiki and euwikisource - T192678 |
[production] |
| 10:26 |
<jdrewniak@tin> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:428299|Bumping portals to master (T128546)]] (duration: 01m 16s) |
[production] |
| 10:25 |
<jdrewniak@tin> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:428299|Bumping portals to master (T128546)]] (duration: 01m 17s) |
[production] |
| 09:56 |
<_joe_> |
restarting memcached on mc1020-1036 at 1 hour intervals - T184854 |
[production] |
| 09:13 |
<godog> |
Flashing Smart Array P840 in Slot 3 [ 4.52 -> 6.30 ] on ms-be2034 - T192721 T141756 |
[production] |
| 09:05 |
<_joe_> |
AMEND: restart memcached on mc1019 (T184854) |
[production] |
| 09:05 |
<_joe_> |
restart memcached on mw1019 (Ttail -f /var/log/etcdmirror-conftool-eqiad-wmnet/syslog.log |
[production] |
| 09:05 |
<vgutierrez> |
restarting pybal on lvs1006 |
[production] |
| 09:02 |
<_joe_> |
restarting etcdmirror on conf2002 after restarting nginx on conf1001 |
[production] |
| 08:59 |
<moritzm> |
reimaging mw1283,mw1285,mw1286 (API servers) to stretch |
[production] |
| 08:57 |
<marostegui> |
Deploy schema change on s6 codfw master (db2039) - this will generate lag on codfw - T191519 T188299 T190148 |
[production] |
| 08:56 |
<gehel> |
rolling restart of blazegraph on wdqs1004, 2004 and 2005 for JVM upgrade |
[production] |
| 08:55 |
<moritzm> |
reimaging mw1270,mw1271,mw1272 (app servers) to stretch |
[production] |
| 08:52 |
<vgutierrez> |
restarting pybal on esams cluster |
[production] |
| 08:50 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1078 (duration: 01m 16s) |
[production] |
| 08:48 |
<_joe_> |
upgrading nginx on the config cluster in eqiad (T164456) |
[production] |
| 08:47 |
<marostegui> |
Drop table logging_pre_1_10 in s5 - T118859 |
[production] |
| 08:47 |
<marostegui> |
Dropped table logging_pre_1_10 in s3 - T118859 |
[production] |
| 08:42 |
<vgutierrez> |
restarting pybal on lvs4006 |
[production] |
| 08:40 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 01m 18s) |
[production] |
| 08:36 |
<vgutierrez> |
restarting pybal on codfw (once at a time) |
[production] |
| 08:33 |
<vgutierrez> |
restart pybal on lvs4007 |
[production] |
| 08:31 |
<vgutierrez> |
restarting pybal on lvs5002 |
[production] |
| 08:30 |
<vgutierrez> |
restarting pybal on lvs5001 |
[production] |
| 08:30 |
<marostegui> |
Drop table logging_pre_1_10 in s4 - T118859 |
[production] |
| 08:27 |
<vgutierrez> |
restarting pybal on lvs4005 |
[production] |
| 08:27 |
<_joe_> |
restarting pybal on lvs5003 |
[production] |
| 08:17 |
<_joe_> |
upgrading nginx on the config cluster in codfw (T164456) |
[production] |
| 08:13 |
<marostegui> |
Drop table logging_pre_1_10 in s7 - T118859 |
[production] |
| 08:08 |
<_joe_> |
restarting memcached in codfw (T184854) |
[production] |
| 08:08 |
<gehel> |
restarting blazegraph on wdqs1003 (crazy number of java threads) |
[production] |
| 08:04 |
<moritzm> |
upgrading terbium to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 07:58 |
<ema> |
cp-misc: upgrade varnish to 5.1.3-1wm7 |
[production] |
| 07:55 |
<marostegui> |
reload haproxy on dbproxy1010 to depool labsdb1010 |
[production] |
| 07:55 |
<marostegui> |
Depool labsdb1010 - T184446 |
[production] |
| 07:50 |
<marostegui> |
Drop table logging_pre_1_10 in s2 - T118859 |
[production] |
| 07:47 |
<marostegui> |
Drop table logging_pre_1_10 in s6 - T118859 |
[production] |
| 07:36 |
<moritzm> |
upgrading remaining API servers to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 07:35 |
<elukey> |
reboot ms-be2034 - stuck in com2 console with "sd 0:1:0:1: rejecting I/O to offline device", not responsive to ssh |
[production] |
| 07:00 |
<moritzm> |
upgrading remaining app servers to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 06:26 |
<marostegui> |
Remove logging_pre_1_10 from codfw - T118859 |
[production] |