|
2017-02-08
ยง
|
| 17:04 |
<jynus> |
rolling restart of replication thread of 29 mysql hosts T111654 |
[production] |
| 17:02 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic20(09|10|11|12).codfw.wmnet |
[production] |
| 16:32 |
<ema> |
cp3045 stuck rebooting, power-cycled |
[production] |
| 16:20 |
<ema> |
cp2017 stuck rebooting, power-cycled |
[production] |
| 16:19 |
<jynus> |
upgrading and restarting db1030 T111654 |
[production] |
| 16:15 |
<ema> |
pybal 1.13.4 built and uploaded to carbon |
[production] |
| 16:12 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: depool db1030 (duration: 00m 41s) |
[production] |
| 16:10 |
<chasemp> |
maintain-views and maintain-meta_p full runs on labsdb1009/10/11 |
[production] |
| 15:46 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: db1073 change IP - T156126 (duration: 00m 40s) |
[production] |
| 15:45 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: db1073 change IP - T156126 (duration: 00m 40s) |
[production] |
| 15:41 |
<ema> |
cp2011 stuck rebooting, power-cycled |
[production] |
| 15:40 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: repool db1026 (duration: 00m 41s) |
[production] |
| 15:28 |
<elukey> |
Eqiad cr1/cr2 - Updated analytics-in4 for new aqs nodes and removed decommed ones |
[production] |
| 15:20 |
<hoo@tin> |
Synchronized php-1.29.0-wmf.10/extensions/Wikidata: Wikibase uses multiple EntityPrefetchers (T157380) (duration: 02m 07s) |
[production] |
| 15:15 |
<hoo@tin> |
Synchronized php-1.29.0-wmf.11/extensions/Wikidata: Wikibase uses multiple EntityPrefetchers (T157380) (duration: 02m 11s) |
[production] |
| 14:54 |
<marostegui> |
Shutdown db1073 for maintenance - https://phabricator.wikimedia.org/T156126 |
[production] |
| 14:31 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic2008.codfw.wmnet |
[production] |
| 14:30 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic2007.codfw.wmnet |
[production] |
| 14:30 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic2006.codfw.wmnet |
[production] |
| 14:29 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic2005.codfw.wmnet |
[production] |
| 14:26 |
<elukey> |
restarting nutcracker in all the codfw mw servers to pick up the new shards |
[production] |
| 14:23 |
<ema> |
cp2022 stuck rebooting, power-cycled |
[production] |
| 14:23 |
<gehel> |
drain shards from elastic20(09|10|11|12) in preparation for reimage - T151326 |
[production] |
| 14:17 |
<jynus> |
upgrading and restarting db1026 T111654 |
[production] |
| 13:46 |
<elukey> |
replacing the codfw memcached/redis shards 12->16 |
[production] |
| 13:41 |
<marostegui> |
Start replication on db1064 - T153743 |
[production] |
| 13:40 |
<marostegui> |
Enable replication between db1095 and db1064 - T153743 |
[production] |
| 13:39 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1026 for maintenance (duration: 00m 41s) |
[production] |
| 13:10 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2008.codfw.wmnet |
[production] |
| 13:10 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2007.codfw.wmnet |
[production] |
| 13:09 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2006.codfw.wmnet |
[production] |
| 13:09 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2005.codfw.wmnet |
[production] |
| 12:43 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1073 - T156126 (duration: 00m 40s) |
[production] |
| 12:38 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1037 after maintenance (duration: 00m 41s) |
[production] |
| 12:17 |
<jynus> |
upgrading and restarting db1037 T111654 |
[production] |
| 12:16 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1037 for maintenance (duration: 00m 40s) |
[production] |
| 12:01 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1045 after maintenance (duration: 00m 42s) |
[production] |
| 10:39 |
<jynus> |
upgrading and restarting db1045 T111654 |
[production] |
| 10:38 |
<moritzm> |
upgrading openssl, libgd, lcms, gnutls, sqlite, libxpm and glibc in codfw mediawiki cluster (so get get effected by the restart during the HHVM upgrade) |
[production] |
| 10:11 |
<moritzm> |
upgrading hhvm on codfw mediawiki cluster |
[production] |
| 09:55 |
<hoo> |
Updated the Wikidata property suggester with data from last Monday's JSON dump and applied the T132839 workarounds |
[production] |
| 09:44 |
<elukey> |
boostrapping aqs1009-b (last new AQS Cassandra instance) |
[production] |
| 09:31 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic2004.codfw.wmnet |
[production] |
| 09:31 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic2003.codfw.wmnet |
[production] |
| 08:56 |
<ema> |
cache_upload: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401 |
[production] |
| 08:42 |
<gehel> |
drain shards from elastic200[5678] in preparation for reimage - T151326 |
[production] |
| 08:18 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2004.codfw.wmnet |
[production] |
| 08:18 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2003.codfw.wmnet |
[production] |
| 08:16 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic2002.codfw.wmnet |
[production] |
| 07:59 |
<marostegui> |
Adding 100G to the lv on dbstore1001 |
[production] |