|
2016-05-18
ยง
|
| 22:00 |
<akosiaris> |
restart cassandra on maps-test200{2,3,4} to get logstash changing working |
[production] |
| 21:39 |
<akosiaris> |
enable puppet on aqs1001 aqs1002 aqs1003 aqs1004 aqs1005 aqs1006 maps-test2001 maps-test2002 maps-test2003 maps-test2004 maps2001 maps2002 maps2003 maps2004 |
[production] |
| 21:32 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: roll back group1 to 1.28.0-wmf.1 |
[production] |
| 21:25 |
<ejegg> |
updated DjangoBannerStats from 8172c4176ea2b78df0dccbfa052064f2739c64bd to 220f80ece976aecbfbfbe1cb476d79910eb48ff4 |
[production] |
| 20:41 |
<urandom> |
Renabling puppet on restbase production cluster : T126629 |
[production] |
| 20:27 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.2 |
[production] |
| 20:26 |
<yurik> |
git deployed tilerator, will restart one by one manually. https://gerrit.wikimedia.org/r/#/c/289542/ |
[production] |
| 20:19 |
<urandom> |
Rolling restart of restbase-test200[1-3].codfw.wmnet : T126629 |
[production] |
| 20:17 |
<urandom> |
restarting Cassandra on cerium.eqiad.wmnet : T126629 |
[production] |
| 20:15 |
<urandom> |
restarting Cassandra on praseodymium.eqiad.wmnet : T126629 |
[production] |
| 20:14 |
<twentyafterfour> |
Deployment of 1.28.0-wmf.2 [T134450] is no longer blocked Preparing to deploy to group1 wikis. |
[production] |
| 20:02 |
<urandom> |
Restarting Cassandra on xenon.eqiad.wmnet post-puppet-run : T126629 |
[production] |
| 19:55 |
<akosiaris> |
disable puppet on restbase1010 restbase1011 restbase1012 restbase1013 restbase1014 restbase1015 as well |
[production] |
| 19:51 |
<akosiaris> |
disabled puppet on aqs1001 aqs1002 aqs1003 aqs1004 aqs1005 aqs1006 maps-test2001 maps-test2002 maps-test2003 maps-test2004 maps2001 maps2002 maps2003 maps2004 restbase1007 restbase1008 restbase1009 restbase2001 restbase2002 restbase2003 restbase2004 restbase2005 restbase2006 restbase2007 restbase2008 restbase2009 for cassandra upgrade |
[production] |
| 18:51 |
<csteipp> |
added oathauth-enable right to Staff group on testwiki. |
[production] |
| 18:40 |
<urandom> |
Starting bootstrap of restbase2008-b.codfw.wmnet : T95253 |
[production] |
| 18:37 |
<csteipp@tin> |
Synchronized wmf-config/CommonSettings.php: Enable OATH on test wikis (duration: 00m 28s) |
[production] |
| 18:36 |
<csteipp@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 34s) |
[production] |
| 18:35 |
<urandom> |
Starting restbase2008-a.codfw.wmnet : T95253 |
[production] |
| 18:32 |
<urandom> |
Stopping restbase2008-a.codfw.wmnet and downgrading Cassandra to 2.1.13 : T95253 |
[production] |
| 18:31 |
<urandom> |
Stopping failed bootstrap of restbase2008-b.codfw.wmnet : T95253 |
[production] |
| 18:14 |
<csteipp> |
created oathauth_users in centralauth db |
[production] |
| 17:45 |
<twentyafterfour@tin> |
Synchronized php-1.28.0-wmf.2/extensions/Renameuser/RenameuserSQL.php: (no message) (duration: 00m 38s) |
[production] |
| 17:33 |
<moritzm> |
rolling restart of hhvm on mediawiki canaries in eqiad to pick up expat security update |
[production] |
| 16:01 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.2/includes/DefaultSettings.php: SWAT: [[gerrit:289438|Increase BotPasswordSessionProvider default priority]] (duration: 00m 26s) |
[production] |
| 15:52 |
<bblack> |
restarting varnish frontends on cache_misc + cache_maps to clear cached X-Cache entries |
[production] |
| 15:43 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.2/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT: [[gerrit:289431|Fix central logout]] (duration: 00m 26s) |
[production] |
| 15:32 |
<hashar> |
Deleted Nodepool snapshot images created around 14:30 apparently they havent been provisioned properly. Started nodepool again. Poke T135631 |
[production] |
| 15:29 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:289413|Set $wgDisableAuthManager = true]] (duration: 00m 25s) |
[production] |
| 15:18 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets/js/index-1a803501de.js: SWAT: manual js sync for portals (duration: 00m 28s) |
[production] |
| 15:14 |
<thcipriani@tin> |
Synchronized portals: (no message) (duration: 00m 31s) |
[production] |
| 15:14 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 30s) |
[production] |
| 15:12 |
<hashar> |
Nodepool / labs instance issue filled as https://phabricator.wikimedia.org/T135631 |
[production] |
| 15:11 |
<thcipriani@tin> |
Synchronized portals: (no message) (duration: 00m 32s) |
[production] |
| 15:10 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 33s) |
[production] |
| 15:10 |
<thcipriani> |
SWAT: running sync-portals for [[gerrit:289403|Updating footer on www.wikipedia.org portal]] |
[production] |
| 15:09 |
<hashar> |
Stopping Nodepool on labnodepool1001 , it can't spawn instances |
[production] |
| 15:02 |
<hashar> |
Zuul/Nodepool is out of instances. Looking |
[production] |
| 14:15 |
<moritzm> |
rolling restart of hhvm in codfw to pick up expat security update |
[production] |
| 13:57 |
<chasemp> |
downtime for dataset1001 puppet runs as T134896 causes failure (temporary for resize) |
[production] |
| 13:55 |
<jynus> |
disabling puppet on all database masters to test replication monitoring change T133337 |
[production] |
| 13:29 |
<chasemp> |
resize volume for nfs dumps per T134896 |
[production] |
| 12:45 |
<gehel> |
elasticsearch eqiad - reducing high watermark to rebalance disk space |
[production] |
| 11:41 |
<gehel> |
starting rolling restart of Elasticsearch codfw for Java update (T135499) |
[production] |
| 11:41 |
<gehel> |
rolling restart of Elasticsearch equiad for Java update completed (T135499) |
[production] |
| 11:17 |
<moritzm> |
install expat security updates |
[production] |
| 10:41 |
<kart_> |
Updated cxserver to 700dac2 |
[production] |
| 10:39 |
<mobrovac> |
zotero restarted it on sca100, was on 50% mem |
[production] |
| 10:38 |
<mobrovac> |
zotero restarted it on sca1002, was on 50% mem and 100% cpu |
[production] |
| 10:37 |
<kart_> |
Updating cxserver to 700dac2 |
[production] |