|
2018-05-08
ยง
|
| 16:27 |
<mutante> |
mw2251,mw2252,mw2201 - reinstall with stretch |
[production] |
| 16:22 |
<herron> |
cleared low count edac counters on hosts mw2205 dbstore1002 db1051 elastic1029 T183177 |
[production] |
| 16:19 |
<urandom> |
force (split) compaction of wikipedia_T_mobile__ng_lead.data, restbase1016 - T192689 |
[production] |
| 16:15 |
<dzahn@neodymium> |
conftool action : set/pooled=yes; selector: name=mw2223.codfw.wmnet |
[production] |
| 16:14 |
<dzahn@neodymium> |
conftool action : set/pooled=yes; selector: name=mw2222.codfw.wmnet |
[production] |
| 16:10 |
<dzahn@neodymium> |
conftool action : set/pooled=yes; selector: name=mw2215.codfw.wmnet |
[production] |
| 16:09 |
<XioNoX> |
failing traffic over lvs2004 - T193677 |
[production] |
| 16:04 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. Codfw only. T167039 (duration: 01m 03s) |
[production] |
| 16:03 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. Codfw only. T167039 |
[production] |
| 16:01 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. Codfw only. T167039 (duration: 00m 42s) |
[production] |
| 16:01 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. Codfw only. T167039 |
[production] |
| 15:53 |
<mutante> |
switching performance.wikimedia.org from graphite to webperf backends - running puppet on cache::misc servers (T158837) |
[production] |
| 15:46 |
<demon@tin> |
Pruned MediaWiki: 1.32.0-wmf.1 [keeping static files] (duration: 01m 47s) |
[production] |
| 15:30 |
<XioNoX> |
starting pybal on lvs2001 - T193677 |
[production] |
| 15:26 |
<godog> |
(un)load edac kernel modules on thumbor1004 to test resetting counters - T183177 |
[production] |
| 15:09 |
<XioNoX> |
stopping pybal on lvs2001 - T193677 |
[production] |
| 15:06 |
<ottomata> |
beginnng Kafka upgrade of main-codfw: T167039 |
[production] |
| 14:53 |
<XioNoX> |
re-enable pybal on lvs2004 - T193677 |
[production] |
| 14:48 |
<XioNoX> |
disabling pybal on lvs2004 - T193677 |
[production] |
| 14:37 |
<mutante> |
LDAP: added 'sbailey' to group 'wmf' (T194091) |
[production] |
| 14:19 |
<ppchelko@tin> |
Started restart [changeprop/deploy@7e86531]: Restart changeprop to try forcing it rebalancing topics |
[production] |
| 14:15 |
<mutante> |
mw2215,mw2222,mw2223 - reinstalling with stretch |
[production] |
| 13:43 |
<zeljkof> |
EU SWAT finished |
[production] |
| 13:42 |
<zfilipin@tin> |
Synchronized php-1.32.0-wmf.2/extensions/Translate: SWAT: [[gerrit:431744|Refactor TranslationUpdateJob to use only primitive types for parameters (T192111)]] (duration: 01m 11s) |
[production] |
| 13:25 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:431628|Enable maps i18n everywhere (T191655)]] (duration: 01m 00s) |
[production] |
| 13:14 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:430388|Enable AdvancedSearch BetaFeature on all wikis (T193182)]] (duration: 01m 00s) |
[production] |
| 13:02 |
<marostegui> |
Manually fail disk #9 on db1073 to get it replaced |
[production] |
| 12:20 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Remove db1055 (duration: 00m 59s) |
[production] |
| 12:19 |
<moritzm> |
reimaging mw2159, mw2160, mw2161 (job runners) to stretch |
[production] |
| 12:18 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Remove db1055 (duration: 00m 59s) |
[production] |
| 12:17 |
<moritzm> |
upgrading app servers in beta to wikidiff 1.6.0 (T190717) |
[production] |
| 12:16 |
<moritzm> |
upgrading app servers in beta to |
[production] |
| 12:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1064 with low load (duration: 00m 59s) |
[production] |
| 11:36 |
<marostegui> |
Deploy schema change on db1103:3314 - T191519 T188299 T190148 |
[production] |
| 11:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 59s) |
[production] |
| 11:18 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Really depool db2092 (duration: 00m 53s) |
[production] |
| 10:29 |
<moritzm> |
reimaging mw1347, mw1348 (API servers) to stretch (last two remaining API servers in eqiad) |
[production] |
| 10:22 |
<jynus> |
stop mariadb on db1055 to clone it to db1064 |
[production] |
| 10:15 |
<moritzm> |
reimaging mw1310, mw1311 (job runners) to stretch |
[production] |
| 09:58 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 54s) |
[production] |
| 09:25 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1121 after alter table (duration: 01m 00s) |
[production] |
| 09:20 |
<elukey> |
forced a BBU re-learn cycle on analytics1032 |
[production] |
| 09:17 |
<gehel> |
reducing replication factor on cassandra v3 (unused) keyspace for maps |
[production] |
| 08:56 |
<moritzm> |
reimaging mw1345, mw1346 (API servers) to stretch |
[production] |
| 08:30 |
<moritzm> |
reimaging mw2156, mw2157, mw2158 (job runners) to stretch |
[production] |
| 08:27 |
<moritzm> |
reimaging mw1308, mw1309 (job runners) to stretch |
[production] |
| 08:03 |
<marostegui> |
Stop MySQL on db1116 to transfer its content to db2092 - T190704 |
[production] |
| 07:59 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2092 T190704 (duration: 00m 57s) |
[production] |
| 07:53 |
<elukey> |
second attempt to remove the cassandra-metrics-collector (+ cleanup) from aqs* |
[production] |
| 07:30 |
<jynus> |
cleaning up maintenance hosts (terbium, etc.) from tendril maintenance files |
[production] |