|
2015-09-22
§
|
| 20:44 |
<mutante> |
cancel backup job of bast1001 on helium because running low on disk |
[production] |
| 20:22 |
<twentyafterfour@tin> |
Started scap: Test 1.26wmf24 |
[production] |
| 16:50 |
<robh> |
all mw servers returned to puppet enabled, puppet swat window over |
[production] |
| 16:23 |
<robh> |
re-enabled puppet on mw hosts, as both redirection changes are good |
[production] |
| 16:14 |
<robh> |
re-enabling puppet on mw hosts, as the new patchset 239278 deployed and tested fine on a single host, deploying to rest |
[production] |
| 16:04 |
<robh> |
disabling puppet across mw hosts for new configuration deployment |
[production] |
| 15:46 |
<godog> |
running puppet on restbase2001 |
[production] |
| 15:41 |
<godog> |
stop puppet on restbase2* pending codfw expansion |
[production] |
| 14:42 |
<cmjohnson1> |
shutting down elastic1005 and elastic1030 to move around within the data center |
[production] |
| 14:19 |
<bblack> |
starting slow restart of varnish + varnish-frontend daemon processes on global text, upload, and mobile clusters for shm_reclen (all randomly blended, no parallelism, ~5 minute spacing, will take ~9 hours - FEs will lose cache data, BEs will not) |
[production] |
| 14:14 |
<chasemp> |
depool elastic nodes for T112559 |
[production] |
| 11:18 |
<aude@tin> |
Synchronized php-1.26wmf23/extensions/Wikidata: Fix autocomment and change handling bugs (duration: 00m 21s) |
[production] |
| 10:42 |
<aude@tin> |
Synchronized arbitraryaccess.dblist: Enable arbitrary access for Wikibooks (duration: 00m 12s) |
[production] |
| 10:42 |
<aude@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks - try again for snapshot hosts (duration: 00m 12s) |
[production] |
| 10:35 |
<aude@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks (duration: 01m 12s) |
[production] |
| 10:17 |
<moritzm> |
enabled ferm on mw1152 (videoscaler) |
[production] |
| 10:03 |
<godog> |
finished stressdisk on restbase200[123] no errors reported |
[production] |
| 10:03 |
<moritzm> |
enabled ferm on mw1259 (videoscaler) |
[production] |
| 04:35 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Sep 22 04:35:46 UTC 2015 (duration 35m 45s) |
[production] |
| 02:22 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf23) at 2015-09-22 02:22:56+00:00 |
[production] |
| 02:19 |
<l10nupdate@tin> |
Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 00s) |
[production] |
| 01:53 |
<mutante> |
sodium - deleted salt key, revoked puppet cert, rm from icinga .. |
[production] |
| 00:32 |
<ori> |
Disabled Puppet for 24h on hafnium and stopped ganglia-monitor. gmond was saturating CPU. |
[production] |
|
2015-09-21
§
|
| 23:29 |
<krinkle@tin> |
Synchronized php-1.26wmf23/extensions/NavigationTiming/modules/ext.navigationTiming.js: T112593 (duration: 00m 14s) |
[production] |
| 23:21 |
<krenair@tin> |
Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/239828/ (duration: 00m 21s) |
[production] |
| 23:07 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238357/ (duration: 00m 13s) |
[production] |
| 22:26 |
<mutante> |
restarting gerrit for ssh config change |
[production] |
| 20:41 |
<mdholloway> |
MobileApps deployed sha1 013044e |
[production] |
| 20:39 |
<chasemp> |
banning elastic1005 for T112559 |
[production] |
| 20:29 |
<subbu> |
deployed parsoid version 9984d221 |
[production] |
| 19:40 |
<urandom> |
temporarily stopping codfw restbase cassandra nodes to test quorum auth |
[production] |
| 19:15 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: Ieccb23f: Enable async secondary writes for mysql-multiwrite cache (on testwiki) (duration: 00m 13s) |
[production] |
| 18:16 |
<cmjohnson1> |
disabling puppet on mw1031 |
[production] |
| 17:58 |
<chasemp> |
banning 1030 from eqiad elastic cluster for T112559#1660068 |
[production] |
| 15:22 |
<thcipriani@tin> |
Synchronized php-1.26wmf23/extensions/ContentTranslation/modules/entrypoint/ext.cx.interlanguagelink.js: SWAT: Revert "Do not call cxserver to display gray interwiki link" [[gerrit:239819]] (duration: 00m 11s) |
[production] |
| 15:10 |
<thcipriani@tin> |
Synchronized robots.txt: SWAT: Remove redundant entries from robots.txt [[gerrit:239403]] (duration: 00m 12s) |
[production] |
| 14:33 |
<ottomata> |
restart eventlogging with mysql consumer replace=True (AKA INSERT IGNORE) |
[production] |
| 14:09 |
<godog> |
rolling restart restbase in production after cassandra credentials change |
[production] |
| 12:53 |
<godog> |
rolling restart cassandra after enabling dc encryption, no nodes in codfw yet |
[production] |
| 12:01 |
<moritzm> |
repooled mw1160 (for T104969) |
[production] |
| 11:54 |
<moritzm> |
depooled mw1160 (for T104969) |
[production] |
| 11:51 |
<moritzm> |
repooled mw1158, mw1159 (for T104969) |
[production] |
| 11:39 |
<moritzm> |
depooled mw1158, mw1159 (for T104969) |
[production] |
| 11:37 |
<moritzm> |
depooled and repooled mw1156, mw1157 (for T104969) |
[production] |
| 11:26 |
<moritzm> |
repooled mw1154, mw1155 (for T104969) |
[production] |
| 11:21 |
<moritzm> |
depooled mw1154, mw1155 (for T104969) |
[production] |
| 10:39 |
<moritzm> |
repooled mw1026-mw1029 and mw1110-mw1113 (for T104968) |
[production] |
| 10:24 |
<moritzm> |
depooled mw1026-mw1029 and mw1110-mw1113 (for T104968) |
[production] |
| 10:17 |
<moritzm> |
repooled mw1100-mw1109 (for T104968) |
[production] |
| 10:17 |
<godog> |
create restbase user on cassandra cluster |
[production] |