2017-01-06
§
|
23:08 |
<godog> |
force puppet run on cache_upload in eqiad to switch thumbs back from codfw |
[production] |
22:38 |
<demon@tin> |
Synchronized multiversion: updateBranchPointers consolidation (duration: 00m 56s) |
[production] |
20:14 |
<demon@tin> |
Synchronized w: Dropping old entry point (duration: 00m 41s) |
[production] |
19:35 |
<demon@tin> |
Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources: I32e0b8f81ca2a2e9ffc0c3a379921e12465815f2 (duration: 00m 40s) |
[production] |
19:29 |
<demon@tin> |
Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources/mw.UploadWizard.js: I32e0b8f81ca2a2e9ffc0c3a379921e12465815f2 (duration: 00m 59s) |
[production] |
19:17 |
<ebernhardson> |
restarting elasticsearch on relforge100[12] to test new search-ltr plugin |
[production] |
19:07 |
<ostriches> |
gerrit: Started full reindex of all changes, should be background but will be watching |
[production] |
18:59 |
<mutante> |
gerrit restarting for config change 308753 - will be back in seconds |
[production] |
18:13 |
<mutante> |
mw1205 - restarted hhvm |
[production] |
18:08 |
<godog> |
force puppet run on eqiad cache_upload to switch thumbs to codfw |
[production] |
16:17 |
<godog> |
bounce swift-proxy on ms-fe100[123] leave ms-fe1004 for investigation |
[production] |
16:14 |
<hashar> |
Restarting Nodepool |
[production] |
16:05 |
<ema> |
wiping codfw caches T154758 |
[production] |
15:29 |
<cmjohnson1> |
powering off mw1239 to reseat DIMM |
[production] |
15:22 |
<papaul> |
elastic2025-elastic2036 - signing puppet certs, salt-key, initial run |
[production] |
14:53 |
<mark> |
papaul powercycled asw-a7-codfw 14:50 |
[production] |
14:16 |
<reedy@tin> |
Finished scap: Rebuild message cache for Echo api messages being missing T154110 (duration: 25m 00s) |
[production] |
13:51 |
<reedy@tin> |
Started scap: Rebuild message cache for Echo api messages being missing T154110 |
[production] |
10:00 |
<ariel@tin> |
Synchronized wmf-config/throttle.php: test, noop (duration: 02m 45s) |
[production] |
09:24 |
<paravoid> |
asw-a7-codfw is down, serial console unresponsive |
[production] |
09:12 |
<ariel@tin> |
Synchronized wmf-config/throttle.php: Adjust throttle rule for Maharashtra 'Edit Wikipedia' workshop (VNGIASS) (duration: 02m 46s) |
[production] |
07:08 |
<moritzm> |
installing crypto++ security updates on trusty hosts |
[production] |
04:37 |
<matt_flaschen> |
Finished FlowFixInconsistentBoards.php (production mode) on all wikis |
[production] |
04:27 |
<matt_flaschen> |
Started FlowFixInconsistentBoards.php (production mode) on all wikis |
[production] |
04:10 |
<mutante> |
Icinga now using Letsencrypt cert and all good |
[production] |
03:42 |
<mutante> |
icinga - debugging issue with cert change |
[production] |
03:05 |
<papaul> |
OS instalaltion on elastic2025-elastic2036 |
[production] |
02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.7) (duration: 13m 23s) |
[production] |
01:27 |
<bd808> |
Restarted logstash on logstash1002 (T154732) |
[production] |
01:06 |
<Krinkle> |
stream.wikimedia.org problems - nginx responds with HTTP 502 Bad Gateway to most requests |
[production] |
01:01 |
<ostriches> |
gerrit: back up from upgrade |
[production] |
01:00 |
<ostriches> |
gerrit: down for upgrade |
[production] |
00:13 |
<mutante> |
analytics1036, ms-fe1003 - ran puppet to fix Icinga |
[production] |