|
2016-06-27
§
|
| 09:06 |
<gehel@palladium> |
conftool action : set/pooled=yes; selector: elastic1032.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch']) |
[production] |
| 09:00 |
<gehel> |
adding new elasticsearch servers in eqiad to LVS |
[production] |
| 08:54 |
<godog> |
swift codfw-prod ms-be202[234] weight 2000 |
[production] |
| 07:15 |
<elukey> |
puppet stopped on analytics1049 to remove it completely from the Hadoop cluster - broken disk |
[production] |
| 02:51 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Jun 27 02:51:41 UTC 2016 (duration 7m 5s) |
[production] |
| 02:44 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.7) (duration: 08m 09s) |
[production] |
| 02:27 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 54s) |
[production] |
|
2016-06-24
§
|
| 18:41 |
<krenair@tin> |
Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295958/4 - fix mobile main page rendering on a bunch of wikis, effectively putting them back to how they were a few days ago (duration: 00m 37s) |
[production] |
| 17:19 |
<mobrovac> |
change-prop deploying df88a75b |
[production] |
| 17:05 |
<_joe_> |
re-started changeprop after disabling the dependency module |
[production] |
| 14:18 |
<paravoid> |
shutting down ms-fe3002 due to on-site work |
[production] |
| 14:05 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.7/includes/OutputPage.php: T138586 hotfix (duration: 00m 47s) |
[production] |
| 14:02 |
<mobrovac> |
scb100x disabled puppet to clear changeprop queues |
[production] |
| 13:22 |
<gehel> |
re-enabling puppet on maps1002 (still in pre-configuration state, only default role) |
[production] |
| 12:34 |
<hashar> |
Random resource loader entries are apparently faulty causing issues with css and/or javascript T138586 |
[production] |
| 12:04 |
<elukey@palladium> |
conftool action : set/pooled=no; selector: aqs1001.eqiad.wmnet |
[production] |
| 12:03 |
<elukey> |
rebooting aqs1001.eqiad.wmnet for kernel upgrades |
[production] |
| 10:55 |
<jynus> |
updated m1-slave dns to be db1001 |
[production] |
| 10:20 |
<hashar> |
gallium: restarted apache2 , potentially stuck proxy |
[production] |
| 10:18 |
<moritzm> |
upgrade nodejs on scb systems in codfw and restart node-based services |
[production] |
| 09:59 |
<ema> |
nginx rolling restart to enable TFO on all tlsproxies (T108827) |
[production] |
| 09:52 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1059 with low weight, increase weight of db1061, db1062 (duration: 00m 33s) |
[production] |
| 09:48 |
<moritzm> |
upgrade nodejs on restbase test systems (xenon/praseodymium/cerium/restbase-test) and restart restbase on those |
[production] |
| 09:09 |
<mobrovac> |
scb100x stopping puppet to stop change-prop and clear the queue |
[production] |
| 08:29 |
<moritzm> |
uploaded nodejs 4.4.6 for jessie-wikimedia to carbon |
[production] |
| 07:10 |
<elukey> |
memcached on mc1007 restarted with growth factor 1.05 (T129963) |
[production] |
| 03:54 |
<robh> |
data copy for labmon1001 verified complete with proper permissions, re-enabling and running puppet to start back up services |
[production] |
| 03:19 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Jun 24 03:19:55 UTC 2016 (duration 7m 4s) |
[production] |
| 03:12 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.7) (duration: 17m 24s) |
[production] |
| 02:38 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.6) (duration: 17m 08s) |
[production] |
| 01:22 |
<bblack> |
stream.wikimedia.org (RCStream) DNS moved to cache_misc termination. If anyone reports bugs with rcstream services, revert https://gerrit.wikimedia.org/r/295385 |
[production] |
|
2016-06-23
§
|
| 23:16 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295600/ (duration: 00m 29s) |
[production] |
| 23:15 |
<maxsem@tin> |
Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295600/ (duration: 00m 28s) |
[production] |
| 22:33 |
<chasemp> |
reimage labstore1005 post io testing |
[production] |
| 22:12 |
<chasemp> |
powercycle labstore1005 |
[production] |
| 21:24 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to wmf.6 |
[production] |
| 21:11 |
<chasemp> |
silence alerts for labstore1004 for setup |
[production] |
| 20:31 |
<ebernhardson> |
synced out latest logstash-plugins via trebuchet |
[production] |
| 20:17 |
<Dereckson> |
Run initSiteStats.php on cebwiki (T138533) |
[production] |
| 20:04 |
<jzerebecki@tin> |
Synchronized wmf-config/CommonSettings.php: Log PHP/HHVM errors in CLI mode to stderr, not stdout T138291 (duration: 00m 28s) |
[production] |
| 20:03 |
<robh> |
labmon1001 data restore at 100gb 50minutes in, 298gb total for restoration |
[production] |