|
2015-08-03
§
|
| 19:07 |
<ottomata> |
stopped a couple of kafka brokers. acknowldeging.. |
[production] |
| 19:02 |
<bblack> |
https://gerrit.wikimedia.org/r/228882 reversion salted + nginx reloaded |
[production] |
| 18:28 |
<gwicke> |
switched restbase1002 and restbase1003 to iojs as well |
[production] |
| 17:36 |
<aude> |
Synchronized usagetracking.dblist: Enable usage tracking on zhwiki (duration: 00m 12s) |
[production] |
| 17:21 |
<legoktm> |
Synchronized php-1.26wmf16/includes/Revision.php: https://gerrit.wikimedia.org/r/228853 (duration: 00m 12s) |
[production] |
| 17:21 |
<ottomata> |
starting kafka partition reassignment to balance all partiions over to 3 new kafka brokers and off of analytics1021 |
[production] |
| 17:21 |
<gwicke> |
switching from node 0.10 to iojs 2.5 on restbase1001 after load testing on xenon went well |
[production] |
| 17:02 |
<legoktm> |
Synchronized wmf-config/logging.php: logging: Enable stacktrace printing (duration: 00m 12s) |
[production] |
| 17:00 |
<hoo> |
Started dumpwikidatajson.sh on snapshot1003 to re-create today's dump |
[production] |
| 16:55 |
<legoktm> |
Synchronized php-1.26wmf16/autoload.php: https://gerrit.wikimedia.org/r/#/c/228850/ (duration: 00m 12s) |
[production] |
| 16:54 |
<legoktm> |
Synchronized php-1.26wmf16/includes/debug/logger/: https://gerrit.wikimedia.org/r/#/c/228850/ (duration: 00m 11s) |
[production] |
| 16:49 |
<hoo> |
Removed today's Wikidata json dump (wikidata-20150803-all.json.gz) because it was incomplete due to the dataset problems earlier |
[production] |
| 16:27 |
<paravoid> |
upgrading junos on cr2-codfw |
[production] |
| 15:34 |
<bblack> |
wiping cp3034 disk cache (upload esams) for ipsec reload testing |
[production] |
| 15:23 |
<thcipriani> |
Synchronized php-1.26wmf16/extensions/MultimediaViewer: SWAT: Track image load time with statsv (touch and re-sync) [[gerrit:228218]] (duration: 00m 12s) |
[production] |
| 15:22 |
<ottomata> |
reinstalling analytics1013,1014 and 1020 with Jessie |
[production] |
| 15:10 |
<thcipriani> |
Synchronized php-1.26wmf16/extensions/MultimediaViewer: SWAT: Track image load time with statsv [[gerrit:228218]] (duration: 00m 12s) |
[production] |
| 14:59 |
<aude> |
Synchronized usagetracking.dblist: Enable usage tracking on trwiki (duration: 00m 12s) |
[production] |
| 14:54 |
<krenair> |
Synchronized php-1.26wmf16/extensions/SemanticResultFormats: https://gerrit.wikimedia.org/r/#/c/228793/ (duration: 00m 13s) |
[production] |
| 14:42 |
<aude> |
Synchronized usagetracking.dblist: Enable usage tracking on thwiki (duration: 00m 12s) |
[production] |
| 14:33 |
<mutante> |
temp. stop puppet on dataset1001 |
[production] |
| 14:27 |
<paravoid> |
upgrading junos on cr1-codfw |
[production] |
| 14:23 |
<moritzm> |
updated iojs on apt.wikimedia.org to 2.5.0 for jessie-wikimedia |
[production] |
| 14:21 |
<ottomata> |
upgrading kernel on analytics1042-1049 from 3.13.0.24.28 to 3.13.0.61.68 because T107698 |
[production] |
| 14:18 |
<aude> |
Synchronized usagetracking.dblist: Enable usage tracking on svwiki (duration: 00m 12s) |
[production] |
| 13:50 |
<bblack> |
re-enabling puppet + ircecho on neon (vast majority of recovery spam is over with) |
[production] |
| 13:17 |
<bblack> |
re-enable agent, restarted apache2 on palladium, strontium, rhodium (fact_values truncated in mysql) |
[production] |
| 13:10 |
<bblack> |
rhodium too (puppetmaster stop) |
[production] |
| 13:05 |
<bblack> |
stopped puppet-agent + apache2 on strontium + palladium (no masters alive, for mysql maintenance) |
[production] |
| 12:59 |
<bblack> |
stopped ircecho + puppet-agent on neon (spam from epic puppetmaster fail) |
[production] |
| 12:52 |
<bblack> |
stop->wait->restart of apache2 service on palladium (seemed dead to puppet reqs) |
[production] |
| 12:21 |
<_joe_> |
bumped ganglia-monitor-aggregator on bast4001, the upstart script needs immediate fixing |
[production] |
| 11:01 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: avoid db1044 SPOF by repooling db1027 and db1015 (duration: 00m 12s) |
[production] |
| 10:56 |
<paravoid> |
switching GeoDNS to GeoIP2 |
[production] |
| 10:45 |
<paravoid> |
upgrading all AuthDNS servers to gdnsd 2.2.0 |
[production] |
| 09:31 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: depool db1035 for maintenance (duration: 00m 12s) |
[production] |
| 05:22 |
<@tin> |
ResourceLoader cache refresh completed at Mon Aug 3 05:22:15 UTC 2015 (duration 22m 14s) |
[production] |
| 02:23 |
<@tin> |
LocalisationUpdate completed (1.26wmf16) at 2015-08-03 02:23:21+00:00 |
[production] |
| 02:20 |
<l10nupdate> |
Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 21s) |
[production] |
| 01:47 |
<springle> |
starting OSC gerrit 228756 s5 wb_items_per_site.ips_site_page |
[production] |
| 00:03 |
<krenair> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/228198/ (duration: 00m 12s) |
[production] |