|
2015-08-03
§
|
| 13:50 |
<bblack> |
re-enabling puppet + ircecho on neon (vast majority of recovery spam is over with) |
[production] |
| 13:17 |
<bblack> |
re-enable agent, restarted apache2 on palladium, strontium, rhodium (fact_values truncated in mysql) |
[production] |
| 13:10 |
<bblack> |
rhodium too (puppetmaster stop) |
[production] |
| 13:05 |
<bblack> |
stopped puppet-agent + apache2 on strontium + palladium (no masters alive, for mysql maintenance) |
[production] |
| 12:59 |
<bblack> |
stopped ircecho + puppet-agent on neon (spam from epic puppetmaster fail) |
[production] |
| 12:52 |
<bblack> |
stop->wait->restart of apache2 service on palladium (seemed dead to puppet reqs) |
[production] |
| 12:21 |
<_joe_> |
bumped ganglia-monitor-aggregator on bast4001, the upstart script needs immediate fixing |
[production] |
| 11:01 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: avoid db1044 SPOF by repooling db1027 and db1015 (duration: 00m 12s) |
[production] |
| 10:56 |
<paravoid> |
switching GeoDNS to GeoIP2 |
[production] |
| 10:45 |
<paravoid> |
upgrading all AuthDNS servers to gdnsd 2.2.0 |
[production] |
| 09:31 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: depool db1035 for maintenance (duration: 00m 12s) |
[production] |
| 05:22 |
<@tin> |
ResourceLoader cache refresh completed at Mon Aug 3 05:22:15 UTC 2015 (duration 22m 14s) |
[production] |
| 02:23 |
<@tin> |
LocalisationUpdate completed (1.26wmf16) at 2015-08-03 02:23:21+00:00 |
[production] |
| 02:20 |
<l10nupdate> |
Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 21s) |
[production] |
| 01:47 |
<springle> |
starting OSC gerrit 228756 s5 wb_items_per_site.ips_site_page |
[production] |
| 00:03 |
<krenair> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/228198/ (duration: 00m 12s) |
[production] |
|
2015-08-01
§
|
| 06:04 |
<_joe_> |
removing some old apache access logs from mw1114 |
[production] |
| 05:06 |
<@tin> |
ResourceLoader cache refresh completed at Sat Aug 1 05:06:46 UTC 2015 (duration 6m 45s) |
[production] |
| 03:53 |
<andrewbogott> |
cleared out nova-conductor.log on labcontrol1001, restarted nova-conductor, graceful’d apache |
[production] |
| 02:23 |
<@tin> |
LocalisationUpdate completed (1.26wmf16) at 2015-08-01 02:23:15+00:00 |
[production] |
| 02:20 |
<l10nupdate> |
Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 11s) |
[production] |
| 00:12 |
<ori> |
Synchronized extract2.php: Ie919881a4: Add an API listing template to the allowed templates in extract2.php |
[production] |
| 00:01 |
<ori> |
Synchronized php-1.26wmf16/includes: Revert I4afaecd8: "Avoiding writing sessions for no reason", and undo several uncommitted live-hacks for debugging T102199 (duration: 00m 16s) |
[production] |
|
2015-07-31
§
|
| 20:14 |
<ori> |
Synchronized php-1.26wmf16/includes/objectcache/ObjectCacheSessionHandler.php: Uncommitted revert of I4afaecd to test impact on T102199 (duration: 00m 12s) |
[production] |
| 20:11 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1008 |
[production] |
| 19:55 |
<ori> |
Synchronized php-1.26wmf16/includes/User.php: More debug logging for T102199 (duration: 00m 13s) |
[production] |
| 19:54 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1007 |
[production] |
| 19:51 |
<ori> |
Synchronized php-1.26wmf16/includes/EditPage.php: More debug logging for T102199 (duration: 00m 12s) |
[production] |
| 19:21 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1006 |
[production] |
| 19:02 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1005 |
[production] |
| 18:44 |
<twentyafterfour> |
oddly, the symptom was that there were logs about apc cache entries that had been on the GC queue for too long, I guess this is due to phd being stuck |
[production] |
| 18:43 |
<twentyafterfour> |
restarted phd on iridium. I had to forcefully kill one stuck repository worker to get the daemons to restart properly. |
[production] |
| 18:36 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1004 |
[production] |
| 18:15 |
<mutante> |
multatuli - installing package upgrades |
[production] |
| 18:08 |
<legoktm> |
made User:Flow talk page manager a 'bot' on all wikis (except loginwiki) |
[production] |
| 18:08 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1003 |
[production] |
| 17:53 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1002 |
[production] |
| 17:41 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1001 T104887 |
[production] |
| 17:11 |
<greg-g> |
follow on to previous to be explicit: it's not deployed, it is queued for Monday morning SWAT |
[production] |
| 17:10 |
<aude> |
wmf/1.26wmf16 core submodule bump for Ic25edf7 (MultimediaViewer) is now on tin |
[production] |
| 17:06 |
<aude> |
Synchronized php-1.26wmf16/extensions/Wikidata: Fix api xml format (duration: 00m 20s) |
[production] |
| 15:52 |
<bd808> |
Rebuilt grafana-dashboards index to have 1 shard/2 replicas in logstash cluster |
[production] |
| 15:46 |
<bd808> |
Rebuilt kibana-int index to have 1 shard/2 replicas in logstash cluster |
[production] |