|
2015-07-31
§
|
| 20:14 |
<ori> |
Synchronized php-1.26wmf16/includes/objectcache/ObjectCacheSessionHandler.php: Uncommitted revert of I4afaecd to test impact on T102199 (duration: 00m 12s) |
[production] |
| 20:11 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1008 |
[production] |
| 19:55 |
<ori> |
Synchronized php-1.26wmf16/includes/User.php: More debug logging for T102199 (duration: 00m 13s) |
[production] |
| 19:54 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1007 |
[production] |
| 19:51 |
<ori> |
Synchronized php-1.26wmf16/includes/EditPage.php: More debug logging for T102199 (duration: 00m 12s) |
[production] |
| 19:21 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1006 |
[production] |
| 19:02 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1005 |
[production] |
| 18:44 |
<twentyafterfour> |
oddly, the symptom was that there were logs about apc cache entries that had been on the GC queue for too long, I guess this is due to phd being stuck |
[production] |
| 18:43 |
<twentyafterfour> |
restarted phd on iridium. I had to forcefully kill one stuck repository worker to get the daemons to restart properly. |
[production] |
| 18:36 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1004 |
[production] |
| 18:15 |
<mutante> |
multatuli - installing package upgrades |
[production] |
| 18:08 |
<legoktm> |
made User:Flow talk page manager a 'bot' on all wikis (except loginwiki) |
[production] |
| 18:08 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1003 |
[production] |
| 17:53 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1002 |
[production] |
| 17:41 |
<godog> |
revert to openjdk8 and restart cassandra on restbase1001 T104887 |
[production] |
| 17:11 |
<greg-g> |
follow on to previous to be explicit: it's not deployed, it is queued for Monday morning SWAT |
[production] |
| 17:10 |
<aude> |
wmf/1.26wmf16 core submodule bump for Ic25edf7 (MultimediaViewer) is now on tin |
[production] |
| 17:06 |
<aude> |
Synchronized php-1.26wmf16/extensions/Wikidata: Fix api xml format (duration: 00m 20s) |
[production] |
| 15:52 |
<bd808> |
Rebuilt grafana-dashboards index to have 1 shard/2 replicas in logstash cluster |
[production] |
| 15:46 |
<bd808> |
Rebuilt kibana-int index to have 1 shard/2 replicas in logstash cluster |
[production] |
| 15:45 |
<andrewbogott> |
rebooting labvirt1005, again (3.16 this time) |
[production] |
| 15:19 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: reverting db1035 load to 10% (duration: 00m 14s) |
[production] |
| 15:03 |
<urandom> |
bouncing restbase1005 (attempting to reproduce GC trends) |
[production] |
| 14:54 |
<Coren> |
turned on alerting of backup status on labstore* with (by design) low limits. Expect alarms, and ignore. |
[production] |
| 14:44 |
<kart_> |
Update cxserver to 9669e19 |
[production] |
| 14:38 |
<andrewbogott> |
bumped the kernel version on labvirt1005, rebooting. |
[production] |
| 14:09 |
<godog> |
restart cassandra on restbase1004 to apply java downgrade, missed from batch downgrade yesterday |
[production] |
| 12:10 |
<godog> |
restbase1008 bootstrap finished successfully |
[production] |
| 10:30 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: returning db1035 to 100% load (duration: 00m 12s) |
[production] |
| 08:19 |
<ori> |
Synchronized wmf-config/CommonSettings.php: I7be6dd2f5: Set $wgAjaxEditStash to false, on suspicion of being implicated in T102199 (duration: 00m 12s) |
[production] |
| 07:35 |
<_joe_> |
powercycling analytics1013, no ssh, console unresponsive |
[production] |
| 04:45 |
<@tin> |
ResourceLoader cache refresh completed at Fri Jul 31 04:45:41 UTC 2015 (duration 45m 40s) |
[production] |
| 04:09 |
<springle> |
upgrade/restart dbstore1001 |
[production] |
| 03:48 |
<krenair> |
Synchronized php-1.26wmf16/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/228197/ (duration: 00m 12s) |
[production] |
| 02:31 |
<@tin> |
LocalisationUpdate completed (1.26wmf16) at 2015-07-31 02:31:20+00:00 |
[production] |
| 02:28 |
<l10nupdate> |
Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 13s) |
[production] |
| 00:35 |
<catrope> |
Synchronized php-1.26wmf16/extensions/Flow/includes/Model/WikiReference.php: debugging (duration: 00m 12s) |
[production] |
| 00:34 |
<catrope> |
Synchronized php-1.26wmf16/extensions/Flow/includes/Model/WikiReference.php: debugging (duration: 00m 12s) |
[production] |
| 00:29 |
<catrope> |
Synchronized php-1.26wmf16/extensions/Flow/includes/Model/WikiReference.php: debugging (duration: 00m 13s) |
[production] |
|
2015-07-30
§
|
| 23:52 |
<catrope> |
Synchronized flow.dblist: remove commons (duration: 00m 14s) |
[production] |
| 23:47 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/195886/ (duration: 00m 11s) |
[production] |
| 23:46 |
<krenair> |
Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/195886/ (duration: 00m 12s) |
[production] |
| 23:41 |
<catrope> |
Synchronized flow.dblist: Enable Flow on plwiki and commonswiki (duration: 00m 11s) |
[production] |
| 23:30 |
<ebernhardson> |
Synchronized php-1.26wmf16/extensions/DonationInterface/: Bump DonationInterfae in 1.26wmf16 again...its uses submodules (duration: 00m 15s) |
[production] |
| 23:29 |
<ebernhardson> |
Synchronized php-1.26wmf16/extensions/DonationInterface/: Bump DonationInterfae in 1.26wmf16 (duration: 00m 16s) |
[production] |
| 23:28 |
<robh> |
disregard log entry about racktables, never offlined |
[production] |
| 23:22 |
<ebernhardson> |
Synchronized php-1.26wmf16/includes/specials/SpecialMIMEsearch.php: (no message) (duration: 00m 12s) |
[production] |
| 23:21 |
<ebernhardson> |
Synchronized php-1.26wmf16/includes/specials/SpecialSearch.php: Fix search-suggest i18n for frwiki in SWAT (duration: 00m 14s) |
[production] |
| 23:21 |
<ebernhardson> |
Synchronized php-1.26wmf16/extensions/SpamBlacklist/: Update SpamBlacklist for SWAT (duration: 00m 11s) |
[production] |
| 23:10 |
<robh> |
killing apache on magnesium to manually trigger an outage of racktables and test catchpoint alert formatting |
[production] |