|
2014-04-22
§
|
| 05:14 |
<springle> |
db68 down. s1-analytics-slave cname to db1007 |
[production] |
| 04:35 |
<paravoid> |
reactivate esams<->HE & eqiad<->HE peerings; issues are confirmed to be resolved |
[production] |
| 03:32 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Apr 22 03:32:29 UTC 2014 (duration 32m 28s) |
[production] |
| 02:43 |
<LocalisationUpdate> |
completed (1.24wmf1) at 2014-04-22 02:43:01+00:00 |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.23wmf22) at 2014-04-22 02:29:34+00:00 |
[production] |
| 01:16 |
<awight> |
nevermind previous update. |
[production] |
| 01:11 |
<awight> |
update from 7dafce5f8fe265fb0ab6c96e01e59fc4362ea5b4 to cfe34fe0b10861167199a8f72bba279b9cac5e6e |
[production] |
| 00:19 |
<catrope> |
synchronized php-1.24wmf1/extensions/MobileFrontend/javascripts/modules/editor/VisualEditorOverlay.js 'Fix JS error on save' |
[production] |
| 00:15 |
<Reedy> |
torrus (on manutius) is down |
[production] |
| 00:05 |
<RoanKattouw> |
Restarted ircecho on ekrem, IRC working again now |
[production] |
| 00:00 |
<K4-713> |
updated payments from 2819549ab853ddd44c02519d4084580cb8572e7b --> 4811f6d3d80d126c8b3c89c11d20cc6416cb58f6 |
[production] |
|
2014-04-21
§
|
| 23:57 |
<RoanKattouw> |
Started ircd on ekrem, startup doesn't seem to be puppetized |
[production] |
| 23:24 |
<catrope> |
synchronized php-1.23wmf22/extensions/MultimediaViewer 'SWAT deploy cherry-picks' |
[production] |
| 23:24 |
<catrope> |
synchronized php-1.24wmf1/extensions/MobileFrontend 'SWAT deploy cherry-picks' |
[production] |
| 23:23 |
<catrope> |
synchronized php-1.24wmf1/extensions/MultimediaViewer 'SWAT deploy cherry-picks' |
[production] |
| 23:23 |
<catrope> |
synchronized php-1.24wmf1/extensions/VisualEditor 'SWAT deploy cherry-picks' |
[production] |
| 23:17 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Lithuanian namespace aliases for betawikiversity' |
[production] |
| 23:11 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'BetaFeatures whitelist' |
[production] |
| 23:11 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Beta Features whitelist' |
[production] |
| 22:51 |
<RoanKattouw> |
Restarted stuck Jenkins |
[production] |
| 21:33 |
<awight> |
updated payments: af35b7b175bbe232 --> 2819549ab853ddd44c025 |
[production] |
| 20:45 |
<andrewbogott> |
rebooted wtp1018 |
[production] |
| 20:45 |
<subbu> |
deployed Parsoid ec51e5d1 (deploy SHA 0dd607fc) |
[production] |
| 20:31 |
<manybubbles> |
rolling restart on remaining Elasticsearch servers to get the plugin (1010, 1011, 1012, 1015) |
[production] |
| 19:45 |
<manybubbles> |
rolling restart on more of the elasticsearch servers to pick up plugins (06, 07, 09) |
[production] |
| 19:15 |
<cmjohnson1> |
shutting down and relocating dobson |
[production] |
| 19:15 |
<cmjohnson1> |
shutting down and relocating linne |
[production] |
| 17:40 |
<cmjohnson> |
shutting down and relocating fenari |
[production] |
| 17:37 |
<cmjohnson> |
shutting down mexia to relocate to 12th floor |
[production] |
| 17:25 |
<aaron> |
synchronized php-1.24wmf1/includes/filerepo/file/LocalFile.php '2026e4aee7ffe00e0192914c0bd4a9ce04681c36' |
[production] |
| 17:22 |
<aaron> |
synchronized wmf-config/PoolCounterSettings-eqiad.php 'Adjust large file download pool counter config to tie up less workers' |
[production] |
| 17:18 |
<aaron> |
synchronized php-1.23wmf22/includes/filerepo/file/LocalFile.php '01ce2888d0d9011514a3fc16c1606b5b42f1ef37' |
[production] |
| 17:13 |
<demon> |
synchronized wmf-config/InitialiseSettings.php 'renderfile-nonstandard throttle config' |
[production] |
| 17:03 |
<aaron> |
synchronized php-1.24wmf1/thumb.php '44c46589ef02a30e60177ea268bc8de27a740434' |
[production] |
| 17:02 |
<aaron> |
synchronized php-1.23wmf22/thumb.php '95913654cbf0a2c35ddbef74d135eeea71600d54' |
[production] |
| 16:34 |
<ottomata> |
reinstalling elastic1013 (elastic1014 is still coming back online, but I don't want there to an extra eligible master for long) |
[production] |
| 16:04 |
<ottomata> |
reinstalling elastic1014 |
[production] |
| 15:22 |
<cmjohnson1> |
dataset2 going down to be relocated to the 12th floor |
[production] |
| 13:41 |
<manybubbles> |
rolling restart on some of the Elasticsearch servers to pick up new plugins. should not cause any trouble. |
[production] |
| 13:05 |
<Reedy> |
De-activated status.wm.o monitor for icinga due to false positive from HTTP auth |
[production] |
| 12:54 |
<paravoid> |
demoting myself, removing Commons crat/admin rights |
[production] |
| 12:41 |
<paravoid> |
escalating myself to Commons bureaucrat/admin, then adding GWToolset privileges |
[production] |
| 12:40 |
<paravoid> |
deleting 29 GWToolset XML under Swift's wikipedia-commons-gwtoolset-metadata container for user Fæ/ |
[production] |
| 11:51 |
<reedy> |
synchronized php-1.23wmf22/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca04' |
[production] |
| 11:51 |
<paravoid> |
deactivating esams<->HE peering, >90% packet loss between lon<->nyc |
[production] |
| 11:49 |
<paravoid> |
deactivating eqiad<->HE peering, >90% packet loss between lon<->nyc |
[production] |
| 11:45 |
<reedy> |
synchronized php-1.23wmf22/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca0467ac70de7' |
[production] |
| 03:55 |
<springle> |
reset pc100* slaves previously replicating from pmtpa |
[production] |
| 03:33 |
<ori> |
5.5k fatals over last 20 hrs, of which 3.5k are calls to doTransform() on a non-object at TimedMediaThumbnail.php:201, and 0.9k are Lua API OOMs at LuaSandbox/Engine.php:264 |
[production] |
| 03:30 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Apr 21 03:30:15 UTC 2014 (duration 30m 14s) |
[production] |