|
2013-11-14
§
|
| 20:49 |
<reedy> |
synchronized php-1.23wmf4/includes/ |
[production] |
| 20:48 |
<mark> |
Raised session_max on eqiad bits caches from 200k to 400k |
[production] |
| 20:45 |
<ori> |
synchronized php-1.23wmf3/resources/startup.js 'touch' |
[production] |
| 20:42 |
<ori> |
synchronized php-1.23wmf3/extensions/UniversalLanguageSelector 'Revert "Update UniversalLanguageSelector to b2f9e4211efc"' |
[production] |
| 20:26 |
<ori> |
synchronized php-1.23wmf3/extensions/UniversalLanguageSelector 'Update UniversalLanguageSelector to b2f9e4211efc' |
[production] |
| 20:19 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Phase 1 wikis to 1.23wmf4 |
[production] |
| 20:14 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf3 |
[production] |
| 20:12 |
<reedy> |
updated /a/common to {{Gerrit|I3034b1e4e}}: Add/update symlinks |
[production] |
| 20:05 |
<reedy> |
Finished syncing Wikimedia installation... : testwiki to 1.23wmf4 and build l10n cache |
[production] |
| 19:34 |
<reedy> |
Started syncing Wikimedia installation... : testwiki to 1.23wmf4 and build l10n cache |
[production] |
| 19:33 |
<mark> |
Changed IP addresses of upload-lb.eqiad.wikimedia.org |
[production] |
| 19:24 |
<RobH> |
18:50 not 13:30 durrrrr |
[production] |
| 19:19 |
<RobH> |
may have spoken too soon about ending, we shall see |
[production] |
| 19:18 |
<RobH> |
logging for record: outage started at approx 13:50 GMT, seems to have ended well, now-ish |
[production] |
| 19:00 |
<paravoid> |
pt-kill on s7 to kill LogPager queries |
[production] |
| 18:40 |
<cmjohnson1> |
dns update |
[production] |
| 18:38 |
<reedy> |
updated /a/common to {{Gerrit|I076d1eddf}}: CirrusSearch secondary for itwiki and plwiktionary |
[production] |
| 18:37 |
<reedy> |
synchronized docroot and w |
[production] |
| 18:36 |
<reedy> |
synchronized php-1.23wmf4/ |
[production] |
| 17:57 |
<reedy> |
synchronized php-1.23wmf3/extensions/Wikibase |
[production] |
| 17:23 |
<manybubbles> |
rebuilding all CirrusSearch indexes in place to get config updates |
[production] |
| 17:19 |
<hashar> |
Jenkins: updated code sniffer style from 574f68d to 0bebf0f7b |
[production] |
| 17:18 |
<manybubbles> |
synchronized wmf-config/InitialiseSettings.php 'plwiktionary and itwiki get cirrus as secondary' |
[production] |
| 17:14 |
<manybubbles> |
synchronized php-1.23wmf3/extensions/CirrusSearch/ 'Update CirrusSearch to master' |
[production] |
| 17:13 |
<manybubbles> |
synchronized php-1.23wmf2/extensions/CirrusSearch/ 'Update CirrusSearch to master' |
[production] |
| 16:45 |
<cmjohnson1> |
powering down analytics1009, analytics1011-13, analtyics1021, analytics1023 to relocate to rack A2 |
[production] |
| 16:06 |
<hashar> |
Gerrit/Git resetting mediawiki/extensions/Scribunto REL1_21 branch from fd1fbb4 to previous b5015a2. hexmode cherry-picked a bunch of changes from another branch/master instead of submitting a merge commit. |
[production] |
| 15:58 |
<apergos> |
preallocated cache files for amssq61 and 62 with ext4, this requires removing logbuf option in fstab |
[production] |
| 13:27 |
<faidon> |
synchronized wmf-config/filebackend.php 'switch multiMaster to swift-eqiad' |
[production] |
| 13:26 |
<faidon> |
updated /a/common to {{Gerrit|I898bfd6f1}}: filebackend: switch multiMaster to swift-eqiad |
[production] |
| 13:09 |
<apergos> |
preallocated cache file on amssq: 51, 52, 54, 56 59, 60 in addition to previously logged ones |
[production] |
| 13:02 |
<apergos> |
bad news: Nov 14 11:17:43 amssq54 kernel: [246380.560873] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) and this host has the preallocated cache file |
[production] |
| 09:36 |
<ori-l> |
Manually replaced jquery.ba-bbq.js on nickel:/srv/org/wikimedia/ganglia-web-latest/js with <https://github.com/cowboy/jquery-bbq/pull/42>. Upstream maintainer unresponsive. |
[production] |
| 07:46 |
<apergos> |
powercycled sq80, accessible but responded 'bus error' or 'i/o error' to all commands |
[production] |
| 07:34 |
<apergos> |
powercycled mw1210, unreachable via mgmt, no messages on console |
[production] |
| 02:36 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Nov 14 02:36:09 UTC 2013 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.23wmf2) at Thu Nov 14 02:13:58 UTC 2013 |
[production] |
| 02:08 |
<LocalisationUpdate> |
completed (1.23wmf3) at Thu Nov 14 02:07:56 UTC 2013 |
[production] |
| 01:27 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'Iaae2da0accd: Enable Bug54847 debug log group and route to ' |
[production] |
| 01:26 |
<ori> |
updated /a/common to {{Gerrit|I27cf69dd1}}: Revert "Rearrange W0 config-pages-supported variables order to be more logical." |
[production] |
| 00:32 |
<ori-l> |
restarted stuck profiler-to-carbon on professor |
[production] |