|
2013-11-14
§
|
| 17:13 |
<manybubbles> |
synchronized php-1.23wmf2/extensions/CirrusSearch/ 'Update CirrusSearch to master' |
[production] |
| 16:45 |
<cmjohnson1> |
powering down analytics1009, analytics1011-13, analtyics1021, analytics1023 to relocate to rack A2 |
[production] |
| 16:06 |
<hashar> |
Gerrit/Git resetting mediawiki/extensions/Scribunto REL1_21 branch from fd1fbb4 to previous b5015a2. hexmode cherry-picked a bunch of changes from another branch/master instead of submitting a merge commit. |
[production] |
| 15:58 |
<apergos> |
preallocated cache files for amssq61 and 62 with ext4, this requires removing logbuf option in fstab |
[production] |
| 13:27 |
<faidon> |
synchronized wmf-config/filebackend.php 'switch multiMaster to swift-eqiad' |
[production] |
| 13:26 |
<faidon> |
updated /a/common to {{Gerrit|I898bfd6f1}}: filebackend: switch multiMaster to swift-eqiad |
[production] |
| 13:09 |
<apergos> |
preallocated cache file on amssq: 51, 52, 54, 56 59, 60 in addition to previously logged ones |
[production] |
| 13:02 |
<apergos> |
bad news: Nov 14 11:17:43 amssq54 kernel: [246380.560873] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) and this host has the preallocated cache file |
[production] |
| 09:36 |
<ori-l> |
Manually replaced jquery.ba-bbq.js on nickel:/srv/org/wikimedia/ganglia-web-latest/js with <https://github.com/cowboy/jquery-bbq/pull/42>. Upstream maintainer unresponsive. |
[production] |
| 07:46 |
<apergos> |
powercycled sq80, accessible but responded 'bus error' or 'i/o error' to all commands |
[production] |
| 07:34 |
<apergos> |
powercycled mw1210, unreachable via mgmt, no messages on console |
[production] |
| 02:36 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Nov 14 02:36:09 UTC 2013 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.23wmf2) at Thu Nov 14 02:13:58 UTC 2013 |
[production] |
| 02:08 |
<LocalisationUpdate> |
completed (1.23wmf3) at Thu Nov 14 02:07:56 UTC 2013 |
[production] |
| 01:27 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'Iaae2da0accd: Enable Bug54847 debug log group and route to ' |
[production] |
| 01:26 |
<ori> |
updated /a/common to {{Gerrit|I27cf69dd1}}: Revert "Rearrange W0 config-pages-supported variables order to be more logical." |
[production] |
| 00:32 |
<ori-l> |
restarted stuck profiler-to-carbon on professor |
[production] |
|
2013-11-13
§
|
| 23:37 |
<mutante> |
power down db61 |
[production] |
| 23:29 |
<mutante> |
DNS update - remove db61 (former OTRS box) |
[production] |
| 22:54 |
<mutante> |
DNS update - remove osm-web in Tampa, replaced with asset tags for wiping |
[production] |
| 21:17 |
<gwicke> |
deployed Parsoid be03c28 |
[production] |
| 20:58 |
<mutante> |
DNS update - fix warnings for analytics IPv6 addresses |
[production] |
| 20:48 |
<mutante> |
DNS update - kill frlabs, merge unmerged config-geo changes |
[production] |
| 20:46 |
<hashar> |
Jenkins: fixing up jobs *-phpcs-HEAD which were not using the proper phpcs path {{gerrit|95268}} |
[production] |
| 19:14 |
<reedy> |
synchronized wmf-config/ 'wgImgAuthDetails to true. Remove wmgUseUsabilityInitiative' |
[production] |
| 19:10 |
<reedy> |
updated /a/common to {{Gerrit|I2a05d4a97}}: Remove usages of $wmgUseUsabilityInitiative |
[production] |
| 19:07 |
<reedy> |
updated /a/common to {{Gerrit|I775790df6}}: Disable and remove CommunityHiring and CommunityApplications |
[production] |
| 19:03 |
<reedy> |
synchronized wmf-config/ 'Disable and remove CommunityHiring and CommunityApplications' |
[production] |
| 17:21 |
<Reedy> |
populateRevisionLength.php running in screen on terbium |
[production] |
| 16:56 |
<apergos> |
recreating esams varnish backend caches, preallocated with fallocate; the following have been done: amssq49, 50, 53, 55, 57, 58 |
[production] |
| 16:48 |
<hashar> |
Jenkins: phpcs jobs successfully running on lanthanum. !!! |
[production] |
| 16:31 |
<hashar> |
Jenkins: applying label 'hasPhpcs' on lanthanum slave, letting php code sniffer jobs to roam there. |
[production] |
| 16:29 |
<hashar> |
Jenkins switched PHP Codesniffer (phpcs) to use the version from integration/phpcs.git instead of the pear one {{gerrit|95172}} |
[production] |
| 15:56 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Set $wgImgAuthDetails to true' |
[production] |
| 14:50 |
<reedy> |
synchronized php-1.23wmf3/ 'Revert I49e34bdf13e8e8c6cd169c362c283fe1034bdc6d' |
[production] |
| 14:49 |
<reedy> |
synchronized php-1.23wmf2/ 'Revert I49e34bdf13e8e8c6cd169c362c283fe1034bdc6d' |
[production] |
| 14:00 |
<paravoid> |
adjusting new api appservers weight from 30 -> 20 to balance the load a little better |
[production] |
| 13:24 |
<hashar> |
deploying phpcs on Jenkins slave using "the nameless wikimedia deployment system" |
[production] |
| 11:43 |
<mark> |
Starting amssq* persistent storage cache fragmentation rework, temporarily disabling puppet in the process |
[production] |
| 10:53 |
<mark> |
Manually set vm.dirty_ratio to 10 on amssq57 |
[production] |
| 08:38 |
<apergos> |
rebooting amssq48, deadlock in kmalloc (started about 10 hours ago) |
[production] |
| 06:31 |
<apergos> |
rebooting amssq52, deadlock in kmem_alloc |
[production] |
| 04:38 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'I5043210cb: Enable 'exception-json' debug log group; route to ' |
[production] |
| 04:37 |
<ori> |
updated /a/common to {{Gerrit|I5043210cb}}: Enable 'exception-json' debug log group; route to $wmfUdp2logDest |
[production] |
| 03:17 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Nov 13 03:17:40 UTC 2013 |
[production] |
| 02:56 |
<springle> |
synchronized wmf-config/db-eqiad.php 'QueryPage::recache traffic' |
[production] |
| 02:42 |
<springle> |
synchronized wmf-config/db-eqiad.php 'move S3 Query::recache traffic to snapshot slave' |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.23wmf2) at Wed Nov 13 02:31:21 UTC 2013 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.23wmf3) at Wed Nov 13 02:17:07 UTC 2013 |
[production] |
| 02:13 |
<springle> |
restarting externallinks.el_id schema change jobs |
[production] |