|
2013-11-14
§
|
| 19:24 |
<RobH> |
18:50 not 13:30 durrrrr |
[production] |
| 19:19 |
<RobH> |
may have spoken too soon about ending, we shall see |
[production] |
| 19:18 |
<RobH> |
logging for record: outage started at approx 13:50 GMT, seems to have ended well, now-ish |
[production] |
| 19:00 |
<paravoid> |
pt-kill on s7 to kill LogPager queries |
[production] |
| 18:40 |
<cmjohnson1> |
dns update |
[production] |
| 18:38 |
<reedy> |
updated /a/common to {{Gerrit|I076d1eddf}}: CirrusSearch secondary for itwiki and plwiktionary |
[production] |
| 18:37 |
<reedy> |
synchronized docroot and w |
[production] |
| 18:36 |
<reedy> |
synchronized php-1.23wmf4/ |
[production] |
| 17:57 |
<reedy> |
synchronized php-1.23wmf3/extensions/Wikibase |
[production] |
| 17:23 |
<manybubbles> |
rebuilding all CirrusSearch indexes in place to get config updates |
[production] |
| 17:19 |
<hashar> |
Jenkins: updated code sniffer style from 574f68d to 0bebf0f7b |
[production] |
| 17:18 |
<manybubbles> |
synchronized wmf-config/InitialiseSettings.php 'plwiktionary and itwiki get cirrus as secondary' |
[production] |
| 17:14 |
<manybubbles> |
synchronized php-1.23wmf3/extensions/CirrusSearch/ 'Update CirrusSearch to master' |
[production] |
| 17:13 |
<manybubbles> |
synchronized php-1.23wmf2/extensions/CirrusSearch/ 'Update CirrusSearch to master' |
[production] |
| 16:45 |
<cmjohnson1> |
powering down analytics1009, analytics1011-13, analtyics1021, analytics1023 to relocate to rack A2 |
[production] |
| 16:06 |
<hashar> |
Gerrit/Git resetting mediawiki/extensions/Scribunto REL1_21 branch from fd1fbb4 to previous b5015a2. hexmode cherry-picked a bunch of changes from another branch/master instead of submitting a merge commit. |
[production] |
| 15:58 |
<apergos> |
preallocated cache files for amssq61 and 62 with ext4, this requires removing logbuf option in fstab |
[production] |
| 13:27 |
<faidon> |
synchronized wmf-config/filebackend.php 'switch multiMaster to swift-eqiad' |
[production] |
| 13:26 |
<faidon> |
updated /a/common to {{Gerrit|I898bfd6f1}}: filebackend: switch multiMaster to swift-eqiad |
[production] |
| 13:09 |
<apergos> |
preallocated cache file on amssq: 51, 52, 54, 56 59, 60 in addition to previously logged ones |
[production] |
| 13:02 |
<apergos> |
bad news: Nov 14 11:17:43 amssq54 kernel: [246380.560873] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) and this host has the preallocated cache file |
[production] |
| 09:36 |
<ori-l> |
Manually replaced jquery.ba-bbq.js on nickel:/srv/org/wikimedia/ganglia-web-latest/js with <https://github.com/cowboy/jquery-bbq/pull/42>. Upstream maintainer unresponsive. |
[production] |
| 07:46 |
<apergos> |
powercycled sq80, accessible but responded 'bus error' or 'i/o error' to all commands |
[production] |
| 07:34 |
<apergos> |
powercycled mw1210, unreachable via mgmt, no messages on console |
[production] |
| 02:36 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Nov 14 02:36:09 UTC 2013 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.23wmf2) at Thu Nov 14 02:13:58 UTC 2013 |
[production] |
| 02:08 |
<LocalisationUpdate> |
completed (1.23wmf3) at Thu Nov 14 02:07:56 UTC 2013 |
[production] |
| 01:27 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'Iaae2da0accd: Enable Bug54847 debug log group and route to ' |
[production] |
| 01:26 |
<ori> |
updated /a/common to {{Gerrit|I27cf69dd1}}: Revert "Rearrange W0 config-pages-supported variables order to be more logical." |
[production] |
| 00:32 |
<ori-l> |
restarted stuck profiler-to-carbon on professor |
[production] |
|
2013-11-13
§
|
| 23:37 |
<mutante> |
power down db61 |
[production] |
| 23:29 |
<mutante> |
DNS update - remove db61 (former OTRS box) |
[production] |
| 22:54 |
<mutante> |
DNS update - remove osm-web in Tampa, replaced with asset tags for wiping |
[production] |
| 21:17 |
<gwicke> |
deployed Parsoid be03c28 |
[production] |
| 20:58 |
<mutante> |
DNS update - fix warnings for analytics IPv6 addresses |
[production] |
| 20:48 |
<mutante> |
DNS update - kill frlabs, merge unmerged config-geo changes |
[production] |
| 20:46 |
<hashar> |
Jenkins: fixing up jobs *-phpcs-HEAD which were not using the proper phpcs path {{gerrit|95268}} |
[production] |
| 19:14 |
<reedy> |
synchronized wmf-config/ 'wgImgAuthDetails to true. Remove wmgUseUsabilityInitiative' |
[production] |
| 19:10 |
<reedy> |
updated /a/common to {{Gerrit|I2a05d4a97}}: Remove usages of $wmgUseUsabilityInitiative |
[production] |
| 19:07 |
<reedy> |
updated /a/common to {{Gerrit|I775790df6}}: Disable and remove CommunityHiring and CommunityApplications |
[production] |
| 19:03 |
<reedy> |
synchronized wmf-config/ 'Disable and remove CommunityHiring and CommunityApplications' |
[production] |
| 17:21 |
<Reedy> |
populateRevisionLength.php running in screen on terbium |
[production] |
| 16:56 |
<apergos> |
recreating esams varnish backend caches, preallocated with fallocate; the following have been done: amssq49, 50, 53, 55, 57, 58 |
[production] |
| 16:48 |
<hashar> |
Jenkins: phpcs jobs successfully running on lanthanum. !!! |
[production] |
| 16:31 |
<hashar> |
Jenkins: applying label 'hasPhpcs' on lanthanum slave, letting php code sniffer jobs to roam there. |
[production] |
| 16:29 |
<hashar> |
Jenkins switched PHP Codesniffer (phpcs) to use the version from integration/phpcs.git instead of the pear one {{gerrit|95172}} |
[production] |
| 15:56 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Set $wgImgAuthDetails to true' |
[production] |
| 14:50 |
<reedy> |
synchronized php-1.23wmf3/ 'Revert I49e34bdf13e8e8c6cd169c362c283fe1034bdc6d' |
[production] |
| 14:49 |
<reedy> |
synchronized php-1.23wmf2/ 'Revert I49e34bdf13e8e8c6cd169c362c283fe1034bdc6d' |
[production] |
| 14:00 |
<paravoid> |
adjusting new api appservers weight from 30 -> 20 to balance the load a little better |
[production] |