|
2014-01-16
§
|
| 19:18 |
<LeslieCarr> |
reseating pem0 on cr2-ulsfo to try and clear problem |
[production] |
| 19:13 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, testwiki, testwikidatawiki and mediawikiwiki to 1.23wmf11 |
[production] |
| 19:12 |
<^d> |
dropped all cirrusSearchLinksUpdate jobs from enwiki job queue. Job queue back to ~330k entries, far better than the ~3mil |
[production] |
| 19:06 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf10 |
[production] |
| 18:50 |
<paravoid> |
powercycling ms-be1002, down, console unresponsive |
[production] |
| 18:44 |
<manybubbles> |
starting Elasticsearch upgrade from 0.90.9 to 0.90.10 |
[production] |
| 18:01 |
<reedy> |
synchronized php-1.23wmf11/includes/specials/SpecialWatchlist.php |
[production] |
| 17:54 |
<reedy> |
synchronized docroot and w |
[production] |
| 17:45 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf9 till window |
[production] |
| 17:29 |
<reedy> |
finished scap: testwiki to 1.23wmf11 and build l10n cache (duration: 23m 24s) |
[production] |
| 17:08 |
<reedy> |
started scap: testwiki to 1.23wmf11 and build l10n cache |
[production] |
| 16:48 |
<cmjohnson1> |
es1007 replacing disk slot 6 |
[production] |
| 16:47 |
<cmjohnson1> |
tungsten replacing failing disk slot 3 |
[production] |
| 16:47 |
<cmjohnson1> |
db1004 replacing failing disk slot 9 |
[production] |
| 16:43 |
<reedy> |
synchronized php-1.23wmf11 |
[production] |
| 16:14 |
<bblack> |
rebooting cp1055 to clear out mess from XFS/kmem_alloc bug |
[production] |
| 15:48 |
<cmjohnson1> |
disabling puppet on db29 |
[production] |
| 11:55 |
<akosiaris> |
fixed the cp1065 puppet freshness check constantly bugging us and reenabled notifications after that |
[production] |
| 09:19 |
<ori> |
updated /a/common to {{Gerrit|Id13e614e5}}: repool db1042 |
[production] |
| 06:01 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1042' |
[production] |
| 05:58 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1033' |
[production] |
| 05:21 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1042' |
[production] |
| 03:29 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-16 03:28:58+00:00 |
[production] |
| 03:00 |
<Coren> |
Deleted all non-current pacct on neon; activity seems to have increased tenfold and /var/log is OOS |
[production] |
| 02:51 |
<LocalisationUpdate> |
completed (1.23wmf9) at 2014-01-16 02:51:29+00:00 |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-16 02:27:00+00:00 |
[production] |
| 01:30 |
<catrope> |
synchronized php-1.23wmf10/extensions/VisualEditor 'Forgot to run git submodule update earlier' |
[production] |
| 01:28 |
<catrope> |
synchronized php-1.23wmf10/resources/startup.js 'touch' |
[production] |
| 01:27 |
<catrope> |
synchronized php-1.23wmf10/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWTransclusionNode.js 'touch' |
[production] |
| 01:27 |
<catrope> |
synchronized php-1.23wmf9/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWTransclusionNode.js 'touch' |
[production] |
| 01:00 |
<aaron> |
synchronized php-1.23wmf10/includes/filebackend/SwiftFileBackend.php '1218cc7e9cba8342d0184523f1d7c1fec608e656' |
[production] |
| 00:49 |
<demon> |
synchronized wmf-config/InitialiseSettings.php 'Config new Cirrus logs' |
[production] |
| 00:43 |
<catrope> |
synchronized php-1.23wmf10/extensions/VisualEditor 'Update VE for cherry-picks' |
[production] |
| 00:43 |
<catrope> |
synchronized php-1.23wmf9/extensions/VisualEditor 'Update VE for cherry-picks' |
[production] |
| 00:29 |
<mflaschen> |
synchronized php-1.23wmf10/extensions/GettingStarted/ 'Sync GettingStarted wmf10 for hotfix' |
[production] |
| 00:29 |
<mflaschen> |
synchronized php-1.23wmf9/extensions/GettingStarted/ 'Sync GettingStarted wmf9 for hotfix' |
[production] |
| 00:16 |
<mflaschen> |
synchronized wmf-config/InitialiseSettings.php 'Deploy GuidedTour to astwiki, fawiki, and ruwiki' |
[production] |
| 00:07 |
<Tim> |
rebooting cp1066 following XFS deadlock |
[production] |
|
2014-01-15
§
|
| 23:33 |
<Tim> |
on cp1066: stopping varnish -- kswapd has gone crazy and is continuously flushing all buffers |
[production] |
| 21:31 |
<hashar> |
Jenkins: generation of MediaWiki documentation with doxygen got broken since Zuul upgrade. Fix: {{gerrit|107718}} |
[production] |
| 21:17 |
<aaron> |
synchronized php-1.23wmf10/includes/filebackend/SwiftFileBackend.php 'bc2a0ddbff4c3e4bff8e931163fd3d1c0340a78e' |
[production] |
| 20:50 |
<^d> |
gerrit: running gc on all repositories. think weekly cron to do this might be broken. |
[production] |
| 20:49 |
<manybubbles> |
deployment-prep updating elasticsearch to 0.90.10 |
[production] |
| 20:26 |
<cmjohnson1> |
db1031 swapping failing disk slot 11 |
[production] |
| 19:37 |
<cmjohnson1> |
replacing failing disk db1031 slot 8 |
[production] |
| 18:22 |
<awight> |
crm updated from 43a7475df8461f59ab49d8a932ccd76257dc6d72 to e82d2c5e64e67bee9bfa1f703fc94602addea2cb |
[production] |
| 17:18 |
<ottomata> |
updated ganglios in apt to 1.3 |
[production] |
| 14:58 |
<reedy> |
updated /a/common to {{Gerrit|I26f2f5322}}: db1040 to full steam |
[production] |
| 14:48 |
<cmjohnson1> |
replacing failing disk db1031 slot 6 |
[production] |
| 14:42 |
<cmjohnson1> |
replacing failing disk on db1029 slot 8 |
[production] |