|
2014-01-17
§
|
| 21:52 |
<ori> |
to restart morebots: ssh to tool labs, become morebots, then: qdel $(qstat | grep production | cut -d' ' -f 1) ; sleep 5 ; jstart -N production /usr/lib/adminbot/adminlogbot.py --config ./confs/production-logbot.py |
[production] |
| 21:51 |
<ori> |
restarted morebots |
[production] |
| 21:51 |
<LeslieCarr> |
removed myself from all mail aliases |
[production] |
| 16:14 |
<hashar> |
synchronized wmf-config/InitialiseSettings.php 'touch, restore $oaiAgentRegex' |
[production] |
| 16:14 |
<hashar> |
synchronized wmf-config/CommonSettings.php 'restore $oaiAgentRegex' |
[production] |
| 09:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 'LB 'dump' s[2-7]' |
[production] |
| 08:50 |
<manybubbles> |
finished elasticsearch upgrade. we're now 0.90.10 all the way. |
[production] |
| 08:50 |
<ori> |
Replacing StatsD on tungsten with txStatsD; see commit I19ecf608d for rationale. |
[production] |
| 08:50 |
<ori> |
restarted morebots. morebots missed Sean's 'repool db1007, depool db1041' @ 6:26 UTC. |
[production] |
|
2014-01-16
§
|
| 19:22 |
<reedy> |
synchronized database lists files: |
[production] |
| 19:20 |
<Reedy> |
Created zhwikivoyage echo tables on db1029 |
[production] |
| 19:18 |
<LeslieCarr> |
reseating pem0 on cr2-ulsfo to try and clear problem |
[production] |
| 19:13 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, testwiki, testwikidatawiki and mediawikiwiki to 1.23wmf11 |
[production] |
| 19:12 |
<^d> |
dropped all cirrusSearchLinksUpdate jobs from enwiki job queue. Job queue back to ~330k entries, far better than the ~3mil |
[production] |
| 19:06 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf10 |
[production] |
| 18:50 |
<paravoid> |
powercycling ms-be1002, down, console unresponsive |
[production] |
| 18:44 |
<manybubbles> |
starting Elasticsearch upgrade from 0.90.9 to 0.90.10 |
[production] |
| 18:01 |
<reedy> |
synchronized php-1.23wmf11/includes/specials/SpecialWatchlist.php |
[production] |
| 17:54 |
<reedy> |
synchronized docroot and w |
[production] |
| 17:45 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf9 till window |
[production] |
| 17:29 |
<reedy> |
finished scap: testwiki to 1.23wmf11 and build l10n cache (duration: 23m 24s) |
[production] |
| 17:08 |
<reedy> |
started scap: testwiki to 1.23wmf11 and build l10n cache |
[production] |
| 16:48 |
<cmjohnson1> |
es1007 replacing disk slot 6 |
[production] |
| 16:47 |
<cmjohnson1> |
tungsten replacing failing disk slot 3 |
[production] |
| 16:47 |
<cmjohnson1> |
db1004 replacing failing disk slot 9 |
[production] |
| 16:43 |
<reedy> |
synchronized php-1.23wmf11 |
[production] |
| 16:14 |
<bblack> |
rebooting cp1055 to clear out mess from XFS/kmem_alloc bug |
[production] |
| 15:48 |
<cmjohnson1> |
disabling puppet on db29 |
[production] |
| 11:55 |
<akosiaris> |
fixed the cp1065 puppet freshness check constantly bugging us and reenabled notifications after that |
[production] |
| 09:19 |
<ori> |
updated /a/common to {{Gerrit|Id13e614e5}}: repool db1042 |
[production] |
| 06:01 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1042' |
[production] |
| 05:58 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1033' |
[production] |
| 05:21 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1042' |
[production] |
| 03:29 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-16 03:28:58+00:00 |
[production] |
| 03:00 |
<Coren> |
Deleted all non-current pacct on neon; activity seems to have increased tenfold and /var/log is OOS |
[production] |
| 02:51 |
<LocalisationUpdate> |
completed (1.23wmf9) at 2014-01-16 02:51:29+00:00 |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-16 02:27:00+00:00 |
[production] |
| 01:30 |
<catrope> |
synchronized php-1.23wmf10/extensions/VisualEditor 'Forgot to run git submodule update earlier' |
[production] |
| 01:28 |
<catrope> |
synchronized php-1.23wmf10/resources/startup.js 'touch' |
[production] |
| 01:27 |
<catrope> |
synchronized php-1.23wmf10/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWTransclusionNode.js 'touch' |
[production] |
| 01:27 |
<catrope> |
synchronized php-1.23wmf9/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWTransclusionNode.js 'touch' |
[production] |
| 01:00 |
<aaron> |
synchronized php-1.23wmf10/includes/filebackend/SwiftFileBackend.php '1218cc7e9cba8342d0184523f1d7c1fec608e656' |
[production] |
| 00:49 |
<demon> |
synchronized wmf-config/InitialiseSettings.php 'Config new Cirrus logs' |
[production] |
| 00:43 |
<catrope> |
synchronized php-1.23wmf10/extensions/VisualEditor 'Update VE for cherry-picks' |
[production] |
| 00:43 |
<catrope> |
synchronized php-1.23wmf9/extensions/VisualEditor 'Update VE for cherry-picks' |
[production] |
| 00:29 |
<mflaschen> |
synchronized php-1.23wmf10/extensions/GettingStarted/ 'Sync GettingStarted wmf10 for hotfix' |
[production] |
| 00:29 |
<mflaschen> |
synchronized php-1.23wmf9/extensions/GettingStarted/ 'Sync GettingStarted wmf9 for hotfix' |
[production] |
| 00:16 |
<mflaschen> |
synchronized wmf-config/InitialiseSettings.php 'Deploy GuidedTour to astwiki, fawiki, and ruwiki' |
[production] |
| 00:07 |
<Tim> |
rebooting cp1066 following XFS deadlock |
[production] |