|
2014-01-18
§
|
| 11:04 |
<springle> |
xtrabackup clone db1007 to db1033 |
[production] |
| 10:52 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1041' |
[production] |
| 10:35 |
<paravoid> |
powercycling ms-be1012; down for 12h, console unresponsive |
[production] |
| 03:52 |
<Coren> |
marc (on neon) |
[production] |
| 03:50 |
<Coren> |
marc truncated /var/log/ganglia/ganglia_parser.log again. Check incoming email to ops-l |
[production] |
| 03:39 |
<ori> |
synchronized php-1.23wmf11/extensions/UniversalLanguageSelector 'Update ULS to master for I862b01e6b (@embed fix)' |
[production] |
| 03:38 |
<ori> |
synchronized php-1.23wmf11/extensions/UniversalLanguageSelector/resources/js/ext.uls.webfonts.js 'Update UniversalLanguageSelector to master for I2da436caa: Wait till rendering thread completion before applying webfonts (Bug: 59958)' |
[production] |
| 03:32 |
<ori> |
synchronized php-1.23wmf10/extensions/UniversalLanguageSelector 'Update ULS to master for I862b01e6b (@embed fix)' |
[production] |
| 02:46 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-18 02:46:09+00:00 |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-18 02:26:05+00:00 |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-18 02:12:58+00:00 |
[production] |
| 01:28 |
<LeslieCarr> |
Leslie Carr is signing off. au revoir! |
[production] |
| 01:23 |
<Reedy> |
[01:18:21] <LeslieCarr> ok !log exporting no routes over XO transit in sdtpa |
[production] |
|
2014-01-17
§
|
| 22:12 |
<awight> |
updated legacy paypal a4748700afaa4a7c5e9b84bfa1f78e3298e04ad1 to 725055ac1e2217806245c595d90b1ab63ec49007 |
[production] |
| 21:52 |
<ori> |
to restart morebots: ssh to tool labs, become morebots, then: qdel $(qstat | grep production | cut -d' ' -f 1) ; sleep 5 ; jstart -N production /usr/lib/adminbot/adminlogbot.py --config ./confs/production-logbot.py |
[production] |
| 21:51 |
<ori> |
restarted morebots |
[production] |
| 21:51 |
<LeslieCarr> |
removed myself from all mail aliases |
[production] |
| 16:14 |
<hashar> |
synchronized wmf-config/InitialiseSettings.php 'touch, restore $oaiAgentRegex' |
[production] |
| 16:14 |
<hashar> |
synchronized wmf-config/CommonSettings.php 'restore $oaiAgentRegex' |
[production] |
| 09:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 'LB 'dump' s[2-7]' |
[production] |
| 08:50 |
<manybubbles> |
finished elasticsearch upgrade. we're now 0.90.10 all the way. |
[production] |
| 08:50 |
<ori> |
Replacing StatsD on tungsten with txStatsD; see commit I19ecf608d for rationale. |
[production] |
| 08:50 |
<ori> |
restarted morebots. morebots missed Sean's 'repool db1007, depool db1041' @ 6:26 UTC. |
[production] |
|
2014-01-16
§
|
| 19:22 |
<reedy> |
synchronized database lists files: |
[production] |
| 19:20 |
<Reedy> |
Created zhwikivoyage echo tables on db1029 |
[production] |
| 19:18 |
<LeslieCarr> |
reseating pem0 on cr2-ulsfo to try and clear problem |
[production] |
| 19:13 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, testwiki, testwikidatawiki and mediawikiwiki to 1.23wmf11 |
[production] |
| 19:12 |
<^d> |
dropped all cirrusSearchLinksUpdate jobs from enwiki job queue. Job queue back to ~330k entries, far better than the ~3mil |
[production] |
| 19:06 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf10 |
[production] |
| 18:50 |
<paravoid> |
powercycling ms-be1002, down, console unresponsive |
[production] |
| 18:44 |
<manybubbles> |
starting Elasticsearch upgrade from 0.90.9 to 0.90.10 |
[production] |
| 18:01 |
<reedy> |
synchronized php-1.23wmf11/includes/specials/SpecialWatchlist.php |
[production] |
| 17:54 |
<reedy> |
synchronized docroot and w |
[production] |
| 17:45 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf9 till window |
[production] |
| 17:29 |
<reedy> |
finished scap: testwiki to 1.23wmf11 and build l10n cache (duration: 23m 24s) |
[production] |
| 17:08 |
<reedy> |
started scap: testwiki to 1.23wmf11 and build l10n cache |
[production] |
| 16:48 |
<cmjohnson1> |
es1007 replacing disk slot 6 |
[production] |
| 16:47 |
<cmjohnson1> |
tungsten replacing failing disk slot 3 |
[production] |
| 16:47 |
<cmjohnson1> |
db1004 replacing failing disk slot 9 |
[production] |
| 16:43 |
<reedy> |
synchronized php-1.23wmf11 |
[production] |
| 16:14 |
<bblack> |
rebooting cp1055 to clear out mess from XFS/kmem_alloc bug |
[production] |
| 15:48 |
<cmjohnson1> |
disabling puppet on db29 |
[production] |
| 11:55 |
<akosiaris> |
fixed the cp1065 puppet freshness check constantly bugging us and reenabled notifications after that |
[production] |
| 09:19 |
<ori> |
updated /a/common to {{Gerrit|Id13e614e5}}: repool db1042 |
[production] |
| 06:01 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1042' |
[production] |
| 05:58 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1033' |
[production] |
| 05:21 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1042' |
[production] |
| 03:29 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-16 03:28:58+00:00 |
[production] |
| 03:00 |
<Coren> |
Deleted all non-current pacct on neon; activity seems to have increased tenfold and /var/log is OOS |
[production] |
| 02:51 |
<LocalisationUpdate> |
completed (1.23wmf9) at 2014-01-16 02:51:29+00:00 |
[production] |