|
2014-03-07
§
|
| 01:18 |
<mholmquist> |
Finished scap: (no message) (duration: 10m 55s) |
[production] |
| 01:07 |
<rdwrer> |
That scap was for ULS, VE, and MobileFrontend fixes and updates. |
[production] |
| 01:07 |
<mholmquist> |
Started scap: (no message) |
[production] |
| 01:03 |
<manybubbles> |
[Elasticsearch upgrade] Reenabling puppet |
[production] |
| 01:03 |
<hashar> |
Jenkins back up |
[production] |
| 01:03 |
<hashar> |
Jenkins backup |
[production] |
| 01:03 |
<manybubbles> |
[Elasticsearch upgrade] All primary shards have started. Waiting on secondary. |
[production] |
| 01:01 |
<manybubbles> |
[Elasticsearch upgrade] Wait for the cluster to recover. |
[production] |
| 01:00 |
<hashar> |
killed wrong jenkins process (+1 for 2am fix up). Restarting jenkins |
[production] |
| 00:59 |
<manybubbles> |
[Elasticsearch upgrade] Verifying versions |
[production] |
| 00:59 |
<manybubbles> |
[Elasticsearch upgrade] Starting Elasticsearch |
[production] |
| 00:58 |
<hashar> |
Killing a duplicate Jenkins java process on gallium (init.d script sucks, I really need to get it fixed one day) |
[production] |
| 00:52 |
<mholmquist> |
updated /a/common to {{Gerrit|Iad8c84a7d}}: Don't use += with $wgJobTypesExcludedFromDefaultQueue |
[production] |
| 00:52 |
<manybubbles> |
[Elasticsearch upgrade] Upgrading Elasticsearch |
[production] |
| 00:49 |
<manybubbles> |
[Elasticsearch upgrade] Shutting down Elasticsearch |
[production] |
| 00:48 |
<manybubbles> |
[Elasticsearch upgrade] Turning off shard reallocation so we don't thrash while Elasticsearch shuts down |
[production] |
| 00:47 |
<manybubbles> |
[Elasticsearch upgrade] Disabling puppet so it doesn't restart Elasticsearch while we're upgrading it |
[production] |
| 00:44 |
<manybubbles> |
[Elasticsearch upgrade] Elasticsearch is now quiescent |
[production] |
| 00:43 |
<mutante> |
gracefull'ing rogue apaches, mw1131,mw1189,mw1190,mw1215 |
[production] |
| 00:41 |
<mutante> |
gracefull'ing rogue apaches, mw1070,mw1089,mw1104,mw1111 |
[production] |
| 00:37 |
<demon> |
synchronized wmf-config/jobqueue-eqiad.php 'Fixing $wgJobTypesExcludedFromDefaultQueue config' |
[production] |
| 00:34 |
<manybubbles> |
[Elasticsearch upgrade] Running puppet everywhere to make sure we have the newest config |
[production] |
| 00:33 |
<mutante> |
graceful mw1040 |
[production] |
| 00:03 |
<manybubbles> |
synchronized wmf-config/InitialiseSettings.php 'Turn Cirrus of for the duration of the upgrade' |
[production] |
| 00:02 |
<manybubbles> |
synchronized wmf-config/jobqueue-eqiad.php 'Pausing Cirrus jobs for the duration of the upgrade.' |
[production] |
| 00:02 |
<manybubbles> |
Starting Elasticsearch upgrade |
[production] |
|
2014-03-06
§
|
| 22:21 |
<^d> |
restarting jenkins on gallium. It's totally hung and nothing's getting done. Jobs will probably need retriggering. |
[production] |
| 21:23 |
<bd808> |
"No space left on device" errors from snapshot1004.eqiad.wmnet during scap |
[production] |
| 21:21 |
<bd808> |
Finished scap: php-1.23wmf17 l10n cache rebuild (duration: 11m 20s) |
[production] |
| 21:13 |
<bblack> |
all imagemagick-related packages updated to latest on image/video scalers |
[production] |
| 21:10 |
<bd808> |
Started scap: php-1.23wmf17 l10n cache rebuild |
[production] |
| 20:31 |
<hoo> |
Syncing to snapshot[1-4] failed |
[production] |
| 20:28 |
<hoo> |
synchronized php-1.23wmf17/extensions/Wikidata/ 'Update Wikidata to fix an uncaught exception in claim html formatting' |
[production] |
| 20:23 |
<bd808> |
Restored pre Ic56177a versions of wmf-config/*pmtpa* config files to srv270, mw31 and mw40 |
[production] |
| 20:03 |
<bd808> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf17 |
[production] |
| 19:48 |
<bd808|deploy> |
on mw31, mw40 and srv270 |
[production] |
| 19:48 |
<bd808|deploy> |
Touched wmf-config/PoolCounterSettings-pmtpa.php and jobqueue-pmtpa.php as more follow up to I332c0e8 |
[production] |
| 19:44 |
<bd808> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf16 |
[production] |
| 19:36 |
<bd808> |
synchronized wmf-config/abusefilter.php 'Simplify the AbuseFilter configuration Ia472af8' |
[production] |
| 19:32 |
<cmjohnson1> |
ms-be1005 down for update |
[production] |
| 19:11 |
<mutante> |
touch /usr/local/apache/common-local/wmf-config/db-ptmpa.php on srv270,mw40,mw31 to fix Apache errors, details in change 117244 |
[production] |
| 18:57 |
<akosiaris> |
lvresize -L +40G terbium/root, resize2fs /dev/mapper/terbium-root. See RT #6984 |
[production] |
| 18:22 |
<bd808> |
Finished scap: testwiki to php-1.23wmf17 and rebuild l10n cache (try #2) (duration: 15m 22s) |
[production] |
| 18:09 |
<mutante> |
shut down sockpuppet - bye for good |
[production] |
| 18:06 |
<bd808> |
Started scap: testwiki to php-1.23wmf17 and rebuild l10n cache (try #2) |
[production] |
| 17:57 |
<bd808> |
scap aborted: testwiki to php-1.23wmf17 and rebuild l10n cache (duration: 00m 41s) |
[production] |
| 17:56 |
<bd808> |
Started scap: testwiki to php-1.23wmf17 and rebuild l10n cache |
[production] |
| 17:20 |
<mutante> |
sockpuppet - disabling puppet,disabling monitoring,remove from stored configs,revoke puppet cert,delete salt key |
[production] |
| 16:18 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 16:17 |
<reedy> |
synchronized docroot and w |
[production] |