|
2013-11-02
§
|
| 19:50 |
<cmjohnson1> |
dns update |
[production] |
| 16:14 |
<reedy> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 12:43 |
<springle> |
start xtrabackup db52->db69 and db39->db71 |
[production] |
| 09:56 |
<apergos> |
restarted twemproxy on ~10 of the api servers in eqiad in the lower memory/core group, was seeing a lot of entries in memcached-serious log and some processes churning on that |
[production] |
| 07:37 |
<apergos> |
restarted a bunch of the apaches on the 12gb memory apis that were maxed out at their 100 client limit |
[production] |
| 07:09 |
<apergos> |
and increased again to 30, after looking at memory on both groups of boxes. |
[production] |
| 06:58 |
<apergos> |
increase weight of mw1189-1208 api servers from 20 to 25, they handle the load better (hopefully) |
[production] |
| 06:03 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'depool db69 and db71 for reassignment during pmtpa decomm' |
[production] |
| 02:00 |
<LocalisationUpdate> |
failed: git pull of extensions failed |
[production] |
| 00:39 |
<awight> |
updated crm from da3448df1404311620adc50b4d4ca28abfd2f3cb to 272966e353244e668a64c54ca4fa6f344079f4bf |
[production] |
|
2013-11-01
§
|
| 23:55 |
<awight> |
updated crm from 4a4c31cf60b783a5269c3e18b4baaf3bde5568ab to da3448df1404311620adc50b4d4ca28abfd2f3cb |
[production] |
| 23:45 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'Ib19944b16: = true on test & test2' |
[production] |
| 22:57 |
<awight> |
crm updated from 28093721d29226e9bd17d8be43db8a0deb51fe22 to 4a4c31cf60b783a5269c3e18b4baaf3bde5568ab |
[production] |
| 22:45 |
<awight> |
crm updated from 21cb97b5083f4a32e67b5b6dac4b58d733b8d7ca to 28093721d29226e9bd17d8be43db8a0deb51fe22 |
[production] |
| 22:23 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Nov 1 22:23:20 UTC 2013 |
[production] |
| 22:16 |
<LocalisationUpdate> |
completed (1.23wmf2) at Fri Nov 1 22:16:51 UTC 2013 |
[production] |
| 21:57 |
<LocalisationUpdate> |
completed (1.23wmf1) at Fri Nov 1 21:57:34 UTC 2013 |
[production] |
| 21:50 |
<reedy> |
Finished syncing Wikimedia installation... : Rebuild l10n cache for MultimediaViewer update |
[production] |
| 21:39 |
<reedy> |
Started syncing Wikimedia installation... : Rebuild l10n cache for MultimediaViewer update |
[production] |
| 21:25 |
<reedy> |
synchronized wmf-config 'touch' |
[production] |
| 21:24 |
<reedy> |
synchronized php-1.23wmf2/ |
[production] |
| 21:22 |
<awight> |
updated crm from dca2c8cc0eaa503e6d0c9921952cb85855e3b110 to 21cb97b5083f4a32e67b5b6dac4b58d733b8d7ca |
[production] |
| 19:59 |
<reedy> |
synchronized php-1.23wmf1/extensions/Wikibase |
[production] |
| 19:49 |
<hashar> |
jenkins: refreshing all *pep8 jobs, pep8 wrapper for puppet pointing to a wrong path |
[production] |
| 19:48 |
<Reedy> |
Added rc_source column to testwikidatawiki.recentchanges |
[production] |
| 19:47 |
<Reedy> |
Added rc_source column to mediawikiwiki.recentchanges |
[production] |
| 19:46 |
<Reedy> |
Added rc_source column to test2wiki.recentchanges |
[production] |
| 19:43 |
<Reedy> |
Added rc_source column to testwiki.recentchanges |
[production] |
| 19:12 |
<Reedy> |
Created betafeatures_user_counts on testwiki and test2wiki |
[production] |
| 19:06 |
<reedy> |
synchronized wmf-config/ 'Enable magic beta extensions' |
[production] |
| 18:58 |
<reedy> |
Finished syncing Wikimedia installation... : scap take 2 to build l10ncache with magical beta things |
[production] |
| 18:49 |
<reedy> |
Started syncing Wikimedia installation... : scap take 2 to build l10ncache with magical beta things |
[production] |
| 18:31 |
<reedy> |
Finished syncing Wikimedia installation... : testwiki and test2wiki to 1.23wmf2 |
[production] |
| 18:17 |
<reedy> |
Started syncing Wikimedia installation... : testwiki and test2wiki to 1.23wmf2 |
[production] |
| 18:14 |
<paravoid> |
reenabling mw1114-mw1148 |
[production] |
| 18:12 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 18:08 |
<paravoid> |
swapoff on all api appservers |
[production] |
| 17:53 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 17:26 |
<paravoid> |
depooling mw1114-mw1148; balancing is unfair, boxes overloaded, mw1189-mw1208 capable of handling the load |
[production] |
| 16:01 |
<apergos> |
shot one more forceSearchIndex on arsenic because we were back in swap, only one left... |
[production] |
| 13:51 |
<hasharEat> |
gallium : Jenkins console log compression completed, saved 80G out of 500G total disk space. |
[production] |
| 10:57 |
<apergos> |
shot one more forceSearchIndex on arsenic, only two left... (using about 12gb between them) |
[production] |
| 10:51 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'depool first batch of pmtpa boxes to be decommissioned/shipped' |
[production] |
| 09:08 |
<hashar> |
Jenkins: fixing up a race condition in MobileFrontend qunit tests. Both variant were using the same path and the first job to complete would break the other one by deleting the mediawiki install. {{gerrit|93030}} |
[production] |
| 08:18 |
<apergos> |
shot two of the forceSearchIndex on arsenic, they were using 7gb between them and arsenic was in swapdeath |
[production] |