1851-1900 of 10000 results (15ms)
2013-11-04 §
04:15 <LocalisationUpdate> failed: git pull of extensions failed [production]
03:12 <Tim> moved udpmcast to chromium since it actually has an external IP address [production]
03:05 <Tim> moved udpmcast.py from dobson to tungsten to work around failure of multicast routing eqiad -> pmtpa [production]
02:01 <LocalisationUpdate> failed: git pull of extensions failed [production]
2013-11-03 §
12:50 <mark> Activated BFD between cr2-eqiad and cr1-sdtpa link [production]
10:07 <mark> Send traffic from Japan and Korea to ulsfo, https://gerrit.wikimedia.org/r/#/c/93309/ [production]
07:26 <ori-l> kaulen's apache was down (killed by oom killer) & refused to start. copied over old config per #5011, restarted apache, & disabled puppet for now. [production]
07:07 <p858snake|l> Bugzilla isn't loading [production]
02:02 <LocalisationUpdate> failed: git pull of extensions failed [production]
2013-11-02 §
19:50 <cmjohnson1> dns update [production]
16:14 <reedy> synchronized wmf-config/CommonSettings.php [production]
12:43 <springle> start xtrabackup db52->db69 and db39->db71 [production]
09:56 <apergos> restarted twemproxy on ~10 of the api servers in eqiad in the lower memory/core group, was seeing a lot of entries in memcached-serious log and some processes churning on that [production]
07:37 <apergos> restarted a bunch of the apaches on the 12gb memory apis that were maxed out at their 100 client limit [production]
07:09 <apergos> and increased again to 30, after looking at memory on both groups of boxes. [production]
06:58 <apergos> increase weight of mw1189-1208 api servers from 20 to 25, they handle the load better (hopefully) [production]
06:03 <springle> synchronized wmf-config/db-pmtpa.php 'depool db69 and db71 for reassignment during pmtpa decomm' [production]
02:00 <LocalisationUpdate> failed: git pull of extensions failed [production]
00:39 <awight> updated crm from da3448df1404311620adc50b4d4ca28abfd2f3cb to 272966e353244e668a64c54ca4fa6f344079f4bf [production]
2013-11-01 §
23:55 <awight> updated crm from 4a4c31cf60b783a5269c3e18b4baaf3bde5568ab to da3448df1404311620adc50b4d4ca28abfd2f3cb [production]
23:45 <ori> synchronized wmf-config/InitialiseSettings.php 'Ib19944b16: = true on test & test2' [production]
22:57 <awight> crm updated from 28093721d29226e9bd17d8be43db8a0deb51fe22 to 4a4c31cf60b783a5269c3e18b4baaf3bde5568ab [production]
22:45 <awight> crm updated from 21cb97b5083f4a32e67b5b6dac4b58d733b8d7ca to 28093721d29226e9bd17d8be43db8a0deb51fe22 [production]
22:23 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Nov 1 22:23:20 UTC 2013 [production]
22:16 <LocalisationUpdate> completed (1.23wmf2) at Fri Nov 1 22:16:51 UTC 2013 [production]
21:57 <LocalisationUpdate> completed (1.23wmf1) at Fri Nov 1 21:57:34 UTC 2013 [production]
21:50 <reedy> Finished syncing Wikimedia installation... : Rebuild l10n cache for MultimediaViewer update [production]
21:39 <reedy> Started syncing Wikimedia installation... : Rebuild l10n cache for MultimediaViewer update [production]
21:25 <reedy> synchronized wmf-config 'touch' [production]
21:24 <reedy> synchronized php-1.23wmf2/ [production]
21:22 <awight> updated crm from dca2c8cc0eaa503e6d0c9921952cb85855e3b110 to 21cb97b5083f4a32e67b5b6dac4b58d733b8d7ca [production]
19:59 <reedy> synchronized php-1.23wmf1/extensions/Wikibase [production]
19:49 <hashar> jenkins: refreshing all *pep8 jobs, pep8 wrapper for puppet pointing to a wrong path [production]
19:48 <Reedy> Added rc_source column to testwikidatawiki.recentchanges [production]
19:47 <Reedy> Added rc_source column to mediawikiwiki.recentchanges [production]
19:46 <Reedy> Added rc_source column to test2wiki.recentchanges [production]
19:43 <Reedy> Added rc_source column to testwiki.recentchanges [production]
19:12 <Reedy> Created betafeatures_user_counts on testwiki and test2wiki [production]
19:06 <reedy> synchronized wmf-config/ 'Enable magic beta extensions' [production]
18:58 <reedy> Finished syncing Wikimedia installation... : scap take 2 to build l10ncache with magical beta things [production]
18:49 <reedy> Started syncing Wikimedia installation... : scap take 2 to build l10ncache with magical beta things [production]
18:31 <reedy> Finished syncing Wikimedia installation... : testwiki and test2wiki to 1.23wmf2 [production]
18:17 <reedy> Started syncing Wikimedia installation... : testwiki and test2wiki to 1.23wmf2 [production]
18:14 <paravoid> reenabling mw1114-mw1148 [production]
18:12 <reedy> synchronized wmf-config/ [production]
18:08 <paravoid> swapoff on all api appservers [production]
17:53 <reedy> synchronized wmf-config/ [production]
17:26 <paravoid> depooling mw1114-mw1148; balancing is unfair, boxes overloaded, mw1189-mw1208 capable of handling the load [production]
16:01 <apergos> shot one more forceSearchIndex on arsenic because we were back in swap, only one left... [production]
13:51 <hasharEat> gallium : Jenkins console log compression completed, saved 80G out of 500G total disk space. [production]