851-900 of 10000 results (17ms)
2012-05-29 §
16:21 <hashar> synchronized wmf-config/CommonSettings.php 'cleanup wgNoticeBanner_Harvard2011 https://gerrit.wikimedia.org/r/#/c/9205/' [production]
16:17 <hashar> /usr/local/apache/common-local is 4G where as / is 7G on srv187. Looks like deploying wmf2 + wmf3 + wmf4 will require partitions to be resized. [production]
16:10 <hashar> srv187 and srv188 are out of disk space [production]
16:10 <hashar> synchronized search-redirect.php 'https://gerrit.wikimedia.org/r/9206 - cleanup search-redirect.php' [production]
14:37 <cmjohnson1> removing disk4 on virt1 for replacement [production]
12:18 <mutante> killing / restarting morebots [production]
02:37 <LocalisationUpdate> completed (1.20wmf4) at Tue May 29 02:37:44 UTC 2012 [production]
02:24 <LocalisationUpdate> completed (1.20wmf3) at Tue May 29 02:24:21 UTC 2012 [production]
2012-05-28 §
23:15 <reedy> synchronized php-1.20wmf4/extensions/UploadWizard/ 'Master UW per Kaldari' [production]
18:25 <reedy> synchronized php-1.20wmf4/extensions/Translate/tag/PageTranslationHooks.php 'Fix Catchable fatal error' [production]
18:08 <reedy> synchronized php-1.20wmf4/LocalSettings.php [production]
18:05 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf4 [production]
16:13 <reedy> synchronizing Wikimedia installation... : Rebuild l10n for php-1.20wmf4 [production]
15:44 <reedy> synchronizing Wikimedia installation... : test2wiki to 1.20wmf4 to build localisation cache [production]
14:39 <reedy> synchronizing Wikimedia installation... : Does running scap on its own with no wikis on that version build l10n for it? I suspect not... [production]
14:22 <reedy> synchronized php-1.20wmf4/ 'Staging php-1.20wmf4' [production]
14:15 <Reedy> sync-dir'ing php-1.20wmf4 [production]
14:12 <Reedy> Copying php-1.20wmf4 from /tmp to /h/w/c on Fenari [production]
13:12 <apergos> doing security updates for a batch of mws in eqiad [production]
11:22 <apergos> updated kernel etc on mw1133, reboot [production]
11:08 <apergos> powercycled mw1133 [production]
09:05 <apergos> rebooted snapshot4, 3 for security updates [production]
08:43 <apergos> rebooted snapshot1002, security updates (will do the same for 1003, 1004 shortly) [production]
08:37 <apergos> rebooted snapshot1001, security updates [production]
02:24 <LocalisationUpdate> completed (1.20wmf3) at Mon May 28 02:24:05 UTC 2012 [production]
2012-05-27 §
22:23 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 37134 - s:cs: site settings' [production]
14:45 <asher> synchronized wmf-config/db.php 'returning db51 to prod as an s4 slave' [production]
14:34 <maplebed> dns and puppet changes for s4 master rotation done. [production]
14:18 <binasher> rebooted db51, reslaved [production]
14:09 <maplebed> new s4 master position post-rotation is master_log_file="db31-bin.000334", master_log_pos=583315125 [production]
14:02 <ben> synchronized wmf-config/db.php 's4 master switch complete; db31 is the new master. turning off read only on s4' [production]
13:25 <Tim> on db31 set global read_only=1 [production]
13:17 <tstarling> synchronized wmf-config/db.php 's4 read-only and taking out db51' [production]
04:27 <binasher> kaulen - temporarily disabled swap and set oom_adj score to 15 for apache [production]
02:24 <LocalisationUpdate> completed (1.20wmf3) at Sun May 27 02:24:13 UTC 2012 [production]
2012-05-26 §
19:55 <apergos> restarting apache2 on kaulen [production]
19:34 <Nemo_bis> bugzilla down again [production]
14:19 <reedy> synchronized wmf-config/CommonSettings.php 'Settting wgBlacklistSettings' [production]
11:05 <apergos> stopping and restarting apache on kaulen blah blah blah [production]
09:12 <Reedy> Bugzilla is down, Kaulen looks to be in swap death again [production]
06:46 <apergos> powercycling kaulen [production]
05:53 <hashar> kaulen dead :-[ [production]
05:47 <hashar> Bugzilla on Kaulen being super slow again [production]
02:22 <LocalisationUpdate> completed (1.20wmf3) at Sat May 26 02:22:37 UTC 2012 [production]
00:40 <K4-713> synchronized payments cluster to DonationInterface 04809f1cf0d [production]
2012-05-25 §
22:57 <maplebed> powercycled kaulen on the mgmt interface [production]
19:09 <maplebed> disabled the outdated /etc/init.d/gmond on spence. use ganglia-monitor instead. [production]
18:57 <RobH> bugzilla appears back online [production]
18:57 <RobH> kaulen is rebooted, it may have had a runaway process or a memory leak, not sure yet, but it was locked up from access [production]
18:55 <RobH> kaulen serial console unresponsive, rebooting [production]