5251-5300 of 10000 results (25ms)
2012-05-28 §
11:22 <apergos> updated kernel etc on mw1133, reboot [production]
11:08 <apergos> powercycled mw1133 [production]
09:05 <apergos> rebooted snapshot4, 3 for security updates [production]
08:43 <apergos> rebooted snapshot1002, security updates (will do the same for 1003, 1004 shortly) [production]
08:37 <apergos> rebooted snapshot1001, security updates [production]
02:24 <LocalisationUpdate> completed (1.20wmf3) at Mon May 28 02:24:05 UTC 2012 [production]
2012-05-27 §
22:23 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 37134 - s:cs: site settings' [production]
14:45 <asher> synchronized wmf-config/db.php 'returning db51 to prod as an s4 slave' [production]
14:34 <maplebed> dns and puppet changes for s4 master rotation done. [production]
14:18 <binasher> rebooted db51, reslaved [production]
14:09 <maplebed> new s4 master position post-rotation is master_log_file="db31-bin.000334", master_log_pos=583315125 [production]
14:02 <ben> synchronized wmf-config/db.php 's4 master switch complete; db31 is the new master. turning off read only on s4' [production]
13:25 <Tim> on db31 set global read_only=1 [production]
13:17 <tstarling> synchronized wmf-config/db.php 's4 read-only and taking out db51' [production]
04:27 <binasher> kaulen - temporarily disabled swap and set oom_adj score to 15 for apache [production]
02:24 <LocalisationUpdate> completed (1.20wmf3) at Sun May 27 02:24:13 UTC 2012 [production]
2012-05-26 §
19:55 <apergos> restarting apache2 on kaulen [production]
19:34 <Nemo_bis> bugzilla down again [production]
14:19 <reedy> synchronized wmf-config/CommonSettings.php 'Settting wgBlacklistSettings' [production]
11:05 <apergos> stopping and restarting apache on kaulen blah blah blah [production]
09:12 <Reedy> Bugzilla is down, Kaulen looks to be in swap death again [production]
06:46 <apergos> powercycling kaulen [production]
05:53 <hashar> kaulen dead :-[ [production]
05:47 <hashar> Bugzilla on Kaulen being super slow again [production]
02:22 <LocalisationUpdate> completed (1.20wmf3) at Sat May 26 02:22:37 UTC 2012 [production]
00:40 <K4-713> synchronized payments cluster to DonationInterface 04809f1cf0d [production]
2012-05-25 §
22:57 <maplebed> powercycled kaulen on the mgmt interface [production]
19:09 <maplebed> disabled the outdated /etc/init.d/gmond on spence. use ganglia-monitor instead. [production]
18:57 <RobH> bugzilla appears back online [production]
18:57 <RobH> kaulen is rebooted, it may have had a runaway process or a memory leak, not sure yet, but it was locked up from access [production]
18:55 <RobH> kaulen serial console unresponsive, rebooting [production]
17:42 <paravoid> rebooting gurvin & yvon with new kernel [production]
17:25 <paravoid> resetting gurvin, load spiking at 370+, SSH unreachable, 214 days of uptime [production]
15:34 <mark> Power cycled kaulen [production]
15:23 <hashar> kaulen (bugzilla) unreacheable :-( [production]
13:56 <RobH> palladium disk replaced [production]
13:52 <cmjohnson1> replacing ps2 on mw1017 [production]
13:50 <RobH> palladium has a bad disk, goign to replace it [production]
13:37 <RobH> updating drac on search18, shouldnt cause system reboot. [production]
10:42 <apergos> restarted apache on kaulen, was seeing page.cgi segfaults in dmesg and he logs, huge cpu wait spikes (why?) [production]
2012-05-24 §
23:34 <awjrichards> synchronized wmf-config/CommonSettings.php [production]
23:32 <awjrichards> synchronized wmf-config/CommonSettings.php 'Updating technical feedback email address for mobile feedback' [production]
23:17 <awjrichards> synchronizing Wikimedia installation... : Picking up changes to hide feedback form to prevent spamming of mobile feedback page - f6ed8ba [production]
23:12 <awjrichards> synchronized wmf-config/CommonSettings.php 'Enabling 'technical feedback' link on mobile feedback form to disable feedback form' [production]
23:00 <binasher> stopped replication on es1002 in order to rsync cluster23 to es1003 [production]
21:53 <aaron> synchronized php-1.20wmf3/extensions/UploadWizard 'deployed 144b58854e38d910210ccd23402225e5b1d2d62d' [production]
21:52 <RoanKattouw> Restarted morebots [production]
2012-05-23 §
21:46 <binasher> shutting down mysql on db12 in able to restart with binlogging disabled [production]
21:45 <asher> synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db60' [production]
21:34 <ottomata1> upgraded udp-filter to 0.2.4 on oxygen, emery, and locke (with maplebed's help) [production]