851-900 of 5802 results (5ms)
2009-12-10 §
00:24 <Andrew> db12 recovered, db26 still barely coping with the load [production]
00:17 <tstarling> synchronized php-1.5/wmf-config/db.php 'unloading general load from db12, excessive lag >800s' [production]
00:09 <Andrew> db12 seems not to be replicating at all. [production]
00:09 <Andrew> make that db12, not db22 [production]
00:06 <Andrew> s1 lagged, db12 282s, db26 33s -- no slave servers up-to-date enough to serve pages [production]
2009-12-09 §
23:53 <fred> synchronized php-1.5/wmf-config/CommonSettings.php 'Adding 2009_Notice_43-44-45 to tracking.' [production]
23:06 <midom> synchronized php-1.5/wmf-config/db.php 'taking down one more server oh noes' [production]
22:59 <Fred> fixed noc.wikimedia.org [production]
21:08 <midom> synchronized php-1.5/wmf-config/db.php 'db14 offline' [production]
20:54 <atglenn> open support call with sun for ms8, won't power up (no POST), even after resetting ILOM [production]
17:28 <andrew> ran sync-common-all [production]
17:18 <Andrew> Updating production LiquidThreads installations to new beta version, branch of current alpha. [production]
17:08 <andrew> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20253 Install Extension:NewUserMessage on strategy wiki' [production]
13:39 <andrew> synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merge r59886' [production]
13:38 <andrew> synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r59885' [production]
13:08 <andrew> synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r59883' [production]
13:05 <andrew> synchronized php-1.5/extensions/LiquidThreads_alpha/jquery/js2.combined.js 'merge r59882' [production]
12:50 <andrew> ran sync-common-all [production]
12:39 <Andrew> Updating LiquidThreads to trunk state, using sync-common-all to deploy to apaches (since scap crashes the cluster) [production]
00:04 <RoanKattouw> Manually reallocating edits from [[User:Until It Sleeps]] to [[User:The Thing That Should Not Be]] on enwiki [production]
2009-12-08 §
23:51 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21718: Allow sysops to add/remove abusefilter too on arwiki' [production]
23:46 <Big_Fred> synchronized php-1.5/wmf-config/CommonSettings.php 'Added tracking for Notice_42' [production]
23:31 <catrope> synchronized php-1.5/includes/DefaultSettings.php 'Deploying r59858' [production]
23:31 <catrope> synchronized php-1.5/includes/Skin.php 'Deploying r59858' [production]
23:31 <catrope> synchronized php-1.5/skins/vector/main-rtl.css 'Deploying r59858' [production]
23:31 <catrope> synchronized php-1.5/skins/vector/main-ltr.css 'Deploying r59858' [production]
23:31 <catrope> synchronized php-1.5/skins/common/ajaxwatch.js 'Deploying r59858' [production]
23:26 <TrevorParscal> Running sync-file repeatedly to deploy r59858 (usability fixes) [production]
23:16 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21718: allow bureaucrats to add&remove abusefilter group on arwiki' [production]
23:12 <catrope> synchronized php-1.5/wmf-config/abusefilter.php 'bug 21718: Restrict abusefilter-modify to abusefilter group on arwiki' [production]
22:13 <tfinc> synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'picking up r59852' [production]
19:09 <Big_Fred> synchronized php-1.5/wmf-config/CommonSettings.php 'added geo-location for Italie.' [production]
16:43 <andrew> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20216 AbuseFilter for itwikiquote' [production]
16:43 <andrew> synchronized php-1.5/wmf-config/abusefilter.php 'Bug 20216 AbuseFilter for itwikiquote' [production]
16:20 <Rob> running diagnostics on srv217, please do not try to put it into service. [production]
16:06 <Rob> srv229 memory replaced, system restarted. [production]
15:58 <Rob> shutting down srv229 to swap out faulty dimm [production]
15:46 <Rob> sq43 disk replaced, system back online and in rotation [production]
15:31 <Rob> replaced bad disk in sq43 [production]
15:23 <Rob> replaced bad disk in sq37, detected, restarted squid backend [production]
15:18 <midom> synchronized php-1.5/wmf-config/db.php [production]
15:09 <Rob> db27 back up with mysql running. [production]
15:07 <midom> synchronized php-1.5/wmf-config/db.php [production]
15:03 <Rob> mysql stopped normally on db27, restarting it. [production]
15:02 <Rob> was able to hotswap the fan, shutdown was not needed [production]
14:58 <Rob> stopping mysql on db27 to shut it down and swap out a bad fan assembly. [production]
14:56 <Rob> disregard, did not receive part for db28, just db27 [production]
14:53 <Rob> db28 powered down already, pulling to replace bad fan [production]
11:12 <rainman-sr> searchidx1 rebooted by domas, restarting indexing [production]
11:00 <rainman-sr> searchidx1 has a "cp -lr" process stuck in I/O, cannot be killed, dmesg says "aacraid: Host adapter reset request. SCSI hang ?". Stopping indexing until someone sorts this out. Hardware check, reboot? [production]