751-800 of 10000 results (17ms)
2012-12-18 §
06:29 <andrewbogott> rsynced all labs homedirs to gluster volumes [production]
02:46 <LocalisationUpdate> completed (1.21wmf5) at Tue Dec 18 02:45:58 UTC 2012 [production]
02:25 <LocalisationUpdate> completed (1.21wmf6) at Tue Dec 18 02:25:22 UTC 2012 [production]
01:28 <mutante> fixing duplicate UID issue on stat1 for maryana [production]
00:57 <LeslieCarr> asw-c-eqiad unreachable due to lacp issue [production]
00:32 <mutante> fixing fenari permissions for gwicke.. (pre-puppet age UID) [production]
00:26 <LeslieCarr> starting upgrade of asw-c-eqiad.mgmt - connectivity to row c machines may be affected [production]
00:11 <notpeter> temp stopping lsave on es1009 and es1010 for upcoming networking downtime [production]
2012-12-17 §
23:12 <LeslieCarr> restarted pybal on lvs1001-1003 in order to restart their bgp peering [production]
22:44 <notpeter> taking fenari down for upgarde to precise (not upgrading, not reimaging) [production]
22:37 <LeslieCarr> cr1-eqiad being upgraded and rebooted [production]
22:30 <Ryan_Lane> labstore1 is locked up, powercycling [production]
22:29 <Nemo_bis> en.wiki job queue spiked from 1 to 3 millions in last 3 hours [production]
21:22 <Ryan_Lane> rebooting labstore4 [production]
21:15 <reedy> synchronized php-1.21wmf6/includes/filebackend/FSFileBackend.php [production]
21:11 <Ryan_Lane> rebooting labstore3 [production]
21:03 <Ryan_Lane> rebooting labstore2 [production]
20:48 <Ryan_Lane> restarting labstore1 [production]
20:33 <cmjohnson1> auth-dns update to add internal ip's for solr1-3 [production]
19:35 <hashar> regenerating Jenkins job mediawiki-core-install-sqlite [production]
19:12 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf6 [production]
18:16 <andrewbogott> beginning labs $HOME migration from NFS to gluster [production]
16:17 <cmjohnson1> auth-dns update adding mgmt for solr1-3, solr1001-3 and internal ip's for solr1001-3 [production]
13:13 <hashar> set Jenkins to use /bin/bash as a default shell (intend of /bin/sh) [production]
04:02 <paravoid> killall -9 convert on imagescalers; uploading 120px generated thumbnail directly to swift [production]
02:50 <LocalisationUpdate> completed (1.21wmf5) at Mon Dec 17 02:50:11 UTC 2012 [production]
02:27 <LocalisationUpdate> completed (1.21wmf6) at Mon Dec 17 02:27:34 UTC 2012 [production]
2012-12-16 §
17:14 <paravoid> "killall -9 convert" on all imagescalers [production]
15:55 <reedy> synchronized wmf-config/InitialiseSettings.php [production]
15:54 <reedy> synchronized images/sul/wikimania.png [production]
15:36 <reedy> synchronized images/sul/ [production]
02:47 <LocalisationUpdate> completed (1.21wmf5) at Sun Dec 16 02:47:42 UTC 2012 [production]
02:45 <LeslieCarr> mwscript.php appears to be broken [production]
02:40 <LeslieCarr> checking memcache servers due to spate of mediawiki servers flapping [production]
02:26 <LocalisationUpdate> completed (1.21wmf6) at Sun Dec 16 02:26:23 UTC 2012 [production]
2012-12-15 §
14:37 <maxsem> synchronized /php-1.21wmf5/includes/Message.php 'https://gerrit.wikimedia.org/r/#/c/38822/' [production]
14:30 <paravoid> powercycling ms-be1 [production]
12:18 <apergos> ms-be1 can't be reached form mgmt console to check on it (Connection refused) [production]
02:47 <LocalisationUpdate> completed (1.21wmf5) at Sat Dec 15 02:47:17 UTC 2012 [production]
02:45 <LeslieCarr> drained all transit traffic from cr1-eqiad -- restore by load override /var/home/lcarr/cr1-eqiad.undrained if traffic levels seem too high [production]
02:25 <LocalisationUpdate> completed (1.21wmf6) at Sat Dec 15 02:25:26 UTC 2012 [production]
00:36 <LeslieCarr> appears that cr1-eqiad's fpc rebooted [production]
2012-12-14 §
23:25 <AaronSchulz> Compiled texvc into uncommon/php-1.21wmf6 dir for all apaches [production]
23:08 <K4-713> synchronized payments to 4f5296e43b1c708 [production]
22:34 <mutante> fixing symlink on kaulen to reflect new bugzilla version 4.2.4 [production]
22:30 <hashar> restarting Zuul on gallium [production]
20:54 <mutante> deleting /var/lib/git/integration on gallium [production]
18:54 <mutante> appyling mini patch on bugzilla per http://bzr.mozilla.org/bugzilla/4.2/revision/8176 [production]
18:31 <reedy> synchronized wmf-config/InitialiseSettings.php 'Enable clicktracking on test2wiki' [production]
16:56 <sbernardin> sq48 going down for HDD controller card replacement-rt4041 [production]