751-800 of 10000 results (11ms)
2012-05-04 §
18:02 <Ryan_Lane> gerrit upgrade is done [production]
17:55 <Ryan_Lane> starting gerrit [production]
17:32 <Ryan_Lane> installing gerrit package on manganese [production]
17:28 <Ryan_Lane> adding gerrit 2.3 package to the repo [production]
17:25 <Ryan_Lane> shutting down gerrit so that everything can be backed up [production]
16:45 <apergos> lighty on dataset2 is running under gdb in screen session as root, if it dies please leave that alone (or look at it if you want to investigate) [production]
16:26 <notpeter> turning off db30 (former s2 db, still on hardy, will ask asher what to do with it) to test noise in DC [production]
15:50 <mutante> rebooting sq67 (bits) [production]
15:42 <mutante> going through sq7x servers (text), full upgrades [production]
15:32 <notpeter> removing srv281 from rending pool until we figure out what's going on with it [production]
15:23 <notpeter> putting srv224 back into pybal pool [production]
15:09 <notpeter> removing srv224 from pybal pool for repartitioning [production]
14:56 <notpeter> putting srv223 back into pybal pool [production]
14:50 <mutante> going through sq6x (text), full upgrades [production]
14:08 <notpeter> removing srv223 from pybal pool for repartitioning [production]
14:02 <notpeter> putting srv222 back into pybal pool [production]
13:50 <notpeter> removing srv222 from pybal pool for repartitioning [production]
13:43 <notpeter> putting srv221 back into pybal pool [production]
13:30 <notpeter> removing srv221 from pybal pool for repartitioning [production]
13:16 <mutante> going through sq80 to sq86 (upload), full upgrade & reboot [production]
12:56 <mutante> maximum uptime in the sq* group down to 171 days, so we have like a month now for the rest. stopping upgrades for the moment being. [production]
12:54 <notpeter> starting script to move /usr/local/apache to /a partition on all remaing non-imagescaler apaches [production]
12:47 <mutante> (just) new kernels & reboot - sq45,sq49 (upload) [production]
12:30 <mark> Sending ALL non-european upload traffic to eqiad [production]
12:23 <mutante> (just) new kernels & reboot - sq63 to sq66 (209 days up) [production]
12:06 <mutante> dist-upgrade & kernel & reboot - sq42,sq43 - rebooting upload squids one by one [production]
11:48 <mutante> powercycling srv266 one more time, but now creating RT for it, once already showed CPU issue before it was reinstalled recently [production]
11:13 <apergos> restarted lighty on dataset2 ... about ... half an hour ago. stupid case sensitivity [production]
10:02 <apergos> tossed knsq1 through 7 from squid_knams dsh nodegroups file, prolly lots more cleanup where that came from [production]
09:34 <mutante> dist-upgrade/kernel/reboot: sq37, sq41. rebooting upload squid sq41 [production]
08:49 <mutante> dist-upgrade & new kernel & reboot: sq33, sq36 [production]
07:47 <mutante> preemptive rebooting of sq* servers identified as having > 200 days of uptime [production]
02:22 <LocalisationUpdate> completed (1.20wmf2) at Fri May 4 02:22:42 UTC 2012 [production]
02:14 <LocalisationUpdate> completed (1.20wmf1) at Fri May 4 02:13:58 UTC 2012 [production]
00:20 <raindrift> synchronizing Wikimedia installation... : [production]
00:18 <raindrift> synchronizing Wikimedia installation... : Syncing the PageTriage extension, but only enabling on testwiki [production]
00:08 <awjrichards> synchronized wmf-config/CommonSettings.php 'Adding 'fr' to language codes for mobile feedback' [production]
00:06 <maplebed> moved ms1-3 from the production cluster to the test cluster [production]
2012-05-03 §
23:29 <LeslieCarr> restarting networking on sq55 [production]
23:29 <tstarling> synchronizing Wikimedia installation... : [production]
23:27 <LeslieCarr> restarting networking on sq54 [production]
23:24 <LeslieCarr> restarting networking on sq53 [production]
23:21 <LeslieCarr> restarting networking on sq52 [production]
23:16 <LeslieCarr> restarting networking on sq51 [production]
21:30 <notpeter> removing srv220 from pybal pool for repartitioning [production]
21:29 <LeslieCarr> switching asw-a4-sdtpa from single uplink to lag [production]
21:19 <notpeter> putting srv219 back into pybal pool [production]
21:14 <asher> synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23' [production]
21:09 <asher> synchronized wmf-config/db.php 'reverting cluster23 change' [production]
21:05 <asher> synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23' [production]