|
2012-05-04
§
|
| 17:55 |
<Ryan_Lane> |
starting gerrit |
[production] |
| 17:32 |
<Ryan_Lane> |
installing gerrit package on manganese |
[production] |
| 17:28 |
<Ryan_Lane> |
adding gerrit 2.3 package to the repo |
[production] |
| 17:25 |
<Ryan_Lane> |
shutting down gerrit so that everything can be backed up |
[production] |
| 16:45 |
<apergos> |
lighty on dataset2 is running under gdb in screen session as root, if it dies please leave that alone (or look at it if you want to investigate) |
[production] |
| 16:26 |
<notpeter> |
turning off db30 (former s2 db, still on hardy, will ask asher what to do with it) to test noise in DC |
[production] |
| 15:50 |
<mutante> |
rebooting sq67 (bits) |
[production] |
| 15:42 |
<mutante> |
going through sq7x servers (text), full upgrades |
[production] |
| 15:32 |
<notpeter> |
removing srv281 from rending pool until we figure out what's going on with it |
[production] |
| 15:23 |
<notpeter> |
putting srv224 back into pybal pool |
[production] |
| 15:09 |
<notpeter> |
removing srv224 from pybal pool for repartitioning |
[production] |
| 14:56 |
<notpeter> |
putting srv223 back into pybal pool |
[production] |
| 14:50 |
<mutante> |
going through sq6x (text), full upgrades |
[production] |
| 14:08 |
<notpeter> |
removing srv223 from pybal pool for repartitioning |
[production] |
| 14:02 |
<notpeter> |
putting srv222 back into pybal pool |
[production] |
| 13:50 |
<notpeter> |
removing srv222 from pybal pool for repartitioning |
[production] |
| 13:43 |
<notpeter> |
putting srv221 back into pybal pool |
[production] |
| 13:30 |
<notpeter> |
removing srv221 from pybal pool for repartitioning |
[production] |
| 13:16 |
<mutante> |
going through sq80 to sq86 (upload), full upgrade & reboot |
[production] |
| 12:56 |
<mutante> |
maximum uptime in the sq* group down to 171 days, so we have like a month now for the rest. stopping upgrades for the moment being. |
[production] |
| 12:54 |
<notpeter> |
starting script to move /usr/local/apache to /a partition on all remaing non-imagescaler apaches |
[production] |
| 12:47 |
<mutante> |
(just) new kernels & reboot - sq45,sq49 (upload) |
[production] |
| 12:30 |
<mark> |
Sending ALL non-european upload traffic to eqiad |
[production] |
| 12:23 |
<mutante> |
(just) new kernels & reboot - sq63 to sq66 (209 days up) |
[production] |
| 12:06 |
<mutante> |
dist-upgrade & kernel & reboot - sq42,sq43 - rebooting upload squids one by one |
[production] |
| 11:48 |
<mutante> |
powercycling srv266 one more time, but now creating RT for it, once already showed CPU issue before it was reinstalled recently |
[production] |
| 11:13 |
<apergos> |
restarted lighty on dataset2 ... about ... half an hour ago. stupid case sensitivity |
[production] |
| 10:02 |
<apergos> |
tossed knsq1 through 7 from squid_knams dsh nodegroups file, prolly lots more cleanup where that came from |
[production] |
| 09:34 |
<mutante> |
dist-upgrade/kernel/reboot: sq37, sq41. rebooting upload squid sq41 |
[production] |
| 08:49 |
<mutante> |
dist-upgrade & new kernel & reboot: sq33, sq36 |
[production] |
| 07:47 |
<mutante> |
preemptive rebooting of sq* servers identified as having > 200 days of uptime |
[production] |
| 02:22 |
<LocalisationUpdate> |
completed (1.20wmf2) at Fri May 4 02:22:42 UTC 2012 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.20wmf1) at Fri May 4 02:13:58 UTC 2012 |
[production] |
| 00:20 |
<raindrift> |
synchronizing Wikimedia installation... : |
[production] |
| 00:18 |
<raindrift> |
synchronizing Wikimedia installation... : Syncing the PageTriage extension, but only enabling on testwiki |
[production] |
| 00:08 |
<awjrichards> |
synchronized wmf-config/CommonSettings.php 'Adding 'fr' to language codes for mobile feedback' |
[production] |
| 00:06 |
<maplebed> |
moved ms1-3 from the production cluster to the test cluster |
[production] |
|
2012-05-03
§
|
| 23:29 |
<LeslieCarr> |
restarting networking on sq55 |
[production] |
| 23:29 |
<tstarling> |
synchronizing Wikimedia installation... : |
[production] |
| 23:27 |
<LeslieCarr> |
restarting networking on sq54 |
[production] |
| 23:24 |
<LeslieCarr> |
restarting networking on sq53 |
[production] |
| 23:21 |
<LeslieCarr> |
restarting networking on sq52 |
[production] |
| 23:16 |
<LeslieCarr> |
restarting networking on sq51 |
[production] |
| 21:30 |
<notpeter> |
removing srv220 from pybal pool for repartitioning |
[production] |
| 21:29 |
<LeslieCarr> |
switching asw-a4-sdtpa from single uplink to lag |
[production] |
| 21:19 |
<notpeter> |
putting srv219 back into pybal pool |
[production] |
| 21:14 |
<asher> |
synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23' |
[production] |
| 21:09 |
<asher> |
synchronized wmf-config/db.php 'reverting cluster23 change' |
[production] |
| 21:05 |
<asher> |
synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23' |
[production] |
| 21:02 |
<binasher> |
about to move ES writes to cluster23 |
[production] |