|
2012-09-12
§
|
| 23:06 |
<hashar> |
synchronized docroot/checkuser/favicon.ico |
[production] |
| 23:06 |
<aaron> |
synchronized php-1.20wmf11/extensions/timeline 'deployed f79a67ac344801acc57edbc95158bf5ecc87481b' |
[production] |
| 23:03 |
<hashar> |
synchronized docroot/bits/favicon |
[production] |
| 23:02 |
<hashar> |
synchronized docroot/bits/favicon |
[production] |
| 22:24 |
<aaron> |
synchronized wmf-config/filebackend.php 'Enabled auto-resyncing for file backends.' |
[production] |
| 22:12 |
<aaron> |
synchronized wmf-config/filebackend.php 'Disabled directory operations on local-temp zones.' |
[production] |
| 22:12 |
<reedy> |
synchronized closed.dblist |
[production] |
| 21:48 |
<aaron> |
synchronized php-1.20wmf11/includes/filebackend/FileBackendMultiWrite.php 'deployed 6a6a0427160e1493617789ce320020b21d0836dd' |
[production] |
| 21:48 |
<aaron> |
synchronized php-1.20wmf11/languages/messages/MessagesEn.php 'deployed 6a6a0427160e1493617789ce320020b21d0836dd' |
[production] |
| 21:47 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Subpagesortkey on ptwikibooks and enwikibooks' |
[production] |
| 21:13 |
<RoanKattouw_> |
On ms7: find /export/upload/math ! -user apache -exec /root/fixownership2 \\{\\} \\; |
[production] |
| 21:10 |
<reedy> |
synchronized php-1.20wmf11/includes/api/ApiQueryLogEvents.php |
[production] |
| 20:51 |
<reedy> |
synchronized php-1.20wmf11/extensions/TimedMediaHandler/ |
[production] |
| 20:50 |
<reedy> |
synchronized php-1.20wmf11/extensions/MwEmbedSupport/ |
[production] |
| 19:05 |
<notpeter> |
reimaging mw1-16 |
[production] |
| 19:00 |
<hashar> |
jenkins: hashar installed a snapshot of Gerrit Trigger plugin |
[production] |
| 18:57 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else over to 1.20wmf11 |
[production] |
| 18:47 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf11 |
[production] |
| 18:09 |
<notpeter> |
adding mw59 to apaches pool |
[production] |
| 18:03 |
<reedy> |
synchronized php-1.20wmf10 |
[production] |
| 17:58 |
<reedy> |
synchronized php-1.20wmf11 |
[production] |
| 17:53 |
<notpeter> |
re-adding srv194-199 to apaches pool |
[production] |
| 17:38 |
<notpeter> |
removing mw1-16 from apaches pool to reimage them as precise jobrunner boxes |
[production] |
| 14:46 |
<paravoid> |
restarted swift on ms-fe1-4, proxy-server was leaking memory and had been swapping |
[production] |
| 02:52 |
<LocalisationUpdate> |
completed (1.20wmf10) at Wed Sep 12 02:52:12 UTC 2012 |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.20wmf11) at Wed Sep 12 02:27:16 UTC 2012 |
[production] |
|
2012-09-11
§
|
| 22:29 |
<notpeter> |
removing srv195-199 from apaches pool for upgarde to precise |
[production] |
| 20:51 |
<RobH> |
authdns-update per rt 1326 |
[production] |
| 18:31 |
<apergos> |
second reboot had whines of device not present/not ready for /dev/sdn1 and /dev/sdh1 (note the list is not the same as the previous list), waited a little and it booted up with all disks mounted |
[production] |
| 18:26 |
<apergos> |
first reboot of msbe10 showed complaints of not ready on devices sdi, k, n but they did eventually mount when the box came up fully. rebooting a second time to see what happens |
[production] |
| 18:13 |
<apergos> |
rebooting ms-be10 after recreation and remount of swift filesystems (except those on ssds), see if they stick |
[production] |
| 17:55 |
<apergos> |
new degradedarray event on ms-be6, booooo |
[production] |
| 17:54 |
<apergos> |
rebooting ms-bs10 to see if we can get some reasonable device names, in prep for trying the same trick on it as on ms-be6 and 7 |
[production] |
| 17:43 |
<Reedy> |
Running sync-common on mw8 |
[production] |
| 17:43 |
<Reedy> |
mw8 seems to have out of date mw files |
[production] |
| 17:42 |
<apergos> |
rebooting ms-be7 after recreating all filesystems (except those on the ssds) and remounting them, to see if the disks stay visible |
[production] |
| 17:38 |
<reedy> |
synchronized php-1.20wmf11/extensions/ConfirmEdit/FancyCaptcha.class.php |
[production] |
| 17:31 |
<apergos> |
puppet disabled on ms-be7 via /etc/default/puppet, swift processes shut down while we test |
[production] |
| 17:10 |
<apergos> |
one more reboot of ms-be6 after raid repaired, see if it and all mounts stay ok |
[production] |
| 15:57 |
<apergos> |
rebuilding the degraded raid array on ms-be6 |
[production] |
| 15:15 |
<apergos> |
stopping swift processes on ms-be6 til we decide what's next; all disks came up and were mounted (but we still do have degraded raid array, not so awesome) |
[production] |
| 15:12 |
<apergos> |
redid other ms-be6 filesystems so labels would be correct with disk layout without ssds. reboot again to see if it comes up properly |
[production] |
| 14:44 |
<apergos> |
manually recreated filesystems and mounted missing drives on ms-be6 (without ssds cabled), seems to have worked without errors. rebooting to see what happens. |
[production] |
| 14:18 |
<apergos> |
stopped swift processes on ms-be6 and disabled puppet (/etc/default/puppet) so it won't restart them. still doing testing of hw over there so we need the box up. |
[production] |
| 04:46 |
<reedy> |
synchronized php-1.20wmf11/extensions/ConfirmEdit/ '9643ccd3964827651e8a23a19a92ba3d240e60ee' |
[production] |
| 02:42 |
<LocalisationUpdate> |
completed (1.20wmf10) at Tue Sep 11 02:41:59 UTC 2012 |
[production] |
| 02:22 |
<LocalisationUpdate> |
completed (1.20wmf11) at Tue Sep 11 02:22:02 UTC 2012 |
[production] |
| 01:30 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf10 |
[production] |
| 01:02 |
<paravoid> |
adjusting spence's firewall to include all eqiad subnets |
[production] |