|
2012-01-16
§
|
| 17:00 |
<Reedy> |
srv221 and srv222 are out of space on / |
[production] |
| 17:00 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable autoconfirmed reupload on incubatorwiki' |
[production] |
| 16:41 |
<reedy> |
synchronized wmf-config/flaggedrevs.php 'Kill labs setting stuff' |
[production] |
| 16:40 |
<reedy> |
synchronized wmf-config/flaggedrevs.php 'Kill labs setting stuff' |
[production] |
| 16:37 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33662 - Change project namespace for lb.wiktionary' |
[production] |
| 16:35 |
<Reedy> |
Ran namespaceDupes on fawikisource |
[production] |
| 16:34 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33662 - Change project namespace for lb.wiktionary' |
[production] |
| 16:28 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33708 - Add alias to fa wikisource' |
[production] |
| 16:26 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33708 - Add alias to fa wikisource' |
[production] |
| 16:06 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33758 - Arabic numerals on Arabic Wiktionary' |
[production] |
| 15:55 |
<cmjohnson1> |
shutting down srv199 for main board replacement |
[production] |
| 15:09 |
<mutante> |
torrus was broken ([[RT:2279]]) and did not start due to corrupted berkeleydb, used db4.8_recover, service started again |
[production] |
| 12:38 |
<mark> |
Running puppet on freshly installed sodium |
[production] |
| 09:15 |
<apergos> |
cleaned up /tmp on srv223... seems like cleanup once an hour by cron isn't often enough any more, scalers are doing too much work |
[production] |
| 02:05 |
<LocalisationUpdate> |
completed (1.18) at Mon Jan 16 02:05:05 UTC 2012 |
[production] |
|
2012-01-13
§
|
| 22:40 |
<Ryan_Lane> |
force running puppet on all instances |
[production] |
| 22:40 |
<Ryan_Lane> |
re-generated certificates for all instances |
[production] |
| 22:40 |
<Ryan_Lane> |
deleted all puppet certificates on all instances |
[production] |
| 22:31 |
<LeslieCarr> |
restarting ganglia1001 |
[production] |
| 21:41 |
<Ryan_Lane> |
changing virt1 to be a cname of virt0 |
[production] |
| 21:17 |
<Ryan_Lane> |
killed pdns on virt1 |
[production] |
| 20:40 |
<Ryan_Lane> |
stopped pdns on virt1 |
[production] |
| 20:32 |
<reedy> |
synchronized php-1.18/extensions/Contest/specials/ '[[rev:108843|r108843]]' |
[production] |
| 20:28 |
<Ryan_Lane> |
changed NS records for wmflabs.org and wmflabs to point to virt0 |
[production] |
| 20:28 |
<Ryan_Lane> |
changed recursor to point wmflabs domain to virt0 |
[production] |
| 20:16 |
<K4-713> |
synchronized payments cluster to [[rev:108833|r108833]] |
[production] |
| 19:30 |
<Ryan_Lane> |
shutting down virt1 to ensure migration was completed |
[production] |
| 19:12 |
<LeslieCarr> |
restarting gmond on cp1043 |
[production] |
| 18:55 |
<LeslieCarr> |
hard powercycling ms1002 |
[production] |
| 18:48 |
<LeslieCarr> |
rebooting ms1002 due to kswapd 100% cpu bug https://bugs.launchpad.net/ubuntu/+bug/721896 |
[production] |
| 16:44 |
<mutante> |
added alswiktionary & alswikibooks to closed.dblist |
[production] |
| 16:43 |
<dzahn> |
synchronized closed.dblist |
[production] |
| 16:33 |
<mutante> |
syncing InitialiseSettings.php after changing as wiki namespace per bz 33507 |
[production] |
| 16:33 |
<dzahn> |
synchronized ./wmf-config/InitialiseSettings.php |
[production] |
| 16:07 |
<mutante> |
updated blog theme and installed a plugin per RT 2271 |
[production] |
| 14:34 |
<mutante> |
srv191 - has now fresh OS, re-issued puppet certs, ran puppet, restart memcached, etc. - all back in monitoring |
[production] |
| 12:59 |
<Guest45399> |
PXE booting srv191, installing OS |
[production] |
| 07:45 |
<Ryan_Lane> |
disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing. |
[production] |
| 07:43 |
<Ryan_Lane> |
added a grant for mediawiki in the database to fix labsconsole mediawiki outage |
[production] |