|
2012-03-18
§
|
| 19:20 |
<Ryan_Lane> |
stopping all labs instances, manually recovering gluster volume |
[production] |
| 15:39 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 35295 - Missing a in abusefilter-hide-log permission for oversighters' |
[production] |
| 10:49 |
<Ryan_Lane> |
rebooting virt4 thanks to defunct libvirt process |
[production] |
| 03:43 |
<Ryan_Lane> |
bringing all labs instances up |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Sun Mar 18 02:18:51 UTC 2012 |
[production] |
| 01:09 |
<Ryan_Lane> |
rebooting all of the virt hosts, gluster is having major issues |
[production] |
| 00:43 |
<Ryan_Lane> |
rebooting virt2 |
[production] |
| 00:40 |
<Ryan_Lane> |
restarting glusterfs on virt2 |
[production] |
| 00:11 |
<Ryan_Lane> |
rebooting virt3 libirt is non-responsive |
[production] |
| 00:00 |
<Ryan_Lane> |
bringing up instances that were downed on virt3 |
[production] |
|
2012-03-17
§
|
| 23:50 |
<Ryan_Lane> |
virt3 crashed, powercycling it |
[production] |
| 23:34 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Remove old comments' |
[production] |
| 23:24 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Remove old comments' |
[production] |
| 23:18 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat' |
[production] |
| 23:11 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat' |
[production] |
| 23:02 |
<catrope> |
synchronizing Wikimedia installation... : Have to scap for that AFTv5 change to propagate i18n change |
[production] |
| 22:52 |
<catrope> |
synchronized php-1.19/extensions/ArticleFeedbackv5/ '[[rev:114087|r114087]]' |
[production] |
| 21:42 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 35289 - Add wikisource logo to mobile wikisource gateway' |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.19) at Sat Mar 17 02:21:03 UTC 2012 |
[production] |
| 01:23 |
<AaronSchulz> |
FindFilesMissingDBRows.php done, list under aaron/output/missingFileDBRows |
[production] |
| 00:11 |
<AaronSchulz> |
Running FindFilesMissingDBRows.php on all wikis |
[production] |
|
2012-03-16
§
|
| 21:21 |
<binasher> |
running enwiki.revision sha1 schema migrations on eqiad side |
[production] |
| 20:12 |
<reedy> |
synchronizing Wikimedia installation... : Rebuild moodbar messages |
[production] |
| 20:03 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Re-enable moodbar on enwiki' |
[production] |
| 19:53 |
<reedy> |
synchronized php-1.19/extensions/MoodBar/ '[[rev:114030|r114030]]' |
[production] |
| 19:15 |
<Reedy> |
Ran namespaceDupes on stewardwiki |
[production] |
| 17:11 |
<RobH> |
hdd in search1017/1018 replaced per rt 2583 |
[production] |
| 16:54 |
<RobH> |
search1017 and search1018 coming down for hdd swap |
[production] |
| 16:53 |
<RobH> |
cp1017 back in service pool |
[production] |
| 16:43 |
<RobH> |
cp1019 back in full service |
[production] |
| 16:22 |
<reedy> |
synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php '[[rev:114021|r114021]]' |
[production] |
| 16:22 |
<RobH> |
cp1017 memory error, coming down for troubleshooting. |
[production] |
| 16:18 |
<RobH> |
cp1019 memory error cleared after reseating, notes on rt 2651 |
[production] |
| 16:09 |
<mark> |
Migrated all varnish3 packages to newer varnish packages from git |
[production] |
| 16:08 |
<RobH> |
cp1019 coming down for memory error troubleshooting |
[production] |
| 15:58 |
<RobH> |
cp1040 repaired per rt 2611 |
[production] |
| 15:48 |
<RobH> |
cp1040 down for memory replacement |
[production] |
| 15:09 |
<reedy> |
synchronized stylize.php 'Test for hume' |
[production] |
| 15:04 |
<root> |
synchronized ufg.sql 'test sync to see if hume is fixed' |
[production] |
| 14:55 |
<reedy> |
synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php |
[production] |
| 14:04 |
<apergos> |
restarted swift-container-auditor on ms-be3, it had died for some reason |
[production] |
| 08:07 |
<mutante> |
i reverted that (star cert for wikitech), no worries i "shred"ded the files |
[production] |
| 07:51 |
<mutante> |
replaced self-signed cert on wikitech with the star cert |
[production] |
| 04:19 |
<mutante> |
on stafford, deleting spence's puppet report files to free some disk space (they are like the largest report files of all) |
[production] |
| 03:09 |
<mutante> |
stafford - - /var/lib/puppet/reports is getting quite large (18G), and we got the first disk space warning, do we want to keep those? |
[production] |
| 02:45 |
<mutante> |
killing nrpe on several hosts where it was running as the wrong user again (somehow through the use of dsh) |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.19) at Fri Mar 16 02:21:35 UTC 2012 |
[production] |
| 01:12 |
<mutante> |
stopping nagios-wm temp. while changing nrpe config (will watch it manually until it's back) |
[production] |
| 00:54 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe' |
[production] |
| 00:51 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe' |
[production] |