|
2013-03-18
§
|
| 21:22 |
<RobH> |
gracefuling all apaches resulted in rendering cluster overload, they didnt restart apache, odd |
[production] |
| 21:21 |
<robh> |
synchronized docroot |
[production] |
| 21:18 |
<LeslieCarr> |
service apache2 start on the mw-eqiad group via dsh |
[production] |
| 21:14 |
<binasher> |
started apache on all imagescalers |
[production] |
| 21:00 |
<robh> |
gracefulled all apaches |
[production] |
| 21:00 |
<RobH> |
troubleshooting erros in apache restart script |
[production] |
| 20:58 |
<robh> |
gracefulled all apaches |
[production] |
| 20:36 |
<RobH> |
authdns-update rt4694 |
[production] |
| 20:20 |
<RobH> |
authdns-update |
[production] |
| 20:04 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: |
[production] |
| 20:00 |
<reedy> |
Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12 |
[production] |
| 19:44 |
<^demon> |
jetty freaked out again, forcing a gerrit restart |
[production] |
| 19:32 |
<binasher> |
upgraded all coredb mariadb replicas to 5.5.30 |
[production] |
| 19:25 |
<reedy> |
Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12 |
[production] |
| 19:07 |
<asher> |
synchronized wmf-config/db-eqiad.php 'returning db1043 db1009' |
[production] |
| 19:04 |
<reedy> |
synchronized docroot |
[production] |
| 19:02 |
<asher> |
synchronized wmf-config/db-eqiad.php 'db1050 for special s1' |
[production] |
| 18:56 |
<asher> |
synchronized wmf-config/db-eqiad.php 'pulling db1043, db1009 for upgrade' |
[production] |
| 18:40 |
<reedy> |
synchronized php-1.21wmf12 'Initial sync of php-1.21wmf12' |
[production] |
| 18:16 |
<awight> |
payments cluster updated from c19cc66 to fe4fb96b |
[production] |
| 18:06 |
<reedy> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 18:03 |
<aaron> |
synchronized wmf-config/PrivateSettings.php 'Removed old testing cruft' |
[production] |
| 16:19 |
<cmjohnson1> |
removing/replacing disk 4 ms-be1004 |
[production] |
| 16:01 |
<hashar> |
Hard restarted Zuul which was dead locked again :-( |
[production] |
| 13:23 |
<paravoid> |
depooled, upgraded, restarted and pooled again all ms-fe[1-4] |
[production] |
| 13:01 |
<paravoid> |
depooling ms-fe1 for new swift version (patched 1.7.4) |
[production] |
| 11:27 |
<apergos> |
restarted both the varnishncsas on niobium, they were giant again |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.21wmf11) at Mon Mar 18 02:30:03 UTC 2013 |
[production] |
|
2013-03-15
§
|
| 23:24 |
<laner> |
synchronized wmf-config/CommonSettings.php 'Disabling aft5' |
[production] |
| 22:45 |
<asher> |
synchronized wmf-config/db-eqiad.php 'returning es1007,10 at lower weights' |
[production] |
| 22:35 |
<binasher> |
rebooted es1007,1010 for kernel upgrades |
[production] |
| 22:33 |
<asher> |
synchronized wmf-config/db-eqiad.php 'temporarily pulling es1007, es1010' |
[production] |
| 22:23 |
<Ryan_Lane> |
hid domain dropdown for account creation and login on wikitech |
[production] |
| 22:23 |
<Ryan_Lane> |
changed wikitech configuration to require email address on account creation |
[production] |
| 22:22 |
<Ryan_Lane> |
deployed change 54093 to wikitech |
[production] |
| 22:03 |
<hashar> |
Zuul : opened a bug about this week slowness, breakage: https://bugzilla.wikimedia.org/show_bug.cgi?id=46176 |
[production] |
| 21:51 |
<aaron> |
synchronized php-1.21wmf11/includes 'deployed 08295adcdc4a4130954f52fb1eb5377638cd97bc' |
[production] |
| 21:36 |
<binasher> |
db59 -> mariadb 5.5.30 |
[production] |
| 21:29 |
<hashar> |
restarted Zuul with the safe version ff79197 |
[production] |
| 21:27 |
<hashar> |
downgrading Zuul to safe version ff79197 (aka stop doing `git remote update` on ref-update events) |
[production] |
| 21:25 |
<binasher> |
added new build of mariadb 5.5.30 to precise-wikimedia on brewster |
[production] |
| 21:20 |
<hashar> |
hard restarted Zuul. It was locked down again :/ |
[production] |