|
2013-03-18
§
|
| 23:11 |
<asher> |
synchronized wmf-config/db-eqiad.php 'returning db10[10-11],[26-28] at low weights' |
[production] |
| 22:45 |
<binasher> |
converted rogue s3 tables (moodbar_feedback, trackbacks, transcache) to innodb |
[production] |
| 22:28 |
<ottomata> |
added python-flask-login-0.1.2-1 to apt |
[production] |
| 22:23 |
<robh> |
gracefulled all apaches |
[production] |
| 22:20 |
<LeslieCarr> |
intradatacenter link is flapping , switching links, this may cause some higher latency |
[production] |
| 22:14 |
<robh> |
gracefulled all apaches |
[production] |
| 22:11 |
<robh> |
gracefulled all apaches |
[production] |
| 21:58 |
<mutante> |
purging wikimediafoundation.org from squid |
[production] |
| 21:57 |
<dzahn> |
gracefulled all apaches |
[production] |
| 21:56 |
<robh> |
gracefulled all apaches |
[production] |
| 21:36 |
<asher> |
synchronized wmf-config/db-eqiad.php 'pulling a db from s3-7 for upgrade' |
[production] |
| 21:30 |
<paravoid> |
authdns-update: removing *.ts.wikimedia.org records |
[production] |
| 21:29 |
<dzahn> |
gracefulled all apaches |
[production] |
| 21:24 |
<binasher> |
ran "ddsh -g apaches -cM '/etc/init.d/apache2 start'" |
[production] |
| 21:24 |
<RobH> |
authdns-update rt4683 |
[production] |
| 21:22 |
<RobH> |
gracefuling all apaches resulted in rendering cluster overload, they didnt restart apache, odd |
[production] |
| 21:21 |
<robh> |
synchronized docroot |
[production] |
| 21:18 |
<LeslieCarr> |
service apache2 start on the mw-eqiad group via dsh |
[production] |
| 21:14 |
<binasher> |
started apache on all imagescalers |
[production] |
| 21:00 |
<robh> |
gracefulled all apaches |
[production] |
| 21:00 |
<RobH> |
troubleshooting erros in apache restart script |
[production] |
| 20:58 |
<robh> |
gracefulled all apaches |
[production] |
| 20:36 |
<RobH> |
authdns-update rt4694 |
[production] |
| 20:20 |
<RobH> |
authdns-update |
[production] |
| 20:04 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: |
[production] |
| 20:00 |
<reedy> |
Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12 |
[production] |
| 19:44 |
<^demon> |
jetty freaked out again, forcing a gerrit restart |
[production] |
| 19:32 |
<binasher> |
upgraded all coredb mariadb replicas to 5.5.30 |
[production] |
| 19:25 |
<reedy> |
Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12 |
[production] |
| 19:07 |
<asher> |
synchronized wmf-config/db-eqiad.php 'returning db1043 db1009' |
[production] |
| 19:04 |
<reedy> |
synchronized docroot |
[production] |
| 19:02 |
<asher> |
synchronized wmf-config/db-eqiad.php 'db1050 for special s1' |
[production] |
| 18:56 |
<asher> |
synchronized wmf-config/db-eqiad.php 'pulling db1043, db1009 for upgrade' |
[production] |
| 18:40 |
<reedy> |
synchronized php-1.21wmf12 'Initial sync of php-1.21wmf12' |
[production] |
| 18:16 |
<awight> |
payments cluster updated from c19cc66 to fe4fb96b |
[production] |
| 18:06 |
<reedy> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 18:03 |
<aaron> |
synchronized wmf-config/PrivateSettings.php 'Removed old testing cruft' |
[production] |
| 16:19 |
<cmjohnson1> |
removing/replacing disk 4 ms-be1004 |
[production] |
| 16:01 |
<hashar> |
Hard restarted Zuul which was dead locked again :-( |
[production] |
| 13:23 |
<paravoid> |
depooled, upgraded, restarted and pooled again all ms-fe[1-4] |
[production] |
| 13:01 |
<paravoid> |
depooling ms-fe1 for new swift version (patched 1.7.4) |
[production] |
| 11:27 |
<apergos> |
restarted both the varnishncsas on niobium, they were giant again |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.21wmf11) at Mon Mar 18 02:30:03 UTC 2013 |
[production] |