|
2014-08-28
§
|
| 16:49 |
<bd808> |
CentralAuth looks broken on http://deployment.wikimedia.beta.wmflabs.org/ |
[releng] |
| 16:49 |
<bd808> |
Apache vhosts look good again |
[releng] |
| 16:34 |
<bd808> |
Restarted varnishes on deployment-cache-text02 |
[releng] |
| 16:13 |
<andrewbogott> |
merging a patch that renames 'labswiki' to 'deploymentwiki' |
[releng] |
| 16:02 |
<ottomata> |
restarted webstats-collector on gadolinium |
[production] |
| 13:18 |
<mark> |
Reactivated cr2-eqiad AS3257 transit link |
[production] |
| 10:44 |
<springle> |
xtrabackup clone db1051 to db1073 |
[production] |
| 10:18 |
<godog> |
restarting mailman on sodium |
[production] |
| 09:21 |
<hashar> |
resetting git repository in /data/project/apache/conf to point to the betaclusterbranch of operations/mediawiki-config.git discarded all local hacks in the process |
[releng] |
| 08:52 |
<godog> |
restarted apache on mw1134 |
[production] |
| 08:03 |
<godog> |
killed stray mailman processes on sodium (no pid file) and restarted mailman |
[production] |
| 06:12 |
<springle> |
xtrabackup clone db1051 to db1072 |
[production] |
|
2014-08-27
§
|
| 23:03 |
<hashar> |
Blacklisting the security audit IP again on deployment-cache bits01 mobile03 and text02 |
[releng] |
| 22:53 |
<hashar> |
removed the blackhole ip route from deployment-cache-text02 and deployment-cache-mobile03 |
[releng] |
| 22:48 |
<hashar> |
the IP is a known security audit. See Chris Steipp. |
[releng] |
| 22:46 |
<hashar> |
blackholed an IP address on deployment-cache-text02 and deployment-cache-mobile03 , it was causing hundred of requests per seconds and overloaded the beta cluster. Use route -n to find the IP |
[releng] |
| 22:37 |
<hashar> |
restarting udp2log-mw on deployment-bastion. It keeps crashing since fiarly recently |
[releng] |
| 22:26 |
<bd808> |
when restarting varnish on deployment-cache-text02, don't forget that there are 2 varnish services (varnish and varnish-frontend) |
[releng] |
| 22:19 |
<bd808> |
restarted varnish (again) on deployment-cache-text02 |
[releng] |
| 22:10 |
<bd808> |
restarted varnish on deployment-cache-text02 |
[releng] |
| 16:22 |
<bd808> |
killing `apt-get update` process running on deployment-bastion since Jun13 |
[releng] |
| 14:59 |
<bd808> |
Resolved puppet git merge conflict on deployment-salt |
[releng] |
| 14:49 |
<bd808> |
Moved hhvm core dumps to /data/project/hhvm-cores |
[releng] |
| 14:42 |
<bd808> |
Root dirve full on deployment-mediawiki02; hhvm core files are the culprit |
[releng] |
|
2014-08-26
§
|
| 21:04 |
<hashar> |
Updating our Jenkins Job Builder fork 0268581..e5c0c61 . Will let us define variables in 'default' section and override them when invoking a job template ( https://review.openstack.org/#/c/100020/ ) |
[production] |
| 19:58 |
<bd808> |
Ran sync-common on mw1053.eqiad.wmnet to recover from failure during last scap |
[production] |
| 19:48 |
<aude> |
Finished scap: Update new messages for Wikibase (duration: 07m 16s) |
[production] |
| 19:41 |
<aude> |
Started scap: Update new messages for Wikibase |
[production] |
| 19:39 |
<aude> |
Synchronized wmf-config/Wikibase.php: add Wikibase badges css setting (duration: 00m 10s) |
[production] |
| 19:26 |
<aude> |
Synchronized wmf-config/Wikibase.php: enable new serialization format for wikidata (duration: 00m 08s) |
[production] |
| 19:10 |
<reedy> |
Synchronized php-1.24wmf18/extensions/Echo/: (no message) (duration: 00m 14s) |
[production] |
| 19:05 |
<aude> |
Synchronized wmf-config/Wikibase.php: enable otherprojects sidebar beta feature (duration: 00m 15s) |
[production] |
| 18:55 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf18 |
[production] |
| 18:53 |
<reedy> |
Synchronized php-1.24wmf18/extensions/MassMessage: (no message) (duration: 00m 14s) |
[production] |
| 18:53 |
<reedy> |
Synchronized php-1.24wmf17/extensions/MassMessage: (no message) (duration: 00m 16s) |
[production] |
| 18:19 |
<jgage> |
Failover from analytics1010-eqiad-wmnet to analytics1004-eqiad-wmnet successful |
[production] |
| 17:47 |
<bd808> |
Synchronized private/PrivateSettings.php: Syncing file rather than symlink (duration: 00m 04s) |
[production] |
| 17:36 |
<bd808> |
mw1010.eqiad.wmnet was out of sync too. I suspect there is something wrong with the fanout update step in scap |
[production] |
| 17:26 |
<bd808> |
/usr/local/apache/common-local out of date on mw1161.eqiad.wmnet; updated via sync-common |
[production] |
| 17:25 |
<bd808> |
sync-* not updating terbium properly; sync-common from terbium manually got several config changes; maybe a problem with mw1161.eqiad.wmnet rsync mirror |
[production] |
| 17:14 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 04s) |
[production] |
| 17:12 |
<demon> |
Synchronized wmf-config/PrivateSettings.php: adjust swift auth url for cirrus (duration: 00m 04s) |
[production] |
| 17:05 |
<cmjohnson> |
swapping failed disk labsdb1003 slot 1 |
[production] |
| 16:42 |
<bd808> |
Ran sync-common on osmium to verify that it now rebuilds l10n cache by default (and it does!) |
[production] |
| 16:36 |
<legoktm> |
running removeOldManualUserPages.php (GlobalCssJs) for users who requested it |
[production] |
| 16:29 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: Again, with feeling (duration: 00m 04s) |
[production] |
| 16:26 |
<bd808> |
Finished scap: no-op scap to test scap code update (duration: 13m 31s) |
[production] |
| 16:20 |
<bd808|DEPLOY> |
Rsync sloooow to fenari "16:18:52 fenari INFO - Finished rsync common (duration: 04m 38s)" |
[production] |
| 16:12 |
<bd808> |
Started scap: no-op scap to test scap code update |
[production] |
| 16:07 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) |
[production] |