|
2014-08-28
§
|
| 23:30 |
<maxsem> |
Synchronized php-1.24wmf18/extensions/GlobalCssJs/: https://gerrit.wikimedia.org/r/#/c/157009/ (duration: 00m 04s) |
[production] |
| 23:27 |
<K4-713> |
Updated fraud filters on payments |
[production] |
| 22:52 |
<aaron> |
Synchronized php-1.24wmf18/maintenance/findMissingFiles.php: (no message) (duration: 00m 07s) |
[production] |
| 22:15 |
<mutante> |
restarted tools.morebots production instance - can i log now? |
[production] |
| 22:13 |
<cmjohnson1> |
ms-fe1003 down for relocation |
[production] |
| 22:13 |
<mutante> |
test |
[production] |
| 21:32 |
<bd808> |
Added "Greg Grossmeier" to UnderNDA sudoers group |
[releng] |
| 21:15 |
<robh> |
bast2001.wikimedia.org now online in codfw. |
[production] |
| 21:15 |
<robh> |
i never admin logged when install2001.wikimedia.org went online the other day, opps. |
[production] |
| 21:15 |
<ori> |
last sync was of Iac37a2369: resourceloader: Don't register raw modules client-side |
[production] |
| 21:14 |
<ori> |
Synchronized php-1.24wmf18/includes/resourceloader/ResourceLoaderStartUpModule.php: (no message) (duration: 00m 03s) |
[production] |
| 20:57 |
<krinkle> |
Synchronized php-1.24wmf19/includes/resourceloader/ResourceLoaderStartUpModule.php: fd5b963458c19 (duration: 00m 06s) |
[production] |
| 20:33 |
<ottomata> |
shutting down elastic1016 |
[production] |
| 20:16 |
<ottomata> |
temporarily disable puppet on gadolinium |
[production] |
| 19:36 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 14s) |
[production] |
| 19:12 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf19 |
[production] |
| 19:09 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf18 |
[production] |
| 19:07 |
<reedy> |
Finished scap: testwiki to 1.24wmf19 (duration: 43m 00s) |
[production] |
| 18:25 |
<godog> |
install build-essential and fakeroot on tin |
[production] |
| 18:24 |
<reedy> |
Started scap: testwiki to 1.24wmf19 |
[production] |
| 17:26 |
<aaron> |
Synchronized rpc: 9564e93ecd4953126d91b99d7728f63401a4dc86 (duration: 00m 07s) |
[production] |
| 17:13 |
<^d> |
elastic: excluded the elastic1016 node from shard allocation, shards draining so we can take it down for disk testing |
[production] |
| 17:12 |
<bd808> |
Changed centralauth db to rename labswiki -> deploymentwiki |
[releng] |
| 16:49 |
<bd808> |
CentralAuth looks broken on http://deployment.wikimedia.beta.wmflabs.org/ |
[releng] |
| 16:49 |
<bd808> |
Apache vhosts look good again |
[releng] |
| 16:34 |
<bd808> |
Restarted varnishes on deployment-cache-text02 |
[releng] |
| 16:13 |
<andrewbogott> |
merging a patch that renames 'labswiki' to 'deploymentwiki' |
[releng] |
| 16:02 |
<ottomata> |
restarted webstats-collector on gadolinium |
[production] |
| 13:18 |
<mark> |
Reactivated cr2-eqiad AS3257 transit link |
[production] |
| 10:44 |
<springle> |
xtrabackup clone db1051 to db1073 |
[production] |
| 10:18 |
<godog> |
restarting mailman on sodium |
[production] |
| 09:21 |
<hashar> |
resetting git repository in /data/project/apache/conf to point to the betaclusterbranch of operations/mediawiki-config.git discarded all local hacks in the process |
[releng] |
| 08:52 |
<godog> |
restarted apache on mw1134 |
[production] |
| 08:03 |
<godog> |
killed stray mailman processes on sodium (no pid file) and restarted mailman |
[production] |
| 06:12 |
<springle> |
xtrabackup clone db1051 to db1072 |
[production] |
|
2014-08-27
§
|
| 23:03 |
<hashar> |
Blacklisting the security audit IP again on deployment-cache bits01 mobile03 and text02 |
[releng] |
| 22:53 |
<hashar> |
removed the blackhole ip route from deployment-cache-text02 and deployment-cache-mobile03 |
[releng] |
| 22:48 |
<hashar> |
the IP is a known security audit. See Chris Steipp. |
[releng] |
| 22:46 |
<hashar> |
blackholed an IP address on deployment-cache-text02 and deployment-cache-mobile03 , it was causing hundred of requests per seconds and overloaded the beta cluster. Use route -n to find the IP |
[releng] |
| 22:37 |
<hashar> |
restarting udp2log-mw on deployment-bastion. It keeps crashing since fiarly recently |
[releng] |
| 22:26 |
<bd808> |
when restarting varnish on deployment-cache-text02, don't forget that there are 2 varnish services (varnish and varnish-frontend) |
[releng] |
| 22:19 |
<bd808> |
restarted varnish (again) on deployment-cache-text02 |
[releng] |
| 22:10 |
<bd808> |
restarted varnish on deployment-cache-text02 |
[releng] |
| 16:22 |
<bd808> |
killing `apt-get update` process running on deployment-bastion since Jun13 |
[releng] |
| 14:59 |
<bd808> |
Resolved puppet git merge conflict on deployment-salt |
[releng] |
| 14:49 |
<bd808> |
Moved hhvm core dumps to /data/project/hhvm-cores |
[releng] |
| 14:42 |
<bd808> |
Root dirve full on deployment-mediawiki02; hhvm core files are the culprit |
[releng] |