|
2012-06-08
§
|
| 19:57 |
<logmsgbot_> |
reedy Finished syncing Wikimedia installation... : Rebuilding localisation cache for message updates |
[production] |
| 19:39 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for message updates |
[production] |
| 18:47 |
<logmsgbot_> |
reedy synchronized php-1.20wmf4/languages/messages/ 'Pushing out updated files upon siebrands request' |
[production] |
| 16:30 |
<cmjohnson1> |
shutting down search32 to swap DIMM around |
[production] |
| 10:19 |
<notpeter> |
ganglia down, restarting apache on nickel. |
[production] |
| 09:32 |
<notpeter> |
stopping indexing on searchidx1001 to re-copy to searchidx2 |
[production] |
| 08:45 |
<notpeter> |
reimaging searchidx2 with correct partitioning |
[production] |
| 06:50 |
<Tim> |
on cp1001: disabled HTCP plugin in gmond for testing, seems to work so I will disable it properly |
[production] |
| 03:55 |
<Tim> |
disabled LastModified extension due to overload on cp1005 |
[production] |
| 03:51 |
<logmsgbot_> |
tstarling synchronized wmf-config/InitialiseSettings.php |
[production] |
| 03:18 |
<Tim> |
restarting squid on cp1005, maybe out of FDs or something, cachemgr shows exactly 1000 open connections to 10.2.1.1 |
[production] |
| 03:08 |
<Tim> |
stopped gmond on cp1001 with kill -STOP for memory leak debugging |
[production] |
| 03:02 |
<Tim> |
on cp1002: killed gmond again, it was leaking memory again, already up to 27GB in the few minutes since I restarted it |
[production] |
| 03:01 |
<Tim> |
on fenari: copied *.text and *.upload from /home/wikipedia/conf/squid/generated/clusters to /etc/dsh/group |
[production] |
| 02:56 |
<Tim> |
cp1001: same as on cp1002, restarted gmond |
[production] |
| 02:53 |
<Tim> |
on cp1002: killed gmond, which was using 100% CPU and 23GB RSS. Restarting squid which had died |
[production] |
| 02:23 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf3) at Fri Jun 8 02:23:09 UTC 2012 |
[production] |
| 02:15 |
<SigmaWP> |
Hello, world! |
[production] |
| 02:14 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf4) at Fri Jun 8 02:14:55 UTC 2012 |
[production] |
|
2012-06-07
§
|
| 23:32 |
<Tim> |
deploying varnish configuration change https://gerrit.wikimedia.org/r/#/c/10672/ on cp1041, cp1042, cp1043, cp1044 |
[production] |
| 20:46 |
<logmsgbot_> |
kaldari synchronized php-1.20wmf4/extensions/LastModified 'syncing LastModified extension' |
[production] |
| 20:32 |
<logmsgbot_> |
kaldari synchronized wmf-config/InitialiseSettings.php 'turning on LastModified and E3Experiemnts for en.wiki' |
[production] |
| 19:54 |
<logmsgbot_> |
kaldari synchronized php-1.20wmf4/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'syncing js file for E3Experiments' |
[production] |
| 18:58 |
<cmjohnson1_> |
shutting down search32 for testing |
[production] |
| 16:52 |
<Ryan_Lane> |
added ldap automount entries for /public/datasets and /public/keys |
[production] |
| 02:19 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf3) at Thu Jun 7 02:19:52 UTC 2012 |
[production] |
| 02:11 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf4) at Thu Jun 7 02:11:49 UTC 2012 |
[production] |
|
2012-06-06
§
|
| 22:29 |
<logmsgbot_> |
aaron synchronized wmf-config/swift.php 'deployed 7dc77e431310580da0dbd368b8b290a293e3ee21' |
[production] |
| 19:55 |
<cmjohnson1> |
shutting down search32 to reseat DIMM B2 |
[production] |
| 19:18 |
<cmjohnson1> |
shutting down storage3 |
[production] |
| 18:05 |
<logmsgbot_> |
aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaining wikis to 1.20wmf4 |
[production] |
| 17:02 |
<Jeff_Green> |
mailman 'site' password changed per RT 3039 |
[production] |
| 11:12 |
<mark> |
Added AAAA record to mobile |
[production] |
| 10:38 |
<mark> |
Wikipedia is IPv6-enabled. |
[production] |
| 10:37 |
<mark> |
Added AAAA records to all non-mobile wiki projects |
[production] |
| 10:19 |
<mark> |
Added AAAA record to bits.wikimedia.org |
[production] |
| 10:02 |
<Ryan_Lane> |
repooling ssl3001 |
[production] |
| 10:00 |
<mark> |
Added AAAA record to upload.wikimedia.org |
[production] |
| 09:51 |
<Ryan_Lane> |
depooling ssl3001 |
[production] |
| 08:41 |
<mark> |
Converted bits.wikimedia.org into a direct geodns record, removed the old bits -> bits-geo CNAME |
[production] |
| 07:53 |
<mark> |
Converted geoiplookup.wikimedia.org into a separate, IPv4-only geodns record |
[production] |
| 07:37 |
<pp-pdf1> |
installed tmpreaper cronjob for /home/pp/mathcache directory |
[production] |
| 07:37 |
<pp-pdf2> |
installed tmpreaper cronjob for /home/pp/mathcache directory |
[production] |
| 07:37 |
<pp-pdf3> |
installed tmpreaper cronjob for /home/pp/mathcache directory |
[production] |
| 02:35 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf3) at Wed Jun 6 02:35:47 UTC 2012 |
[production] |
| 02:11 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf4) at Wed Jun 6 02:10:58 UTC 2012 |
[production] |