|
2012-06-11
§
|
| 16:58 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Running scap to ensure consistency |
[production] |
| 16:53 |
<logmsgbot_> |
reedy synchronized php-1.20wmf5/cache/l10n/ |
[production] |
| 16:28 |
<mutante> |
installing security upgrades on sodium |
[production] |
| 16:19 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5 |
[production] |
| 16:17 |
<logmsgbot_> |
reedy synchronized wmf-config/ExtensionMessages-1.20wmf5.php |
[production] |
| 16:02 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5 |
[production] |
| 15:55 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5 |
[production] |
| 15:54 |
<logmsgbot_> |
reedy Finished syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5 |
[production] |
| 15:46 |
<mutante> |
hume /usr/local/apache is out of disk (just 5GB but more branches now). (LVM vg "tank" lv "tank-apache" ) but no free extents. could take from /archive but unsure about shrinking the xfs. |
[production] |
| 15:35 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5 |
[production] |
| 15:30 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf5 |
[production] |
| 15:28 |
<logmsgbot_> |
reedy synchronized php-1.20wmf5/ |
[production] |
| 15:20 |
<Reedy> |
running sync-dir php-1.20wmf5 |
[production] |
| 15:13 |
<Reedy> |
Copying checkout of 1.20wmf5 onto NFS |
[production] |
| 14:53 |
<mutante> |
running puppet on stat1. installs plotting packages |
[production] |
| 09:56 |
<apergos> |
shut down mysqld on db1047, reparing tables |
[production] |
| 06:54 |
<pp-pdf2> |
upgraded mwlib to 0.13.8 |
[production] |
| 06:54 |
<pp-pdf3> |
upgraded mwlib to 0.13.8 |
[production] |
| 06:54 |
<pp-pdf1> |
upgraded mwlib to 0.13.8 |
[production] |
| 02:24 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf4) at Mon Jun 11 02:24:06 UTC 2012 |
[production] |
|
2012-06-08
§
|
| 20:32 |
<Reedy> |
Updated php to point to php-1.20wmf4 rather than php-1.20wmf3 |
[production] |
| 20:09 |
<logmsgbot_> |
reedy synchronized wikimedia.dblist |
[production] |
| 20:08 |
<logmsgbot_> |
reedy synchronized all.dblist |
[production] |
| 20:04 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 2 wikimedia wikis to 1.20wmf4 |
[production] |
| 19:57 |
<logmsgbot_> |
reedy Finished syncing Wikimedia installation... : Rebuilding localisation cache for message updates |
[production] |
| 19:39 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for message updates |
[production] |
| 18:47 |
<logmsgbot_> |
reedy synchronized php-1.20wmf4/languages/messages/ 'Pushing out updated files upon siebrands request' |
[production] |
| 16:30 |
<cmjohnson1> |
shutting down search32 to swap DIMM around |
[production] |
| 10:19 |
<notpeter> |
ganglia down, restarting apache on nickel. |
[production] |
| 09:32 |
<notpeter> |
stopping indexing on searchidx1001 to re-copy to searchidx2 |
[production] |
| 08:45 |
<notpeter> |
reimaging searchidx2 with correct partitioning |
[production] |
| 06:50 |
<Tim> |
on cp1001: disabled HTCP plugin in gmond for testing, seems to work so I will disable it properly |
[production] |
| 03:55 |
<Tim> |
disabled LastModified extension due to overload on cp1005 |
[production] |
| 03:51 |
<logmsgbot_> |
tstarling synchronized wmf-config/InitialiseSettings.php |
[production] |
| 03:18 |
<Tim> |
restarting squid on cp1005, maybe out of FDs or something, cachemgr shows exactly 1000 open connections to 10.2.1.1 |
[production] |
| 03:08 |
<Tim> |
stopped gmond on cp1001 with kill -STOP for memory leak debugging |
[production] |
| 03:02 |
<Tim> |
on cp1002: killed gmond again, it was leaking memory again, already up to 27GB in the few minutes since I restarted it |
[production] |
| 03:01 |
<Tim> |
on fenari: copied *.text and *.upload from /home/wikipedia/conf/squid/generated/clusters to /etc/dsh/group |
[production] |
| 02:56 |
<Tim> |
cp1001: same as on cp1002, restarted gmond |
[production] |
| 02:53 |
<Tim> |
on cp1002: killed gmond, which was using 100% CPU and 23GB RSS. Restarting squid which had died |
[production] |