|
2012-08-16
§
|
| 16:52 |
<cmjohnson1> |
shutting down mw18 and mw19 to replace DIMM |
[production] |
| 16:29 |
<ottomata> |
Stopping puppet on brewster again. Sigh. PARTMAAAAN! |
[production] |
| 16:22 |
<reedy> |
synchronized php-1.20wmf9/includes/QueryPage.php |
[production] |
| 16:07 |
<cmjohnson1> |
shutting down mw8 to replace bad DIMM |
[production] |
| 15:59 |
<ottomata> |
starting puppet on brewster. (Woo, partman looking better!) |
[production] |
| 14:15 |
<ottomata> |
stopping puppet on brewster to continue partman troubleshooting for analytics dells |
[production] |
| 13:26 |
<mark> |
Enabled and started SIS deduplication on home_pmtpa on nas1-a |
[production] |
| 12:05 |
<mark> |
Removed OSPF metric on xe-5/2/1.0 on cr2-eqiad, to move eqiad->pmtpa traffic to the lower latency link |
[production] |
| 11:27 |
<pp-pdf2> |
restarted all services |
[production] |
| 11:27 |
<pp-pdf1> |
restarted all services |
[production] |
| 11:27 |
<pp-pdf3> |
restarted all services |
[production] |
| 11:26 |
<pp-pdf3> |
updated mwlib.epub to 0.14.1 |
[production] |
| 11:26 |
<pp-pdf2> |
updated mwlib.epub to 0.14.1 |
[production] |
| 11:25 |
<pp-pdf1> |
updated mwlib.epub to 0.14.1 |
[production] |
| 10:50 |
<mark> |
Setup semi-sync snapmirror from nas1-a:home_pmtpa to nas1001-a:home_pmtpa |
[production] |
| 10:29 |
<pp-pdf3> |
restarted all services |
[production] |
| 10:29 |
<pp-pdf1> |
restarted all services |
[production] |
| 10:29 |
<pp-pdf2> |
restarted all services |
[production] |
| 10:29 |
<pp-pdf1> |
update mwlib.epub to 0.14.0 |
[production] |
| 10:28 |
<pp-pdf2> |
update mwlib.epub to 0.14.0 |
[production] |
| 10:28 |
<pp-pdf3> |
update mwlib.epub to 0.14.0 |
[production] |
| 10:28 |
<pp-pdf2> |
update mwlib.rl to 0.14.0 |
[production] |
| 10:28 |
<pp-pdf1> |
update mwlib.rl to 0.14.0 |
[production] |
| 10:28 |
<pp-pdf3> |
update mwlib.rl to 0.14.0 |
[production] |
| 10:28 |
<pp-pdf2> |
update mwlib to 0.14.0 |
[production] |
| 10:28 |
<pp-pdf3> |
update mwlib to 0.14.0 |
[production] |
| 10:27 |
<pp-pdf1> |
update mwlib to 0.14.0 |
[production] |
| 03:57 |
<mutante> |
nagios back up after adding missing monitor groups misc_pmtpa in appserver role (srv194) |
[production] |
| 02:56 |
<K4-713> |
synchronized payments cluster to 37e31eddf5a4 |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.20wmf9) at Thu Aug 16 02:28:33 UTC 2012 |
[production] |
| 02:12 |
<Ryan_Lane> |
pushed in large puppet change for ldap, openstack, gerrit and ldap pdns to make it more modular and to add support for eqiad region |
[production] |
| 02:10 |
<Ryan_Lane> |
fixed arecord issues with labsconsole by adding an exception handling live hack for the jobs |
[production] |
| 01:21 |
<tstarling> |
synchronized wmf-config/InitialiseSettings.php 'remove wikimedia philippines event exemption' |
[production] |
| 00:41 |
<mutante> |
installing package upgrades on singer |
[production] |
| 00:36 |
<aaron> |
synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed c51a9a288b6dd5c0023a77f324c04707b23501c6' |
[production] |
| 00:24 |
<binasher> |
running "mwscript purgeParserCache.php --wiki=$db --age=1814400" instead |
[production] |
| 00:19 |
<binasher> |
running mwscript purgeParserCache.php --wiki=enwiki --age=1209600 |
[production] |
| 00:02 |
<RoanKattouw> |
find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec /root/fixownership2 \\{\\} \\; where fixownership2 = chown -h apache $1 |
[production] |
| 00:02 |
<RoanKattouw> |
Restarting the find-chown, this time with -h so symlinks are handled correctly (for some reason there are a bunch of broken symlinks with weird characters out there...) |
[production] |
|
2012-08-15
§
|
| 23:52 |
<binasher> |
powercycled srv259 |
[production] |
| 23:44 |
<mutante> |
brought nagios back up by removing "misc_pmtpa" host and servicegroups from srv194 manually |
[production] |
| 23:35 |
<mutante> |
restarted pdns-recursor on ns0 |
[production] |
| 23:11 |
<mutante> |
authdns-update, moving zirconium to public IP and removing selenium |
[production] |
| 22:33 |
<reedy> |
synchronized php-1.20wmf9/extensions/OpenSearchXml/ |
[production] |
| 21:19 |
<RoanKattouw> |
find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec chown apache \\{\\} \\; |
[production] |
| 21:17 |
<RoanKattouw> |
chown on ms7 has finished, running another find to find remaining bad files |
[production] |
| 20:57 |
<K4-713> |
synchronized payments cluster to 79275e152b76f |
[production] |
| 20:26 |
<ottomata> |
staring puppet on brewster. PARTMAN IS THE ENEMY. will continue the fight tomorrow. |
[production] |
| 20:22 |
<mutante> |
http://en.wikipedia.org/w/ now redirects to Main_page on all languages. It was a 403 before. |
[production] |
| 20:20 |
<mutante> |
apache-graceful-all |
[production] |