|
2013-09-24
ยง
|
| 18:54 |
<cmjohnson> |
apache-graceful on mw1046 |
[production] |
| 18:53 |
<bblack> |
varnish updated to -wm16 on all eqiad mobile caches now, yell at me if something starts looking "off" there... |
[production] |
| 18:53 |
<cmjohnson> |
sync'ing mw1046 |
[production] |
| 18:43 |
<bsitu> |
Started syncing Wikimedia installation... : Update Echo and Thanks to master |
[production] |
| 18:09 |
<bblack> |
upgraded varnish pkg to -wm16 on cp1046 |
[production] |
| 17:59 |
<RobH> |
bast4001 reinstall |
[production] |
| 17:58 |
<manybubbles> |
finished building search indexes on closed wikis |
[production] |
| 17:49 |
<LeslieCarr> |
Hurricane electric peering moved to 10GE port. the spice packets must flow... |
[production] |
| 17:14 |
<MaxSem> |
Recreating GeoData index |
[production] |
| 16:38 |
<manybubbles> |
rebuilt search indecies in place for all other wikis runnig cirrus except cawiki and enwikisource. skipped them because they are already up to date. |
[production] |
| 16:27 |
<cmjohnson> |
adding mw1046 back to pybal |
[production] |
| 16:24 |
<manybubbles> |
rebuilding search indecies in place for all other wikis running cirrus |
[production] |
| 16:19 |
<mutante> |
DNS update, add wikipediastories domains |
[production] |
| 16:18 |
<mutante> |
sync-apache, graceful for wikipediastories |
[production] |
| 16:16 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable CleanChanges on meta' |
[production] |
| 16:10 |
<manybubbles> |
building search indecies for all closed wikis |
[production] |
| 16:06 |
<manybubbles> |
synchronized wmf-config/InitialiseSettings.php 'Cirrus on closed wikis' |
[production] |
| 15:25 |
<demon> |
synchronized php-1.22wmf18/extensions/CirrusSearch |
[production] |
| 15:24 |
<mutante> |
freshly activated http://wikimediastories.org/ (and .com,.net) |
[production] |
| 15:21 |
<mutante> |
sync-apache and graceful-all for wikimediastories and other cleanup |
[production] |
| 15:21 |
<yurik> |
synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/ |
[production] |
| 15:13 |
<mutante> |
during sync-apache: mw1072 - read-only file system |
[production] |
| 15:07 |
<cmjohnson> |
db1053 removing/replacing disk 3 |
[production] |
| 15:05 |
<yurik> |
synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/ |
[production] |
| 14:59 |
<cmjohnson> |
db1001 removing/replacing disk 7 |
[production] |
| 14:48 |
<cmjohnson> |
db1031 removing/replacing disk 7 |
[production] |
| 14:38 |
<cmjohnson> |
replacing mw1046 hard drive --requires reinstall |
[production] |
| 14:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Sep 24 14:21:10 UTC 2013 |
[production] |
| 14:15 |
<LocalisationUpdate> |
completed (1.22wmf17) at Tue Sep 24 14:15:56 UTC 2013 |
[production] |
| 14:15 |
<LocalisationUpdate> |
completed (1.22wmf18) at Tue Sep 24 14:15:16 UTC 2013 |
[production] |
| 14:13 |
<demon> |
synchronized php-1.22wmf18/extensions/Elastica 'Sync message fixes' |
[production] |
| 12:26 |
<mutante> |
shutting down sq45 |
[production] |
| 11:04 |
<mutante> |
powercycled sq45 |
[production] |
| 10:08 |
<p858snake|l> |
[20:01AEST] <mark> upped to weight 5 |
[production] |
| 10:08 |
<p858snake|l> |
[19:51AEST] <mark> !log Pooled Varnish Text caches in eqiad with weight 1 (affecting wikidata, wikivoyage, *.wikimedia.org) |
[production] |
| 10:07 |
<p858snake|l> |
[19:36AEST] <akosiaris> !log rebooted labstore4, controller configuration lost ? |
[production] |
| 10:07 |
<p858snake|l> |
[19:36AEST] <akosiaris> !log rebooted labstore3 (thought the 14days NFS issue was resolved???) |
[production] |
| 10:04 |
<akosiaris> |
rebooted labstore4, controller configuration lost ? |
[production] |
| 10:04 |
<akosiaris> |
rebooted labstore3 (thought the 14days NFS issue was resolved???) |
[production] |
| 08:30 |
<hashar> |
restarting Jenkins again for one last plugin upgrade |
[production] |
| 08:30 |
<springle> |
labstore4 disk issues, /a failed mount in console, ssh key bouncing. awaiting input from ryan or coren |
[production] |
| 08:30 |
<hashar> |
8:20am INFO: Jenkins is fully up and running |
[production] |
| 08:20 |
<hashar> |
restarting Jenkins |
[production] |
| 08:12 |
<hashar> |
jenkins: upgrading plugins |
[production] |
| 08:12 |
<hashar> |
8:12am INFO: Jenkins is fully up and running |
[production] |
| 08:06 |
<springle> |
labstore4 locked up, power cycled |
[production] |
| 07:43 |
<hashar> |
upgrading Jenkins on gallium from 1.509.2 to 1.509.3 |
[production] |
| 06:36 |
<springle> |
upgrading db1002 to precise and mariadb |
[production] |
| 06:13 |
<^demon|zzz> |
ES: enwikisource & cawiki done indexing. Really need to revisit batch indexing, way too slow. |
[production] |
| 03:42 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1002 for upgrade, db1036 take up slack' |
[production] |