|
2012-03-29
§
|
| 17:59 |
<RobH> |
search1021 coming back up, done with tests |
[production] |
| 17:53 |
<RobH> |
search1021 coming down for ssd fit test |
[production] |
| 17:07 |
<notpeter> |
disabling notifications for search lvs nagios checks for 24 hours to test fix |
[production] |
| 15:42 |
<notpeter> |
finished clearning up all pmtpa search hosts. hey look! they all have lots of space now! |
[production] |
| 15:15 |
<notpeter> |
restarting lsearchd on search3 |
[production] |
| 15:02 |
<RobH> |
brewster puppet re-enabled |
[production] |
| 15:02 |
<RobH> |
virt1001 pxe boots via dhcp and fails tftp download, i have to hold off on further troubleshooting until i have a network admin |
[production] |
| 14:47 |
<RobH> |
did virt1001 wrong, reupdating dns |
[production] |
| 14:39 |
<RobH> |
all nameservers still online after udpate |
[production] |
| 14:37 |
<RobH> |
updating dns for virt1001 testing |
[production] |
| 14:29 |
<RobH> |
stopping puppet runs on brewster so my hacking at the dhcpd.conf file won't get overwritten until I have it working right |
[production] |
| 14:01 |
<Jeff_Green> |
restarted varnish on on cp3002 because it was thrashing futiley |
[production] |
| 13:45 |
<notpeter> |
rebooting (mostly) down cp3001 |
[production] |
| 13:17 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Add participation namespace to metawiki per request' |
[production] |
| 13:11 |
<notpeter> |
trimming logs and such on search1-20 |
[production] |
| 09:59 |
<mutante> |
srv221, disabling ureadahead, installing package upgrades and new kernel, rebooting |
[production] |
| 09:40 |
<mutante> |
kill and start lsearchd on search7 |
[production] |
| 09:36 |
<mutante> |
restarted defunct lsearchd on search6 |
[production] |
| 09:10 |
<mutante> |
gallium - added demon,hashar,reedy to group jenkins as it's a problem using puppet when users and groups already exist |
[production] |
| 06:25 |
<mutante> |
powercycling sq40 |
[production] |
| 06:21 |
<mutante> |
installed more package upgrades on sodium |
[production] |
| 05:58 |
<mutante> |
installed security upgrades on brewser, cadmium, capella (apache,mysql,ruby,apt..) |
[production] |
| 05:49 |
<mutante> |
db42 - mysql did not autostart after boot, added using update-rc.d |
[production] |
| 05:42 |
<mutante> |
db42 - reboot worked despite the grub warning about unreliable blocklists |
[production] |
| 05:37 |
<mutante> |
rebooting db42 to finish upgrades |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Thu Mar 29 02:17:53 UTC 2012 |
[production] |
|
2012-03-28
§
|
| 23:27 |
<Tim> |
running apt-get upgrade on mw22,mw66,srv193,srv250,srv253,srv236 |
[production] |
| 23:25 |
<Tim> |
cleaned up stuck apt-get process on srv236 |
[production] |
| 23:22 |
<Tim> |
cleaned up stuck apt-get processes on mw22,mw66,srv193,srv250,srv253 |
[production] |
| 21:44 |
<awjrichards> |
synchronized wmf-config/CommonSettings.php 'Bumping mobile frontend resrouce version' |
[production] |
| 21:43 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/javascripts/application.min.js '[[rev:114576|r114576]]' |
[production] |
| 21:43 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/javascripts/banner.min.js '[[rev:114576|r114576]]' |
[production] |
| 21:43 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFrontend.body.php '[[rev:114576|r114576]]' |
[production] |
| 21:43 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php '[[rev:114576|r114576]]' |
[production] |
| 21:42 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/javascripts/application.js '[[rev:114576|r114576]]' |
[production] |
| 21:42 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/javascripts/banner.js '[[rev:114576|r114576]]' |
[production] |
| 20:43 |
<LocalisationUpdate> |
completed (1.19) at Wed Mar 28 20:43:20 UTC 2012 |
[production] |
| 20:29 |
<notpeter> |
restarted search1020. nothing conspicuous in logs |
[production] |
| 19:56 |
<RoanKattouw> |
Running a patched version of l10nupdate that rebuilds the localization cache |
[production] |
| 18:49 |
<catrope> |
synchronizing Wikimedia installation... : Bugfixes for ArticleFeedbackv5, ArticleFeedback and ClickTracking |
[production] |
| 16:47 |
<cmjohnson1> |
msw1-d1-pmtpa replacement complete |
[production] |
| 16:34 |
<cmjohnson1> |
replacing msw-d1-pmtpa per rt2639 |
[production] |
| 15:36 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech' |
[production] |
| 15:34 |
<Reedy> |
srv221 is full |
[production] |
| 15:34 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech' |
[production] |
| 14:39 |
<RobH> |
restarted morebots in screen on wikitech, no longer as catrope, as roan has root on that box |
[production] |
| 14:36 |
<RobH> |
got virt1001 to pxe, but dhcp doesnt know how to handle, need subnet details. |
[production] |
| 14:34 |
<notpeter> |
lucene hosed on search9 and search15. restarting, then will look after cause |
[production] |
| 13:14 |
<Jeff_Green> |
restarting puppet/puppetmaster on stafford to experiment with report settings |
[production] |
| 02:10 |
<LocalisationUpdate> |
completed (1.19) at Wed Mar 28 02:10:34 UTC 2012 |
[production] |