|
2012-03-06
§
|
| 20:22 |
<notpeter> |
temp stopping puppet on brewster |
[production] |
| 20:21 |
<reedy> |
synchronized php-1.19/resources/mediawiki.action/mediawiki.action.edit.js '[[rev:113175|r113175]]' |
[production] |
| 20:20 |
<reedy> |
synchronized php-1.19/maintenance/populateRevisionSha1.php '[[rev:113175|r113175]]' |
[production] |
| 20:19 |
<reedy> |
synchronized php-1.19/includes/specials/SpecialContributions.php '[[rev:113175|r113175]]' |
[production] |
| 20:18 |
<reedy> |
synchronized php-1.19/includes/specials/SpecialUserlogin.php '[[rev:113176|r113176]]' |
[production] |
| 20:00 |
<pp-pdf1> |
installed log-wikimedia-operations (which can be used for automated logging to #wikimedia-operations) |
[production] |
| 19:53 |
<Ryan_Lane> |
restarting labs mysql to allow for more connections |
[production] |
| 19:26 |
<Ryan_Lane> |
installing nova-api on virt0 |
[production] |
| 19:09 |
<Ryan_Lane> |
upping FLAGS.sql_max_pool_size for nova-api |
[production] |
| 18:47 |
<catrope> |
synchronized php-1.19/resources/startup.js 'touch' |
[production] |
| 18:46 |
<Ryan_Lane> |
rebooting all instances |
[production] |
| 18:34 |
<Ryan_Lane> |
restarting nova-network on virt2 |
[production] |
| 18:19 |
<Ryan_Lane> |
rebooting virt1 |
[production] |
| 18:15 |
<Ryan_Lane> |
rebooting virt2 |
[production] |
| 18:11 |
<Ryan_Lane> |
rebooting virt3 |
[production] |
| 18:07 |
<Ryan_Lane> |
rebooting virt4 |
[production] |
| 17:58 |
<Ryan_Lane> |
taking the opportunity to apply security updates to virt0-4 |
[production] |
| 16:25 |
<catrope> |
synchronized docroot/foundation/FrameResize.html 'Put Jobvite frame resize file in foundationwiki docroot per Erik' |
[production] |
| 11:40 |
<tstarling> |
rebuilt wikiversions.cdb and synchronized wikiversions files: switching sr* to 1.19 |
[production] |
| 11:15 |
<hashar> |
synchronized php-1.19/languages/messages/MessagesSa.php '[[rev:1113039|r1113039]] for [[bugzilla:34938|bug 34938]] : title is sometime empty on Sanskrit wikis' |
[production] |
| 11:13 |
<tstarling> |
synchronized php-1.19/includes/OutputPage.php '[[rev:113128|r113128]]' |
[production] |
| 10:41 |
<tstarling> |
rebuilt wikiversions.cdb and synchronized wikiversions files: switching zh* from 1.18 to 1.19 |
[production] |
| 08:36 |
<mutante> |
on hooper: puppet broken due to dependency Package[libapache2-mod-php5] for Service[apache2] |
[production] |
| 03:33 |
<mutante> |
rebooting bast1001 for kernel upgrade |
[production] |
| 03:32 |
<mutante> |
upgrading apache2 packages, base-files, kernel, several libs on bast1001 |
[production] |
| 03:27 |
<mutante> |
installing a couple upgrades on fenari (apache2-utils, update-manager-core, cvs, ruby, libxml*, libopenssl-ruby*...) |
[production] |
| 02:37 |
<LocalisationUpdate> |
completed (1.18) at Tue Mar 6 02:37:06 UTC 2012 |
[production] |
| 02:36 |
<tstarling> |
synchronizing Wikimedia installation... : updating to [[rev:113119|r113119]] |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Tue Mar 6 02:18:13 UTC 2012 |
[production] |
| 01:27 |
<Jeff_Green> |
manually updated packages and restarted apache on srv198, srv229, srv262, srv268, mw40 because their apache redirect configs failed to update after sync-apache and restart |
[production] |
| 01:07 |
<Jeff_Green> |
another adjustment to redirects.conf and apache-graceful-all for RT#2488 |
[production] |
|
2012-03-05
§
|
| 22:24 |
<Jeff_Green> |
modified redirects.conf per RT #2488 |
[production] |
| 21:21 |
<Reedy> |
Ran foreachwiki cleanupUploadStash.php |
[production] |
| 20:36 |
<maplebed> |
enabled swift for 100% of thumbnails in production |
[production] |
| 18:18 |
<nikerabbit> |
synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy [[rev:113058|r113058]]' |
[production] |
| 18:11 |
<nikerabbit> |
synchronized wmf-config/InitialiseSettings.php 'WebFonts: bugwiki [[bugzilla:34550|bug 34550]]; sawikisource [[bugzilla:34159|bug 34159]]; amwiktionary amwikiquote [[bugzilla:34700|bug 34700]]' |
[production] |
| 18:01 |
<mark> |
Raised MTU between cr1-sdtpa - (csw1-sdtpa) - cr2-pmtpa to 9192 |
[production] |
| 17:35 |
<Jeff_Green> |
removed 3GB db30:/tmp/gmond.log and force-restarted gmond b/c the init script failed to restart it |
[production] |
| 17:16 |
<Jeff_Green> |
adjusted LVS partitions on hume, moved /usr/local/apache to a new 5GB mount |
[production] |
| 15:18 |
<mark> |
Fixed DNS resolving on the core routers by allowing DNS replies in the loopback filter |
[production] |
| 14:44 |
<reedy> |
synchronized php-1.19/includes/Title.php '[[rev:113036|r113036]]' |
[production] |
| 14:44 |
<reedy> |
synchronized php-1.19/includes/AjaxResponse.php '[[rev:113036|r113036]]' |
[production] |
| 14:35 |
<reedy> |
synchronized php-1.19/extensions/CentralAuth/ '[[rev:113035|r113035]]' |
[production] |
| 14:34 |
<reedy> |
synchronized php-1.19/extensions/CategoryTree/ '[[rev:113035|r113035]]' |
[production] |
| 13:50 |
<mark> |
Set increased OSPF/OSPFv3 metric 30 on both directions of the link cr1-eqiad:xe-5/2/1 <--> cr1-sdtpa:xe-0/0/1, to combat higher than normal jitter and packet loss on the link |
[production] |
| 12:53 |
<mark> |
Upgraded observium to latest version |
[production] |
| 09:41 |
<mutante> |
restarting memcached on marmontel |
[production] |
| 09:40 |
<mutante> |
restarting squid backend on knsq25 |
[production] |
| 06:52 |
<Ryan_Lane> |
all of the instances are accessing the file descriptors of files inside of the _base directory, and fuse has an issue with this. gluster can't recreate the base directory because of the processes holding open the old one. |
[production] |
| 06:50 |
<Ryan_Lane> |
I've corrupted the _base directory on the instance's glusterfs share. I'm recovering the files from file descriptors using lsof. Not totally sure how I'm going to get the _base directory back, yet. |
[production] |