|
2012-04-23
§
|
| 14:18 |
<notpeter> |
temp stopping puppet on cp1042-1044 |
[production] |
| 13:09 |
<mutante> |
powercycling frozen mw25, looks like mw21 above but no console output to paste here |
[production] |
| 13:07 |
<mutante> |
fix puppet run on spence by removing searchidx1 resources from db9 (was in weird state being in site but also decommissioned) |
[production] |
| 11:23 |
<mutante> |
mw21 powercycling mw21 - it died with this http://etherpad.wikimedia.org/mw21 |
[production] |
| 10:55 |
<mutante> |
force-reload ircecho on manganese to make gerrit-wm rejoin #mediawiki |
[production] |
| 10:48 |
<hashar> |
banned CIA bots from #mediawiki IRC channel. It started spamming us with notifications from KDE and mandriva projects. See http://permalink.gmane.org/gmane.science.linguistics.wikipedia.technical/60905 |
[production] |
| 10:30 |
<mutante> |
searchidx1 was in site.pp and decom.pp at the same time. breaks puppet runs on spence. cannot override local resource. removing from site |
[production] |
| 10:27 |
<mutante> |
killed a couple morebots processes on wikitech and it came back by itself :p |
[production] |
|
2012-04-20
§
|
| 22:03 |
<logmsgbot_> |
aaron synchronized wmf-config/CommonSettings.php 'Switched test2wiki to use the new LocalRepo config style.' |
[production] |
| 22:01 |
<logmsgbot_> |
aaron synchronized wmf-config/CommonSettings.php 'Switched testwiki to use the new LocalRepo config style.' |
[production] |
| 21:52 |
<logmsgbot_> |
aaron synchronized wmf-config/CommonSettings.php 'Added NFS backends for local/shared repos; they are not used yet.' |
[production] |
| 21:12 |
<LeslieCarr> |
starting swift delete script on ms-be2 |
[production] |
| 20:02 |
<logmsgbot_> |
aaron synchronized php-1.20wmf1/includes/filerepo/file/LocalFile.php 'deployed c77fbd394cda701758ad4523113f567bff7ede66' |
[production] |
| 19:45 |
<apergos> |
powercycled mw4, it was unresponsive to pings and via mgmt |
[production] |
| 18:48 |
<logmsgbot_> |
catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449' |
[production] |
| 18:48 |
<logmsgbot_> |
catrope synchronized php-1.19/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449' |
[production] |
| 18:07 |
<notpeter> |
restarting nginx on ssl1002 and ssl1004 as they are not back up |
[production] |
| 18:01 |
<logmsgbot_> |
awjrichards synchronizing Wikimedia installation... : |
[production] |
| 17:31 |
<logmsgbot_> |
catrope synchronized wmf-config/CommonSettings.php 'Remoev wgArticleFeedbackv5OversightEmails override that was messing things up' |
[production] |
| 17:15 |
<notpeter> |
stopping puppet on locke and emery. just to be safe... |
[production] |
| 17:11 |
<RoanKattouw> |
Fixed ownership of /h/w/common/php-1.20wmf1/cache/l10n , should be owned by l10nupdate but was owned by reedy |
[production] |
| 17:02 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36124 - Deploy ProofreadPage extension on test2' |
[production] |
| 17:00 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'Giving test2wiki moar namespaces' |
[production] |
| 16:11 |
<mutante> |
add missing memcached servicegroup to nagios, restarted |
[production] |
| 15:10 |
<mutante> |
apache error log on stafford has ruby exceptions re: phusion_passenger |
[production] |
| 15:01 |
<mark> |
Converted OSPF directly connected redistributed routes from type 2 to type 1 |
[production] |
| 14:51 |
<mutante> |
starting swift-container-auditor on ms-be1 |
[production] |
| 14:30 |
<mark> |
Disabled down-pref of Tampa AS2828 routes |
[production] |
| 13:14 |
<logmsgbot_> |
demon synchronized php-1.20wmf1/maintenance/backupTextPass.inc 'Pushing out Idb58ce27 for Ariel/Chris for dumps' |
[production] |
| 13:10 |
<mark> |
Sending India upload traffic to upload-lb.eqiad |
[production] |
| 12:40 |
<mark> |
Disabled iptables firewalls on internal prod swift cluster servers as it's dropping packets |
[production] |
| 12:22 |
<mutante> |
restarted pdns on ns2 |
[production] |
| 11:19 |
<mark> |
Sending US upload traffic to eqiad as well |
[production] |
| 10:27 |
<mark> |
Sending Brazil upload traffic to eqiad |
[production] |
| 08:39 |
<hashar> |
Gave up running l10nupdate script it has some file permissions issues. Opened {{bug|36119}} and {{bug|36120}} |
[production] |
| 08:36 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 08:36:53 UTC 2012 |
[production] |
| 08:27 |
<logmsgbot_> |
LocalisationUpdate completed (1.19) at Fri Apr 20 08:27:36 UTC 2012 |
[production] |
| 08:13 |
<hashar> |
rerunning l10nupdate for {{bug|34938}} |
[production] |
| 08:02 |
<hashar> |
running l10nupdate for {{bug|34938}} |
[production] |
| 06:27 |
<pgehres> |
re-eanabled PayPal on donatewiki and wmfwiki and resumed queue consumer on Aluminium |
[production] |
| 05:32 |
<LeslieCarr> |
flushing mobile varnish cache |
[production] |
| 04:56 |
<pgehres> |
disabled paypal on donatewiki and disabled queue consumer for duration of PayPal outage |
[production] |
| 02:33 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 02:33:02 UTC 2012 |
[production] |
| 02:23 |
<logmsgbot_> |
LocalisationUpdate completed (1.19) at Fri Apr 20 02:23:57 UTC 2012 |
[production] |
| 01:47 |
<logmsgbot_> |
awjrichards synchronizing Wikimedia installation... : [[rev:114983|r114983]] on wikis still running 1.19 |
[production] |