4951-5000 of 10000 results (34ms)
2012-04-23 §
13:09 <mutante> powercycling frozen mw25, looks like mw21 above but no console output to paste here [production]
13:07 <mutante> fix puppet run on spence by removing searchidx1 resources from db9 (was in weird state being in site but also decommissioned) [production]
11:23 <mutante> mw21 powercycling mw21 - it died with this http://etherpad.wikimedia.org/mw21 [production]
10:55 <mutante> force-reload ircecho on manganese to make gerrit-wm rejoin #mediawiki [production]
10:48 <hashar> banned CIA bots from #mediawiki IRC channel. It started spamming us with notifications from KDE and mandriva projects. See http://permalink.gmane.org/gmane.science.linguistics.wikipedia.technical/60905 [production]
10:30 <mutante> searchidx1 was in site.pp and decom.pp at the same time. breaks puppet runs on spence. cannot override local resource. removing from site [production]
10:27 <mutante> killed a couple morebots processes on wikitech and it came back by itself :p [production]
2012-04-21 §
02:29 <logmsgbot_> LocalisationUpdate completed (1.20wmf1) at Sat Apr 21 02:29:40 UTC 2012 [production]
02:15 <logmsgbot_> LocalisationUpdate completed (1.19) at Sat Apr 21 02:15:20 UTC 2012 [production]
2012-04-20 §
22:03 <logmsgbot_> aaron synchronized wmf-config/CommonSettings.php 'Switched test2wiki to use the new LocalRepo config style.' [production]
22:01 <logmsgbot_> aaron synchronized wmf-config/CommonSettings.php 'Switched testwiki to use the new LocalRepo config style.' [production]
21:52 <logmsgbot_> aaron synchronized wmf-config/CommonSettings.php 'Added NFS backends for local/shared repos; they are not used yet.' [production]
21:12 <LeslieCarr> starting swift delete script on ms-be2 [production]
20:02 <logmsgbot_> aaron synchronized php-1.20wmf1/includes/filerepo/file/LocalFile.php 'deployed c77fbd394cda701758ad4523113f567bff7ede66' [production]
19:45 <apergos> powercycled mw4, it was unresponsive to pings and via mgmt [production]
18:48 <logmsgbot_> catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449' [production]
18:48 <logmsgbot_> catrope synchronized php-1.19/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449' [production]
18:07 <notpeter> restarting nginx on ssl1002 and ssl1004 as they are not back up [production]
18:01 <logmsgbot_> awjrichards synchronizing Wikimedia installation... : [production]
17:31 <logmsgbot_> catrope synchronized wmf-config/CommonSettings.php 'Remoev wgArticleFeedbackv5OversightEmails override that was messing things up' [production]
17:15 <notpeter> stopping puppet on locke and emery. just to be safe... [production]
17:11 <RoanKattouw> Fixed ownership of /h/w/common/php-1.20wmf1/cache/l10n , should be owned by l10nupdate but was owned by reedy [production]
17:02 <logmsgbot_> reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36124 - Deploy ProofreadPage extension on test2' [production]
17:00 <logmsgbot_> reedy synchronized wmf-config/InitialiseSettings.php 'Giving test2wiki moar namespaces' [production]
16:11 <mutante> add missing memcached servicegroup to nagios, restarted [production]
15:10 <mutante> apache error log on stafford has ruby exceptions re: phusion_passenger [production]
15:01 <mark> Converted OSPF directly connected redistributed routes from type 2 to type 1 [production]
14:51 <mutante> starting swift-container-auditor on ms-be1 [production]
14:30 <mark> Disabled down-pref of Tampa AS2828 routes [production]
13:14 <logmsgbot_> demon synchronized php-1.20wmf1/maintenance/backupTextPass.inc 'Pushing out Idb58ce27 for Ariel/Chris for dumps' [production]
13:10 <mark> Sending India upload traffic to upload-lb.eqiad [production]
12:40 <mark> Disabled iptables firewalls on internal prod swift cluster servers as it's dropping packets [production]
12:22 <mutante> restarted pdns on ns2 [production]
11:19 <mark> Sending US upload traffic to eqiad as well [production]
10:27 <mark> Sending Brazil upload traffic to eqiad [production]
08:39 <hashar> Gave up running l10nupdate script it has some file permissions issues. Opened {{bug|36119}} and {{bug|36120}} [production]
08:36 <logmsgbot_> LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 08:36:53 UTC 2012 [production]
08:27 <logmsgbot_> LocalisationUpdate completed (1.19) at Fri Apr 20 08:27:36 UTC 2012 [production]
08:13 <hashar> rerunning l10nupdate for {{bug|34938}} [production]
08:02 <hashar> running l10nupdate for {{bug|34938}} [production]
06:27 <pgehres> re-eanabled PayPal on donatewiki and wmfwiki and resumed queue consumer on Aluminium [production]
05:32 <LeslieCarr> flushing mobile varnish cache [production]
04:56 <pgehres> disabled paypal on donatewiki and disabled queue consumer for duration of PayPal outage [production]
02:33 <logmsgbot_> LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 02:33:02 UTC 2012 [production]
02:23 <logmsgbot_> LocalisationUpdate completed (1.19) at Fri Apr 20 02:23:57 UTC 2012 [production]
01:47 <logmsgbot_> awjrichards synchronizing Wikimedia installation... : [[rev:114983|r114983]] on wikis still running 1.19 [production]
2012-04-19 §
23:33 <binasher> powercycled es1004 [production]
21:08 <Jeff_Green> changed nagios contactgroup fundraising from tfinc/awrichards --> jgreen [production]
21:03 <RoanKattouw> Scap is broken in some weird way, it just stops running after the scap1-skins step. Doesn't run scap-1 (which does the actual sync), doesn't log "sync done", doesn't update graphite [production]
21:01 <logmsgbot_> catrope synchronizing Wikimedia installation... : Running scap again, AFTv5 is acting up [production]