1001-1050 of 10000 results (14ms)
2012-04-20 §
20:02 <logmsgbot_> aaron synchronized php-1.20wmf1/includes/filerepo/file/LocalFile.php 'deployed c77fbd394cda701758ad4523113f567bff7ede66' [production]
19:45 <apergos> powercycled mw4, it was unresponsive to pings and via mgmt [production]
18:48 <logmsgbot_> catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449' [production]
18:48 <logmsgbot_> catrope synchronized php-1.19/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449' [production]
18:07 <notpeter> restarting nginx on ssl1002 and ssl1004 as they are not back up [production]
18:01 <logmsgbot_> awjrichards synchronizing Wikimedia installation... : [production]
17:31 <logmsgbot_> catrope synchronized wmf-config/CommonSettings.php 'Remoev wgArticleFeedbackv5OversightEmails override that was messing things up' [production]
17:15 <notpeter> stopping puppet on locke and emery. just to be safe... [production]
17:11 <RoanKattouw> Fixed ownership of /h/w/common/php-1.20wmf1/cache/l10n , should be owned by l10nupdate but was owned by reedy [production]
17:02 <logmsgbot_> reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36124 - Deploy ProofreadPage extension on test2' [production]
17:00 <logmsgbot_> reedy synchronized wmf-config/InitialiseSettings.php 'Giving test2wiki moar namespaces' [production]
16:11 <mutante> add missing memcached servicegroup to nagios, restarted [production]
15:10 <mutante> apache error log on stafford has ruby exceptions re: phusion_passenger [production]
15:01 <mark> Converted OSPF directly connected redistributed routes from type 2 to type 1 [production]
14:51 <mutante> starting swift-container-auditor on ms-be1 [production]
14:30 <mark> Disabled down-pref of Tampa AS2828 routes [production]
13:14 <logmsgbot_> demon synchronized php-1.20wmf1/maintenance/backupTextPass.inc 'Pushing out Idb58ce27 for Ariel/Chris for dumps' [production]
13:10 <mark> Sending India upload traffic to upload-lb.eqiad [production]
12:40 <mark> Disabled iptables firewalls on internal prod swift cluster servers as it's dropping packets [production]
12:22 <mutante> restarted pdns on ns2 [production]
11:19 <mark> Sending US upload traffic to eqiad as well [production]
10:27 <mark> Sending Brazil upload traffic to eqiad [production]
08:39 <hashar> Gave up running l10nupdate script it has some file permissions issues. Opened {{bug|36119}} and {{bug|36120}} [production]
08:36 <logmsgbot_> LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 08:36:53 UTC 2012 [production]
08:27 <logmsgbot_> LocalisationUpdate completed (1.19) at Fri Apr 20 08:27:36 UTC 2012 [production]
08:13 <hashar> rerunning l10nupdate for {{bug|34938}} [production]
08:02 <hashar> running l10nupdate for {{bug|34938}} [production]
06:27 <pgehres> re-eanabled PayPal on donatewiki and wmfwiki and resumed queue consumer on Aluminium [production]
05:32 <LeslieCarr> flushing mobile varnish cache [production]
04:56 <pgehres> disabled paypal on donatewiki and disabled queue consumer for duration of PayPal outage [production]
02:33 <logmsgbot_> LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 02:33:02 UTC 2012 [production]
02:23 <logmsgbot_> LocalisationUpdate completed (1.19) at Fri Apr 20 02:23:57 UTC 2012 [production]
01:47 <logmsgbot_> awjrichards synchronizing Wikimedia installation... : [[rev:114983|r114983]] on wikis still running 1.19 [production]
2012-04-19 §
23:33 <binasher> powercycled es1004 [production]
21:08 <Jeff_Green> changed nagios contactgroup fundraising from tfinc/awrichards --> jgreen [production]
21:03 <RoanKattouw> Scap is broken in some weird way, it just stops running after the scap1-skins step. Doesn't run scap-1 (which does the actual sync), doesn't log "sync done", doesn't update graphite [production]
21:01 <logmsgbot_> catrope synchronizing Wikimedia installation... : Running scap again, AFTv5 is acting up [production]
19:34 <logmsgbot_> catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates [production]
19:29 <RoanKattouw> Running scap to deploy AFTv5 updates, and running AFTv5 schema changes on enwiki at the same time [production]
18:50 <logmsgbot_> catrope synchronized wmf-config/InitialiseSettings.php 'Set wmgArticleFeedbackv5OversightEmails for enwiki' [production]
18:25 <notpeter> nothing obvious in logs on db1005, starting mysql [production]
18:15 <notpeter> rebooting db1005. it's dead, jim. [production]
17:53 <RoanKattouw> Running schema changes for AFTv5 on testwiki [production]
17:52 <Jeff_Green> discovered nfs1 had ~1K redundant iptables rules, removed extras and reloaded [production]
17:42 <Jeff_Green> discovered sanger had ~7K redundant iptables rules, removed extras and reloaded [production]
08:35 <mutante> emery - "udp2log_age" says some squid logfiles have not been written to in 6 hours, but from the filenames looks like this isnt a reason to worry, right [production]
07:49 <mutante> stat1 - this also needs udp2log stuff fixed. currently Could not find class misc::udp2log::udp-filter [production]
07:47 <mutante> gilman - what's up with it? closes SSH, does not like mgmt pass, was running jenkins but broken [production]
07:43 <mutante> owa[1-3] They dont have real puppet freshness issues, it's rather firewalling and the snmp traps [production]
02:30 <logmsgbot_> LocalisationUpdate completed (1.20wmf1) at Thu Apr 19 02:30:33 UTC 2012 [production]