|
2012-04-20
§
|
| 19:45 |
<apergos> |
powercycled mw4, it was unresponsive to pings and via mgmt |
[production] |
| 18:48 |
<logmsgbot_> |
catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449' |
[production] |
| 18:48 |
<logmsgbot_> |
catrope synchronized php-1.19/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449' |
[production] |
| 18:07 |
<notpeter> |
restarting nginx on ssl1002 and ssl1004 as they are not back up |
[production] |
| 18:01 |
<logmsgbot_> |
awjrichards synchronizing Wikimedia installation... : |
[production] |
| 17:31 |
<logmsgbot_> |
catrope synchronized wmf-config/CommonSettings.php 'Remoev wgArticleFeedbackv5OversightEmails override that was messing things up' |
[production] |
| 17:15 |
<notpeter> |
stopping puppet on locke and emery. just to be safe... |
[production] |
| 17:11 |
<RoanKattouw> |
Fixed ownership of /h/w/common/php-1.20wmf1/cache/l10n , should be owned by l10nupdate but was owned by reedy |
[production] |
| 17:02 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36124 - Deploy ProofreadPage extension on test2' |
[production] |
| 17:00 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'Giving test2wiki moar namespaces' |
[production] |
| 16:11 |
<mutante> |
add missing memcached servicegroup to nagios, restarted |
[production] |
| 15:10 |
<mutante> |
apache error log on stafford has ruby exceptions re: phusion_passenger |
[production] |
| 15:01 |
<mark> |
Converted OSPF directly connected redistributed routes from type 2 to type 1 |
[production] |
| 14:51 |
<mutante> |
starting swift-container-auditor on ms-be1 |
[production] |
| 14:30 |
<mark> |
Disabled down-pref of Tampa AS2828 routes |
[production] |
| 13:14 |
<logmsgbot_> |
demon synchronized php-1.20wmf1/maintenance/backupTextPass.inc 'Pushing out Idb58ce27 for Ariel/Chris for dumps' |
[production] |
| 13:10 |
<mark> |
Sending India upload traffic to upload-lb.eqiad |
[production] |
| 12:40 |
<mark> |
Disabled iptables firewalls on internal prod swift cluster servers as it's dropping packets |
[production] |
| 12:22 |
<mutante> |
restarted pdns on ns2 |
[production] |
| 11:19 |
<mark> |
Sending US upload traffic to eqiad as well |
[production] |
| 10:27 |
<mark> |
Sending Brazil upload traffic to eqiad |
[production] |
| 08:39 |
<hashar> |
Gave up running l10nupdate script it has some file permissions issues. Opened {{bug|36119}} and {{bug|36120}} |
[production] |
| 08:36 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 08:36:53 UTC 2012 |
[production] |
| 08:27 |
<logmsgbot_> |
LocalisationUpdate completed (1.19) at Fri Apr 20 08:27:36 UTC 2012 |
[production] |
| 08:13 |
<hashar> |
rerunning l10nupdate for {{bug|34938}} |
[production] |
| 08:02 |
<hashar> |
running l10nupdate for {{bug|34938}} |
[production] |
| 06:27 |
<pgehres> |
re-eanabled PayPal on donatewiki and wmfwiki and resumed queue consumer on Aluminium |
[production] |
| 05:32 |
<LeslieCarr> |
flushing mobile varnish cache |
[production] |
| 04:56 |
<pgehres> |
disabled paypal on donatewiki and disabled queue consumer for duration of PayPal outage |
[production] |
| 02:33 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 02:33:02 UTC 2012 |
[production] |
| 02:23 |
<logmsgbot_> |
LocalisationUpdate completed (1.19) at Fri Apr 20 02:23:57 UTC 2012 |
[production] |
| 01:47 |
<logmsgbot_> |
awjrichards synchronizing Wikimedia installation... : [[rev:114983|r114983]] on wikis still running 1.19 |
[production] |
|
2012-04-19
§
|
| 23:33 |
<binasher> |
powercycled es1004 |
[production] |
| 21:08 |
<Jeff_Green> |
changed nagios contactgroup fundraising from tfinc/awrichards --> jgreen |
[production] |
| 21:03 |
<RoanKattouw> |
Scap is broken in some weird way, it just stops running after the scap1-skins step. Doesn't run scap-1 (which does the actual sync), doesn't log "sync done", doesn't update graphite |
[production] |
| 21:01 |
<logmsgbot_> |
catrope synchronizing Wikimedia installation... : Running scap again, AFTv5 is acting up |
[production] |
| 19:34 |
<logmsgbot_> |
catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates |
[production] |
| 19:29 |
<RoanKattouw> |
Running scap to deploy AFTv5 updates, and running AFTv5 schema changes on enwiki at the same time |
[production] |
| 18:50 |
<logmsgbot_> |
catrope synchronized wmf-config/InitialiseSettings.php 'Set wmgArticleFeedbackv5OversightEmails for enwiki' |
[production] |
| 18:25 |
<notpeter> |
nothing obvious in logs on db1005, starting mysql |
[production] |
| 18:15 |
<notpeter> |
rebooting db1005. it's dead, jim. |
[production] |
| 17:53 |
<RoanKattouw> |
Running schema changes for AFTv5 on testwiki |
[production] |
| 17:52 |
<Jeff_Green> |
discovered nfs1 had ~1K redundant iptables rules, removed extras and reloaded |
[production] |
| 17:42 |
<Jeff_Green> |
discovered sanger had ~7K redundant iptables rules, removed extras and reloaded |
[production] |
| 08:35 |
<mutante> |
emery - "udp2log_age" says some squid logfiles have not been written to in 6 hours, but from the filenames looks like this isnt a reason to worry, right |
[production] |
| 07:49 |
<mutante> |
stat1 - this also needs udp2log stuff fixed. currently Could not find class misc::udp2log::udp-filter |
[production] |
| 07:47 |
<mutante> |
gilman - what's up with it? closes SSH, does not like mgmt pass, was running jenkins but broken |
[production] |
| 07:43 |
<mutante> |
owa[1-3] They dont have real puppet freshness issues, it's rather firewalling and the snmp traps |
[production] |
| 02:30 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf1) at Thu Apr 19 02:30:33 UTC 2012 |
[production] |
| 02:21 |
<logmsgbot_> |
LocalisationUpdate completed (1.19) at Thu Apr 19 02:21:31 UTC 2012 |
[production] |