|
2012-04-16
§
|
| 18:37 |
<mutante> |
manually added iptables nat rules on nfs2 |
[production] |
| 18:13 |
<notpeter> |
upgrade of udp2log on nfs1/2 complete. should be operating normally now. |
[production] |
| 17:41 |
<mutante> |
LDAP on nfs2 warnings - opendj was _just_ started there when puppet was fixed with an unrelated issue |
[production] |
| 17:38 |
<mutante> |
restarting opendj on nfs2 because it refused connections |
[production] |
| 17:08 |
<logmsgbot_> |
preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ 'zero and mobile changes' |
[production] |
| 16:07 |
<notpeter> |
upgrading and restarting udp2log on nfs1/2 |
[production] |
| 15:04 |
<mutante> |
puppet fresh on nfs[12] after removing nonexistent misc::mediawiki-logger class |
[production] |
| 14:46 |
<mark> |
Shutdown db24 for memory testing by Chris |
[production] |
| 13:27 |
<mark> |
Sending European bits traffic back to pmtpa |
[production] |
| 12:24 |
<mark> |
Sending European bits traffic back to esams |
[production] |
| 12:06 |
<mark> |
Testing sess_leak_fix2 patch with a snapshot varnish build on cp3001 |
[production] |
| 11:56 |
<Reedy> |
Ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -- "cd /usr/local/apache/common && sudo -u mwdeploy ln -s php php-1.18" to create symlink for php-1.18 |
[production] |
| 11:51 |
<Reedy> |
Killing php-1.18 again |
[production] |
| 11:48 |
<mutante> |
sq34 - System halted! Error: Internal Storage Slot, powered down, -> RT |
[production] |
| 11:45 |
<logmsgbot_> |
reedy synchronized php-1.18/ 'Symlink php-1.18 back to php (our current main running version) as lots of requests on bits are for 1.18 resources' |
[production] |
| 11:44 |
<mutante> |
sq34 was broken and died when connecting to mgmt, powercycling |
[production] |
| 11:37 |
<mutante> |
nfs1 - Could not find class misc::mediawiki-logger for nfs1 |
[production] |
| 10:57 |
<Krinkle> |
bits.wikimedia.org back up, mark fixed it. |
[production] |
| 10:33 |
<Krinkle> |
bits.wikimedia.org serving Error 503 Service Unavailable on all load.php requests for mediawiki.org and nl.wikipedia.org, maybe more |
[production] |
| 09:45 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnableJavaScriptTest to true for test2wiki' |
[production] |
| 02:26 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf1) at Mon Apr 16 02:26:58 UTC 2012 |
[production] |
| 02:18 |
<logmsgbot_> |
LocalisationUpdate completed (1.19) at Mon Apr 16 02:17:57 UTC 2012 |
[production] |
|
2012-04-14
§
|
| 18:14 |
<mark> |
Shifting european bits traffic back from esams to pmtpa, session leak is still there |
[production] |
| 17:08 |
<mark> |
Shifting european bits traffic back from pmtpa to esams |
[production] |
| 15:31 |
<mark> |
Reverted varnish to 3.0.2-2wm4 on cp3001; the race condition patch did not fix the problem |
[production] |
| 14:56 |
<mark> |
Sending European bits traffic to pmtpa for testing |
[production] |
| 13:52 |
<mark> |
Backported varnish bug #897 patch to varnish 3.0.2, testing a snapshot build on cp3001 |
[production] |
| 11:37 |
<mark> |
Raised session_max to 300000 (runtime) on cp3001/cp3002 |
[production] |
| 05:58 |
<K4-713> |
re-enabled the queue consumer on aluminium |
[production] |
| 02:26 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf1) at Sat Apr 14 02:26:55 UTC 2012 |
[production] |
| 02:17 |
<logmsgbot_> |
LocalisationUpdate completed (1.19) at Sat Apr 14 02:17:34 UTC 2012 |
[production] |
| 02:16 |
<K4-713> |
updated prod civi to [[rev:1616|r1616]] |
[production] |
| 01:36 |
<K4-713> |
turned off queue consumption on prod civicrm |
[production] |
| 01:36 |
<K4-713> |
updated production civicrm to [[rev:1614|r1614]] |
[production] |
|
2012-04-13
§
|
| 20:53 |
<mark> |
Rebooting cp3002 |
[production] |
| 20:37 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFrontend.body.php '[[rev:114889|r114889]]' |
[production] |
| 17:54 |
<Jeff_Green> |
created new repo operations/debs/wikimedia-search-qa to stay within package naming conventions |
[production] |
| 17:31 |
<notpeter> |
upgrading udplog on locke to 1.8-2 and restarting, etc |
[production] |
| 17:27 |
<Jeff_Green> |
created new operations/debs/search-qa repo for packaging search qa scripts |
[production] |
| 17:17 |
<notpeter> |
restarting udp2log on emery |
[production] |
| 12:53 |
<notpeter> |
restopping puppet on locke/emery |
[production] |
| 12:09 |
<mark> |
Deploying varnish 3.0.2-2wm4 and enabling persistent storage on all even numbered eqiad upload varnish hosts |
[production] |
| 11:46 |
<mark> |
Imported varnish 3.0.2-2wm4 into the Wikimedia APT repository |
[production] |
| 02:48 |
<tstarling> |
synchronizing Wikimedia installation... : |
[production] |
| 02:39 |
<LocalisationUpdate> |
completed (1.20wmf1) at Fri Apr 13 02:39:01 UTC 2012 |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.19) at Fri Apr 13 02:20:35 UTC 2012 |
[production] |