|
2011-11-03
§
|
| 19:33 |
<notpeter> |
putting srv290-srv301 back into api pybal pool |
[production] |
| 19:12 |
<catrope> |
synchronized php-1.18/README 'Dummy sync to confirm that sync logging is fixed now' |
[production] |
| 18:39 |
<awjr> |
disabling jenkins Donations queue consume job on grosley |
[production] |
| 17:50 |
<Jeff_Green> |
flipped DNS for civicrm and other fundraising services from grosley to aluminium |
[production] |
| 17:00 |
<Jeff_Green> |
stopping apache on grosley and aluminium |
[production] |
| 15:58 |
<notpeter> |
restarting nagios on spence |
[production] |
| 11:35 |
<Jeff_Green> |
stopping mysql on storage3 |
[production] |
| 10:30 |
<mutante> |
tried to apache-graceful-all to make them pick up config change, but getting a lot of "Too many authentication failures" |
[production] |
|
2011-11-02
§
|
| 23:50 |
<Ryan_Lane> |
reverted Roan's dsh.conf changes |
[production] |
| 23:47 |
<Ryan_Lane> |
updated Roan's changes to dsh.conf, wrapping the options in quotes |
[production] |
| 22:30 |
<Ryan_Lane> |
rebooting pad1.pmtpa.wmflabs |
[production] |
| 21:11 |
<RoanKattouw> |
Added -o SetupTimeout=10 to ssh options in /etc/dsh/dsh.conf |
[production] |
| 19:19 |
<notpeter> |
depooling srv290-srv301 for actual upgrades to lucid |
[production] |
| 19:14 |
<RoanKattouw> |
Running scap for real now, was broken. Fixed in puppet |
[production] |
| 19:04 |
<RoanKattouw> |
Scapping to deploy editor engagement changes |
[production] |
| 18:44 |
<notpeter> |
reimageing srv290, srv291, srv292 to lucid. this time for real |
[production] |
| 16:56 |
<notpeter> |
adding new version of php-wikidiff2 package to brewster (that will be pushed out to all apaches...) |
[production] |
| 16:44 |
<notpeter> |
installing new version of php-wikidiff2 on mw6 for testing |
[production] |
| 14:28 |
<RoanKattouw> |
(morebots had to be restarted too, got killed in a netsplit earlier) |
[production] |
| 14:28 |
<RoanKattouw> |
RoanKattouw\t!log logmsgbot will be broken until Ryan shows up to save us all. I just e-mailed him about it |
[production] |
| 14:27 |
<RoanKattouw> |
apergos\t!log restarted logmsgbot though it's apparently broken atm |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.18) at Wed Nov 2 02:21:36 UTC 2011 |
[production] |
|
2011-11-01
§
|
| 21:36 |
<awjr> |
Deployed thank_you module fixes to CiviCRM on grosley [[rev:696|r696]] |
[production] |
| 21:21 |
<Jeff_Green> |
changed donate.wikimedia.org from CNAME to A record so we can use MX records |
[production] |
| 21:13 |
<awjr> |
Configured DonationInterface to only load i18n files on the cluster, enabled on donatewiki and foundation (CommonSettings @ [[rev:2560|r2560]], InitialiseSettings @ [[rev:2561|r2561]]) |
[production] |
| 21:10 |
<awjr> |
Running scap to pick up new DonationInterface set up for the cluster and new message files ([[rev:101527|r101527]]) |
[production] |
| 19:55 |
<awjr> |
Fixed borked config change in CommonSettings for LandingCheck, [[rev:2559|r2559]] |
[production] |
| 19:53 |
<awjr> |
Config change to CommonSettings to make LandingCheck always point to foundationwiki for now [[rev:2558|r2558]] |
[production] |
| 19:45 |
<binasher> |
note, locke and emery changes only made to the running kernel for testing, not in /etc/sysctl* |
[production] |
| 19:43 |
<binasher> |
made the same kernel changes on emery as locke, udp packet loss dropped from 6% to 0 |
[production] |
| 19:33 |
<awjr> |
deploying LandingCheck updates (@ [[rev:101515|r101515]]) via sync-file as well as config change to CommonSettings.php (@ [[rev:2557|r2557]]) |
[production] |
| 18:52 |
<notpeter> |
restarting lighttpd on lily |
[production] |
| 18:43 |
<notpeter> |
restarting lighttpd on lily |
[production] |
| 18:30 |
<binasher> |
locke: disabled bbu checks, set per5 to writeback caching |
[production] |
| 18:15 |
<binasher> |
increased def recv buf on locke to 128M (just experimenting) |
[production] |
| 18:10 |
<RobH> |
mgmt fixed on mw1027. mw1033, mw1038, mw1039 |
[production] |
| 18:03 |
<Ryan_Lane> |
re-enabling LDAP plugin on labsconsole |
[production] |
| 18:02 |
<Ryan_Lane> |
temporarily disabling LDAP plugin on labsconsole to give admin rights to Andrew Bogott (stupid bugs) |
[production] |
| 17:49 |
<binasher> |
on locke: set rmem_default to 64M and rmem_max to 128M, retarted udp2log. packet loss has stopped per /proc/net/udp which is definitive vs. seq numbers |
[production] |
| 17:21 |
<RobH> |
mw1018 mw1021 firmware updated, mgmt working |
[production] |
| 16:52 |
<RoanKattouw> |
Synchronized php-1.18/cache/interwiki{,-pr}.cdb , updating interwiki cache for nsowiki addition ([[bugzilla:30882|bug 30882]]) |
[production] |
| 16:51 |
<RoanKattouw> |
sync-file logging is broken due to my changes in puppet, will get Ryan to deploy the fixes |
[production] |
| 15:28 |
<notpeter> |
scratch that, mw5 |
[production] |
| 15:26 |
<notpeter> |
trying again to replace php-wikidiff2 with php5-wikidiff2 on mw4. then graceful |
[production] |
| 14:32 |
<notpeter> |
restarting lighttpd on lily for rt 1752 |
[production] |
| 03:19 |
<Reedy> |
Bug 32042 - Narayam for Assamese Wikipedia |
[production] |
| 03:12 |
<Reedy> |
sync-file is giving "/usr/local/bin/dologmsg: 2: cannot create /var/log/logmsg: Permission denied" and not logging to IRC |
[production] |
| 03:11 |
<Reedy> |
Sync [[rev:101448|r101448]] |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.18) at Tue Nov 1 02:22:51 UTC 2011 |
[production] |
| 01:42 |
<awjr> |
Oops forgot to log config change - removed Hungary from list of priority countries for LandingCheck extension |
[production] |