|
2011-11-04
§
|
| 02:51 |
<Tim> |
stopping puppet and xinetd and disabling extdist crontab on fenari to update extensions directory |
[production] |
| 02:32 |
<Tim> |
on db10: removed the deleted relay logs from mysqld-relay-bin.index with a text editor, then ran "reset slave" and "change master" to get replication working again |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.18) at Fri Nov 4 02:29:26 UTC 2011 |
[production] |
| 02:16 |
<Tim> |
db10 ran out of disk space on /a with 27GB of relay logs created in the last 2 hours. It wouldn't respond to "stop slave" or shut down gracefully, so I had to delete the relay logs manually to free up enough space for a graceful shutdown |
[production] |
| 02:07 |
<Tim> |
fixing db10, is out of disk space |
[production] |
| 02:00 |
<awjr> |
Enabling ParserFunctions string functions on donatewiki (config [[rev:2566|r2566]] and [[rev:2567|r2567]]) |
[production] |
| 01:22 |
<awjr> |
Synchronizing payments [[rev:101939|r101939]] |
[production] |
|
2011-11-03
§
|
| 23:36 |
<tstarling> |
synchronized docroot/secure/keys.txt |
[production] |
| 23:31 |
<tstarling> |
synchronized docroot/secure/keys.html |
[production] |
| 22:25 |
<LeslieCarr> |
restarting powerdns on ns2 |
[production] |
| 22:11 |
<LeslieCarr> |
reinstalling mw1019 to test out systems |
[production] |
| 21:56 |
<awjr> |
Updating payments boxes to [[rev:101890|r101890]] |
[production] |
| 21:36 |
<awjr> |
reenabling Donations queue consume in Jenkins on aluminium |
[production] |
| 21:06 |
<awjrichards> |
synchronized wmf-config/contribution-tracking-setup.php |
[production] |
| 20:54 |
<nikerabbit> |
synchronized php-1.18/extensions/WebFonts/ 'static webfonts test' |
[production] |
| 20:34 |
<notpeter> |
restarted gammu-smsd on spence |
[production] |
| 19:33 |
<notpeter> |
putting srv290-srv301 back into api pybal pool |
[production] |
| 19:12 |
<catrope> |
synchronized php-1.18/README 'Dummy sync to confirm that sync logging is fixed now' |
[production] |
| 18:39 |
<awjr> |
disabling jenkins Donations queue consume job on grosley |
[production] |
| 17:50 |
<Jeff_Green> |
flipped DNS for civicrm and other fundraising services from grosley to aluminium |
[production] |
| 17:00 |
<Jeff_Green> |
stopping apache on grosley and aluminium |
[production] |
| 15:58 |
<notpeter> |
restarting nagios on spence |
[production] |
| 11:35 |
<Jeff_Green> |
stopping mysql on storage3 |
[production] |
| 10:30 |
<mutante> |
tried to apache-graceful-all to make them pick up config change, but getting a lot of "Too many authentication failures" |
[production] |
|
2011-11-02
§
|
| 23:50 |
<Ryan_Lane> |
reverted Roan's dsh.conf changes |
[production] |
| 23:47 |
<Ryan_Lane> |
updated Roan's changes to dsh.conf, wrapping the options in quotes |
[production] |
| 22:30 |
<Ryan_Lane> |
rebooting pad1.pmtpa.wmflabs |
[production] |
| 21:11 |
<RoanKattouw> |
Added -o SetupTimeout=10 to ssh options in /etc/dsh/dsh.conf |
[production] |
| 19:19 |
<notpeter> |
depooling srv290-srv301 for actual upgrades to lucid |
[production] |
| 19:14 |
<RoanKattouw> |
Running scap for real now, was broken. Fixed in puppet |
[production] |
| 19:04 |
<RoanKattouw> |
Scapping to deploy editor engagement changes |
[production] |
| 18:44 |
<notpeter> |
reimageing srv290, srv291, srv292 to lucid. this time for real |
[production] |
| 16:56 |
<notpeter> |
adding new version of php-wikidiff2 package to brewster (that will be pushed out to all apaches...) |
[production] |
| 16:44 |
<notpeter> |
installing new version of php-wikidiff2 on mw6 for testing |
[production] |
| 14:28 |
<RoanKattouw> |
(morebots had to be restarted too, got killed in a netsplit earlier) |
[production] |
| 14:28 |
<RoanKattouw> |
RoanKattouw\t!log logmsgbot will be broken until Ryan shows up to save us all. I just e-mailed him about it |
[production] |
| 14:27 |
<RoanKattouw> |
apergos\t!log restarted logmsgbot though it's apparently broken atm |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.18) at Wed Nov 2 02:21:36 UTC 2011 |
[production] |
|
2011-11-01
§
|
| 21:36 |
<awjr> |
Deployed thank_you module fixes to CiviCRM on grosley [[rev:696|r696]] |
[production] |
| 21:21 |
<Jeff_Green> |
changed donate.wikimedia.org from CNAME to A record so we can use MX records |
[production] |
| 21:13 |
<awjr> |
Configured DonationInterface to only load i18n files on the cluster, enabled on donatewiki and foundation (CommonSettings @ [[rev:2560|r2560]], InitialiseSettings @ [[rev:2561|r2561]]) |
[production] |
| 21:10 |
<awjr> |
Running scap to pick up new DonationInterface set up for the cluster and new message files ([[rev:101527|r101527]]) |
[production] |
| 19:55 |
<awjr> |
Fixed borked config change in CommonSettings for LandingCheck, [[rev:2559|r2559]] |
[production] |
| 19:53 |
<awjr> |
Config change to CommonSettings to make LandingCheck always point to foundationwiki for now [[rev:2558|r2558]] |
[production] |
| 19:45 |
<binasher> |
note, locke and emery changes only made to the running kernel for testing, not in /etc/sysctl* |
[production] |
| 19:43 |
<binasher> |
made the same kernel changes on emery as locke, udp packet loss dropped from 6% to 0 |
[production] |
| 19:33 |
<awjr> |
deploying LandingCheck updates (@ [[rev:101515|r101515]]) via sync-file as well as config change to CommonSettings.php (@ [[rev:2557|r2557]]) |
[production] |
| 18:52 |
<notpeter> |
restarting lighttpd on lily |
[production] |
| 18:43 |
<notpeter> |
restarting lighttpd on lily |
[production] |
| 18:30 |
<binasher> |
locke: disabled bbu checks, set per5 to writeback caching |
[production] |