|
2009-11-08
§
|
| 09:28 |
<apergos> |
cleaned out some space from /tmp and ran apt-get clean on srv218, these should probably become a cron job |
[production] |
| 09:27 |
<andrew> |
synchronized php-1.5/includes/ImagePage.php 'Merge r58692' |
[production] |
| 09:23 |
<Andrew> |
scapping to update GlobalUsage |
[production] |
| 09:20 |
<brion> |
srv218: rsync: write failed on "/apache/common/php-1.5/wmf-config/InitialiseSettings.php": No space left on device (28) |
[production] |
| 09:19 |
<brion> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding TimedText namespace on Commons for mdale subtitle awesomeness' |
[production] |
| 08:43 |
<andrew> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Activated LiquidThreads on MediaWiki.org' |
[production] |
| 08:39 |
<Andrew> |
Sticking LiquidThreads in opt-in mode on mediawiki.org |
[production] |
| 05:54 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'putting db1 and db21 live with jafrru dataset' |
[production] |
| 03:58 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'taking out db1 and db21 for new image load' |
[production] |
| 01:04 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'putting db21 back live' |
[production] |
|
2009-11-07
§
|
| 22:14 |
<mark> |
Started puppet on sq44 |
[production] |
| 22:12 |
<mark> |
Unstuck puppet on srv149 |
[production] |
| 22:02 |
<mark> |
Fixed puppet on ixia |
[production] |
| 21:48 |
<mark> |
Installed puppet on yongle |
[production] |
| 21:26 |
<mark> |
Rolling dist-upgrade finished |
[production] |
| 21:12 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'removing db21 from rotation - will be used for migration source' |
[production] |
| 18:35 |
<mark> |
Moved planet.wikimedia.org DNS CNAMEs to singer |
[production] |
| 18:35 |
<mark> |
Setup planet.wikimedia.org instance on singer |
[production] |
| 17:09 |
<mark> |
Moved noc.wikimedia.org CNAME to fenari |
[production] |
| 17:09 |
<mark> |
Setup noc.wikimedia.org instance on fenari, using puppet |
[production] |
| 14:57 |
<mark> |
Doing rolling dist-upgrade of all application servers |
[production] |
| 14:13 |
<mark> |
Removed optsview APT repository on srv225 |
[production] |
| 14:12 |
<mark> |
Removed optsview APT repository on srv224 |
[production] |
| 13:36 |
<mark> |
Manually upgrading puppet across the cluster to help itself get past a catalog bug |
[production] |
| 13:03 |
<mark> |
Setup puppet to automatically upgrade wikimedia-task-appserver on all application servers; in this case it will roll out package php5-geoip across the cluster |
[production] |
| 09:12 |
<ariel> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable Collection on strategy wiki per bug 21361' |
[production] |
|
2009-11-06
§
|
| 23:45 |
<tfinc> |
synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 'dropping 2008 theme' |
[production] |
| 23:39 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php 'dropping 2008 theme' |
[production] |
| 23:19 |
<mark> |
Replaced the Webrick puppetmaster setup on sockpuppet by an Apache/Mongrel setup with 4 children to deal with scalability problems |
[production] |
| 21:29 |
<mark> |
Increased active record DB connection pool size from 5 to 50... by just editing the ruby library file |
[production] |
| 18:54 |
<mark> |
db10 replication caught up with db9 |
[production] |
| 18:08 |
<mark> |
Set up puppet to run "apt-get update" on every run (for now, may make that somewhat less aggressive if needed) |
[production] |
| 17:10 |
<andrew> |
synchronized php-1.5/wmf-config/ExtensionMessages.php |
[production] |
| 17:08 |
<mark> |
Setup puppet to exchange all hostkeys between all servers |
[production] |
| 17:07 |
<andrew> |
synchronized php-1.5/includes/LinksUpdate.php |
[production] |
| 17:05 |
<Andrew> |
Scapping |
[production] |
| 17:05 |
<Andrew> |
Set up GlobalUsage on test.wikipedia.org |
[production] |
| 16:55 |
<mark> |
Set up puppet to upgrade itself everywhere |
[production] |
| 16:40 |
<mark> |
Fixed permission problem in the MySQL data dir, restarted replication on db10 |
[production] |
| 16:37 |
<mark> |
Fixed LVM problem on db10 |
[production] |
| 16:02 |
<mark> |
Rebooting db10: OOM errors |
[production] |
| 15:59 |
<mark> |
Replication db9->db10 broken again |
[production] |
| 15:56 |
<mark> |
Set up MySQL database 'puppet' on db9/db10 |
[production] |
|
2009-11-05
§
|
| 19:59 |
<tfinc> |
synchronized php-1.5/extensions/DonationInterface/donate_interface/donate_interface.i18n.php |
[production] |
| 19:51 |
<tfinc> |
synchronized php-1.5/extensions/DonationInterface/donate_interface/donate_interface.php |
[production] |
| 18:48 |
<mark> |
Setup puppet Stored Configurations with Stomp queuing; made a custom /usr/local/sbin/puppetqd since it was missing in the puppetmaster package. |
[production] |
| 17:58 |
<Rob> |
transcode1-3 drac setup, servers racked, not in racktables yet, no os yet. |
[production] |
| 17:42 |
<mark> |
Upgraded distribution of sockpuppet to 9.10 (Karmic) |
[production] |
| 17:35 |
<Rob> |
pushing dns changes for mgmt on transcode1-3 |
[production] |
| 16:54 |
<mark> |
Upgraded puppetmaster on sockpuppet to version 0.25.1 |
[production] |