|
2009-11-07
§
|
| 22:14 |
<mark> |
Started puppet on sq44 |
[production] |
| 22:12 |
<mark> |
Unstuck puppet on srv149 |
[production] |
| 22:02 |
<mark> |
Fixed puppet on ixia |
[production] |
| 21:48 |
<mark> |
Installed puppet on yongle |
[production] |
| 21:26 |
<mark> |
Rolling dist-upgrade finished |
[production] |
| 21:12 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'removing db21 from rotation - will be used for migration source' |
[production] |
| 18:35 |
<mark> |
Moved planet.wikimedia.org DNS CNAMEs to singer |
[production] |
| 18:35 |
<mark> |
Setup planet.wikimedia.org instance on singer |
[production] |
| 17:09 |
<mark> |
Moved noc.wikimedia.org CNAME to fenari |
[production] |
| 17:09 |
<mark> |
Setup noc.wikimedia.org instance on fenari, using puppet |
[production] |
| 14:57 |
<mark> |
Doing rolling dist-upgrade of all application servers |
[production] |
| 14:13 |
<mark> |
Removed optsview APT repository on srv225 |
[production] |
| 14:12 |
<mark> |
Removed optsview APT repository on srv224 |
[production] |
| 13:36 |
<mark> |
Manually upgrading puppet across the cluster to help itself get past a catalog bug |
[production] |
| 13:03 |
<mark> |
Setup puppet to automatically upgrade wikimedia-task-appserver on all application servers; in this case it will roll out package php5-geoip across the cluster |
[production] |
| 09:12 |
<ariel> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable Collection on strategy wiki per bug 21361' |
[production] |
|
2009-11-06
§
|
| 23:45 |
<tfinc> |
synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 'dropping 2008 theme' |
[production] |
| 23:39 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php 'dropping 2008 theme' |
[production] |
| 23:19 |
<mark> |
Replaced the Webrick puppetmaster setup on sockpuppet by an Apache/Mongrel setup with 4 children to deal with scalability problems |
[production] |
| 21:29 |
<mark> |
Increased active record DB connection pool size from 5 to 50... by just editing the ruby library file |
[production] |
| 18:54 |
<mark> |
db10 replication caught up with db9 |
[production] |
| 18:08 |
<mark> |
Set up puppet to run "apt-get update" on every run (for now, may make that somewhat less aggressive if needed) |
[production] |
| 17:10 |
<andrew> |
synchronized php-1.5/wmf-config/ExtensionMessages.php |
[production] |
| 17:08 |
<mark> |
Setup puppet to exchange all hostkeys between all servers |
[production] |
| 17:07 |
<andrew> |
synchronized php-1.5/includes/LinksUpdate.php |
[production] |
| 17:05 |
<Andrew> |
Scapping |
[production] |
| 17:05 |
<Andrew> |
Set up GlobalUsage on test.wikipedia.org |
[production] |
| 16:55 |
<mark> |
Set up puppet to upgrade itself everywhere |
[production] |
| 16:40 |
<mark> |
Fixed permission problem in the MySQL data dir, restarted replication on db10 |
[production] |
| 16:37 |
<mark> |
Fixed LVM problem on db10 |
[production] |
| 16:02 |
<mark> |
Rebooting db10: OOM errors |
[production] |
| 15:59 |
<mark> |
Replication db9->db10 broken again |
[production] |
| 15:56 |
<mark> |
Set up MySQL database 'puppet' on db9/db10 |
[production] |
|
2009-11-05
§
|
| 19:59 |
<tfinc> |
synchronized php-1.5/extensions/DonationInterface/donate_interface/donate_interface.i18n.php |
[production] |
| 19:51 |
<tfinc> |
synchronized php-1.5/extensions/DonationInterface/donate_interface/donate_interface.php |
[production] |
| 18:48 |
<mark> |
Setup puppet Stored Configurations with Stomp queuing; made a custom /usr/local/sbin/puppetqd since it was missing in the puppetmaster package. |
[production] |
| 17:58 |
<Rob> |
transcode1-3 drac setup, servers racked, not in racktables yet, no os yet. |
[production] |
| 17:42 |
<mark> |
Upgraded distribution of sockpuppet to 9.10 (Karmic) |
[production] |
| 17:35 |
<Rob> |
pushing dns changes for mgmt on transcode1-3 |
[production] |
| 16:54 |
<mark> |
Upgraded puppetmaster on sockpuppet to version 0.25.1 |
[production] |
| 15:35 |
<mark> |
Killed all Nagios processes on bart |
[production] |
| 15:28 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabled InitialiseSettings.php again on all wikis, except testwiki. Pending HTTP proxy setup and security review.' |
[production] |
| 15:19 |
<mark> |
Moved secure.wikimedia.org service IP from bart to singer |
[production] |
| 15:13 |
<brion> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling upload by URL for sysops on all wikis' |
[production] |
| 15:09 |
<mark> |
Moved NameVirtualHost entries to /etc/apache2/conf.d/namevirtualhost on singer, to stop Apache from complaining |
[production] |
| 15:06 |
<mark> |
Set up secure.wikimedia.org vhost on singer, SSL proxying to the apaches |
[production] |
| 14:49 |
<mark> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Add singer to the SquidServers list' |
[production] |