6351-6400 of 10000 results (19ms)
2011-02-08 §
13:04 <RoanKattouw> Rerunning sync-common-all [production]
13:03 <mark> Upgraded rsync to a version with higher tcp backlog on nfs2 [production]
13:01 <RoanKattouw> sync-common-all failed for a few servers due to NFS problems. Mark is fixing it [production]
12:58 <catrope> ran sync-common-all [production]
12:57 <RoanKattouw> Moved old wmf-deployment/ and php-1.17/ back on /home , now syncing that out to the cluster [production]
10:38 <mark> Power cycled amssq37, 43, 49 [production]
10:34 <mark> Power cycled amssq35 [production]
10:15 <mark> Deploying special case for test.wikipedia.org (=> srv193) on bits.pmtpa (varnish) [production]
10:12 <catrope> synchronized live-1.5/load.php 'Revert failed hack' [production]
10:01 <catrope> synchronized live-1.5/load.php 'Temp hack to make load.php work on test' [production]
09:28 <root> ran sync-common-all [production]
08:58 <root> ran sync-common-all [production]
08:41 <mark> Unstuck SMS modem [production]
07:50 <Tim> restarted apache on srv174 [production]
07:22 <mark> Started Nagios bot [production]
06:15 <^demon> fixed svn permissions for wmf-deployment group [production]
2011-02-07 §
22:20 <Ryan_Lane> upgrading nova-* on nova-*.tesla [production]
20:02 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '27195 - Grant anonymous users 'createpage' right and enable Extension' [production]
19:55 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '27209 - Request Extension of Quiz Addon to English Wikibooks' [production]
17:18 <demon> synchronized php-1.5/wmf-config/CommonSettings.php 'Removing proof-of-concept MWReleases extension' [production]
17:18 <demon> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling proof-of-concept MWReleases extension' [production]
13:18 <Tim> running schema updates on all old masters. db29 already done. [production]
13:17 <Tim> new master positions: db22-bin.000001:106, db18-bin.000001:106, db34-bin.000001:106 [production]
13:16 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:16 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:15 <Tim> doing master switch on s3 from db39 to db34 [production]
13:15 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:14 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:14 <Tim> doing master switch on s7 from db37 to db18 [production]
13:13 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:13 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:12 <Tim> doing master switch on s4 from db31 to db22 [production]
13:11 <Tim> new positions: db21-bin.001:79, db32-bin.000001:106 [production]
13:07 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:07 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:06 <Tim> doing master switch on s1 from db36 to db32 [production]
13:03 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:03 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
13:02 <Tim> doing master switch of s6 from db29 to db21 [production]
08:02 <Tim> running upgrade-1.17wmf1-initial.php again to upgrade db13 and db23 [production]
07:52 <Tim> s5: New position: db14-bin.001:79 [production]
07:51 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
07:51 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
07:51 <Tim> switching master on s5 from db23 to db14 [production]
07:41 <Tim> s2: New position: db30-bin.000001:106 [production]
07:41 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
07:40 <tstarling> synchronized php-1.5/wmf-config/db.php [production]
07:39 <Tim> switching master on s2 from db13 to db30 [production]
04:35 <apergos> running rebuild of the first of 4 history dump pieces on snapshot3, root, screen session [production]
2011-02-06 §
21:27 <apergos> spliting up stubs into pieces in prep for resume of truncated history dumps fo en wiki; running on snapshot3 in screen session as root (see "part1,8,9,10.sh" in /backups-atg for what's running) [production]