2101-2150 of 10000 results (18ms)
2014-02-06 §
05:48 <ori> varnish on cp1054: CPU wait spiked at 05:27. dmesg|tail: XFS: possible memory allocation deadlock in kmem_alloc. not investigating further. [production]
05:45 <ori> finished scap: no new code. testing scap changes. (again.) (duration: 05m 00s) [production]
05:40 <ori> started scap: no new code. testing scap changes. (again.) [production]
05:15 <springle> restart labsdb1003 mariadb instances [production]
05:09 <Ryan_Lane> scratch that, will redploy parsoid/deploy in about an hour [production]
05:08 <Ryan_Lane> redeploying parsoid/deploy on wtp* [production]
04:28 <ori> finished scap: no new code. testing scap changes. (duration: 04m 35s) [production]
04:24 <ori> started scap: no new code. testing scap changes. [production]
03:22 <LocalisationUpdate> ResourceLoader cache refresh completed at 2014-02-06 03:22:26+00:00 [production]
03:10 <springle> xtrabackup clone db1018 to db1024 [production]
02:45 <springle> xtrabackup clone db1034 to db1009 [production]
02:44 <LocalisationUpdate> completed (1.23wmf11) at 2014-02-06 02:44:10+00:00 [production]
02:23 <springle> synchronized wmf-config/db-eqiad.php 's2 depool db1009 schema changes' [production]
02:21 <LocalisationUpdate> completed (1.23wmf12) at 2014-02-06 02:21:47+00:00 [production]
02:01 <springle> synchronized wmf-config/db-eqiad.php 's2 repool db1034 warm up' [production]
01:06 <mutante> apt-get remove libnet-stomp-perl on neon, i just removed that from puppet but didn't think it should stay in as an "absent" package forever [production]
01:01 <ori> finished scap: no-diff scap to test script changes (duration: 02m 34s) [production]
00:58 <ori> started scap: no-diff scap to test script changes [production]
00:44 <ebernhardson> finished deploying Special:Notifications fix to Echo and Flow [production]
00:44 <ebernhardson> synchronized php-1.23wmf11/extensions/Flow/ 'Update flow for Special:Notifications fix' [production]
00:43 <ebernhardson> synchronized php-1.23wmf11/extensions/Echo/ 'Update echo for Special:Notifications fix' [production]
00:38 <ebernhardson> synchronized php-1.23wmf12/extensions/Flow/ 'Update flow for Special:Notifications fix' [production]
00:37 <ebernhardson> synchronized php-1.23wmf12/extensions/Echo/ 'Update echo for Special:Notifications fix' [production]
00:04 <ori> finished scap: (no message) (duration: 29m 26s) [production]
00:04 <ori> rebuilt wikiversions.cdb and synchronized wikiversions files: [production]
2014-02-05 §
23:34 <ori> started scap: (no message) [production]
22:11 <ottomata> stopping puppet on analytics1021. Trying to get it to catch up on replica lag [production]
20:13 <mwalker> updated fundraising civicrm from ec1e1fc75193ac8c0e6237fc0b80f58b3f8d55fc to 309a69545b4c89b323bffc500f71bdee82c35e42 for logging updates [production]
18:57 <yurik> synchronized php-1.23wmf12/extensions/ZeroRatedMobileAccess/ [production]
18:49 <yurik> synchronized php-1.23wmf11/extensions/ZeroRatedMobileAccess/ [production]
18:38 <yurik> synchronized docroot/bits/WikipediaMobileFirefoxOS [production]
14:11 <paravoid> ms-be1002/sdd: megacli -DiscardPreservedCache, -CfgEachDskRaid0, puppet run [production]
13:57 <paravoid> remove XO-Level3 avoided as-path from cr1/2-eqiad despite no ticket reply; seems to work now [production]
13:57 <hashar> compressing Jenkins console logs on gallium.wikimedia.org using gzip -9 [production]
08:47 <ori> finished scap: (no message) (duration: 03m 31s) [production]
08:47 <ori> rebuilt wikiversions.cdb and synchronized wikiversions files: [production]
08:44 <ori> started scap: (no message) [production]
08:17 <Ryan_Lane> restart parsoid on wtp1012 [production]
08:14 <Ryan_Lane> stopping parsoid on wtp1012 shortly [production]
07:53 <ori> rebuilt wikiversions.cdb and synchronized wikiversions files: [production]
07:51 <ori> rebuilt wikiversions.cdb and synchronized wikiversions files: [production]
07:15 <ori> rebuilt wikiversions.cdb and synchronized wikiversions files: [production]
06:59 <ori> finished scap: (no message) (duration: 00m 35s) [production]
06:59 <ori> rebuilt wikiversions.cdb and synchronized wikiversions files: [production]
06:59 <ori> started scap: (no message) [production]
05:52 <springle> synchronized wmf-config/db-pmtpa.php 'update pmtpa config for s1 master change' [production]
05:47 <springle> synchronized wmf-config/db-eqiad.php 'switch s1 master to db1052' [production]
05:32 <springle> synchronized wmf-config/db-eqiad.php 'prepare for s1 master rotation db1056 to db1052' [production]
05:18 <LocalisationUpdate> completed (1.23wmf11) at 2014-02-05 05:18:46+00:00 [production]
05:07 <ori> Last sync-file: connect to host mw1163 port 22: Connection timed out [production]