7651-7700 of 10000 results (39ms)
2015-08-22 §
16:45 <chasemp> scratch that as we have mpm_prefork enabled :) [production]
16:32 <chasemp> raising values in mpm_worker.conf for iridium to to debug and hopefully head off further crashing [production]
14:44 <twentyafterfour> restarted apache2 on iridium. Segfault again. This time I at least got one clue in the log: "zend_mm_heap corrupted" [production]
09:18 <twentyafterfour> phabricator seems stable now, restarting apache2 on iridium did the trick, unfortunately we didn't learn why [production]
08:36 <twentyafterfour> restarted phd on iridium [production]
08:35 <twentyafterfour> restarted apache2 on iridium [production]
02:20 <l10nupdate@tin> Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 09s) [production]
00:26 <mutante> deleting blog.sh and blog_pageviews crontab from stat1003 [production]
2015-08-21 §
23:34 <urandom> restarting Cassandra on restbase1001 to restore baseline settings [production]
23:11 <yurik> synced kartotherian [production]
22:35 <mutante> deleting held messages on mailman that are older than 1 year [production]
21:44 <mutante> had to reset list creator password for mailman - ask me if you think you should have it and don't (this is not the master pass) [production]
20:37 <ori@tin> Synchronized php-1.26wmf19/includes: I1eb8dfc: Revert Count API and hook calls, with 1:1000 sampling (duration: 01m 09s) [production]
16:06 <jynus> checksumming dewiki database, higher write rate/dbstore lag expected temporarily [production]
15:10 <ottomata> rebooting kafka broker analytics1021 to hopefully reload /dev/sdg with new disk, also will turn on hyperthreading [production]
14:13 <ottomata> rebooting analytics1056 after upgrading kernel to linux-image-3.13.0-61-generic [production]
13:57 <urandom> restarting restbase1001 to apply temporary GC setting [production]
13:33 <ottomata> stopping kafka broker on analytics1021 due to bad disk. [production]
13:30 <bblack> wiped ganglia apache access log on uranium, to free up half of the (full) rootfs [production]
10:07 <godog> enable puppet on ms-fe1/ms-be1 [production]
09:49 <godog> disable puppet on ms-fe1/ms-be1 before merging https://gerrit.wikimedia.org/r/#/c/231240/ [production]
07:06 <_joe_> restarting gitblit, because it will be decommissioned "soon"... [production]
02:34 <l10nupdate@tin> Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 11m 19s) [production]
2015-08-20 §
23:40 <ebernhardson@tin> Synchronized php-1.26wmf19/extensions/CirrusSearch/: Fix some cirrussearch logspam (duration: 00m 13s) [production]
23:30 <ebernhardson@tin> Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s) [production]
23:29 <ebernhardson@tin> Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s) [production]
23:23 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232854/ (duration: 00m 13s) [production]
23:22 <krenair@tin> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232671/ (duration: 00m 12s) [production]
23:15 <krenair@tin> Synchronized php-1.26wmf19/extensions/LiquidThreads/classes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/232783/ (duration: 00m 12s) [production]
23:13 <krinkle@tin> Synchronized php-1.26wmf19/includes/resourceloader/ResourceLoaderFileModule.php: T102578 (duration: 00m 13s) [production]
23:08 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232782/ (duration: 00m 12s) [production]
22:47 <ori@tin> Synchronized php-1.26wmf19/includes/libs/CSSMin.php: Icc1c23a2: CSSMin: remove dot segments in relative local URLs (duration: 00m 12s) [production]
21:36 <cscott> updated Parsoid to version db6e6404f67a9f971b4fbefe9de239735426c738 [production]
21:25 <matt_flaschen> Ran FlowUpdateRevContentModelFromOccupyPages.php on all wikis [production]
20:41 <twentyafterfour> scap failed to sync to mw2180.codwf.wmnet [production]
20:41 <twentyafterfour@tin> rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf19 [production]
20:38 <twentyafterfour@tin> Synchronized php-1.26wmf19: Silence the undefined index error in CirrusSearch (duration: 06m 24s) [production]
19:40 <chasemp> moving enwiki_content_1432182861 elastic shard from 1022 to 1004 due to space (1022 is at 91%) [production]
18:57 <mutante> no log bot [production]
18:56 <mutante> labvirt1007 "only" 29G space left - but since we have 2.2T there that means 99% full [production]
17:39 <ottomata> stopping kafka on analytics1018 and bringing it down for reinstall as kafka1018 with Jessie [production]
16:38 <YuviPanda> puppet swat done [production]
15:44 <akosiaris> uploaded to apt.wikimedia.org jessie-wikimedia: etherpad-lite_1.5.7-1 [production]
15:43 <krenair@tin> Synchronized php-1.26wmf18/extensions/ContentTranslation/modules/tools/ext.cx.tools.reference.js: https://gerrit.wikimedia.org/r/#/c/232729/ (duration: 00m 12s) [production]
15:42 <krenair@tin> Synchronized php-1.26wmf19/extensions/ContentTranslation/modules/tools/ext.cx.tools.reference.js: https://gerrit.wikimedia.org/r/#/c/232730/ (duration: 00m 13s) [production]
15:39 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/206480/ (duration: 00m 13s) [production]
15:38 <krenair@tin> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/206480/ (duration: 00m 13s) [production]
15:32 <krenair@tin> Synchronized php-1.26wmf18/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: https://gerrit.wikimedia.org/r/#/c/232687/ (duration: 00m 13s) [production]
15:31 <krenair@tin> Synchronized php-1.26wmf19/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: https://gerrit.wikimedia.org/r/#/c/232688/ (duration: 00m 11s) [production]
15:27 <greg-g> on mw2187: rsync: failed to set times on "/srv/mediawiki/wmf-config": Read-only file system (30) [production]