1201-1250 of 10000 results (25ms)
2014-10-20 §
20:11 <subbu> deployed Parsoid version d4567e9f [production]
20:01 <paravoid> cr1-ulsfo: reenable ospf/ospf3 (GTT is stable) [production]
19:13 <mutante> importing schema, data, users into mysql for iegreview [production]
17:42 <apergos> all trusty and lucid hosts now running salt 2014.1.11 (this includes labcontrol2001, salt master for future codfw labs) [production]
16:48 <reedy> Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 14s) [production]
16:47 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: orwikisource [production]
16:44 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 15s) [production]
16:44 <reedy> Synchronized database lists: orwikisource (duration: 00m 13s) [production]
16:43 <cmjohnson> reseating pem3 cr2-eqiad [production]
16:43 <apergos> all precise hosts salt updated to 2014.1.11, this includes tin (deployment) and virt1000 (salt master for labs). Not updated: virt1006 (inaccessible) [production]
15:58 <anomie> Synchronized wmf-config: SWAT: Enable wgSecurePollUseNamespace for testwiki [[gerrit:167592]] (duration: 00m 10s) [production]
15:57 <anomie> Finished scap: (no message) (duration: 18m 22s) [production]
15:39 <anomie> Started scap: (no message) [production]
15:39 <anomie> Synchronized php-1.25wmf4/extensions/SecurePoll/: SWAT: Update SecurePoll for testing on testwiki [[gerrit:167586]] (duration: 00m 10s) [production]
15:20 <_joe_> disabling puppet on mw1189 to do some hhvm testing [production]
15:20 <anomie> Synchronized php-1.25wmf3/resources/lib/oojs-ui/: SWAT: OOJS-UI bug fixes [[gerrit:167344]] (duration: 00m 12s) [production]
15:19 <anomie> Synchronized php-1.25wmf3/extensions/VisualEditor/: SWAT: VE bug fixes [[gerrit:167344]] (duration: 00m 10s) [production]
15:08 <anomie> Synchronized php-1.25wmf4/extensions/VisualEditor/: SWAT: VE bug fixes [[gerrit:167577]] (duration: 00m 10s) [production]
14:26 <paravoid> cr1-ulsfo: deactivating ospf/ospf3 on GTT ulsfo-eqiad link [production]
13:26 <paravoid> cr2-ulsfo: "request chassis mic {off,on}line fpc-slot 1 mic-slot 1" to reboot broken card [production]
12:28 <apergos> upgraded salt master (plus minion) on palladium to 2014.1.11, all neww precise installs will get this version now, other minion upgrades to follow shortly [production]
11:51 <godog> temporarily stopped ircecho/icinga-wm on neon, shower of alarms [production]
11:42 <godog> killed stray/old copy of diamond that was filling up conntrack on virt1000 [production]
09:53 <akosiaris> restarted ocg on ocg1001, ocg1002, ocg1003 [production]
07:07 <_joe_> rolling restart of ocg services [production]
04:29 <springle> removed old /var/log/ocg* on ocg1001 and ocg1003 and forced logrotate, / space critical [production]
03:42 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Oct 20 03:42:24 UTC 2014 (duration 42m 23s) [production]
02:50 <springle> Synchronized wmf-config/db-eqiad.php: repool db1042 (duration: 00m 06s) [production]
02:34 <springle> Synchronized wmf-config/db-eqiad.php: depool db1066 (duration: 00m 06s) [production]
02:28 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-20 02:28:35+00:00 [production]
02:16 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-20 02:16:31+00:00 [production]
2014-10-19 §
03:36 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Oct 19 03:36:03 UTC 2014 (duration 36m 2s) [production]
02:25 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-19 02:25:44+00:00 [production]
02:14 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-19 02:14:23+00:00 [production]
2014-10-18 §
15:09 <springle> Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 06s) [production]
13:49 <_joe_> restarted apache on the puppetmasters [production]
03:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Oct 18 03:51:18 UTC 2014 (duration 51m 17s) [production]
02:33 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-18 02:33:01+00:00 [production]
02:20 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-18 02:20:20+00:00 [production]
2014-10-17 §
23:46 <bd808> Ran trebuchet to create initial tag for iegreview/iegreview [production]
23:35 <ori> pooled mw1114 (hhvm api server) to test whether new package resolves overload behavior [production]
23:29 <ori> updating hhvm app servers to 3.3.0-20140925+wmf3 [production]
22:18 <mutante> graceful Apache on antimony - svn fixed, gitblit behind varnish [production]
22:09 <mutante> graceful Apache on neon - icinga and tendril done, ishmael = misc-web [production]
21:47 <mutante> graceful Apache on stat1001 [production]
21:25 <springle> Synchronized wmf-config/db-eqiad.php: repool es1004, es1007, es1010 (duration: 00m 07s) [production]
21:19 <mutante> graceful Apache on netmon1001 [production]
21:13 <mutante> graceful Apache on puppetmaster [production]
20:34 <andrewbogott> removed some stray .zip files from /tmp on ocg1002 [production]
20:16 <ori> disabled puppet on osmium to debug hhvm [production]