1951-2000 of 10000 results (24ms)
2014-10-24 §
22:40 <akosiaris> puppet disabled on uranium, do not enable [production]
20:52 <andrewbogott> revived virt1006 on a probationary basis. It's running compute but is disabled so new instances won't be scheduled there. I've moved a few test instances there to see how it behaves. [production]
20:36 <andrew> Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 04s) [production]
20:29 <Reedy> sync-common on mw1088 [production]
20:23 <mutante> mw1088 - gzipping core dump files, disabled core dumps, restarted apache [production]
20:15 <mutante> mw1088 - gzip other_vhosts_access.log.1 - Avail. 38G [production]
20:15 <Reedy> / full on mw1088 due to apache core dumps [production]
20:09 <Reedy> running sync-common on mw1041 [production]
20:04 <mutante> powercycled mw1041 [production]
20:03 <reedy> Synchronized php-1.25wmf5/extensions/SemanticForms/: noop for prod (duration: 00m 17s) [production]
20:01 <Reedy> mw1041 is down [production]
20:01 <Reedy> mw1088 has a full / [production]
20:00 <reedy> Synchronized php-1.25wmf4/extensions/SemanticForms/: noop for prod (duration: 00m 16s) [production]
19:53 <bblack> nickel's basically dead, uranium has been promoted to prod ganglia a little early for now [production]
19:22 <awight> updated payments from 6fa864d4aaa22b9f271de4bc662be68bb0b40b56 to 525988487d6bbd08ddad50badd88e34e34104292 [production]
18:55 <ori> repooled mw1189 to do heap profiling on production api workload. [production]
17:58 <mutante> stat1001 - Duplicate declaration: Package[nodejs] [production]
17:08 <cmjohnson> getting ready to replace a failed disk on ganglia (server:nickel)...it will be offline for a few minutes [production]
17:05 <ejegg> updated dash from 58fda9403dd33e4d47238f119b6bb2b2905856b1 to 69c9330d6983873ffa9bb87fcd783be03382bdfc [production]
15:50 <awight> campaigns reenabled [production]
15:40 <awight> payments db migrated to 1.23 schema [production]
15:37 <awight> updated payments to REL1_23, 6fa864d4aaa22b9f271de4bc662be68bb0b40b56 [production]
15:18 <awight> payments in maintenance mode [production]
14:57 <robh> francium going offline, ignore any icinga warning [production]
14:37 <andrewbogott> running sync-common on virt1000 [production]
14:36 <andrew> Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 03s) [production]
14:35 <andrew> Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 02s) [production]
13:39 <akosiaris> disabled puppet on uranium. Testing ganglia with SSDs [production]
12:10 <akosiaris> restarted gmetad on nickel, it was not responding on port 8654 [production]
05:04 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Oct 24 05:04:33 UTC 2014 (duration 4m 32s) [production]
03:22 <LocalisationUpdate> completed (1.25wmf5) at 2014-10-24 03:22:36+00:00 [production]
02:46 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-24 02:46:15+00:00 [production]
2014-10-23 §
23:39 <catrope> Synchronized php-1.25wmf4/extensions/CentralAuth/: SWAT (duration: 00m 04s) [production]
23:38 <catrope> Synchronized php-1.25wmf4/extensions/AntiSpoof/: SWAT (duration: 00m 06s) [production]
23:29 <catrope> Synchronized php-1.25wmf5/extensions/TimedMediaHandler/: SWAT (duration: 00m 04s) [production]
23:29 <catrope> Synchronized php-1.25wmf5/extensions/UploadWizard/: SWAT (duration: 00m 06s) [production]
23:28 <catrope> Synchronized php-1.25wmf4/extensions/UploadWizard/: SWAT (duration: 00m 04s) [production]
23:28 <catrope> Synchronized php-1.25wmf4/extensions/TimedMediaHandler/: SWAT (duration: 00m 04s) [production]
23:27 <catrope> Synchronized php-1.25wmf4/includes/api/ApiFormatBase.php: SWAT (duration: 00m 04s) [production]
22:42 <hashar> Jenkins is all good now. [production]
22:36 <hashar> Jenkins restarting [production]
22:28 <hashar> preparing jenkins for restart [production]
21:54 <hashar> Jenkins the Gearman plugin is holding a lock on deployment-bastion slave that prevents it from running any job :-/ [production]
21:51 <ejegg> updated civicrm from ad3386cd0f9b776e2fded7c4e6b1195e05ed669c to 937df4dacae0dd620ae9e8fed13566d51c1b18a4 [production]
21:46 <springle> Synchronized wmf-config/db-eqiad.php: depool db1018 (duration: 00m 06s) [production]
21:43 <hashar> Jenkins: disabling / reenabling Gearman plugin [production]
21:35 <hashar> Jenkins: disconnected / reconnected slave node deployment-bastion.eqiad [production]
21:15 <awight> updated crm from 03b15f7dad58ce61894d632e8fbebd2ae76ae4d0 to ad3386cd0f9b776e2fded7c4e6b1195e05ed669c [production]
21:04 <ejegg> updated civicrm from 0a3ab0f18ce726898d14adcbe6ab08411c9e3e82 to 03b15f7dad58ce61894d632e8fbebd2ae76ae4d0 [production]
19:29 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf5 [production]