|
2014-10-24
§
|
| 22:40 |
<akosiaris> |
puppet disabled on uranium, do not enable |
[production] |
| 20:52 |
<andrewbogott> |
revived virt1006 on a probationary basis. It's running compute but is disabled so new instances won't be scheduled there. I've moved a few test instances there to see how it behaves. |
[production] |
| 20:36 |
<andrew> |
Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 04s) |
[production] |
| 20:29 |
<Reedy> |
sync-common on mw1088 |
[production] |
| 20:23 |
<mutante> |
mw1088 - gzipping core dump files, disabled core dumps, restarted apache |
[production] |
| 20:15 |
<mutante> |
mw1088 - gzip other_vhosts_access.log.1 - Avail. 38G |
[production] |
| 20:15 |
<Reedy> |
/ full on mw1088 due to apache core dumps |
[production] |
| 20:09 |
<Reedy> |
running sync-common on mw1041 |
[production] |
| 20:04 |
<mutante> |
powercycled mw1041 |
[production] |
| 20:03 |
<reedy> |
Synchronized php-1.25wmf5/extensions/SemanticForms/: noop for prod (duration: 00m 17s) |
[production] |
| 20:01 |
<Reedy> |
mw1041 is down |
[production] |
| 20:01 |
<Reedy> |
mw1088 has a full / |
[production] |
| 20:00 |
<reedy> |
Synchronized php-1.25wmf4/extensions/SemanticForms/: noop for prod (duration: 00m 16s) |
[production] |
| 19:53 |
<bblack> |
nickel's basically dead, uranium has been promoted to prod ganglia a little early for now |
[production] |
| 19:22 |
<awight> |
updated payments from 6fa864d4aaa22b9f271de4bc662be68bb0b40b56 to 525988487d6bbd08ddad50badd88e34e34104292 |
[production] |
| 18:55 |
<ori> |
repooled mw1189 to do heap profiling on production api workload. |
[production] |
| 17:58 |
<mutante> |
stat1001 - Duplicate declaration: Package[nodejs] |
[production] |
| 17:08 |
<cmjohnson> |
getting ready to replace a failed disk on ganglia (server:nickel)...it will be offline for a few minutes |
[production] |
| 17:05 |
<ejegg> |
updated dash from 58fda9403dd33e4d47238f119b6bb2b2905856b1 to 69c9330d6983873ffa9bb87fcd783be03382bdfc |
[production] |
| 15:50 |
<awight> |
campaigns reenabled |
[production] |
| 15:40 |
<awight> |
payments db migrated to 1.23 schema |
[production] |
| 15:37 |
<awight> |
updated payments to REL1_23, 6fa864d4aaa22b9f271de4bc662be68bb0b40b56 |
[production] |
| 15:18 |
<awight> |
payments in maintenance mode |
[production] |
| 14:57 |
<robh> |
francium going offline, ignore any icinga warning |
[production] |
| 14:37 |
<andrewbogott> |
running sync-common on virt1000 |
[production] |
| 14:36 |
<andrew> |
Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 03s) |
[production] |
| 14:35 |
<andrew> |
Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 02s) |
[production] |
| 13:39 |
<akosiaris> |
disabled puppet on uranium. Testing ganglia with SSDs |
[production] |
| 12:10 |
<akosiaris> |
restarted gmetad on nickel, it was not responding on port 8654 |
[production] |
| 05:04 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Oct 24 05:04:33 UTC 2014 (duration 4m 32s) |
[production] |
| 03:22 |
<LocalisationUpdate> |
completed (1.25wmf5) at 2014-10-24 03:22:36+00:00 |
[production] |
| 02:46 |
<LocalisationUpdate> |
completed (1.25wmf4) at 2014-10-24 02:46:15+00:00 |
[production] |
|
2014-10-23
§
|
| 23:39 |
<catrope> |
Synchronized php-1.25wmf4/extensions/CentralAuth/: SWAT (duration: 00m 04s) |
[production] |
| 23:38 |
<catrope> |
Synchronized php-1.25wmf4/extensions/AntiSpoof/: SWAT (duration: 00m 06s) |
[production] |
| 23:29 |
<catrope> |
Synchronized php-1.25wmf5/extensions/TimedMediaHandler/: SWAT (duration: 00m 04s) |
[production] |
| 23:29 |
<catrope> |
Synchronized php-1.25wmf5/extensions/UploadWizard/: SWAT (duration: 00m 06s) |
[production] |
| 23:28 |
<catrope> |
Synchronized php-1.25wmf4/extensions/UploadWizard/: SWAT (duration: 00m 04s) |
[production] |
| 23:28 |
<catrope> |
Synchronized php-1.25wmf4/extensions/TimedMediaHandler/: SWAT (duration: 00m 04s) |
[production] |
| 23:27 |
<catrope> |
Synchronized php-1.25wmf4/includes/api/ApiFormatBase.php: SWAT (duration: 00m 04s) |
[production] |
| 22:42 |
<hashar> |
Jenkins is all good now. |
[production] |
| 22:36 |
<hashar> |
Jenkins restarting |
[production] |
| 22:28 |
<hashar> |
preparing jenkins for restart |
[production] |
| 21:54 |
<hashar> |
Jenkins the Gearman plugin is holding a lock on deployment-bastion slave that prevents it from running any job :-/ |
[production] |
| 21:51 |
<ejegg> |
updated civicrm from ad3386cd0f9b776e2fded7c4e6b1195e05ed669c to 937df4dacae0dd620ae9e8fed13566d51c1b18a4 |
[production] |
| 21:46 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1018 (duration: 00m 06s) |
[production] |
| 21:43 |
<hashar> |
Jenkins: disabling / reenabling Gearman plugin |
[production] |
| 21:35 |
<hashar> |
Jenkins: disconnected / reconnected slave node deployment-bastion.eqiad |
[production] |
| 21:15 |
<awight> |
updated crm from 03b15f7dad58ce61894d632e8fbebd2ae76ae4d0 to ad3386cd0f9b776e2fded7c4e6b1195e05ed669c |
[production] |