|
2014-06-10
§
|
| 19:47 |
<hashar> |
Jenkins restarted apparently properly. Any breakage would probably be related to the version switch :-D |
[production] |
| 19:45 |
<ottomata> |
power cycling analytics1012, attempting to reinstall as kafka broker with new kafka partman recipe |
[production] |
| 19:42 |
<hashar> |
Jenkins upgraded from 1.532.2 to 1.554.2 (i.e. bumped to a new LTS version). |
[production] |
| 19:37 |
<hashar> |
Broke Jenkins by silently upgrading it :-( |
[production] |
| 19:09 |
<Krinkle> |
git-deploy: Deploying integration/slave-scripts I9521890b911714edf2 |
[production] |
| 18:59 |
<reedy> |
Synchronized php-1.24wmf8/skins/vector/components/tabs.less: (no message) (duration: 00m 14s) |
[production] |
| 18:58 |
<mutante> |
shutting down ekrem |
[production] |
| 18:18 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: Enable data transclusion for wikiquote (duration: 00m 14s) |
[production] |
| 18:15 |
<reedy> |
Synchronized docroot and w: Update non Wikipedias to 1.24wmf8 (duration: 00m 16s) |
[production] |
| 18:15 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Update non Wikipedias to 1.24wmf8 |
[production] |
| 18:14 |
<reedy> |
Synchronized php-1.24wmf8/extensions/Wikidata/: (no message) (duration: 00m 16s) |
[production] |
| 17:29 |
<_joe|away> |
restarted profiler-to-carbon, stuck waiting data from mwprof |
[production] |
| 15:21 |
<mutante> |
ekrem - rm from stored configs/icinga |
[production] |
| 15:12 |
<mutante> |
ekrem - revoke salt,puppet keys, stop agents/minion |
[production] |
| 07:42 |
<springle> |
enabled pt-slave-delay for dbstore1001, 24h all shards |
[production] |
| 06:12 |
<springle> |
xtrabackup clone db1043 to db1048 |
[production] |
| 04:57 |
<springle> |
db1048 down for upgrade |
[production] |
| 03:40 |
<springle> |
switched mchenry to use m2-master/m2-slave for OTRS address lookups |
[production] |
| 03:25 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jun 10 03:24:19 UTC 2014 (duration 24m 18s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-10 02:28:14+00:00 |
[production] |
| 02:27 |
<springle> |
switched traffic db1048 to db1020. broke gerrit briefly; see ops email |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf7) at 2014-06-10 02:14:41+00:00 |
[production] |
| 01:33 |
<chasemp> |
restarted gerrit on ytterbium |
[production] |
| 01:01 |
<manybubbles> |
upgraded all elasticsearch servers in production to 1.2.1. They are just restoring the last few shards on the last node now and they'll spend a few hours tonight rebalancing after the upgrade but otherwise I'm done. |
[production] |
| 00:41 |
<mwalker> |
updating donationinterface on payments from b4c5cf1bceb70d65eae28cdd0873036dc33c8992 to 6d74002f2634f41f7038daa7357ff6de55ee4880 for worldpay form error |
[production] |
|
2014-06-09
§
|
| 23:58 |
<manybubbles> |
lied - upgrading elastic1014 |
[production] |
| 23:57 |
<manybubbles> |
upgrading elastic1015 |
[production] |
| 23:30 |
<Krinkle> |
Reloading Zuul to deploy 6727b8b |
[production] |
| 23:12 |
<maxsem> |
Synchronized php-1.24wmf8/extensions/MobileApp: (no message) (duration: 00m 03s) |
[production] |
| 23:11 |
<maxsem> |
Synchronized php-1.24wmf7/extensions/MobileApp: (no message) (duration: 00m 03s) |
[production] |
| 23:03 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://bugzilla.wikimedia.org/66377 (duration: 00m 04s) |
[production] |
| 20:42 |
<manybubbles> |
upgraded elastic1007-elastic1010 without issue - starting elastic1010 |
[production] |
| 20:08 |
<subbu> |
deployed Parsoid 9b673587 (deploy sha 7d0097a1) |
[production] |
| 19:23 |
<ottomata> |
disabling puppet on analytics1012 |
[production] |
| 19:00 |
<ottomata> |
decomissioning analytics1012 in hadoop cluster, this will become a Kafka broker |
[production] |
| 17:59 |
<manybubbles> |
elastic1004-1006 upgraded without trouble - cluster is working on filling elatic1006 before moving on to 1007, and the rest |
[production] |
| 17:04 |
<andrewbogott> |
switching labs to puppet3 |
[production] |
| 17:03 |
<awight> |
update crm from b38497a9d0ef75fe2b20b03b649ac13a5e3f47a7 to b6815d29de97b80a0ab65db576213a604f0c7cb9 |
[production] |
| 16:30 |
<manybubbles> |
upgrading elastic1003 - upgrade is going well so far so I'm going to stop watching it as closely and let it be more automated |
[production] |
| 15:28 |
<manybubbles> |
elastic1001 went well, doing 1002 by hand again |
[production] |
| 15:17 |
<anomie> |
Synchronized php-1.24wmf8/extensions/Wikidata: SWAT: Wikidata entity suggester bug fixes [[gerrit:138339]] (duration: 00m 16s) |
[production] |
| 15:12 |
<greg-g> |
mw1151 still "permission denied" during deploys |
[production] |
| 15:12 |
<anomie> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateData GUI on Portuguese Wikipedia [[gerrit:137986]] (duration: 00m 14s) |
[production] |
| 15:09 |
<anomie> |
Synchronized php-1.24wmf7/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: SWAT: VE fix for focus regression [[gerrit:137978]] (duration: 00m 15s) |
[production] |
| 15:06 |
<andrewbogott> |
beta updating all instances to puppet 3 via a cherry-pick of https://gerrit.wikimedia.org/r/#/c/137898/ on deployment-salt |
[production] |
| 15:05 |
<anomie> |
Synchronized php-1.24wmf8/extensions/VisualEditor/modules/ve-mw/: SWAT: VE fix for focus regression and alignment issues [[gerrit:137971]] [[gerrit:138122]] (duration: 00m 14s) |
[production] |
| 15:01 |
<manybubbles> |
successfully synced plugins, upgrading elastic1001 to make sure everything is working ok with it - then we'll run through the others more quickly |
[production] |
| 14:57 |
<manybubbles> |
syncing elasticsearch plugins for 1.2.1 - any elasticsearch restart from here on out needs to come with 1.2.1 or the node will break. |
[production] |
| 14:54 |
<manybubbles> |
starting Elasticsearch upgrade with elastic1001 |
[production] |
| 07:14 |
<springle> |
disabled puppet on analytics1021 to avoid kafka broker restarting with missing mount |
[production] |