|
2014-06-11
§
|
| 11:32 |
<godog> |
restarted uwsgi on tungsten, a lot of accesses to reqstats.edits.*.submits |
[production] |
| 10:45 |
<godog> |
restarted uwsgi on tungsten, hung on fetching many metrics |
[production] |
| 09:54 |
<_joe_> |
restarted apache on palladium - passenger crashed |
[production] |
| 05:26 |
<paravoid> |
restarting all swift daemons across the cluster to fix runaway threads due to rsyslog restart |
[production] |
| 05:04 |
<springle> |
beginning schema changes bug 49193 page_content_model |
[production] |
| 03:29 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jun 11 03:28:14 UTC 2014 (duration 28m 13s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-11 02:28:18+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf7) at 2014-06-11 02:14:43+00:00 |
[production] |
|
2014-06-10
§
|
| 23:34 |
<andrewbogott> |
updated labs Trusty image w/puppet3, made default |
[production] |
| 23:19 |
<mutante> |
rebooting unresponsive ms-be1003 |
[production] |
| 21:10 |
<RobH> |
montly sms credit check: 1,447.36 SMS credits. will check again in 30 days |
[production] |
| 19:47 |
<hashar> |
Jenkins restarted apparently properly. Any breakage would probably be related to the version switch :-D |
[production] |
| 19:45 |
<ottomata> |
power cycling analytics1012, attempting to reinstall as kafka broker with new kafka partman recipe |
[production] |
| 19:42 |
<hashar> |
Jenkins upgraded from 1.532.2 to 1.554.2 (i.e. bumped to a new LTS version). |
[production] |
| 19:37 |
<hashar> |
Broke Jenkins by silently upgrading it :-( |
[production] |
| 19:09 |
<Krinkle> |
git-deploy: Deploying integration/slave-scripts I9521890b911714edf2 |
[production] |
| 18:59 |
<reedy> |
Synchronized php-1.24wmf8/skins/vector/components/tabs.less: (no message) (duration: 00m 14s) |
[production] |
| 18:58 |
<mutante> |
shutting down ekrem |
[production] |
| 18:18 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: Enable data transclusion for wikiquote (duration: 00m 14s) |
[production] |
| 18:15 |
<reedy> |
Synchronized docroot and w: Update non Wikipedias to 1.24wmf8 (duration: 00m 16s) |
[production] |
| 18:15 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Update non Wikipedias to 1.24wmf8 |
[production] |
| 18:14 |
<reedy> |
Synchronized php-1.24wmf8/extensions/Wikidata/: (no message) (duration: 00m 16s) |
[production] |
| 17:29 |
<_joe|away> |
restarted profiler-to-carbon, stuck waiting data from mwprof |
[production] |
| 15:21 |
<mutante> |
ekrem - rm from stored configs/icinga |
[production] |
| 15:12 |
<mutante> |
ekrem - revoke salt,puppet keys, stop agents/minion |
[production] |
| 07:42 |
<springle> |
enabled pt-slave-delay for dbstore1001, 24h all shards |
[production] |
| 06:12 |
<springle> |
xtrabackup clone db1043 to db1048 |
[production] |
| 04:57 |
<springle> |
db1048 down for upgrade |
[production] |
| 03:40 |
<springle> |
switched mchenry to use m2-master/m2-slave for OTRS address lookups |
[production] |
| 03:25 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jun 10 03:24:19 UTC 2014 (duration 24m 18s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-10 02:28:14+00:00 |
[production] |
| 02:27 |
<springle> |
switched traffic db1048 to db1020. broke gerrit briefly; see ops email |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf7) at 2014-06-10 02:14:41+00:00 |
[production] |
| 01:33 |
<chasemp> |
restarted gerrit on ytterbium |
[production] |
| 01:01 |
<manybubbles> |
upgraded all elasticsearch servers in production to 1.2.1. They are just restoring the last few shards on the last node now and they'll spend a few hours tonight rebalancing after the upgrade but otherwise I'm done. |
[production] |
| 00:41 |
<mwalker> |
updating donationinterface on payments from b4c5cf1bceb70d65eae28cdd0873036dc33c8992 to 6d74002f2634f41f7038daa7357ff6de55ee4880 for worldpay form error |
[production] |
|
2014-06-09
§
|
| 23:58 |
<manybubbles> |
lied - upgrading elastic1014 |
[production] |
| 23:57 |
<manybubbles> |
upgrading elastic1015 |
[production] |
| 23:30 |
<Krinkle> |
Reloading Zuul to deploy 6727b8b |
[production] |
| 23:12 |
<maxsem> |
Synchronized php-1.24wmf8/extensions/MobileApp: (no message) (duration: 00m 03s) |
[production] |
| 23:11 |
<maxsem> |
Synchronized php-1.24wmf7/extensions/MobileApp: (no message) (duration: 00m 03s) |
[production] |
| 23:03 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://bugzilla.wikimedia.org/66377 (duration: 00m 04s) |
[production] |
| 20:42 |
<manybubbles> |
upgraded elastic1007-elastic1010 without issue - starting elastic1010 |
[production] |
| 20:08 |
<subbu> |
deployed Parsoid 9b673587 (deploy sha 7d0097a1) |
[production] |
| 19:23 |
<ottomata> |
disabling puppet on analytics1012 |
[production] |
| 19:00 |
<ottomata> |
decomissioning analytics1012 in hadoop cluster, this will become a Kafka broker |
[production] |
| 17:59 |
<manybubbles> |
elastic1004-1006 upgraded without trouble - cluster is working on filling elatic1006 before moving on to 1007, and the rest |
[production] |
| 17:04 |
<andrewbogott> |
switching labs to puppet3 |
[production] |
| 17:03 |
<awight> |
update crm from b38497a9d0ef75fe2b20b03b649ac13a5e3f47a7 to b6815d29de97b80a0ab65db576213a604f0c7cb9 |
[production] |
| 16:30 |
<manybubbles> |
upgrading elastic1003 - upgrade is going well so far so I'm going to stop watching it as closely and let it be more automated |
[production] |