|
2014-06-11
§
|
| 19:17 |
<greg-g> |
mw1151 *still* giving permission denied errors (publickey), what's the status, yo? |
[production] |
| 19:03 |
<bblack> |
rebooting lvs3002 for 3.13 kernel + XPS |
[production] |
| 18:59 |
<marktraceur> |
Started scap: MultimediaViewer fixes for cards 630, 429, and 697 |
[production] |
| 18:44 |
<ottomata> |
disabling puppet on analytics1012 to allow for more replica threads to catch up with current broker replicas...maybe :) |
[production] |
| 18:41 |
<awight> |
updated crm from b6815d29de97b80a0ab65db576213a604f0c7cb9 to c38296add61421f87e12cb5b4f3dd68bdf2340db |
[production] |
| 18:03 |
<Krinkle> |
Reloading Zuul to deploy I5d154a4002d08 |
[production] |
| 16:43 |
<bblack> |
shutting off lvs3002.esams pybal to test XPS balancing of live traffic on lvs3004.esams + 3.13 |
[production] |
| 16:30 |
<bblack> |
rebooting lvs3004 (inactive uploads LVS) for 3.13 again |
[production] |
| 14:52 |
<hashar> |
Jenkins restarting (plugin upgrades) |
[production] |
| 14:48 |
<bblack> |
rebooting lvs3004.esams (inactive uploads LVS) for 3.13 kernel |
[production] |
| 14:41 |
<_joe_> |
manually ran 'planet' on en.planet to restore technews |
[production] |
| 14:40 |
<hashar> |
Jenkins updating plugins |
[production] |
| 13:56 |
<paravoid> |
upgrading mw1153-mw1160, tmh1001-tmh1002 for USN-2244-1 |
[production] |
| 12:21 |
<_joe_> |
set up a secondary remote named 'readonly' in /a/common on tin, to use with the icinga check for unmerged commits |
[production] |
| 11:40 |
<akosiaris> |
manually cleaning librenms tables. db1001 is going to have increased load for some time. The approach is automatable, see http://jira.observium.org/browse/OBSERVIUM-757 |
[production] |
| 11:32 |
<godog> |
restarted uwsgi on tungsten, a lot of accesses to reqstats.edits.*.submits |
[production] |
| 10:45 |
<godog> |
restarted uwsgi on tungsten, hung on fetching many metrics |
[production] |
| 09:54 |
<_joe_> |
restarted apache on palladium - passenger crashed |
[production] |
| 05:26 |
<paravoid> |
restarting all swift daemons across the cluster to fix runaway threads due to rsyslog restart |
[production] |
| 05:04 |
<springle> |
beginning schema changes bug 49193 page_content_model |
[production] |
| 03:29 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jun 11 03:28:14 UTC 2014 (duration 28m 13s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-11 02:28:18+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf7) at 2014-06-11 02:14:43+00:00 |
[production] |
|
2014-06-10
§
|
| 23:34 |
<andrewbogott> |
updated labs Trusty image w/puppet3, made default |
[production] |
| 23:19 |
<mutante> |
rebooting unresponsive ms-be1003 |
[production] |
| 21:10 |
<RobH> |
montly sms credit check: 1,447.36 SMS credits. will check again in 30 days |
[production] |
| 19:47 |
<hashar> |
Jenkins restarted apparently properly. Any breakage would probably be related to the version switch :-D |
[production] |
| 19:45 |
<ottomata> |
power cycling analytics1012, attempting to reinstall as kafka broker with new kafka partman recipe |
[production] |
| 19:42 |
<hashar> |
Jenkins upgraded from 1.532.2 to 1.554.2 (i.e. bumped to a new LTS version). |
[production] |
| 19:37 |
<hashar> |
Broke Jenkins by silently upgrading it :-( |
[production] |
| 19:09 |
<Krinkle> |
git-deploy: Deploying integration/slave-scripts I9521890b911714edf2 |
[production] |
| 18:59 |
<reedy> |
Synchronized php-1.24wmf8/skins/vector/components/tabs.less: (no message) (duration: 00m 14s) |
[production] |
| 18:58 |
<mutante> |
shutting down ekrem |
[production] |
| 18:18 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: Enable data transclusion for wikiquote (duration: 00m 14s) |
[production] |
| 18:15 |
<reedy> |
Synchronized docroot and w: Update non Wikipedias to 1.24wmf8 (duration: 00m 16s) |
[production] |
| 18:15 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Update non Wikipedias to 1.24wmf8 |
[production] |
| 18:14 |
<reedy> |
Synchronized php-1.24wmf8/extensions/Wikidata/: (no message) (duration: 00m 16s) |
[production] |
| 17:29 |
<_joe|away> |
restarted profiler-to-carbon, stuck waiting data from mwprof |
[production] |
| 15:21 |
<mutante> |
ekrem - rm from stored configs/icinga |
[production] |
| 15:12 |
<mutante> |
ekrem - revoke salt,puppet keys, stop agents/minion |
[production] |
| 07:42 |
<springle> |
enabled pt-slave-delay for dbstore1001, 24h all shards |
[production] |
| 06:12 |
<springle> |
xtrabackup clone db1043 to db1048 |
[production] |
| 04:57 |
<springle> |
db1048 down for upgrade |
[production] |
| 03:40 |
<springle> |
switched mchenry to use m2-master/m2-slave for OTRS address lookups |
[production] |
| 03:25 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jun 10 03:24:19 UTC 2014 (duration 24m 18s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-10 02:28:14+00:00 |
[production] |
| 02:27 |
<springle> |
switched traffic db1048 to db1020. broke gerrit briefly; see ops email |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf7) at 2014-06-10 02:14:41+00:00 |
[production] |
| 01:33 |
<chasemp> |
restarted gerrit on ytterbium |
[production] |
| 01:01 |
<manybubbles> |
upgraded all elasticsearch servers in production to 1.2.1. They are just restoring the last few shards on the last node now and they'll spend a few hours tonight rebalancing after the upgrade but otherwise I'm done. |
[production] |