|
2014-08-21
§
|
| 20:57 |
<bd808> |
Started salt-minon on deployment-elastic04 |
[releng] |
| 20:57 |
<bd808> |
Started salt-minon on deployment-analytics01 |
[releng] |
| 20:55 |
<bd808> |
Started salt-minon on deployment-cache-upload02 |
[releng] |
| 20:54 |
<bd808> |
Started salt-minon on deployment-memc04 |
[releng] |
| 20:54 |
<bd808> |
Started salt-minon on deployment-parsoid04 |
[releng] |
| 20:49 |
<bd808> |
Started salt-minon on deployment-memc05 |
[releng] |
| 20:48 |
<bd808> |
Started salt-minon on deployment-db2 |
[releng] |
| 20:48 |
<twentyafterfour> |
Started salt-minion on deployment-cache-text02 |
[releng] |
| 20:47 |
<twentyafterfour> |
Started salt-minion on deployment-memc03 |
[releng] |
| 20:47 |
<bd808> |
Started salt-minon on deployment-cxserver01 |
[releng] |
| 20:44 |
<godog> |
rolling restart of swift-proxy on ms-fe1* |
[production] |
| 20:12 |
<bd808> |
List of broken salt minions can be obtained with `sudo salt-run manage.down` on deployment-salt |
[releng] |
| 20:11 |
<godog> |
restarted swift-proxy on ms-fe1001 |
[production] |
| 19:55 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 13s) |
[production] |
| 19:55 |
<bd808> |
Fixed salt on deployment-memc02 |
[releng] |
| 19:52 |
<bd808> |
Salt minions are broken all over beta. Hung grain-ensure calls, hung test.ping calls, downed minions |
[releng] |
| 19:50 |
<bd808> |
Killed dozens of grain-ensure calls and started salt-minion on deployment-cache-mobile03 |
[releng] |
| 19:49 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf18 |
[production] |
| 19:47 |
<bd808> |
Killed hung salt-call and started salt-minion on deployment-cache-bits01 |
[releng] |
| 19:46 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf17 |
[production] |
| 19:44 |
<reedy> |
Finished scap: testwiki to 1.24wmf18 (duration: 34m 01s) |
[production] |
| 19:31 |
<mutante> |
disabled mw1178 in pybal |
[production] |
| 19:28 |
<bd808> |
Deployed cherry-pick of Iea7217a for scap |
[releng] |
| 19:27 |
<bd808> |
Restarted salt-minion on deployment-jobrunner01 & deployment-videoscaler01 |
[releng] |
| 19:27 |
<godog> |
restarted memcached on ms-fe1004 |
[production] |
| 19:27 |
<bd808> |
Killed rogue salt-master process on deployment-bastion |
[releng] |
| 19:26 |
<bd808> |
Deleted salt keys for retired apache0[12] minions |
[releng] |
| 19:23 |
<reedy|webirc> |
mw1178 returned [255]: ssh: connect to host mw1178 port 22: Connection timed out |
[production] |
| 19:23 |
<reedy|webirc> |
mw1019 returned [127]: bash: sync-common: command not found |
[production] |
| 19:10 |
<reedy> |
Started scap: testwiki to 1.24wmf18 |
[production] |
| 18:28 |
<manybubbles> |
*victim* |
[production] |
| 18:28 |
<manybubbles> |
trying to recover from weird Elasticsearch upgrade failure by redoing the upgrade on one node while also blowing away the data directory during the upgrade. elastic1005, you are my first victem. |
[production] |
| 17:29 |
<cmjohnson1> |
removing mw1130 from pybal |
[production] |
| 14:54 |
<hashar> |
Jenkins: updated PHP CodeSniffer MediaWiki standard on all slaves. |
[production] |
| 14:36 |
<hashar_> |
Jenkins: updating mediawiki code sniffer repo bf82117..bc4e590 |
[production] |
| 10:02 |
<hashar> |
Jenkins installed plugin [https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin Throttle Concurrent Builds]. |
[production] |
| 03:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Aug 21 03:20:47 UTC 2014 (duration 20m 46s) |
[production] |
| 02:36 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-21 02:34:56+00:00 |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-21 02:19:26+00:00 |
[production] |
| 00:13 |
<bd808> |
Upgraded elasticsearch to 1.3.2 on deployment-logstash1 |
[releng] |
| 00:08 |
<MatmaRex> |
(manybubbles contd.) …a single node going down but I expect the cluster to stay "yellow" during the process- no alerts. |
[production] |
| 00:07 |
<manybubbles> |
bd808 needs to plan a logstash upgrade soon - let it be logged |
[production] |
| 00:00 |
<manybubbles> |
unattended rolling restart of Elasticsearch cluster is going just fine - adding the 30 minute sleep between servers and turning down the replication rate makes it pretty boring. |
[production] |