|
2014-08-22
§
|
| 15:18 |
<springle> |
upgrade & restart db1053, fs check |
[production] |
| 15:08 |
<bd808> |
Still no apache2.log on fluorine or in logstash. Log seems to be available on fenari. |
[production] |
| 14:51 |
<springle> |
switched s1 sanitarium and labsdb replication to db1069:3311 mariadb 10 |
[production] |
| 14:39 |
<mark> |
Removed IPv6 subnets 2620:0:860:1::/64 (squid subnet) and 2620:0:860:3::/64 (sandbox subnet) from cr2-pmtpa configuration |
[production] |
| 04:11 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Aug 22 04:10:47 UTC 2014 (duration 10m 46s) |
[production] |
| 03:19 |
<LocalisationUpdate> |
completed (1.24wmf18) at 2014-08-22 03:18:44+00:00 |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-22 02:32:11+00:00 |
[production] |
| 00:41 |
<bd808> |
cherry-picked scap change https://gerrit.wikimedia.org/r/#/c/155677/ for testing |
[releng] |
| 00:03 |
<ori> |
Finished scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01 (duration: 06m 29s) |
[production] |
|
2014-08-21
§
|
| 23:57 |
<ori> |
Started scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01 |
[production] |
| 21:58 |
<ori> |
Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.js: I8d27442d1: Workaround for bug introduced by Icf6ede09b (duration: 00m 03s) |
[production] |
| 21:57 |
<manybubbles> |
performing elasticsearch upgrade on elastic1015 |
[production] |
| 21:49 |
<bd808> |
Trebuchet happier after all the salt-minion restarts; still have deleted hosts showing in the expected minion list for scap deploys |
[releng] |
| 21:02 |
<ori> |
Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.util.js: Touch resources/src/mediawiki/mediawiki.util.js (duration: 00m 06s) |
[production] |
| 21:01 |
<twentyafterfour> |
Started salt-minion on deployment-redis01 |
[releng] |
| 21:01 |
<bd808> |
Started salt-minon on deployment-upload |
[releng] |
| 21:00 |
<bd808> |
Started salt-minon on deployment-fluoride |
[releng] |
| 21:00 |
<bd808> |
Started salt-minon on deployment-db1 |
[releng] |
| 20:59 |
<bd808> |
Started salt-minon on deployment-elastic01 |
[releng] |
| 20:59 |
<twentyafterfour> |
Started salt-minion on deployment-eventlogging02 |
[releng] |
| 20:58 |
<bd808> |
Started salt-minon on deployment-elastic02 |
[releng] |
| 20:58 |
<bd808> |
Started salt-minon on deployment-elastic03 |
[releng] |
| 20:57 |
<bd808> |
Started salt-minon on deployment-elastic04 |
[releng] |
| 20:57 |
<bd808> |
Started salt-minon on deployment-analytics01 |
[releng] |
| 20:55 |
<bd808> |
Started salt-minon on deployment-cache-upload02 |
[releng] |
| 20:54 |
<bd808> |
Started salt-minon on deployment-memc04 |
[releng] |
| 20:54 |
<bd808> |
Started salt-minon on deployment-parsoid04 |
[releng] |
| 20:49 |
<bd808> |
Started salt-minon on deployment-memc05 |
[releng] |
| 20:48 |
<bd808> |
Started salt-minon on deployment-db2 |
[releng] |
| 20:48 |
<twentyafterfour> |
Started salt-minion on deployment-cache-text02 |
[releng] |
| 20:47 |
<twentyafterfour> |
Started salt-minion on deployment-memc03 |
[releng] |
| 20:47 |
<bd808> |
Started salt-minon on deployment-cxserver01 |
[releng] |
| 20:44 |
<godog> |
rolling restart of swift-proxy on ms-fe1* |
[production] |
| 20:12 |
<bd808> |
List of broken salt minions can be obtained with `sudo salt-run manage.down` on deployment-salt |
[releng] |
| 20:11 |
<godog> |
restarted swift-proxy on ms-fe1001 |
[production] |
| 19:55 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 13s) |
[production] |
| 19:55 |
<bd808> |
Fixed salt on deployment-memc02 |
[releng] |
| 19:52 |
<bd808> |
Salt minions are broken all over beta. Hung grain-ensure calls, hung test.ping calls, downed minions |
[releng] |
| 19:50 |
<bd808> |
Killed dozens of grain-ensure calls and started salt-minion on deployment-cache-mobile03 |
[releng] |
| 19:49 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf18 |
[production] |
| 19:47 |
<bd808> |
Killed hung salt-call and started salt-minion on deployment-cache-bits01 |
[releng] |
| 19:46 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf17 |
[production] |
| 19:44 |
<reedy> |
Finished scap: testwiki to 1.24wmf18 (duration: 34m 01s) |
[production] |
| 19:31 |
<mutante> |
disabled mw1178 in pybal |
[production] |
| 19:28 |
<bd808> |
Deployed cherry-pick of Iea7217a for scap |
[releng] |
| 19:27 |
<bd808> |
Restarted salt-minion on deployment-jobrunner01 & deployment-videoscaler01 |
[releng] |
| 19:27 |
<godog> |
restarted memcached on ms-fe1004 |
[production] |
| 19:27 |
<bd808> |
Killed rogue salt-master process on deployment-bastion |
[releng] |
| 19:26 |
<bd808> |
Deleted salt keys for retired apache0[12] minions |
[releng] |
| 19:23 |
<reedy|webirc> |
mw1178 returned [255]: ssh: connect to host mw1178 port 22: Connection timed out |
[production] |