|
2014-02-24
§
|
| 23:36 |
<mutante> |
DNS update - removing kaulen |
[production] |
| 23:33 |
<RobH> |
all neon services no longer using wildcard, and wildcard shredded off system |
[production] |
| 23:19 |
<RobH> |
updating ishmael to use its own ssl cert |
[production] |
| 23:01 |
<RobH> |
icinga and icinga-admin now using their own certs |
[production] |
| 22:56 |
<mutante> |
kaulen - revoke puppet cert, revoke salt key, stored configs,... |
[production] |
| 22:50 |
<mutante> |
kaulen - shutdown -h now |
[production] |
| 21:58 |
<RobH> |
updated blog.w.o to wp3.8.1 |
[production] |
| 21:51 |
<demon> |
synchronized php-1.23wmf15/extensions/Wikidata 'Updating wikidata build to fix test.wikidata' |
[production] |
| 21:50 |
<aaron> |
synchronized php-1.23wmf15/includes/filebackend 'd52a8af6c2f730d017e87e7217b6a0b299ab85be' |
[production] |
| 21:49 |
<aaron> |
synchronized php-1.23wmf15/includes/filebackend 'efb1e99fdf1e91bef4fc086b945b7933049e2a50' |
[production] |
| 21:44 |
<mutante> |
kaulen - stopping services, disabling monitoring |
[production] |
| 21:26 |
<RobH> |
techblog.w.o redirects now work without certificate errors |
[production] |
| 21:25 |
<RobH> |
updating blog apache configs to use techblog.w.o https cert |
[production] |
| 21:24 |
<gwicke> |
deployed Parsoid 51c71eb / deploy b684fea |
[production] |
| 21:17 |
<mutante> |
restarting parsoid on wtp1002 |
[production] |
| 20:33 |
<bd808> |
Restarted elasticsearch on logstash1002 in attempt to clear stuck reallocations likely caused by OOM while running recovery |
[production] |
| 19:30 |
<Coren> |
remmoting virt1001 (sick stuck on bad mounts) |
[production] |
| 17:06 |
<bd808> |
Logs on logstash1003 show elasticsearch split brain starting at 2014-02-23T00:00:12. logstash1001 and logstash1003 both thought they were master. logstash1001 not responding to logstash1003's requests to become authoritative. |
[production] |
| 16:38 |
<bd808> |
Logstash elasticsearch split-brain resulted in loss of all logs for 2014-02-24 from 00:00Z to ~16:30Z |
[production] |
| 16:16 |
<bd808> |
Restarted elasticsearch on logstash1001 |
[production] |
| 03:28 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-24 03:27:55+00:00 |
[production] |
| 02:44 |
<LocalisationUpdate> |
completed (1.23wmf15) at 2014-02-24 02:44:44+00:00 |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.23wmf14) at 2014-02-24 02:31:50+00:00 |
[production] |
| 01:16 |
<tstarling> |
updated /a/common/php-1.23wmf15 to {{Gerrit|I268599be9}}: [1.23wmf15] Make SiteStats (re)initializing more sane |
[production] |
| 01:16 |
<tstarling> |
synchronized php-1.23wmf14/includes/SiteStats.php |
[production] |
|
2014-02-23
§
|
| 20:29 |
<Tim> |
updated ss_active_users on plwiki master to not be -1 |
[production] |
| 20:14 |
<springle> |
killed SiteStatsInit from both wikiuser and wikiadmin on all s2 slaves |
[production] |
| 20:01 |
<Tim> |
killed SiteStatsInit queries on db1060 |
[production] |
| 19:57 |
<tstarling> |
synchronized php-1.23wmf15/includes/SiteStats.php |
[production] |
| 19:56 |
<tstarling> |
synchronized php-1.23wmf14/includes/SiteStats.php |
[production] |
| 19:48 |
<RobH> |
operations folks are looking into site issues at present |
[production] |
| 19:38 |
<greg-g> |
< paravoid> something that has to do with SiteStatsInit, probably |
[production] |
| 19:33 |
<greg-g> |
< paravoid> it's all plwiki |
[production] |
| 19:33 |
<greg-g> |
< paravoid> tons of SELECT /* SiteStatsInit::edits */ COUNT(*) FROM `revision` LIMIT 1 |
[production] |
| 19:32 |
<greg-g> |
< paravoid> it's s2 |
[production] |
| 02:08 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-23 02:08:36+00:00 |
[production] |
| 02:02 |
<LocalisationUpdate> |
completed (1.23wmf15) at 2014-02-23 02:02:44+00:00 |
[production] |
| 02:02 |
<LocalisationUpdate> |
completed (1.23wmf14) at 2014-02-23 02:01:56+00:00 |
[production] |