|
2014-02-25
§
|
| 08:42 |
<hashar> |
deployment-prep Upgrading all varnishes. |
[production] |
| 03:38 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-25 03:38:50+00:00 |
[production] |
| 03:06 |
<springle> |
synchronized wmf-config/db-eqiad.php 's1 direct api traffic to db1043' |
[production] |
| 02:53 |
<LocalisationUpdate> |
completed (1.23wmf15) at 2014-02-25 02:53:09+00:00 |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.23wmf14) at 2014-02-25 02:27:26+00:00 |
[production] |
| 02:24 |
<awight> |
crm updated from c2e57365220e3ddbdea8cf99a052adbfa7a5bac8 to 37bb48ea2cf2bff656e97b8455115d38fa5c3883 |
[production] |
| 02:15 |
<awight> |
updated crm from 41dce289bc15ea1ca638c37b29ff2e3e709a2251 to c2e57365220e3ddbdea8cf99a052adbfa7a5bac8 |
[production] |
| 00:31 |
<catrope> |
synchronized php-1.23wmf15/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWSettingsPage.js 'Fix adding redirects in VE' |
[production] |
| 00:29 |
<catrope> |
synchronized php-1.23wmf15/includes/filebackend/FileBackendStore.php 'd52a8af6 for real this time' |
[production] |
| 00:27 |
<catrope> |
synchronized php-1.23wmf14/includes/filebackend/FileBackendStore.php 'efb1e99fd for real this time' |
[production] |
| 00:23 |
<mutante> |
Bugzilla - replacing custom template with exiting hook (gerrit 114145) |
[production] |
| 00:17 |
<mutante> |
deploying new hook in Bugzilla's edit.html.tmpl for bug 36064 |
[production] |
| 00:09 |
<mutante> |
restarting gitblit on antimony |
[production] |
| 00:09 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Fix popup video size by ordering transcode settings properly' |
[production] |
| 00:08 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Enable VE in the Recherce namespace on frwikiversity' |
[production] |
| 00:07 |
<catrope> |
synchronized visualeditor-default.dblist 'Enable VE by default on ptwikibooks' |
[production] |
|
2014-02-24
§
|
| 23:36 |
<mutante> |
DNS update - removing kaulen |
[production] |
| 23:33 |
<RobH> |
all neon services no longer using wildcard, and wildcard shredded off system |
[production] |
| 23:19 |
<RobH> |
updating ishmael to use its own ssl cert |
[production] |
| 23:01 |
<RobH> |
icinga and icinga-admin now using their own certs |
[production] |
| 22:56 |
<mutante> |
kaulen - revoke puppet cert, revoke salt key, stored configs,... |
[production] |
| 22:50 |
<mutante> |
kaulen - shutdown -h now |
[production] |
| 21:58 |
<RobH> |
updated blog.w.o to wp3.8.1 |
[production] |
| 21:51 |
<demon> |
synchronized php-1.23wmf15/extensions/Wikidata 'Updating wikidata build to fix test.wikidata' |
[production] |
| 21:50 |
<aaron> |
synchronized php-1.23wmf15/includes/filebackend 'd52a8af6c2f730d017e87e7217b6a0b299ab85be' |
[production] |
| 21:49 |
<aaron> |
synchronized php-1.23wmf15/includes/filebackend 'efb1e99fdf1e91bef4fc086b945b7933049e2a50' |
[production] |
| 21:44 |
<mutante> |
kaulen - stopping services, disabling monitoring |
[production] |
| 21:26 |
<RobH> |
techblog.w.o redirects now work without certificate errors |
[production] |
| 21:25 |
<RobH> |
updating blog apache configs to use techblog.w.o https cert |
[production] |
| 21:24 |
<gwicke> |
deployed Parsoid 51c71eb / deploy b684fea |
[production] |
| 21:17 |
<mutante> |
restarting parsoid on wtp1002 |
[production] |
| 20:33 |
<bd808> |
Restarted elasticsearch on logstash1002 in attempt to clear stuck reallocations likely caused by OOM while running recovery |
[production] |
| 19:30 |
<Coren> |
remmoting virt1001 (sick stuck on bad mounts) |
[production] |
| 17:06 |
<bd808> |
Logs on logstash1003 show elasticsearch split brain starting at 2014-02-23T00:00:12. logstash1001 and logstash1003 both thought they were master. logstash1001 not responding to logstash1003's requests to become authoritative. |
[production] |
| 16:38 |
<bd808> |
Logstash elasticsearch split-brain resulted in loss of all logs for 2014-02-24 from 00:00Z to ~16:30Z |
[production] |
| 16:16 |
<bd808> |
Restarted elasticsearch on logstash1001 |
[production] |
| 03:28 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-02-24 03:27:55+00:00 |
[production] |
| 02:44 |
<LocalisationUpdate> |
completed (1.23wmf15) at 2014-02-24 02:44:44+00:00 |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.23wmf14) at 2014-02-24 02:31:50+00:00 |
[production] |
| 01:16 |
<tstarling> |
updated /a/common/php-1.23wmf15 to {{Gerrit|I268599be9}}: [1.23wmf15] Make SiteStats (re)initializing more sane |
[production] |
| 01:16 |
<tstarling> |
synchronized php-1.23wmf14/includes/SiteStats.php |
[production] |
|
2014-02-23
§
|
| 20:29 |
<Tim> |
updated ss_active_users on plwiki master to not be -1 |
[production] |
| 20:14 |
<springle> |
killed SiteStatsInit from both wikiuser and wikiadmin on all s2 slaves |
[production] |
| 20:01 |
<Tim> |
killed SiteStatsInit queries on db1060 |
[production] |
| 19:57 |
<tstarling> |
synchronized php-1.23wmf15/includes/SiteStats.php |
[production] |
| 19:56 |
<tstarling> |
synchronized php-1.23wmf14/includes/SiteStats.php |
[production] |
| 19:48 |
<RobH> |
operations folks are looking into site issues at present |
[production] |
| 19:38 |
<greg-g> |
< paravoid> something that has to do with SiteStatsInit, probably |
[production] |
| 19:33 |
<greg-g> |
< paravoid> it's all plwiki |
[production] |
| 19:33 |
<greg-g> |
< paravoid> tons of SELECT /* SiteStatsInit::edits */ COUNT(*) FROM `revision` LIMIT 1 |
[production] |