2001-2050 of 10000 results (54ms)
2017-06-29 §
10:30 <ema> repooling acamar T168462 [production]
09:29 <godog> silence paging alerts for *.svc.codfw.wmnet for two hours - T168462 [production]
08:34 <marostegui> Shutdown MySQL and reboot db1034 for maintenance [production]
08:29 <XioNoX> asw-a-codfw upgrade started - T168462 [production]
08:25 <ema> failover codfw LVSs to secondaries T168462 [production]
08:19 <elukey> restart pdfrender on scb1004 - xpra issue [production]
08:16 <volans@neodymium> conftool action : set/pooled=false; selector: name=codfw,dnsdisc=restbase-async [production]
08:15 <volans@neodymium> conftool action : set/pooled=false; selector: name=codfw,dnsdisc=citoid [production]
08:14 <volans@neodymium> conftool action : set/ttl=60; selector: dnsdisc=restbase-async [production]
08:14 <volans@neodymium> conftool action : set/ttl=60; selector: dnsdisc=citoid [production]
08:08 <volans@neodymium> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=citoid [production]
08:07 <volans@neodymium> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=restbase-async [production]
08:05 <ema> bounce pybal on codfw secondary LVSs (lvs2004-2006) [production]
07:57 <volans> switching citoid and restbase-async temporarily to eqiad for T168462 [production]
07:47 <XioNoX> Route cache traffic around codfw - T168462 [production]
07:46 <ema@neodymium> conftool action : set/pooled=no; selector: name=acamar.wikimedia.org,service=pdns_recursor [production]
07:44 <XioNoX> codfw depooled from DNS - T168462 [production]
07:36 <elukey> depooled kafka2001.codfw.wmnet for T168462 [production]
07:19 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment (duration: 02m 55s) [production]
07:18 <marostegui> Disable event scheduler temporarily on dbstore1001 - T169050 [production]
07:16 <elukey@tin> Started deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment [production]
07:13 <marostegui> Deploy alter table on s7 - db1028 - T166208 [production]
07:12 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment (duration: 02m 36s) [production]
07:10 <elukey@tin> Started deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment [production]
07:10 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1028 - T166208 (duration: 00m 47s) [production]
07:09 <joal@tin> Finished deploy [analytics/refinery@f6cccf9]: (no justification provided) (duration: 05m 55s) [production]
07:03 <joal@tin> Started deploy [analytics/refinery@f6cccf9]: (no justification provided) [production]
04:33 <Krinkle> 'service hhvm restart' on mwdebug1001 and mwdebug1002 (T168540) [production]
03:56 <Krinkle> 'service hhvm restart' on mwdebug1001 and mwdebug1002 to help investigate T168540 [production]
03:09 <kartik@tin> Finished deploy [cxserver/deploy@6f0e9a7]: Update cxserver to e69353b (duration: 02m 28s) [production]
03:07 <kartik@tin> Started deploy [cxserver/deploy@6f0e9a7]: Update cxserver to e69353b [production]
02:52 <l10nupdate@tin> ResourceLoader cache refresh completed at Thu Jun 29 02:52:57 UTC 2017 (duration 6m 52s) [production]
02:46 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.7) (duration: 07m 43s) [production]
02:26 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.6) (duration: 09m 24s) [production]
01:08 <mutante> mwlog1001 - deleted /srv/xenon/logs from 2015 and 2016 as requested by Krinkle. Also merged https://gerrit.wikimedia.org/r/#/c/362114/ so now logs are retained for 14 days [production]
00:23 <krinkle@tin> Synchronized wmf-config/InitialiseSettings.php: I8ce28a4ce7 - test2wiki config cleanup (duration: 00m 47s) [production]
2017-06-28 §
23:44 <thcipriani@tin> Synchronized php-1.30.0-wmf.7/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [[gerrit:361950|Adding ssclick events for sister-search results]] T168916 (duration: 00m 46s) [production]
23:36 <thcipriani@tin> Synchronized php-1.30.0-wmf.6/extensions/MobileFrontend/includes/specials/SpecialMobileDiff.php: SWAT: [[gerrit:361944|Revert "Run DiffViewHeader in mobile mode, too"]] T169024 (duration: 00m 46s) [production]
23:35 <thcipriani@tin> Synchronized php-1.30.0-wmf.7/extensions/MobileFrontend/includes/specials/SpecialMobileDiff.php: SWAT: [[gerrit:361945|Revert "Run DiffViewHeader in mobile mode, too"]] T169024 (duration: 00m 47s) [production]
22:10 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: rm more stupid logging, wow this stuff has piled up (duration: 00m 46s) [production]
22:09 <ppchelko@tin> Finished deploy [eventstreams/deploy@ba71a84]: redeploy to pick up config changes (duration: 02m 01s) [production]
22:07 <ppchelko@tin> Started deploy [eventstreams/deploy@ba71a84]: redeploy to pick up config changes [production]
22:06 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: kill temp-debug (duration: 00m 46s) [production]
21:50 <robh> wtp1025-1048 are having icinga reporting errors, they are new installs on stretch [production]
21:48 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: kill weird testwiki logging (duration: 00m 47s) [production]
21:38 <ppchelko@tin> Finished deploy [eventstreams/deploy@05bcc8f]: redeploy to pick up config changes (duration: 00m 20s) [production]
21:37 <ppchelko@tin> Started deploy [eventstreams/deploy@05bcc8f]: redeploy to pick up config changes [production]
21:34 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: kill oai logging channel (duration: 00m 47s) [production]
20:17 <twentyafterfour@tin> Synchronized php-1.30.0-wmf.7/extensions/VisualEditor/VisualEditor.hooks.php: sync https://gerrit.wikimedia.org/r/#/c/361941/ refs T169132 T167536 (duration: 00m 47s) [production]
20:08 <mutante> migrating servermon to stretch on netmon1002 is currently blocked by "python-django-south" package not existing anymore [production]