601-650 of 10000 results (44ms)
2020-03-30 §
09:19 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:15 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:30 <vgutierrez> pool cp2029 - T248816 [production]
08:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
08:12 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:53 <vgutierrez> depool & decommission cp2002 - T248818 [production]
07:48 <marostegui> Run cloudcontrol1003:~# wmcs-wikireplica-dns to promote dbproxy1018 to wikireplicas active proxy T231520 [production]
07:40 <marostegui> Replace dbproxy1010 with dbproxy1011 for wiki replicas, analytics - T231520 [production]
07:28 <marostegui> Deploy schema change on labswiki (wikitech) - T248333 [production]
07:27 <elukey> run /usr/local/bin/refine_sanitize_eventlogging_analytics_immediate --ignore_failure_flag=true --since=72 --verbose --table_whitelist_regex="ResourceTiming" refine_sanitize_eventlogging_analytics_immediate to fix _REFINE_FAILED events [analytics]
07:26 <marostegui> Deploy schema change on s4 codfw, this will generate lag on codfw - T248333 [production]
07:17 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
07:17 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:16 <elukey> run eventlogging refine manually for schemas "EditorActivation|EditorJourney|HomepageVisit|VisualEditorFeatureUse|WikibaseTermboxInteraction|UploadWizardErrorFlowEvent|MobileWikiAppiOSReadingLists|ContentTranslationCTA|QuickSurveysResponses|MobileWikiAppiOSSessions to fix _REFINE_FAILED events [analytics]
07:10 <vgutierrez> depool and decommission cp2001 - T248815 [production]
06:52 <vgutierrez> pool cp2028 - T247340 [production]
06:29 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1074 after schema change', diff saved to https://phabricator.wikimedia.org/P10813 and previous config saved to /var/cache/conftool/dbconfig/20200330-062858-marostegui.json [production]
06:26 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:07 <Jayprakash12345> Add pre-generated category while uploading (T242190) [tools.qrcode-generator]
06:04 <marostegui> Deploy schema change on db1074 with replication, this will generate lag on s2 labs [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 for schema change', diff saved to https://phabricator.wikimedia.org/P10812 and previous config saved to /var/cache/conftool/dbconfig/20200330-060338-marostegui.json [production]
05:40 <vgutierrez> pool cp2027 - T247340 [production]
05:13 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:55 <vgutierrez> Enable TLS Session tickets in ulsfo - T245616 [production]
04:32 <vgutierrez> upgrade ATS to version 8.0.6-1wm4 on ulsfo - T245616 [production]
2020-03-29 §
21:34 <hashar> Restoring job mediawiki-core-doxygen-docker-publish deleted by Krinkle at 17:41 [releng]
09:56 <Framawiki> pip installed pathlib2 and upgraded requests to unstuck the bot [tools.totoazero]
08:44 <elukey> blacklist TwoColConflictExit from Eventlogging Refine to avoid alarm spam [analytics]
08:24 <elukey> powercycle elastic1059 - mgmt/serial console stuck, no ssh - racadm getsel shows a lot of OEM errors occurred, nothing specific [production]
2020-03-28 §
20:48 <wm-bot> <lucaswerkmeister> deployed 99ed67d128 (fix rotated JPEGs some more) [tools.wd-image-positions]
16:54 <elukey> restart yarn on analytics1071 [production]
16:54 <elukey> restart yarn nodemanger on analytics1071 - network errors in the logs [analytics]
14:26 <wm-bot> <lucaswerkmeister> deployed 152af31cca (fix rotated JPEGs) [tools.wd-image-positions]
12:05 <vgutierrez> preemptive restart of ats-tls on cp1081 and cp3062 - T248736 [production]
11:32 <vgutierrez> restart ats-tls on cp1077 - T248736 [production]
08:34 <vgutierrez> pool cp1089 [production]
08:30 <vgutierrez> restarting ats-tls on cp1089 [production]
2020-03-27 §
21:38 <bd808> Replaced A records for huggle-{rc,wl}.wmflabs.org with CNAME records pointing to {rc,wl}.huggle.wmcloud.org [wmflabsdotorg]
21:30 <bd808> Created wl.huggle.wmcloud.org A 185.15.56.49 Designate recordset [huggle]
21:28 <bd808> Created rc.huggle.wmcloud.org A 185.15.56.24 Designate recordset [huggle]
21:28 <bd808> Created huggle.wmcloud.org Designate zone and allocated it to the huggle project [admin]
21:22 <bstorm_> removed puppet prefix tools-docker-builder T248703 [tools]
21:19 <bd808> Added BryanDavis (self) to project as an admin to look at DNS record stuff [huggle]
21:15 <bstorm_> deleted tools-docker-builder-06 T248703 [tools]
20:51 <ejegg> updated payments-wiki from db618f429d to 1640f5e21e [production]
20:49 <bd808> Created huggle-rc.wmflabs.org. A 185.15.56.24 [wmflabsdotorg]