101-150 of 10000 results (45ms)
2020-03-30 §
07:17 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:16 <elukey> run eventlogging refine manually for schemas "EditorActivation|EditorJourney|HomepageVisit|VisualEditorFeatureUse|WikibaseTermboxInteraction|UploadWizardErrorFlowEvent|MobileWikiAppiOSReadingLists|ContentTranslationCTA|QuickSurveysResponses|MobileWikiAppiOSSessions to fix _REFINE_FAILED events [analytics]
07:10 <vgutierrez> depool and decommission cp2001 - T248815 [production]
06:52 <vgutierrez> pool cp2028 - T247340 [production]
06:29 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1074 after schema change', diff saved to https://phabricator.wikimedia.org/P10813 and previous config saved to /var/cache/conftool/dbconfig/20200330-062858-marostegui.json [production]
06:26 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:07 <Jayprakash12345> Add pre-generated category while uploading (T242190) [tools.qrcode-generator]
06:04 <marostegui> Deploy schema change on db1074 with replication, this will generate lag on s2 labs [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 for schema change', diff saved to https://phabricator.wikimedia.org/P10812 and previous config saved to /var/cache/conftool/dbconfig/20200330-060338-marostegui.json [production]
05:40 <vgutierrez> pool cp2027 - T247340 [production]
05:13 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:55 <vgutierrez> Enable TLS Session tickets in ulsfo - T245616 [production]
04:32 <vgutierrez> upgrade ATS to version 8.0.6-1wm4 on ulsfo - T245616 [production]
2020-03-29 §
21:34 <hashar> Restoring job mediawiki-core-doxygen-docker-publish deleted by Krinkle at 17:41 [releng]
09:56 <Framawiki> pip installed pathlib2 and upgraded requests to unstuck the bot [tools.totoazero]
08:44 <elukey> blacklist TwoColConflictExit from Eventlogging Refine to avoid alarm spam [analytics]
08:24 <elukey> powercycle elastic1059 - mgmt/serial console stuck, no ssh - racadm getsel shows a lot of OEM errors occurred, nothing specific [production]
2020-03-28 §
20:48 <wm-bot> <lucaswerkmeister> deployed 99ed67d128 (fix rotated JPEGs some more) [tools.wd-image-positions]
16:54 <elukey> restart yarn on analytics1071 [production]
16:54 <elukey> restart yarn nodemanger on analytics1071 - network errors in the logs [analytics]
14:26 <wm-bot> <lucaswerkmeister> deployed 152af31cca (fix rotated JPEGs) [tools.wd-image-positions]
12:05 <vgutierrez> preemptive restart of ats-tls on cp1081 and cp3062 - T248736 [production]
11:32 <vgutierrez> restart ats-tls on cp1077 - T248736 [production]
08:34 <vgutierrez> pool cp1089 [production]
08:30 <vgutierrez> restarting ats-tls on cp1089 [production]
2020-03-27 §
21:38 <bd808> Replaced A records for huggle-{rc,wl}.wmflabs.org with CNAME records pointing to {rc,wl}.huggle.wmcloud.org [wmflabsdotorg]
21:30 <bd808> Created wl.huggle.wmcloud.org A 185.15.56.49 Designate recordset [huggle]
21:28 <bd808> Created rc.huggle.wmcloud.org A 185.15.56.24 Designate recordset [huggle]
21:28 <bd808> Created huggle.wmcloud.org Designate zone and allocated it to the huggle project [admin]
21:22 <bstorm_> removed puppet prefix tools-docker-builder T248703 [tools]
21:19 <bd808> Added BryanDavis (self) to project as an admin to look at DNS record stuff [huggle]
21:15 <bstorm_> deleted tools-docker-builder-06 T248703 [tools]
20:51 <ejegg> updated payments-wiki from db618f429d to 1640f5e21e [production]
20:49 <bd808> Created huggle-rc.wmflabs.org. A 185.15.56.24 [wmflabsdotorg]
19:51 <jeh> start haproxy on cloudcontrol2003-dev.wikimedia.org [admin]
18:54 <bstorm_> launching tools-docker-imagebuilder-01 T248703 [tools]
15:15 <andrew@deploy1001> Finished deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens (duration: 03m 35s) [production]
15:12 <andrew@deploy1001> Started deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens [production]
14:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1129 after schema change', diff saved to https://phabricator.wikimedia.org/P10807 and previous config saved to /var/cache/conftool/dbconfig/20200327-144125-marostegui.json [production]
14:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1129 for schema change', diff saved to https://phabricator.wikimedia.org/P10806 and previous config saved to /var/cache/conftool/dbconfig/20200327-142240-marostegui.json [production]
14:19 <moritzm> updating linux-image-4.9.0-11-amd64 where applicable [production]
13:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1076 after schema change', diff saved to https://phabricator.wikimedia.org/P10805 and previous config saved to /var/cache/conftool/dbconfig/20200327-133022-marostegui.json [production]
13:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1076 for schema change', diff saved to https://phabricator.wikimedia.org/P10804 and previous config saved to /var/cache/conftool/dbconfig/20200327-130706-marostegui.json [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1105:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10803 and previous config saved to /var/cache/conftool/dbconfig/20200327-130542-marostegui.json [production]
12:52 <arturo> install python3-pykube on tools-k8s-control-3 for some tests interaction with the API from python [tools]
12:49 <Amir1> ladsgroup@mwmaint1002:~$ mwscript createAndPromote.php --wiki=labswiki --force "Ladsgroup" --interface-admin [production]
12:26 <arturo> add myself as project admin [clouddb-services]
12:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10802 and previous config saved to /var/cache/conftool/dbconfig/20200327-122144-marostegui.json [production]