201-250 of 10000 results (69ms)
2020-03-30 §
10:33 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
10:33 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
10:30 <arturo> remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use [toolsbeta]
09:59 <hoo> Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata JSON dumps start at 9:59 UTC today (T248612) [production]
09:56 <hoo@deploy1001> Synchronized php-1.35.0-wmf.25/extensions/Wikibase/repo/maintenance/DumpEntities.php: DumpEntities: Fix DB group default override (T248612) (duration: 01m 02s) [production]
09:19 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:15 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:30 <vgutierrez> pool cp2029 - T248816 [production]
08:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
08:12 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:53 <vgutierrez> depool & decommission cp2002 - T248818 [production]
07:48 <marostegui> Run cloudcontrol1003:~# wmcs-wikireplica-dns to promote dbproxy1018 to wikireplicas active proxy T231520 [production]
07:40 <marostegui> Replace dbproxy1010 with dbproxy1011 for wiki replicas, analytics - T231520 [production]
07:28 <marostegui> Deploy schema change on labswiki (wikitech) - T248333 [production]
07:27 <elukey> run /usr/local/bin/refine_sanitize_eventlogging_analytics_immediate --ignore_failure_flag=true --since=72 --verbose --table_whitelist_regex="ResourceTiming" refine_sanitize_eventlogging_analytics_immediate to fix _REFINE_FAILED events [analytics]
07:26 <marostegui> Deploy schema change on s4 codfw, this will generate lag on codfw - T248333 [production]
07:17 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
07:17 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:16 <elukey> run eventlogging refine manually for schemas "EditorActivation|EditorJourney|HomepageVisit|VisualEditorFeatureUse|WikibaseTermboxInteraction|UploadWizardErrorFlowEvent|MobileWikiAppiOSReadingLists|ContentTranslationCTA|QuickSurveysResponses|MobileWikiAppiOSSessions to fix _REFINE_FAILED events [analytics]
07:10 <vgutierrez> depool and decommission cp2001 - T248815 [production]
06:52 <vgutierrez> pool cp2028 - T247340 [production]
06:29 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1074 after schema change', diff saved to https://phabricator.wikimedia.org/P10813 and previous config saved to /var/cache/conftool/dbconfig/20200330-062858-marostegui.json [production]
06:26 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:07 <Jayprakash12345> Add pre-generated category while uploading (T242190) [tools.qrcode-generator]
06:04 <marostegui> Deploy schema change on db1074 with replication, this will generate lag on s2 labs [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 for schema change', diff saved to https://phabricator.wikimedia.org/P10812 and previous config saved to /var/cache/conftool/dbconfig/20200330-060338-marostegui.json [production]
05:40 <vgutierrez> pool cp2027 - T247340 [production]
05:13 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:55 <vgutierrez> Enable TLS Session tickets in ulsfo - T245616 [production]
04:32 <vgutierrez> upgrade ATS to version 8.0.6-1wm4 on ulsfo - T245616 [production]
2020-03-29 §
21:34 <hashar> Restoring job mediawiki-core-doxygen-docker-publish deleted by Krinkle at 17:41 [releng]
09:56 <Framawiki> pip installed pathlib2 and upgraded requests to unstuck the bot [tools.totoazero]
08:44 <elukey> blacklist TwoColConflictExit from Eventlogging Refine to avoid alarm spam [analytics]
08:24 <elukey> powercycle elastic1059 - mgmt/serial console stuck, no ssh - racadm getsel shows a lot of OEM errors occurred, nothing specific [production]
2020-03-28 §
20:48 <wm-bot> <lucaswerkmeister> deployed 99ed67d128 (fix rotated JPEGs some more) [tools.wd-image-positions]
16:54 <elukey> restart yarn on analytics1071 [production]
16:54 <elukey> restart yarn nodemanger on analytics1071 - network errors in the logs [analytics]
14:26 <wm-bot> <lucaswerkmeister> deployed 152af31cca (fix rotated JPEGs) [tools.wd-image-positions]
12:05 <vgutierrez> preemptive restart of ats-tls on cp1081 and cp3062 - T248736 [production]
11:32 <vgutierrez> restart ats-tls on cp1077 - T248736 [production]
08:34 <vgutierrez> pool cp1089 [production]
08:30 <vgutierrez> restarting ats-tls on cp1089 [production]
2020-03-27 §
21:38 <bd808> Replaced A records for huggle-{rc,wl}.wmflabs.org with CNAME records pointing to {rc,wl}.huggle.wmcloud.org [wmflabsdotorg]
21:30 <bd808> Created wl.huggle.wmcloud.org A 185.15.56.49 Designate recordset [huggle]
21:28 <bd808> Created rc.huggle.wmcloud.org A 185.15.56.24 Designate recordset [huggle]
21:28 <bd808> Created huggle.wmcloud.org Designate zone and allocated it to the huggle project [admin]