|
2020-03-30
§
|
| 10:33 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 10:33 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 10:30 |
<arturo> |
remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use |
[toolsbeta] |
| 09:59 |
<hoo> |
Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata JSON dumps start at 9:59 UTC today (T248612) |
[production] |
| 09:56 |
<hoo@deploy1001> |
Synchronized php-1.35.0-wmf.25/extensions/Wikibase/repo/maintenance/DumpEntities.php: DumpEntities: Fix DB group default override (T248612) (duration: 01m 02s) |
[production] |
| 09:19 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:15 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:30 |
<vgutierrez> |
pool cp2029 - T248816 |
[production] |
| 08:12 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:12 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 08:12 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 08:10 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:53 |
<vgutierrez> |
depool & decommission cp2002 - T248818 |
[production] |
| 07:48 |
<marostegui> |
Run cloudcontrol1003:~# wmcs-wikireplica-dns to promote dbproxy1018 to wikireplicas active proxy T231520 |
[production] |
| 07:40 |
<marostegui> |
Replace dbproxy1010 with dbproxy1011 for wiki replicas, analytics - T231520 |
[production] |
| 07:28 |
<marostegui> |
Deploy schema change on labswiki (wikitech) - T248333 |
[production] |
| 07:27 |
<elukey> |
run /usr/local/bin/refine_sanitize_eventlogging_analytics_immediate --ignore_failure_flag=true --since=72 --verbose --table_whitelist_regex="ResourceTiming" refine_sanitize_eventlogging_analytics_immediate to fix _REFINE_FAILED events |
[analytics] |
| 07:26 |
<marostegui> |
Deploy schema change on s4 codfw, this will generate lag on codfw - T248333 |
[production] |
| 07:17 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:17 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:16 |
<elukey> |
run eventlogging refine manually for schemas "EditorActivation|EditorJourney|HomepageVisit|VisualEditorFeatureUse|WikibaseTermboxInteraction|UploadWizardErrorFlowEvent|MobileWikiAppiOSReadingLists|ContentTranslationCTA|QuickSurveysResponses|MobileWikiAppiOSSessions to fix _REFINE_FAILED events |
[analytics] |
| 07:10 |
<vgutierrez> |
depool and decommission cp2001 - T248815 |
[production] |
| 06:52 |
<vgutierrez> |
pool cp2028 - T247340 |
[production] |
| 06:29 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1074 after schema change', diff saved to https://phabricator.wikimedia.org/P10813 and previous config saved to /var/cache/conftool/dbconfig/20200330-062858-marostegui.json |
[production] |
| 06:26 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:07 |
<Jayprakash12345> |
Add pre-generated category while uploading (T242190) |
[tools.qrcode-generator] |
| 06:04 |
<marostegui> |
Deploy schema change on db1074 with replication, this will generate lag on s2 labs |
[production] |
| 06:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074 for schema change', diff saved to https://phabricator.wikimedia.org/P10812 and previous config saved to /var/cache/conftool/dbconfig/20200330-060338-marostegui.json |
[production] |
| 05:40 |
<vgutierrez> |
pool cp2027 - T247340 |
[production] |
| 05:13 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:10 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:55 |
<vgutierrez> |
Enable TLS Session tickets in ulsfo - T245616 |
[production] |
| 04:32 |
<vgutierrez> |
upgrade ATS to version 8.0.6-1wm4 on ulsfo - T245616 |
[production] |