|
2020-03-04
§
|
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1098:3316 and db1098:3317 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10607 and previous config saved to /var/cache/conftool/dbconfig/20200304-071443-marostegui.json |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1098:3316 and db1098:3317 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10606 and previous config saved to /var/cache/conftool/dbconfig/20200304-070048-marostegui.json |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1098:3316 and db1098:3317 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10605 and previous config saved to /var/cache/conftool/dbconfig/20200304-064520-marostegui.json |
[production] |
| 06:30 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:28 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:22 |
<cdanis> |
✔️ cdanis@prometheus2004.codfw.wmnet ~ 🕝☕ sudo systemctl restart prometheus@ops |
[production] |
| 06:21 |
<cdanis> |
✔️ cdanis@prometheus2004.codfw.wmnet ~ 🕝☕ sudo systemctl reload prometheus@ops |
[production] |
| 06:10 |
<marostegui> |
Stop MySQL on db1098:3316, db1098:3317 for upgrade - T246604 |
[production] |
| 01:56 |
<mutante> |
mw2178 - systemctl reset-failed to clear (CRITICAL: Status of the systemd unit php7.2-fpm_check_restart) |
[production] |
| 01:55 |
<mutante> |
mw2290 - systemctl reset-failed to clear (CRITICAL: Status of the systemd unit php7.2-fpm_check_restart) |
[production] |
| 01:48 |
<mutante> |
mw1315 - restarted php-fpm and apache (was alerting in Icinga with 503 for 12 hours), log showed failed coredumps, restarts recovered it |
[production] |
| 01:35 |
<wm-bot> |
<bd808> Updated to b58a6dc (Filter using keyword subfields) |
[tools.csp-report] |
| 01:34 |
<James_F> |
Beta Cluster: Created deployment-parsoid11 in Horizon T246854 to test 576493. |
[releng] |
| 01:31 |
<mutante> |
ganeti2003 - DRAC reset failed with "ipmi_cmd_cold_reset: BMC busy" |
[production] |
| 01:30 |
<mutante> |
ganeti2003 - mgmt interface stopped responding on SSH, resetting DRAC via bmc-device from the host |
[production] |
| 01:27 |
<wm-bot> |
<bd808> Updated to 91c10d8 (Update for change in search results) and switch to elastic7 backend cluster |
[tools.csp-report] |
| 01:23 |
<Krenair> |
Made James a deployment-prep projectadmin |
[releng] |
| 00:25 |
<ebernhardson@deploy1001> |
Synchronized php-1.35.0-wmf.21/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: [cirrus] Match fallback config key with the one used in cirrus (duration: 01m 03s) |
[production] |
| 00:23 |
<ebernhardson@deploy1001> |
Synchronized php-1.35.0-wmf.22/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: [cirrus] Match fallback config key with the one used in cirrus (duration: 01m 04s) |
[production] |
| 00:15 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/SearchSettingsForWikibase.php: [cirrus] move similarity settings to IS.php (duration: 01m 05s) |
[production] |
| 00:13 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [cirrus] move similarity settings to IS.php (duration: 01m 04s) |
[production] |
| 00:06 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [cirrus] configure wgCirrusSearchMaxShardsPerNode per cluster (duration: 01m 05s) |
[production] |
| 00:06 |
<ebernhardson> |
post-deployment restart mjolnir-kafka-bulk-daemon across eqiad and codfw |
[production] |
| 00:05 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@1c97543]: Bump mjolnir to master: Revert stream gzip decompression (duration: 05m 25s) |
[production] |
| 00:00 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@1c97543]: Bump mjolnir to master: Revert stream gzip decompression |
[production] |
|
2020-03-03
§
|
| 23:13 |
<wm-bot> |
<bd808> Updated to f249e23c (Update for ElasticSearch 7 backend) and switched to http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud backend |
[tools.bash] |
| 22:51 |
<James_F> |
Zuul: Add gabrielchihonglee to CI allow list |
[releng] |
| 22:43 |
<James_F> |
Docker: Publish mediawiki-phan-testrun 0.1.4 |
[releng] |
| 22:28 |
<wm-bot> |
<zppix1> add wikibugs to ignore list and restart bot to deploy |
[tools.zppixbot] |
| 22:23 |
<bstorm_> |
deleted all resources on the old cluster T246519 |
[tools.paws-public] |
| 22:20 |
<bd808> |
Added bstorm_, chicocvenancio, and bd808 as co-maintainers |
[tools.paws-public] |
| 22:20 |
<bstorm_> |
created ingress for paws-public.wmflabs.org and deleted old service object T246519 |
[tools.paws-public] |
| 22:07 |
<bstorm_> |
removing the service object on the old cluster again T246519 |
[tools.paws-public] |
| 22:00 |
<thcipriani> |
reloading zuul to deploy https://gerrit.wikimedia.org/r/575575 |
[releng] |
| 21:57 |
<bstorm_> |
recreated the service on the old cluster because it didn't work right away? T246519 |
[tools.paws-public] |
| 21:55 |
<bstorm_> |
launched ingress on the new cluster, removing the service object on the old cluster T246519 |
[tools.paws-public] |
| 21:48 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 01m 04s) |
[production] |
| 21:46 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [wikidatawiki] Note that MostRevisions and MostLinked have been disabled (duration: 01m 05s) |
[production] |
| 21:33 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 21:33 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 21:19 |
<joal> |
Kill-restart actor jobs |
[analytics] |
| 21:17 |
<joal> |
kill-restart mediawiki-history-check_denormalize-coord |
[analytics] |
| 21:16 |
<joal> |
Kill-restart mediawiki-history job |
[analytics] |
| 21:13 |
<thcipriani@deploy1001> |
Synchronized php-1.35.0-wmf.22/includes/Defines.php: [[gerrit:576439|Update MW_VERSION to 1.35.0-wmf.22]] (duration: 01m 06s) |
[production] |
| 21:10 |
<joal> |
Kill Wikidataplaceholder failling coord |
[analytics] |
| 21:08 |
<joal> |
Kill restart wikidata-specialentitydata_metrics-coord |
[analytics] |
| 21:07 |
<joal> |
Start Wikidataplaceholder job |
[analytics] |
| 21:06 |
<joal> |
Kill/restart edit_hourly job |
[analytics] |
| 21:04 |
<joal> |
Start wikidata_item_page_link coordinator |
[analytics] |
| 20:59 |
<vgutierrez> |
Starting pybal on lvs1013 |
[production] |