|
2019-03-22
§
|
| 06:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 49s) |
[production] |
| 06:05 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2096 after onsite maintenance (duration: 00m 51s) |
[production] |
| 03:09 |
<bstorm_> |
T217280 depooled and rebooted 15 other nodes. Entire stretch grid is in a good state for now. |
[tools] |
| 03:08 |
<legoktm> |
rebuilding mediawiki-phan for https://gerrit.wikimedia.org/r/498297 |
[releng] |
| 02:31 |
<bstorm_> |
T217280 depooled and rebooted tools-sgeexec-0908 since it had no jobs but very high load from an NFS event that was no longer happening |
[tools] |
| 02:09 |
<bstorm_> |
T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0924 |
[tools] |
| 01:31 |
<bd808> |
labweb: upgraded mariadb packages installed on labweb100[12] |
[production] |
| 01:19 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@b4bcd08]: Update python wheels (duration: 01m 00s) |
[production] |
| 01:18 |
<bd808@deploy1001> |
Started deploy [striker/deploy@b4bcd08]: Update python wheels |
[production] |
| 00:54 |
<bd808> |
Striker down following upgrade. scap3 did not rebuild venv as expected. Manually resolved, but not having mysql library issues. |
[production] |
| 00:47 |
<Krinkle> |
krinkle@mwmaint1002 Fixing corrupt 'log_params' field of kawiki.logging row where log_id=1021367; T93110 |
[production] |
| 00:39 |
<bstorm_> |
T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0902 |
[tools] |
| 00:36 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/498276 / T218963 |
[releng] |
| 00:36 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/498276 / T215562) |
[releng] |
| 00:35 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@c4726e3]: Django upgrade and various bug fixes (T192487, T182142, T176325, T217932) (duration: 01m 15s) |
[production] |
| 00:34 |
<bd808@deploy1001> |
Started deploy [striker/deploy@c4726e3]: Django upgrade and various bug fixes (T192487, T182142, T176325, T217932) |
[production] |
| 00:32 |
<James_F> |
SWAT done, 12 minutes ago. |
[production] |
| 00:20 |
<jforrester@deploy1001> |
Finished scap: SWAT: Full scap for i18n rebuild for 498259 and 498113 (duration: 24m 49s) |
[production] |
|
2019-03-21
§
|
| 23:57 |
<gtirloni> |
downtimed systemd check in labweb1001/1002 (T218935) |
[production] |
| 23:56 |
<jforrester@deploy1001> |
Started scap: SWAT: Full scap for i18n rebuild for 498259 and 498113 |
[production] |
| 23:53 |
<gtirloni> |
downtimed systemd check in labwen1001 (T210818) |
[production] |
| 23:34 |
<legoktm> |
rebuilding mediawiki-phan docker image for https://gerrit.wikimedia.org/r/498266 |
[releng] |
| 23:32 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.22/extensions/ContentTranslation/api/ApiQueryContentTranslationSuggestions.php: SWAT T218902 CX: Return API error on anonymous suggestions queries (duration: 00m 51s) |
[production] |
| 23:28 |
<bstorm_> |
T217280 depooled, reloaded and repooled tools-sgeexec-0938 |
[tools] |
| 23:08 |
<jforrester@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT T217730 Add wikimaniawiki to another special group in Wikibase client (duration: 00m 49s) |
[production] |
| 22:34 |
<legoktm> |
onlined integration-slave-jessie-1002 |
[releng] |
| 22:33 |
<legoktm> |
legoktm@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf * |
[releng] |
| 22:33 |
<jijiki> |
Restarting pdfrender on scb1003 |
[production] |
| 22:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 22:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 22:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 22:14 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=0) |
[production] |
| 22:02 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable WikimediaEditorTasks on the Beta Cluster (duration: 00m 49s) |
[production] |
| 21:56 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Add WikimediaEditorTasks labs config to InitializeSettings-labs.php (duration: 00m 47s) |
[production] |
| 21:54 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add WikimediaEditorTasks default config to InitializeSettings.php (duration: 00m 49s) |
[production] |
| 21:53 |
<bstorm_> |
T217280 rebooted and cleared "unknown status" from tools-sgeexec-0914 after depooling |
[tools] |
| 21:53 |
<jijiki> |
Restarting pdfrender on scb1004 |
[production] |
| 21:52 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/extension-list: Add WikimediaEditorTasks to extension-list (duration: 00m 50s) |
[production] |
| 21:51 |
<bstorm_> |
T217280 rebooted and cleared "unknown status" from tools-sgeexec-0909 after depooling |
[tools] |
| 21:45 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 21:39 |
<XioNoX> |
Ping offload - replace test IP with text-lb.codfw IP on cr1/2-codfw - T190090 |
[production] |
| 21:26 |
<bstorm_> |
T217280 cleared error state from a couple queues and rebooted tools-sgeexec-0901 and 04 to clear other issues related |
[tools] |
| 21:11 |
<XioNoX> |
remove peering sessions to AS7385 on cr4-ulsfo |
[production] |
| 21:08 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 21:08 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 21:08 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 21:03 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/498178 https://gerrit.wikimedia.org/r/498185 https://gerrit.wikimedia.org/r/498187 https://gerrit.wikimedia.org/r/498189 https://gerrit.wikimedia.org/r/498182 |
[releng] |
| 20:55 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 20:55 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 20:55 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |