|
2019-03-22
§
|
| 14:33 |
<gehel> |
upgrading to elasticsearch-curator 5.6.0 on all elasticsearch nodes (including logstash) - T218991 |
[production] |
| 11:22 |
<ema> |
lvs1002: bounce pybal to clear backends health icinga warning T218133 |
[production] |
| 11:18 |
<ema> |
lvs1005: bounce pybal to clear backends health icinga warning T218133 |
[production] |
| 10:24 |
<mutante> |
scandium - apt autoremove |
[production] |
| 10:20 |
<mutante> |
scandium - manually removing all php* packages to let puppet reinstall 7.2 instead of 7.0 |
[production] |
| 10:05 |
<ema> |
cp2005: repooled, serving traffic via ATS T213263 |
[production] |
| 10:00 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp2005.codfw.wmnet,service=varnish-fe |
[production] |
| 10:00 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp2005.codfw.wmnet,service=nginx |
[production] |
| 09:48 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2005.codfw.wmnet,service=varnish-fe |
[production] |
| 09:48 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2005.codfw.wmnet,service=nginx |
[production] |
| 09:47 |
<ema> |
cp2005: depool varnish-fe in preparation of traffic switch to ATS T213263 |
[production] |
| 09:42 |
<moritzm> |
rebooting pool counters in codfw to pick up SSBD-enabled qemu |
[production] |
| 09:04 |
<elukey> |
start tcpdump on mc1022 to gather traffic for analysis |
[production] |
| 06:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1094 (duration: 00m 50s) |
[production] |
| 06:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 49s) |
[production] |
| 06:05 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2096 after onsite maintenance (duration: 00m 51s) |
[production] |
| 01:31 |
<bd808> |
labweb: upgraded mariadb packages installed on labweb100[12] |
[production] |
| 01:19 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@b4bcd08]: Update python wheels (duration: 01m 00s) |
[production] |
| 01:18 |
<bd808@deploy1001> |
Started deploy [striker/deploy@b4bcd08]: Update python wheels |
[production] |
| 00:54 |
<bd808> |
Striker down following upgrade. scap3 did not rebuild venv as expected. Manually resolved, but not having mysql library issues. |
[production] |
| 00:47 |
<Krinkle> |
krinkle@mwmaint1002 Fixing corrupt 'log_params' field of kawiki.logging row where log_id=1021367; T93110 |
[production] |
| 00:35 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@c4726e3]: Django upgrade and various bug fixes (T192487, T182142, T176325, T217932) (duration: 01m 15s) |
[production] |
| 00:34 |
<bd808@deploy1001> |
Started deploy [striker/deploy@c4726e3]: Django upgrade and various bug fixes (T192487, T182142, T176325, T217932) |
[production] |
| 00:32 |
<James_F> |
SWAT done, 12 minutes ago. |
[production] |
| 00:20 |
<jforrester@deploy1001> |
Finished scap: SWAT: Full scap for i18n rebuild for 498259 and 498113 (duration: 24m 49s) |
[production] |
|
2019-03-21
§
|
| 23:57 |
<gtirloni> |
downtimed systemd check in labweb1001/1002 (T218935) |
[production] |
| 23:56 |
<jforrester@deploy1001> |
Started scap: SWAT: Full scap for i18n rebuild for 498259 and 498113 |
[production] |
| 23:53 |
<gtirloni> |
downtimed systemd check in labwen1001 (T210818) |
[production] |
| 23:32 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.22/extensions/ContentTranslation/api/ApiQueryContentTranslationSuggestions.php: SWAT T218902 CX: Return API error on anonymous suggestions queries (duration: 00m 51s) |
[production] |
| 23:08 |
<jforrester@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT T217730 Add wikimaniawiki to another special group in Wikibase client (duration: 00m 49s) |
[production] |
| 22:33 |
<jijiki> |
Restarting pdfrender on scb1003 |
[production] |
| 22:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 22:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 22:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 22:14 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=0) |
[production] |
| 22:02 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable WikimediaEditorTasks on the Beta Cluster (duration: 00m 49s) |
[production] |
| 21:56 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Add WikimediaEditorTasks labs config to InitializeSettings-labs.php (duration: 00m 47s) |
[production] |
| 21:54 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add WikimediaEditorTasks default config to InitializeSettings.php (duration: 00m 49s) |
[production] |
| 21:53 |
<jijiki> |
Restarting pdfrender on scb1004 |
[production] |
| 21:52 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/extension-list: Add WikimediaEditorTasks to extension-list (duration: 00m 50s) |
[production] |
| 21:45 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 21:39 |
<XioNoX> |
Ping offload - replace test IP with text-lb.codfw IP on cr1/2-codfw - T190090 |
[production] |
| 21:11 |
<XioNoX> |
remove peering sessions to AS7385 on cr4-ulsfo |
[production] |
| 21:08 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 21:08 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 21:08 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 20:55 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 20:55 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 20:55 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 20:43 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |