2019-03-22
§
|
18:21 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
18:16 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
18:16 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
18:16 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --set main_app.kafka_broker_list=kafka-jumbo1003.eqiad.wmnet:9092 stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
18:13 |
<tzatziki> |
removing 5 files for legal compliance |
[production] |
18:13 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --set main_app.kafka_broker_list=kafka-jumbo1003.eqiad.wmnet:9092 stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
18:01 |
<bd808> |
Disable daily nag emails. Anyone who is not aware at this point is unlikely to suddenly notice the 20th warning email. |
[tools.trusty-tools] |
17:51 |
<paladox> |
deploy new session plugin to gerrit-test3 T218739 |
[git] |
17:29 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --set main_app.kafka_broker_list=kafka-jumbo1003.eqiad.wmnet:9092 stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
17:16 |
<andrewbogott> |
switching all instances to use ldap-ro.eqiad.wikimedia.org as both primary and secondary ldap server |
[tools] |
16:12 |
<bstorm_> |
cleared errored out stretch grid queues |
[tools] |
16:06 |
<jijiki> |
Restart ferm on db2096 |
[production] |
15:58 |
<James_F> |
UBN hot-deploy for T218918: Only load latest revision in MessageCache::loadFromDB |
[production] |
15:56 |
<bd808> |
Rebooting tools-static-12 |
[tools] |
15:26 |
<gehel> |
restarting elasticsearch on elastic1046 for logging configuration change - T218994 |
[production] |
14:34 |
<mutante> |
scandium - apt-get remove --purge php* ; apt autoremove ; letting puppet reinstall php 7.2 one more time using mediawiki::profile::php now |
[production] |
14:33 |
<gehel> |
upgrading to elasticsearch-curator 5.6.0 on all elasticsearch nodes (including logstash) - T218991 |
[production] |
13:59 |
<arturo> |
create VMs arturo-sgeexec-sssd-test-[12] for testing T218126 |
[toolsbeta] |
11:22 |
<ema> |
lvs1002: bounce pybal to clear backends health icinga warning T218133 |
[production] |
11:18 |
<ema> |
lvs1005: bounce pybal to clear backends health icinga warning T218133 |
[production] |
10:24 |
<mutante> |
scandium - apt autoremove |
[production] |
10:20 |
<mutante> |
scandium - manually removing all php* packages to let puppet reinstall 7.2 instead of 7.0 |
[production] |
10:05 |
<ema> |
cp2005: repooled, serving traffic via ATS T213263 |
[production] |
10:00 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp2005.codfw.wmnet,service=varnish-fe |
[production] |
10:00 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp2005.codfw.wmnet,service=nginx |
[production] |
09:48 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2005.codfw.wmnet,service=varnish-fe |
[production] |
09:48 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2005.codfw.wmnet,service=nginx |
[production] |
09:47 |
<ema> |
cp2005: depool varnish-fe in preparation of traffic switch to ATS T213263 |
[production] |
09:42 |
<moritzm> |
rebooting pool counters in codfw to pick up SSBD-enabled qemu |
[production] |
09:04 |
<elukey> |
start tcpdump on mc1022 to gather traffic for analysis |
[production] |
06:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1094 (duration: 00m 50s) |
[production] |
06:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 49s) |
[production] |
06:05 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2096 after onsite maintenance (duration: 00m 51s) |
[production] |
03:09 |
<bstorm_> |
T217280 depooled and rebooted 15 other nodes. Entire stretch grid is in a good state for now. |
[tools] |
03:08 |
<legoktm> |
rebuilding mediawiki-phan for https://gerrit.wikimedia.org/r/498297 |
[releng] |
02:31 |
<bstorm_> |
T217280 depooled and rebooted tools-sgeexec-0908 since it had no jobs but very high load from an NFS event that was no longer happening |
[tools] |
02:09 |
<bstorm_> |
T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0924 |
[tools] |
01:31 |
<bd808> |
labweb: upgraded mariadb packages installed on labweb100[12] |
[production] |
01:19 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@b4bcd08]: Update python wheels (duration: 01m 00s) |
[production] |
01:18 |
<bd808@deploy1001> |
Started deploy [striker/deploy@b4bcd08]: Update python wheels |
[production] |
00:54 |
<bd808> |
Striker down following upgrade. scap3 did not rebuild venv as expected. Manually resolved, but not having mysql library issues. |
[production] |
00:47 |
<Krinkle> |
krinkle@mwmaint1002 Fixing corrupt 'log_params' field of kawiki.logging row where log_id=1021367; T93110 |
[production] |
00:39 |
<bstorm_> |
T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0902 |
[tools] |
00:36 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/498276 / T218963 |
[releng] |
00:36 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/498276 / T215562) |
[releng] |
00:35 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@c4726e3]: Django upgrade and various bug fixes (T192487, T182142, T176325, T217932) (duration: 01m 15s) |
[production] |
00:34 |
<bd808@deploy1001> |
Started deploy [striker/deploy@c4726e3]: Django upgrade and various bug fixes (T192487, T182142, T176325, T217932) |
[production] |
00:32 |
<James_F> |
SWAT done, 12 minutes ago. |
[production] |
00:20 |
<jforrester@deploy1001> |
Finished scap: SWAT: Full scap for i18n rebuild for 498259 and 498113 (duration: 24m 49s) |
[production] |