|
2019-03-22
ยง
|
| 19:36 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 19:34 |
<thcipriani> |
updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/494548/ |
[releng] |
| 19:24 |
<thcipriani> |
clean old docker-pkg image from contint1001 |
[releng] |
| 18:53 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 18:53 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 18:53 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 18:41 |
<krinkle@deploy1001> |
Synchronized php-1.33.0-wmf.22/extensions/Collection/: I2c4f5d005fc25c52 / T217835 (duration: 00m 52s) |
[production] |
| 18:21 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 18:21 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 18:21 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 18:16 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 18:16 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 18:16 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --set main_app.kafka_broker_list=kafka-jumbo1003.eqiad.wmnet:9092 stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 18:13 |
<tzatziki> |
removing 5 files for legal compliance |
[production] |
| 18:13 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --set main_app.kafka_broker_list=kafka-jumbo1003.eqiad.wmnet:9092 stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 18:01 |
<bd808> |
Disable daily nag emails. Anyone who is not aware at this point is unlikely to suddenly notice the 20th warning email. |
[tools.trusty-tools] |
| 17:51 |
<paladox> |
deploy new session plugin to gerrit-test3 T218739 |
[git] |
| 17:29 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --set main_app.kafka_broker_list=kafka-jumbo1003.eqiad.wmnet:9092 stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 17:16 |
<andrewbogott> |
switching all instances to use ldap-ro.eqiad.wikimedia.org as both primary and secondary ldap server |
[tools] |
| 16:12 |
<bstorm_> |
cleared errored out stretch grid queues |
[tools] |
| 16:06 |
<jijiki> |
Restart ferm on db2096 |
[production] |
| 15:58 |
<James_F> |
UBN hot-deploy for T218918: Only load latest revision in MessageCache::loadFromDB |
[production] |
| 15:56 |
<bd808> |
Rebooting tools-static-12 |
[tools] |
| 15:26 |
<gehel> |
restarting elasticsearch on elastic1046 for logging configuration change - T218994 |
[production] |
| 14:34 |
<mutante> |
scandium - apt-get remove --purge php* ; apt autoremove ; letting puppet reinstall php 7.2 one more time using mediawiki::profile::php now |
[production] |
| 14:33 |
<gehel> |
upgrading to elasticsearch-curator 5.6.0 on all elasticsearch nodes (including logstash) - T218991 |
[production] |
| 13:59 |
<arturo> |
create VMs arturo-sgeexec-sssd-test-[12] for testing T218126 |
[toolsbeta] |
| 11:22 |
<ema> |
lvs1002: bounce pybal to clear backends health icinga warning T218133 |
[production] |
| 11:18 |
<ema> |
lvs1005: bounce pybal to clear backends health icinga warning T218133 |
[production] |
| 10:24 |
<mutante> |
scandium - apt autoremove |
[production] |
| 10:20 |
<mutante> |
scandium - manually removing all php* packages to let puppet reinstall 7.2 instead of 7.0 |
[production] |
| 10:05 |
<ema> |
cp2005: repooled, serving traffic via ATS T213263 |
[production] |
| 10:00 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp2005.codfw.wmnet,service=varnish-fe |
[production] |
| 10:00 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp2005.codfw.wmnet,service=nginx |
[production] |
| 09:48 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2005.codfw.wmnet,service=varnish-fe |
[production] |
| 09:48 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2005.codfw.wmnet,service=nginx |
[production] |
| 09:47 |
<ema> |
cp2005: depool varnish-fe in preparation of traffic switch to ATS T213263 |
[production] |
| 09:42 |
<moritzm> |
rebooting pool counters in codfw to pick up SSBD-enabled qemu |
[production] |
| 09:04 |
<elukey> |
start tcpdump on mc1022 to gather traffic for analysis |
[production] |
| 06:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1094 (duration: 00m 50s) |
[production] |
| 06:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 49s) |
[production] |
| 06:05 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2096 after onsite maintenance (duration: 00m 51s) |
[production] |
| 03:09 |
<bstorm_> |
T217280 depooled and rebooted 15 other nodes. Entire stretch grid is in a good state for now. |
[tools] |
| 03:08 |
<legoktm> |
rebuilding mediawiki-phan for https://gerrit.wikimedia.org/r/498297 |
[releng] |
| 02:31 |
<bstorm_> |
T217280 depooled and rebooted tools-sgeexec-0908 since it had no jobs but very high load from an NFS event that was no longer happening |
[tools] |
| 02:09 |
<bstorm_> |
T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0924 |
[tools] |
| 01:31 |
<bd808> |
labweb: upgraded mariadb packages installed on labweb100[12] |
[production] |
| 01:19 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@b4bcd08]: Update python wheels (duration: 01m 00s) |
[production] |
| 01:18 |
<bd808@deploy1001> |
Started deploy [striker/deploy@b4bcd08]: Update python wheels |
[production] |
| 00:54 |
<bd808> |
Striker down following upgrade. scap3 did not rebuild venv as expected. Manually resolved, but not having mysql library issues. |
[production] |