|
2020-04-21
§
|
| 21:50 |
<urandom> |
bootstrapping restbase2014-c — T250050 |
[production] |
| 21:46 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@64c5ec4]: Analytics: tiny follow-up on weekly train [analytics/refinery@64c5ec4] |
[production] |
| 21:38 |
<milimetric> |
deployed twice because analytics1030 failed with "OSError {}" but seems ok after the second deploy |
[analytics] |
| 21:38 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@35781db]: Regular Analytics weekly train deploy [analytics/refinery@35781db] try 2 (analytics1030 failed with OSError the first time) (duration: 00m 13s) |
[production] |
| 21:37 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@35781db]: Regular Analytics weekly train deploy [analytics/refinery@35781db] try 2 (analytics1030 failed with OSError the first time) |
[production] |
| 21:21 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@35781db]: Regular Analytics weekly train deploy [analytics/refinery@35781db] (duration: 16m 19s) |
[production] |
| 21:07 |
<wm-bot> |
<lucaswerkmeister> deployed 6634452b4c (increase uWSGI buffer) |
[tools.lexeme-forms] |
| 21:05 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@35781db]: Regular Analytics weekly train deploy [analytics/refinery@35781db] |
[production] |
| 21:05 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@35781db] (thin): Regular Analytics weekly train deploy THIN [analytics/refinery@35781db] (duration: 00m 08s) |
[production] |
| 21:05 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@35781db] (thin): Regular Analytics weekly train deploy THIN [analytics/refinery@35781db] |
[production] |
| 19:09 |
<rzl> |
mcrouter certs renewed on puppetmaster1001 (again); puppet re-enabled on mcrouter hosts and will update certs naturally over the next 30m T248093 |
[production] |
| 19:02 |
<urandom> |
bootstrapping restbase2014-b — T250050 |
[production] |
| 18:28 |
<hoo> |
Updated the Wikidata property suggester with data from the 2020-04-06 JSON dump and applied the T132839 workarounds |
[production] |
| 18:19 |
<rzl> |
disabling puppet on all mcrouter hosts for cert renewal T248093 |
[production] |
| 17:19 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:16 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:49 |
<urandom> |
bootstrapping restbase2014-a — T250050 |
[production] |
| 16:01 |
<jeh> |
restart cloudceph mon and osd services for openssl upgrades |
[admin] |
| 15:40 |
<cmjohnson1> |
replacing mgmt switch on a6-eqiad T250652 |
[production] |
| 15:38 |
<hashar> |
CI is back, patches would need to be rechecked by commenting "recheck" in Gerrit. |
[production] |
| 15:32 |
<hashar> |
Restarting Gerrit T250820 T246973 |
[production] |
| 15:26 |
<hashar> |
CI / Zuul does not get any events for some reason :/ |
[production] |
| 15:05 |
<Krinkle> |
install 'qemu-system-x86' package on integration-agent-qemu-1001 |
[releng] |
| 14:59 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:59 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:51 |
<hashar> |
contint2001: manually dropping /var/lib/docker (we now use /srv/docker ) |
[production] |
| 14:48 |
<jbond42> |
restart haproxy on dns-auth |
[production] |
| 14:48 |
<hashar> |
restarting docker on contint2001 |
[production] |
| 14:48 |
<Krinkle> |
Creating integration-agent-qemu-1001 to experiment with VM-based CI jobs – T250808 |
[releng] |
| 14:47 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@d101234]: Deploy design/style-guide: (duration: 00m 09s) |
[production] |
| 14:47 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@d101234]: Deploy design/style-guide: |
[production] |
| 14:45 |
<jbond42> |
puppet enabled again |
[production] |
| 14:40 |
<moritzm> |
restarting apache on miscweb |
[production] |
| 14:37 |
<moritzm> |
restarting apache on netbox1001 |
[production] |
| 14:36 |
<jbond42> |
disable puppet fleet wide to restart puppemaster |
[production] |
| 14:28 |
<moritzm> |
installing OpenSSL security updates |
[production] |
| 14:27 |
<elukey> |
add motd to notebook100[3,4] to alert about host deprecation (in favor of stat100x) |
[analytics] |
| 14:17 |
<vgutierrez> |
rolling upgrade of ats to version 8.0.7-1wm1 |
[production] |
| 14:16 |
<moritzm> |
installing OpenSSL updates on caches |
[production] |
| 14:08 |
<hashar> |
contint1001: rm /var/log/apache2/doc_* # service has been moved to doc1001.eqiad.wmnet |
[production] |
| 13:43 |
<vgutierrez> |
upload trafficserver 8.0.7-1wm1 to apt.wm.o (buster) |
[production] |
| 13:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 13:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:51 |
<elukey> |
manually add SUCCESS flags under /wmf/data/wmf/banner_activity/daily/year=2020/month=1 and /wmf/data/wmf/banner_activity/daily/year=2019/month=12 to unblock druid banner monthly indexations |
[analytics] |
| 11:15 |
<mutante> |
recreating cert for contint/integration to add integration.mediawiki.org in addition to integration.wikimedia.org |
[production] |
| 11:06 |
<mutante> |
https://integration.wikimedia.org now also using TLS between ATS and contint1001 using envoy (T210411) |
[production] |
| 10:49 |
<_joe_> |
mwdebug1001:~# iptables -A INPUT -s 10.64.32.208 -m statistic --mode random --probability 0.1 -j DROP (T240684) |
[production] |
| 08:52 |
<ema> |
purged: rolling restart with 4 frontend workers |
[production] |
| 07:54 |
<ema> |
cp3050: restart purged with 4 frontend workers |
[production] |
| 07:47 |
<kormat> |
dropping old data and optimizing tables on pc1010 and pc2010 T247787 |
[production] |