|
2020-04-21
§
|
| 23:25 |
<mstyles@deploy1001> |
Started deploy [wdqs/wdqs@4e0d55f]: v0.3.23 |
[production] |
| 23:19 |
<maryum> |
begin deploy of WDQS v 0.3.23 on deploy1001 |
[production] |
| 22:41 |
<eileen> |
process-control config revision is 6294adfbaa |
[production] |
| 22:24 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@64c5ec4]: Analytics: tiny follow-up on weekly train [analytics/refinery@64c5ec4] (duration: 37m 05s) |
[production] |
| 21:56 |
<andrewbogott> |
rebooting cloudvirt1004, total raid controller failure |
[production] |
| 21:50 |
<urandom> |
bootstrapping restbase2014-c — T250050 |
[production] |
| 21:46 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@64c5ec4]: Analytics: tiny follow-up on weekly train [analytics/refinery@64c5ec4] |
[production] |
| 21:38 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@35781db]: Regular Analytics weekly train deploy [analytics/refinery@35781db] try 2 (analytics1030 failed with OSError the first time) (duration: 00m 13s) |
[production] |
| 21:37 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@35781db]: Regular Analytics weekly train deploy [analytics/refinery@35781db] try 2 (analytics1030 failed with OSError the first time) |
[production] |
| 21:21 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@35781db]: Regular Analytics weekly train deploy [analytics/refinery@35781db] (duration: 16m 19s) |
[production] |
| 21:05 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@35781db]: Regular Analytics weekly train deploy [analytics/refinery@35781db] |
[production] |
| 21:05 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@35781db] (thin): Regular Analytics weekly train deploy THIN [analytics/refinery@35781db] (duration: 00m 08s) |
[production] |
| 21:05 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@35781db] (thin): Regular Analytics weekly train deploy THIN [analytics/refinery@35781db] |
[production] |
| 19:09 |
<rzl> |
mcrouter certs renewed on puppetmaster1001 (again); puppet re-enabled on mcrouter hosts and will update certs naturally over the next 30m T248093 |
[production] |
| 19:02 |
<urandom> |
bootstrapping restbase2014-b — T250050 |
[production] |
| 18:28 |
<hoo> |
Updated the Wikidata property suggester with data from the 2020-04-06 JSON dump and applied the T132839 workarounds |
[production] |
| 18:19 |
<rzl> |
disabling puppet on all mcrouter hosts for cert renewal T248093 |
[production] |
| 17:19 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:16 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:49 |
<urandom> |
bootstrapping restbase2014-a — T250050 |
[production] |
| 15:40 |
<cmjohnson1> |
replacing mgmt switch on a6-eqiad T250652 |
[production] |
| 15:38 |
<hashar> |
CI is back, patches would need to be rechecked by commenting "recheck" in Gerrit. |
[production] |
| 15:32 |
<hashar> |
Restarting Gerrit T250820 T246973 |
[production] |
| 15:26 |
<hashar> |
CI / Zuul does not get any events for some reason :/ |
[production] |
| 14:59 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:59 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:51 |
<hashar> |
contint2001: manually dropping /var/lib/docker (we now use /srv/docker ) |
[production] |
| 14:48 |
<jbond42> |
restart haproxy on dns-auth |
[production] |
| 14:48 |
<hashar> |
restarting docker on contint2001 |
[production] |
| 14:47 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@d101234]: Deploy design/style-guide: (duration: 00m 09s) |
[production] |
| 14:47 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@d101234]: Deploy design/style-guide: |
[production] |
| 14:45 |
<jbond42> |
puppet enabled again |
[production] |
| 14:40 |
<moritzm> |
restarting apache on miscweb |
[production] |
| 14:37 |
<moritzm> |
restarting apache on netbox1001 |
[production] |
| 14:36 |
<jbond42> |
disable puppet fleet wide to restart puppemaster |
[production] |
| 14:28 |
<moritzm> |
installing OpenSSL security updates |
[production] |
| 14:17 |
<vgutierrez> |
rolling upgrade of ats to version 8.0.7-1wm1 |
[production] |
| 14:16 |
<moritzm> |
installing OpenSSL updates on caches |
[production] |
| 14:08 |
<hashar> |
contint1001: rm /var/log/apache2/doc_* # service has been moved to doc1001.eqiad.wmnet |
[production] |
| 13:43 |
<vgutierrez> |
upload trafficserver 8.0.7-1wm1 to apt.wm.o (buster) |
[production] |
| 13:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 13:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:15 |
<mutante> |
recreating cert for contint/integration to add integration.mediawiki.org in addition to integration.wikimedia.org |
[production] |
| 11:06 |
<mutante> |
https://integration.wikimedia.org now also using TLS between ATS and contint1001 using envoy (T210411) |
[production] |
| 10:49 |
<_joe_> |
mwdebug1001:~# iptables -A INPUT -s 10.64.32.208 -m statistic --mode random --probability 0.1 -j DROP (T240684) |
[production] |
| 08:52 |
<ema> |
purged: rolling restart with 4 frontend workers |
[production] |
| 07:54 |
<ema> |
cp3050: restart purged with 4 frontend workers |
[production] |
| 07:47 |
<kormat> |
dropping old data and optimizing tables on pc1010 and pc2010 T247787 |
[production] |
| 07:26 |
<ema> |
cp4032: restart ats-tls and ats-be |
[production] |
| 07:06 |
<ema> |
cp4026: restart ats-tls and ats-be |
[production] |