601-650 of 10000 results (24ms)
2020-05-04 §
10:43 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:594128| Bumping portals to master (563985)]] (duration: 01m 29s) [production]
10:39 <vgutierrez> rolling upgrade of ATS to version 8.0.7-1wm3 [production]
10:36 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:33 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:30 <arturo> running `aborrero@apt1001:~ $ sudo -i reprepro --delete clearvanished` to cleanup buster-wikimedia|thirdparty/kubeadm-k8s (T250866) [production]
09:46 <vgutierrez> upload trafficserver 8.0.7-1wm2 to apt.wm.o (buster) [production]
09:22 <kormat> reimaging db1101 to buster T250666 [production]
08:50 <XioNoX> configure BGP peering with AS132203 [production]
08:20 <godog> add 50G to prometheus-ops on prometheus100[34] [production]
08:17 <marostegui> Deploy schema change on s5 codfw - T251188 [production]
07:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3317 and db1101:3318 for reimage', diff saved to https://phabricator.wikimedia.org/P11113 and previous config saved to /var/cache/conftool/dbconfig/20200504-075148-marostegui.json [production]
07:31 <marostegui> Drop unused flagged* tables from mediawikiwiki - T248298 [production]
07:26 <moritzm> removed jmorgan from cn=wmf [production]
07:24 <marostegui> Install 10.1.43-2 on s5 (db110) and s6 (db1131) masters in preparations for tomorrow's restart - T251154 [production]
07:24 <moritzm> removed Kerberos principal for lexnasser and jmorgan [production]
07:23 <moritzm> removed lexnasser from cn=nda [production]
07:07 <elukey> execute ifdown eno1; ifup eno1 on analytics1052 - interface neg speed flapping [production]
06:41 <elukey> upload prometheus-druid-exporter 0.8-1 to stretch-wikimedia [production]
2020-05-03 §
22:52 <Krinkle> scap pull mwmaint1002 and mw2001 for noc.wm.o. – https://gerrit.wikimedia.org/r/593929 [production]
22:42 <Krinkle> scap pull mwmaint1002 and mw2001 for noc.wm.o. – https://gerrit.wikimedia.org/r/591459 [production]
21:37 <bmansurov@deploy1001> Finished deploy [recommendation-api/deploy@0c68d62]: Update the recommendation API service (duration: 04m 22s) [production]
21:32 <bmansurov@deploy1001> Started deploy [recommendation-api/deploy@0c68d62]: Update the recommendation API service [production]
2020-05-02 §
07:49 <oblivian@cumin1001> conftool action : set/pooled=yes; selector: name=mw13(49|5[0-9]|6[0-2])\.eqiad\.wmnet [production]
07:08 <XioNoX> asw2-d-eqiad> request virtual-chassis vc-port delete pic-slot 1 port 0 member 1 [production]
07:07 <XioNoX> asw2-d-eqiad> request virtual-chassis vc-port delete pic-slot 0 port 1 member 1 [production]
02:36 <volker-e@deploy1001> Finished deploy [design/style-guide@f0d467b]: Deploy design/style-guide: (duration: 00m 07s) [production]
02:36 <volker-e@deploy1001> Started deploy [design/style-guide@f0d467b]: Deploy design/style-guide: [production]
2020-05-01 §
19:56 <rzl@cumin1001> conftool action : set/pooled=no; selector: name=mw13(5[6-9]|6[0-2]).eqiad.wmnet [production]
18:57 <gehel> restart blazegraph on wdqs1006 - T242453 [production]
14:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11110 and previous config saved to /var/cache/conftool/dbconfig/20200501-142354-marostegui.json [production]
14:18 <hknust> holger@mwmaint1002 finished renameInvalidUsernames.php (fail) as part of T219279 [production]
14:06 <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11109 and previous config saved to /var/cache/conftool/dbconfig/20200501-140603-marostegui.json [production]
13:47 <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11108 and previous config saved to /var/cache/conftool/dbconfig/20200501-134707-marostegui.json [production]
13:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly warm up db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11107 and previous config saved to /var/cache/conftool/dbconfig/20200501-132804-marostegui.json [production]
13:06 <hknust> holger@mwmaint1002 Starting renameInvalidUsernames.php as part of T219279 [production]
13:01 <vgutierrez> rolling restart of ats-tls in text@esams - T249335 [production]
12:24 <mutante> mw230* - rolling restart of php-fpm - icinga warnings about opcache health in codfw [production]
12:20 <mutante> mw2376 - restarting php-fpm - icinga warnings about opcache health in codfw [production]
12:07 <mutante> notebook1004 - puppet was failed due to removal of jmorgan while one of his processes was still running. "change to absent failed.. user jmorgan currently used by process 29038". killing 29038, running puppet T251560 [production]
12:05 <mutante> notebook1003 - puppet was failed due to removal of jmorgan while one of his processeswas still running. "change to absent failed.. user jmorgan currently used by porcess 3288". killing 3288, running puppet T251560 [production]
11:52 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
11:51 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
11:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
11:50 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
11:31 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:31 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:31 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:31 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:54 <_joe_> depooled all servers in the app pool in rack D1 [production]
08:54 <oblivian@cumin1001> conftool action : set/pooled=no:weight=30; selector: name=mw13(49|5[0-5])\.eqiad\.wmnet [production]