3601-3650 of 10000 results (67ms)
2020-03-06 ยง
16:40 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:40 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:40 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:40 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:11 <moritzm> installing libtimedate-perl updates from Stretch point release [production]
15:07 <reedy@deploy1001> Synchronized langlist-labs: T247091 (duration: 01m 05s) [production]
14:53 <elukey@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
14:50 <elukey@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
14:44 <XioNoX> add cloud-out4 firewall filter in codfw - T246887 [production]
11:56 <akosiaris> T238658. kubernetes1001 pooled for eventstreams, weight=1 which should account for 2.1% of traffic [production]
11:51 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: dc=eqiad,service=eventstreams,name=kubernetes1001.* [production]
11:50 <akosiaris@cumin1001> conftool action : set/weight=1; selector: dc=eqiad,service=eventstreams,name=kube.* [production]
10:21 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
10:16 <elukey@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) [production]
10:10 <moritzm> rolling restart of Exim on mx* to pick up libidn security updates [production]
10:06 <elukey@cumin1001> START - Cookbook sre.presto.roll-restart-workers [production]
10:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1074', diff saved to https://phabricator.wikimedia.org/P10648 and previous config saved to /var/cache/conftool/dbconfig/20200306-100628-marostegui.json [production]
10:03 <moritzm> rolling restart of labweb* to pick up libidn security updates [production]
09:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2084:3314, db2084:3315 after reimage to buster - T246604', diff saved to https://phabricator.wikimedia.org/P10647 and previous config saved to /var/cache/conftool/dbconfig/20200306-095407-marostegui.json [production]
09:52 <moritzm> rolling restart of slapd on LDAP replicas to pick up libidn security updates [production]
09:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1074', diff saved to https://phabricator.wikimedia.org/P10646 and previous config saved to /var/cache/conftool/dbconfig/20200306-095115-marostegui.json [production]
09:46 <elukey@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
09:45 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:43 <elukey@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
09:42 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:21 <marostegui> Stop MySQL on db2084:3315, db2084:3314 for reimage T246604 [production]
09:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2084:3314, db2084:3315 for reimage to buster - T246604', diff saved to https://phabricator.wikimedia.org/P10645 and previous config saved to /var/cache/conftool/dbconfig/20200306-092103-marostegui.json [production]
09:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1074', diff saved to https://phabricator.wikimedia.org/P10644 and previous config saved to /var/cache/conftool/dbconfig/20200306-092026-marostegui.json [production]
09:12 <moritzm> rolling restart of mw canaries to pick up libidn security updates [production]
09:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1074', diff saved to https://phabricator.wikimedia.org/P10643 and previous config saved to /var/cache/conftool/dbconfig/20200306-090328-marostegui.json [production]
09:00 <moritzm> installing libidn security updates [production]
08:56 <moritzm> rolling restart of kartotherian/tilerator/tileratorui to pick up OpenJPEG security updates [production]
08:56 <marostegui> Stop MySQL on db1074 for upgrade T239791 [production]
08:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 for upgrade T239791', diff saved to https://phabricator.wikimedia.org/P10642 and previous config saved to /var/cache/conftool/dbconfig/20200306-085435-marostegui.json [production]
08:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1113:3315, db1113:3316 after upgrade - T239791', diff saved to https://phabricator.wikimedia.org/P10641 and previous config saved to /var/cache/conftool/dbconfig/20200306-085332-marostegui.json [production]
08:47 <marostegui> Stop mysql for db1113:3315, db1113:3316 for upgrade T239791 [production]
08:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1113:3315, db1113:3316 for upgrade - T239791', diff saved to https://phabricator.wikimedia.org/P10640 and previous config saved to /var/cache/conftool/dbconfig/20200306-084439-marostegui.json [production]
08:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1078 T246604', diff saved to https://phabricator.wikimedia.org/P10639 and previous config saved to /var/cache/conftool/dbconfig/20200306-084141-marostegui.json [production]
08:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2085:3311, db2085:3318 after reimage to buster - T246604', diff saved to https://phabricator.wikimedia.org/P10638 and previous config saved to /var/cache/conftool/dbconfig/20200306-082858-marostegui.json [production]
08:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1078 T246604', diff saved to https://phabricator.wikimedia.org/P10637 and previous config saved to /var/cache/conftool/dbconfig/20200306-082549-marostegui.json [production]
08:19 <moritzm> installing openjpeg2 security updates [production]
08:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:50 <marostegui> Stop MySQL on db2085:3311, db2085:3318 for reimage to buster T246604 [production]
07:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2085:3311, db2085:3318 for reimage to buster - T246604', diff saved to https://phabricator.wikimedia.org/P10636 and previous config saved to /var/cache/conftool/dbconfig/20200306-074427-marostegui.json [production]
07:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1078 T246604', diff saved to https://phabricator.wikimedia.org/P10635 and previous config saved to /var/cache/conftool/dbconfig/20200306-073707-marostegui.json [production]
07:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1078 T246604', diff saved to https://phabricator.wikimedia.org/P10634 and previous config saved to /var/cache/conftool/dbconfig/20200306-070538-marostegui.json [production]
06:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Install 10.4 instead of 10.3 on db1078', diff saved to https://phabricator.wikimedia.org/P10633 and previous config saved to /var/cache/conftool/dbconfig/20200306-064800-marostegui.json [production]
01:38 <mutante> added 9 more appservers to codfw pool split between appserver and API appservers, weight 15 (like all in codfw) T247021 [production]
01:37 <mutante> added 9 more appservers to codfw pool [production]