301-350 of 10000 results (47ms)
2019-12-02 §
10:54 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:52 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:44 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:43 <moritzm> installing python-psutil security updates [production]
10:42 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:42 <effie> reimage mw1299.eqiad.wmnet [production]
10:18 <effie> reimage mw1290.eqiad.wmnet [production]
10:18 <effie> reimage mw1275.eqiad.wmnet [production]
10:15 <moritzm> installing file/libmagic regresssion update for jessie [production]
10:08 <filippo@cumin1001> conftool action : set/pooled=no; selector: name=mw2229.codfw.wmnet [production]
09:52 <godog> swift eqiad-prod: more weight to ms-be105[7-9] - T237438 [production]
09:51 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:49 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:49 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:46 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:41 <joal@deploy1001> Finished deploy [analytics/refinery@8991301] (thin): Regular analytics deploy - late from last week (thin) (duration: 00m 08s) [production]
09:41 <joal@deploy1001> Started deploy [analytics/refinery@8991301] (thin): Regular analytics deploy - late from last week (thin) [production]
09:40 <joal@deploy1001> Finished deploy [analytics/refinery@8991301]: Regular analytics deploy - late from last week (duration: 18m 22s) [production]
09:23 <effie> reimage mw1300.eqiad.wmnet [production]
09:23 <effie> reimage mw1300.eqiad.wmne [production]
09:22 <joal@deploy1001> Started deploy [analytics/refinery@8991301]: Regular analytics deploy - late from last week [production]
09:16 <moritzm> installing libvpx security updates [production]
09:14 <godog> extend graphite LVs on graphite1004 / graphite2003 by 200G [production]
08:39 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:37 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:33 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:31 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:14 <effie> reimage mw1287.eqiad.wmnet mw1288.eqiad.wmnet mw1289.eqiad.wmnet [production]
08:08 <effie> reimage mw1301.eqiad.wmnet [production]
08:03 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:18 <andrewbogott> forcing a reboot of cloudstore1008 via mgmt console — it seems to have locked up [production]
06:43 <Urbanecm> Clear account creation throttle for several IPs (T239465) [production]
06:38 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: New throttle rule for cawiki workshop (T239465) (duration: 01m 03s) [production]
06:00 <marostegui> Compress s8 codfw master (lag might appear on codfw s8) [production]
06:00 <marostegui> Compress s4 codfw master (lag might appear on codfw s4) [production]
05:56 <marostegui> Deploy schema change on db1075 [production]
05:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1075 for schema change', diff saved to https://phabricator.wikimedia.org/P9791 and previous config saved to /var/cache/conftool/dbconfig/20191202-055546-marostegui.json [production]
05:53 <marostegui> Compress db1099:3318 T235599 [production]
05:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3318 for compression', diff saved to https://phabricator.wikimedia.org/P9790 and previous config saved to /var/cache/conftool/dbconfig/20191202-055245-marostegui.json [production]
2019-12-01 §
23:27 <ladsgroup@deploy1001> Started restart [mobileapps/deploy@70154b4]: Rolling restart of mobileapps [production]
23:20 <bblack> restarting AQS services in eqiad [production]
23:15 <eileen> process-control config revision is 9750c318a0 - jobs disabled [production]
21:39 <andrewbogott> restarted nova conductor and api on cloudcontrol1003 and 1004 to free up db connections (T239168) [production]
2019-11-30 §
15:47 <Urbanecm> Reset email of SUL user Hayk.arabaget (T239462) [production]
07:40 <vgutierrez> repooling cp3057 - T239502 [production]
07:30 <vgutierrez@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3057.esams.wmnet [production]
07:30 <vgutierrez> depool and powercycle cp3057 - T239502 [production]
2019-11-29 §
22:02 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:00 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]