1051-1100 of 10000 results (61ms)
2019-12-02 §
09:14 <godog> extend graphite LVs on graphite1004 / graphite2003 by 200G [production]
08:39 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:37 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:33 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:31 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:14 <effie> reimage mw1287.eqiad.wmnet mw1288.eqiad.wmnet mw1289.eqiad.wmnet [production]
08:08 <effie> reimage mw1301.eqiad.wmnet [production]
08:03 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:18 <andrewbogott> forcing a reboot of cloudstore1008 via mgmt console — it seems to have locked up [production]
06:43 <Urbanecm> Clear account creation throttle for several IPs (T239465) [production]
06:38 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: New throttle rule for cawiki workshop (T239465) (duration: 01m 03s) [production]
06:00 <marostegui> Compress s8 codfw master (lag might appear on codfw s8) [production]
06:00 <marostegui> Compress s4 codfw master (lag might appear on codfw s4) [production]
05:56 <marostegui> Deploy schema change on db1075 [production]
05:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1075 for schema change', diff saved to https://phabricator.wikimedia.org/P9791 and previous config saved to /var/cache/conftool/dbconfig/20191202-055546-marostegui.json [production]
05:53 <marostegui> Compress db1099:3318 T235599 [production]
05:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3318 for compression', diff saved to https://phabricator.wikimedia.org/P9790 and previous config saved to /var/cache/conftool/dbconfig/20191202-055245-marostegui.json [production]
2019-12-01 §
23:27 <ladsgroup@deploy1001> Started restart [mobileapps/deploy@70154b4]: Rolling restart of mobileapps [production]
23:20 <bblack> restarting AQS services in eqiad [production]
23:15 <eileen> process-control config revision is 9750c318a0 - jobs disabled [production]
21:39 <andrewbogott> restarted nova conductor and api on cloudcontrol1003 and 1004 to free up db connections (T239168) [production]
2019-11-30 §
15:47 <Urbanecm> Reset email of SUL user Hayk.arabaget (T239462) [production]
07:40 <vgutierrez> repooling cp3057 - T239502 [production]
07:30 <vgutierrez@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3057.esams.wmnet [production]
07:30 <vgutierrez> depool and powercycle cp3057 - T239502 [production]
2019-11-29 §
22:02 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:00 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:38 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:36 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:12 <effie> reimage mw1302.eqiad.wmnet [production]
20:50 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:47 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:15 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:13 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:44 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:42 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:19 <effie> reimage mw1284.eqiad.wmnet [production]
19:19 <effie> reimage mw1303.eqiad.wmnet mw1283.eqiad.wmnet [production]
17:26 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:24 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:51 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:49 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:42 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:40 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:22 <filippo@cumin1001> conftool action : set/pooled=yes; selector: name=mw2228.codfw.wmnet [production]
16:17 <effie> reimage mw1274.eqiad.wmnet [production]
16:14 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:12 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:40 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]