2651-2700 of 10000 results (39ms)
2020-10-19 §
07:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2125 ', diff saved to https://phabricator.wikimedia.org/P13022 and previous config saved to /var/cache/conftool/dbconfig/20201019-071614-marostegui.json [production]
06:46 <elukey@deploy1001> Finished deploy [analytics/turnilo/deploy@334627e]: Upgrade to 1.27 (duration: 00m 10s) [production]
06:45 <elukey@deploy1001> Started deploy [analytics/turnilo/deploy@334627e]: Upgrade to 1.27 [production]
2020-10-17 §
13:22 <Urbanecm> [urbanecm@mwmaint2001 ~/uploads]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Fæ . # T264529 [production]
2020-10-16 §
21:46 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:43 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
20:27 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:25 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:39 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:37 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:43 <thcipriani> restarting gerrit due to gc thrashing [production]
16:25 <andrew@deploy1001> Finished deploy [horizon/deploy@89b308c]: prevent creation of VMs with non-ceph flavors (duration: 04m 08s) [production]
16:21 <andrew@deploy1001> Started deploy [horizon/deploy@89b308c]: prevent creation of VMs with non-ceph flavors [production]
15:36 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
15:36 <ayounsi@cumin1001> START - Cookbook sre.network.cf [production]
15:11 <bblack@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
15:01 <bblack@cumin1001> START - Cookbook sre.hosts.decommission [production]
13:41 <effie> pooling mw2279.codfw.wmnet T264698 [production]
12:11 <jiji@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:09 <jiji@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:35 <reedy@deploy1001> Synchronized php-1.36.0-wmf.13/extensions/ProofreadPage/: Revert excessive escaping T265571 (duration: 01m 12s) [production]
09:23 <ema> text@esams (except for cp3050/cp3052): upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 [production]
09:19 <ema> upload@esams: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 [production]
09:08 <ema> upload@eqsin: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 [production]
09:03 <XioNoX> eqsin, push CR 634473 [production]
09:01 <ema> text@eqsin: upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 [production]
08:53 <ema> upload@codfw: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 [production]
08:52 <XioNoX> add BGP_IXP_RS_in to eqsin RS BGP sessions [production]
08:48 <ema> text@codfw: upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 [production]
08:29 <ema> upload@eqiad: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 [production]
08:24 <ema> text@eqiad: upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 [production]
08:09 <elukey> reboot stat1005/stat1008 to pick up correct GPU settings [production]
08:08 <ema> upload@ulsfo: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 [production]
07:59 <ema> text@ulsfo: upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 [production]
07:19 <dcausse@deploy1001> Finished deploy [wikimedia/discovery/analytics@27d0b01]: cirrus namespace map: Align output columns with table (duration: 04m 22s) [production]
07:15 <dcausse@deploy1001> Started deploy [wikimedia/discovery/analytics@27d0b01]: cirrus namespace map: Align output columns with table [production]
06:57 <XioNoX> enable cr2-eqdfw:xe-0/1/2 [production]
02:14 <eileen> civicrm revision changed from 585eb835d8 to 3c3dcf80ae, config revision is f76d7849bc [production]
01:01 <ryankemper> Cleaning up a dangling no-longer-puppet-managed udev elasticsearch-readahead rule across all cirrus instances: `sudo cumin -b 36 C:profile::elasticsearch::cirrus 'sudo rm -fv /etc/udev/rules.d/elasticsearch-readahead.rules && sudo /sbin/udevadm control --reload && sudo /sbin/udevadm trigger'` [production]
00:56 <cdanis@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
00:56 <cdanis@cumin1001> START - Cookbook sre.network.cf [production]
2020-10-15 §
23:49 <ryankemper> Began in-place reindex of `eqiad`, `codfw`, and `cloudelastic`. Running on `ryankemper@mwmaint2001` under tmux sessions `inplace_reindex_[eqiad, codfw, cloudelastic]` [production]
23:00 <krinkle@deploy1001> Synchronized wmf-config/env.php: I245e84e0b8c (duration: 01m 10s) [production]
22:09 <cdanis> previous sre.network.cf invocation was a no-op; just checking status [production]
22:08 <cdanis@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
22:08 <cdanis@cumin1001> START - Cookbook sre.network.cf [production]
22:06 <mutante> depooled remaining wtp* servers in codfw. old parsoid servers, new servers are parse2* (T265558) [production]
22:05 <dzahn@cumin1001> conftool action : set/pooled=no; selector: dc=codfw,name=wtp2020.codfw.wmnet [production]
22:05 <dzahn@cumin1001> conftool action : set/pooled=no; selector: dc=codfw,name=wtp201[6-9].codfw.wmnet [production]
21:35 <dzahn@cumin1001> conftool action : set/pooled=no; selector: dc=codfw,name=wtp201[0-5].codfw.wmnet [production]