951-1000 of 10000 results (20ms)
2020-07-28 ยง
11:07 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:56 <hashar@deploy1001> Finished deploy [integration/docroot@ba85bdf]: Catch up with HEAD and support DOCUMENT_ROOT being a symbolic link for T149924 (duration: 00m 06s) [production]
10:56 <hashar@deploy1001> Started deploy [integration/docroot@ba85bdf]: Catch up with HEAD and support DOCUMENT_ROOT being a symbolic link for T149924 [production]
10:55 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:53 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:50 <jayme@cumin1001> conftool action : set/pooled=inactive; selector: name=wtp1033.eqiad.wmnet [production]
10:48 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet [production]
10:48 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet [production]
10:47 <jayme@cumin1001> conftool action : set/pooled=inactive; selector: name=wtp1032.eqiad.wmnet [production]
10:47 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet [production]
10:33 <jayme@cumin1001> conftool action : set/pooled=inactive; selector: name=wtp1031.eqiad.wmnet [production]
10:32 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet [production]
10:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1082', diff saved to https://phabricator.wikimedia.org/P12074 and previous config saved to /var/cache/conftool/dbconfig/20200728-102342-marostegui.json [production]
10:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12072 and previous config saved to /var/cache/conftool/dbconfig/20200728-100412-marostegui.json [production]
09:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:57 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:55 <XioNoX> standardize cr2-esams interfaces [production]
09:52 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:50 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:49 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:47 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:45 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:43 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:40 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:38 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:35 <moritzm> imported libmysqlclient18 to component/cloudera T258768 [production]
09:31 <jayme@cumin1001> conftool action : set/pooled=inactive; selector: name=wtp1030.eqiad.wmnet [production]
09:28 <jayme@cumin1001> conftool action : set/pooled=inactive; selector: name=wtp1029.eqiad.wmnet [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12070 and previous config saved to /var/cache/conftool/dbconfig/20200728-092606-marostegui.json [production]
09:24 <jayme@cumin1001> conftool action : set/pooled=inactive; selector: name=wtp1028.eqiad.wmnet [production]
09:19 <XioNoX> standardize cr3-eqsin interfaces [production]
09:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12069 and previous config saved to /var/cache/conftool/dbconfig/20200728-091849-marostegui.json [production]
09:18 <jayme@cumin1001> conftool action : set/pooled=inactive; selector: name=wtp1027.eqiad.wmnet [production]
09:10 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet [production]
09:07 <ema> cp3050: restart varnishmtail.service, stuck on "Condition(c->offset <= c->vtx->len) not true." [production]
08:39 <XioNoX> standardize cr2-eqsin interfaces [production]
08:38 <godog> temporary downgrade prometheus-snmp-exporter on netmon2001 [production]
08:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1143 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12067 and previous config saved to /var/cache/conftool/dbconfig/20200728-083336-marostegui.json [production]
08:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P12066 and previous config saved to /var/cache/conftool/dbconfig/20200728-083209-marostegui.json [production]
08:20 <liw@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.2 (duration: 53m 11s) [production]
08:09 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:07 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:06 <godog> failover librenms/smokeping to netmon2001 - T247967 [production]
08:04 <marostegui> Reduce labsdb1009 weight [production]
07:56 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:48 <jayme> depooled wtp1026.eqiad.wmnet for reimage [production]
07:48 <moritzm> switched superset to CAS [production]
07:47 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
07:46 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
07:43 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]