4001-4050 of 10000 results (33ms)
2021-02-01 ยง
12:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 60%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14099 and previous config saved to /var/cache/conftool/dbconfig/20210201-121301-root.json [production]
12:12 <mlitn@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 9836287e0, 424efdcdb: [WikibaseMediaInfo] Set wgMediaInfoMediaSearchHasLtrPlugin & wgMediaInfoMediaSearchConceptChipsSimpleHeuristics (duration: 01m 10s) [production]
11:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14098 and previous config saved to /var/cache/conftool/dbconfig/20210201-115757-root.json [production]
11:50 <oblivian@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=swift-ro,name=codfw [production]
11:49 <oblivian@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=^swift,name=codfw [production]
11:47 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:660807| Bumping portals to master (T128546)]] (duration: 01m 04s) [production]
11:46 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:660807| Bumping portals to master (T128546)]] (duration: 01m 14s) [production]
11:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 30%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14097 and previous config saved to /var/cache/conftool/dbconfig/20210201-114254-root.json [production]
11:28 <XioNoX> push pfw policies - T272073 [production]
11:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14096 and previous config saved to /var/cache/conftool/dbconfig/20210201-112750-root.json [production]
11:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 20%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14095 and previous config saved to /var/cache/conftool/dbconfig/20210201-111246-root.json [production]
11:04 <hashar@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.28 [production]
11:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14094 and previous config saved to /var/cache/conftool/dbconfig/20210201-110102-root.json [production]
10:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 15%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14093 and previous config saved to /var/cache/conftool/dbconfig/20210201-105743-root.json [production]
10:54 <hashar@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.28 (duration: 07m 48s) [production]
10:46 <hashar@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.28 [production]
10:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14092 and previous config saved to /var/cache/conftool/dbconfig/20210201-104559-root.json [production]
10:45 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1047.eqiad.wmnet with reason: reboot [production]
10:45 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on ms-be1047.eqiad.wmnet with reason: reboot [production]
10:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 12%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14091 and previous config saved to /var/cache/conftool/dbconfig/20210201-104240-root.json [production]
10:42 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.28/includes/: Fixing T273317 T273296 (duration: 01m 01s) [production]
10:41 <urbanecm@deploy1001> sync-file aborted: Fixing T273317 T273296 (duration: 00m 12s) [production]
10:39 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.28/includes/user//User.php: Fixing T273317 T273296 (duration: 00m 58s) [production]
10:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 60%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14090 and previous config saved to /var/cache/conftool/dbconfig/20210201-103055-root.json [production]
10:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 10%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14089 and previous config saved to /var/cache/conftool/dbconfig/20210201-102736-root.json [production]
10:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14088 and previous config saved to /var/cache/conftool/dbconfig/20210201-101552-root.json [production]
10:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 9%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14087 and previous config saved to /var/cache/conftool/dbconfig/20210201-101233-root.json [production]
10:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 30%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14086 and previous config saved to /var/cache/conftool/dbconfig/20210201-100048-root.json [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 8%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14085 and previous config saved to /var/cache/conftool/dbconfig/20210201-095729-root.json [production]
09:52 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool es4 into writes T266483 (duration: 00m 56s) [production]
09:46 <marostegui> Restart mysql on es1021 T266483 [production]
09:45 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool es4 from writes T266483 (duration: 01m 04s) [production]
09:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14084 and previous config saved to /var/cache/conftool/dbconfig/20210201-094545-root.json [production]
09:42 <elukey@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 7%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14083 and previous config saved to /var/cache/conftool/dbconfig/20210201-094226-root.json [production]
09:39 <elukey@cumin1001> START - Cookbook sre.dns.netbox [production]
09:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 20%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14082 and previous config saved to /var/cache/conftool/dbconfig/20210201-093041-root.json [production]
09:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 6%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14081 and previous config saved to /var/cache/conftool/dbconfig/20210201-092722-root.json [production]
09:27 <dcausse> restarting blazegraph on wdqs1013 [production]
09:24 <XioNoX> renumber gr-3/3/0.1 local endpoint on cr1-eqiad [production]
09:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 15%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14080 and previous config saved to /var/cache/conftool/dbconfig/20210201-091538-root.json [production]
09:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 5%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14079 and previous config saved to /var/cache/conftool/dbconfig/20210201-091218-root.json [production]
09:04 <gilles@deploy1001> Finished deploy [performance/navtiming@3215510]: T271208 browser_minor is needed for Mobile Safari allowlist (duration: 00m 05s) [production]
09:04 <gilles@deploy1001> Started deploy [performance/navtiming@3215510]: T271208 browser_minor is needed for Mobile Safari allowlist [production]
09:03 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1054.eqiad.wmnet with reason: reboot [production]
09:03 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on ms-be1054.eqiad.wmnet with reason: reboot [production]
09:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 12%: Slowly pooling db1175 for the first time', diff saved to https://phabricator.wikimedia.org/P14078 and previous config saved to /var/cache/conftool/dbconfig/20210201-090034-root.json [production]
09:00 <elukey@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
08:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 3%: Slowly pooling db1166 for the first time', diff saved to https://phabricator.wikimedia.org/P14077 and previous config saved to /var/cache/conftool/dbconfig/20210201-085714-root.json [production]
08:56 <marostegui> Stop MySQL on db1089 - T273417 [production]