|
2020-01-14
ยง
|
| 14:03 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:03 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:54 |
<marostegui> |
Upgrade db1080 |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1080 for upgrade', diff saved to https://phabricator.wikimedia.org/P10142 and previous config saved to /var/cache/conftool/dbconfig/20200114-135238-marostegui.json |
[production] |
| 12:16 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir3002.esams.wmnet |
[production] |
| 12:16 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir3001.esams.wmnet |
[production] |
| 12:14 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir4001.ulsfo.wmnet |
[production] |
| 12:14 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir4002.ulsfo.wmnet |
[production] |
| 12:04 |
<arturo> |
icinga downtime toolchecker for 2 hours for openstack upgrades T241347 |
[admin] |
| 12:02 |
<arturo> |
icinga downtime cloud* labs* hosts for 2 hours for openstack upgrades T241347 |
[admin] |
| 12:02 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:02 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:02 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:01 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:51 |
<vgutierrez> |
restarting pybal on lvs4005 (high-traffic1 LVS) - T242321 |
[production] |
| 11:49 |
<vgutierrez> |
restarting pybal on lvs4007 (secondary LVS) - T242321 |
[production] |
| 11:48 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: service=nginx,name=ncredir4002.ulsfo.wmnet |
[production] |
| 11:47 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: service=nginx,name=ncredir4001.ulsfo.wmnet |
[production] |
| 11:15 |
<vgutierrez> |
Updating puppet-compiler facts |
[production] |
| 10:40 |
<vgutierrez> |
upgrade ats to 8.0.5-1wm12 in cp4026 and cp4032 - T242620 |
[production] |
| 10:07 |
<moritzm> |
installing remaining cyrus-sasl security updates |
[production] |
| 09:44 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.14/extensions/Wikibase/lib/includes/Store/Sql/Terms: [[gerrit:564555|wbterms: Add Statsd metrics in critical parts of the new term store]] (duration: 00m 57s) |
[production] |
| 09:27 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.mwpackages] |
| 09:26 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.ipchanges] |
| 09:23 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.extreg-wos] |
| 09:21 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.extreg-wos] |
| 09:19 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.fab-proxy] |
| 09:19 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.extreg-wos] |
| 09:17 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.dump-torrents] |
| 09:16 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.coverme] |
| 09:14 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.coverage] |
| 09:13 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.contentcontributor] |
| 09:12 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.checker] |
| 07:33 |
<XioNoX> |
add peering to AS26744 in eqiad, eqord and eqdfw |
[production] |
| 06:25 |
<marostegui> |
Deploy schema change on flowdb (x1) directly on the master T242688 |
[production] |
| 06:23 |
<marostegui> |
Deploy schema change on labswiki (wikitech) T242688 |
[production] |
| 06:20 |
<marostegui> |
Deploy schema change on s3 master for officewiki and techconductwiki T242688 |
[production] |
| 06:05 |
<zhuyifei1999_> |
applied T242355-ver2.patch T242355 |
[quarry] |
| 06:01 |
<marostegui> |
Remove partitions from revision table on db1103:3312 |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3312 - T239453', diff saved to https://phabricator.wikimedia.org/P10141 and previous config saved to /var/cache/conftool/dbconfig/20200114-060116-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1105:3312 after removing partitions from revision table', diff saved to https://phabricator.wikimedia.org/P10140 and previous config saved to /var/cache/conftool/dbconfig/20200114-060003-marostegui.json |
[production] |
| 05:29 |
<andrewbogott> |
rebooting cloudservices1004 to make sure all my upgrades are sustainable |
[production] |
| 04:26 |
<andrewbogott> |
upgrading designate on cloudservices1003/1004 |
[admin] |
| 02:25 |
<andrewbogott> |
rebooting snuggle-enwiki-01 to resolve hangs associated with an old NFS failure |
[snuggle] |
| 02:24 |
<andrewbogott> |
rebooting wp1 to resolve hangs associated with an old NFS failure |
[mwoffliner] |
| 02:23 |
<andrewbogott> |
rebooting tools-paws-worker-1006 to resolve hangs associated with an old NFS failure |
[tools] |
| 02:22 |
<andrewbogott> |
rebooting wikidata-lexeme-lua to resolve hangs associated with an old NFS failure |
[wikidata-dev] |
| 02:20 |
<andrewbogott> |
rebooting mathosphere to resolve hangs associated with an old NFS failure |
[math] |
| 02:17 |
<andrewbogott> |
rebooting fridolin and login-test to resolve hangs associated with an old NFS failure |
[catgraph] |
| 02:15 |
<andrewbogott> |
rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch |
[toolsbeta] |