|
2020-01-14
ยง
|
| 15:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1080 for tranfer', diff saved to https://phabricator.wikimedia.org/P10144 and previous config saved to /var/cache/conftool/dbconfig/20200114-150223-marostegui.json |
[production] |
| 15:00 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:51 |
<liw@deploy1001> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_44869219" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 03m 55s) |
[production] |
| 14:47 |
<liw@deploy1001> |
Started scap: testwiki to php-1.35.0-wmf.15 and rebuild l10n cache |
[production] |
| 14:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1080', diff saved to https://phabricator.wikimedia.org/P10143 and previous config saved to /var/cache/conftool/dbconfig/20200114-144341-marostegui.json |
[production] |
| 14:26 |
<marostegui> |
Move db1114 under db1080 |
[production] |
| 14:24 |
<marostegui> |
Stop db1080 and db1107 replication in sync |
[production] |
| 14:23 |
<Zppix> |
delete pod to attempt force restart due to openstack upgrade (T241347) |
[tools.zppixbot] |
| 14:21 |
<XioNoX> |
push firewall policies to pfw3-eqiad - T242681 |
[production] |
| 14:15 |
<XioNoX> |
push firewall policies to pfw3-codfw - T242681 |
[production] |
| 14:12 |
<liw> |
branch cut for 1.35.0-wmf.15 |
[production] |
| 14:09 |
<vgutierrez> |
upgrade ats to 8.0.5-1wm12 in cp5006 and cp5012 - T242620 |
[production] |
| 14:03 |
<arturo> |
icinga downtime all cloudvirts for another 2h for fixing some icinga checks |
[admin] |
| 14:03 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:03 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:54 |
<marostegui> |
Upgrade db1080 |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1080 for upgrade', diff saved to https://phabricator.wikimedia.org/P10142 and previous config saved to /var/cache/conftool/dbconfig/20200114-135238-marostegui.json |
[production] |
| 12:16 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir3002.esams.wmnet |
[production] |
| 12:16 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir3001.esams.wmnet |
[production] |
| 12:14 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir4001.ulsfo.wmnet |
[production] |
| 12:14 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir4002.ulsfo.wmnet |
[production] |
| 12:04 |
<arturo> |
icinga downtime toolchecker for 2 hours for openstack upgrades T241347 |
[admin] |
| 12:02 |
<arturo> |
icinga downtime cloud* labs* hosts for 2 hours for openstack upgrades T241347 |
[admin] |
| 12:02 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:02 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:02 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:01 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:51 |
<vgutierrez> |
restarting pybal on lvs4005 (high-traffic1 LVS) - T242321 |
[production] |
| 11:49 |
<vgutierrez> |
restarting pybal on lvs4007 (secondary LVS) - T242321 |
[production] |
| 11:48 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: service=nginx,name=ncredir4002.ulsfo.wmnet |
[production] |
| 11:47 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: service=nginx,name=ncredir4001.ulsfo.wmnet |
[production] |
| 11:15 |
<vgutierrez> |
Updating puppet-compiler facts |
[production] |
| 10:40 |
<vgutierrez> |
upgrade ats to 8.0.5-1wm12 in cp4026 and cp4032 - T242620 |
[production] |
| 10:07 |
<moritzm> |
installing remaining cyrus-sasl security updates |
[production] |
| 09:44 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.14/extensions/Wikibase/lib/includes/Store/Sql/Terms: [[gerrit:564555|wbterms: Add Statsd metrics in critical parts of the new term store]] (duration: 00m 57s) |
[production] |
| 09:27 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.mwpackages] |
| 09:26 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.ipchanges] |
| 09:23 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.extreg-wos] |
| 09:21 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.extreg-wos] |
| 09:19 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.fab-proxy] |
| 09:19 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.extreg-wos] |
| 09:17 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.dump-torrents] |
| 09:16 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.coverme] |
| 09:14 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.coverage] |
| 09:13 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.contentcontributor] |
| 09:12 |
<wm-bot> |
<legoktm> Switched over to use new k8s cluster |
[tools.checker] |
| 07:33 |
<XioNoX> |
add peering to AS26744 in eqiad, eqord and eqdfw |
[production] |
| 06:25 |
<marostegui> |
Deploy schema change on flowdb (x1) directly on the master T242688 |
[production] |
| 06:23 |
<marostegui> |
Deploy schema change on labswiki (wikitech) T242688 |
[production] |
| 06:20 |
<marostegui> |
Deploy schema change on s3 master for officewiki and techconductwiki T242688 |
[production] |