|
2020-01-14
ยง
|
| 17:43 |
<vgutierrez> |
repooling cp4027 |
[production] |
| 17:39 |
<vgutierrez> |
depooling cp4027 for some ats-tls parent balancing tests |
[production] |
| 17:21 |
<_joe_> |
upload docker-report 0.0.2 to {buster,stretch}-wikimedia T242604 |
[production] |
| 16:53 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.35.0-wmf.15 |
[production] |
| 16:46 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:44 |
<liw> |
branch is cut for 1.35.0-wmv.15; train window is closed, but I'll continue train since the next time slot seems to not have anything |
[production] |
| 16:44 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:41 |
<marostegui> |
Enable puppet back on install1002 and install2002 - T242481 |
[production] |
| 16:31 |
<liw@deploy1001> |
Finished scap: testwiki to php-1.34.0-wmf.15 and rebuild l10n cache (try 2) (duration: 43m 29s) |
[production] |
| 16:26 |
<marostegui> |
Disable temporarily puppet on install1002 and install2002 - T242481 |
[production] |
| 16:08 |
<volans@deploy1001> |
Finished deploy [debmonitor/deploy@e72911c]: Release v0.2.4 (duration: 01m 09s) |
[production] |
| 16:07 |
<volans@deploy1001> |
Started deploy [debmonitor/deploy@e72911c]: Release v0.2.4 |
[production] |
| 15:47 |
<liw@deploy1001> |
Started scap: testwiki to php-1.34.0-wmf.15 and rebuild l10n cache (try 2) |
[production] |
| 15:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:02 |
<marostegui> |
Copy data from db1080 to db1107 T242702 |
[production] |
| 15:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1080 for tranfer', diff saved to https://phabricator.wikimedia.org/P10144 and previous config saved to /var/cache/conftool/dbconfig/20200114-150223-marostegui.json |
[production] |
| 15:00 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:51 |
<liw@deploy1001> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_44869219" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 03m 55s) |
[production] |
| 14:47 |
<liw@deploy1001> |
Started scap: testwiki to php-1.35.0-wmf.15 and rebuild l10n cache |
[production] |
| 14:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1080', diff saved to https://phabricator.wikimedia.org/P10143 and previous config saved to /var/cache/conftool/dbconfig/20200114-144341-marostegui.json |
[production] |
| 14:26 |
<marostegui> |
Move db1114 under db1080 |
[production] |
| 14:24 |
<marostegui> |
Stop db1080 and db1107 replication in sync |
[production] |
| 14:21 |
<XioNoX> |
push firewall policies to pfw3-eqiad - T242681 |
[production] |
| 14:15 |
<XioNoX> |
push firewall policies to pfw3-codfw - T242681 |
[production] |
| 14:12 |
<liw> |
branch cut for 1.35.0-wmf.15 |
[production] |
| 14:09 |
<vgutierrez> |
upgrade ats to 8.0.5-1wm12 in cp5006 and cp5012 - T242620 |
[production] |
| 14:03 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:03 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:54 |
<marostegui> |
Upgrade db1080 |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1080 for upgrade', diff saved to https://phabricator.wikimedia.org/P10142 and previous config saved to /var/cache/conftool/dbconfig/20200114-135238-marostegui.json |
[production] |
| 12:16 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir3002.esams.wmnet |
[production] |
| 12:16 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir3001.esams.wmnet |
[production] |
| 12:14 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir4001.ulsfo.wmnet |
[production] |
| 12:14 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=nginx,name=ncredir4002.ulsfo.wmnet |
[production] |
| 12:02 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:02 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:02 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:01 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:51 |
<vgutierrez> |
restarting pybal on lvs4005 (high-traffic1 LVS) - T242321 |
[production] |
| 11:49 |
<vgutierrez> |
restarting pybal on lvs4007 (secondary LVS) - T242321 |
[production] |
| 11:48 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: service=nginx,name=ncredir4002.ulsfo.wmnet |
[production] |
| 11:47 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: service=nginx,name=ncredir4001.ulsfo.wmnet |
[production] |
| 11:15 |
<vgutierrez> |
Updating puppet-compiler facts |
[production] |
| 10:40 |
<vgutierrez> |
upgrade ats to 8.0.5-1wm12 in cp4026 and cp4032 - T242620 |
[production] |
| 10:07 |
<moritzm> |
installing remaining cyrus-sasl security updates |
[production] |
| 09:44 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.14/extensions/Wikibase/lib/includes/Store/Sql/Terms: [[gerrit:564555|wbterms: Add Statsd metrics in critical parts of the new term store]] (duration: 00m 57s) |
[production] |
| 07:33 |
<XioNoX> |
add peering to AS26744 in eqiad, eqord and eqdfw |
[production] |
| 06:25 |
<marostegui> |
Deploy schema change on flowdb (x1) directly on the master T242688 |
[production] |
| 06:23 |
<marostegui> |
Deploy schema change on labswiki (wikitech) T242688 |
[production] |
| 06:20 |
<marostegui> |
Deploy schema change on s3 master for officewiki and techconductwiki T242688 |
[production] |