|
2020-11-30
§
|
| 12:21 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=0; selector: dc=eqiad,cluster=maps,service=kartotherian,name=maps1005.eqiad.wmnet |
[production] |
| 12:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: 2922abe7b810f1b53b446af783dc9d51e6585225: Remove wgContentTranslationRESTBase config (T266213) (duration: 00m 57s) |
[production] |
| 11:43 |
<marostegui> |
Sanitize clouddb1016:3318 - T267090 |
[production] |
| 11:38 |
<ema> |
A:cp upgrade fifo-log-demux to 0.6.2 T268883 |
[production] |
| 11:36 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:644196| Bumping portals to master (T128546)]] (duration: 00m 57s) |
[production] |
| 11:35 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:644196| Bumping portals to master (T128546)]] (duration: 01m 01s) |
[production] |
| 11:32 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@e8c6267]: allow page content fixup script to write output files to arbitrary dir (duration: 00m 04s) |
[production] |
| 11:32 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@e8c6267]: allow page content fixup script to write output files to arbitrary dir |
[production] |
| 11:28 |
<ema> |
upload fifo-log-demux 0.6.2 to buster-wikimedia T268883 |
[production] |
| 11:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13473 and previous config saved to /var/cache/conftool/dbconfig/20201130-111321-root.json |
[production] |
| 11:00 |
<hnowlan> |
bootstrapping maps1005 cassandra |
[production] |
| 10:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13472 and previous config saved to /var/cache/conftool/dbconfig/20201130-105818-root.json |
[production] |
| 10:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13471 and previous config saved to /var/cache/conftool/dbconfig/20201130-104314-root.json |
[production] |
| 10:29 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 10:29 |
<marostegui> |
Compare data between clouddb1014:3312 clouddb1018:3312 labsdb1012 T267090 |
[production] |
| 10:29 |
<marostegui> |
Compare data between clouddb1012:3312 clouddb1018:3312 labsdb1012 T267090 |
[production] |
| 10:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13470 and previous config saved to /var/cache/conftool/dbconfig/20201130-102811-root.json |
[production] |
| 10:24 |
<akosiaris> |
applying https://gerrit.wikimedia.org/r/q/topic:%22k8s_config%22 series of patches |
[production] |
| 10:18 |
<ema@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 10:18 |
<ema> |
cp4031: reboot to test atsmtail/fifo-log-demux service dependencies -- https://gerrit.wikimedia.org/r/c/operations/puppet/+/643922 T256467 |
[production] |
| 10:11 |
<ema> |
cp4032: upgrade varnish to 6.0.7-1wm1 T268736 |
[production] |
| 10:06 |
<moritzm> |
installing NSS security updates |
[production] |
| 09:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1134 for schema change', diff saved to https://phabricator.wikimedia.org/P13469 and previous config saved to /var/cache/conftool/dbconfig/20201130-095729-marostegui.json |
[production] |
| 09:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1089 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13468 and previous config saved to /var/cache/conftool/dbconfig/20201130-095621-root.json |
[production] |
| 09:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1089 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13467 and previous config saved to /var/cache/conftool/dbconfig/20201130-094117-root.json |
[production] |
| 09:40 |
<marostegui> |
Stop MySQL on db1087 to clone clouddb1016:3318 T267090) |
[production] |
| 09:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 from s8 and pool db1092 instead temporarily on vslow T267090', diff saved to https://phabricator.wikimedia.org/P13466 and previous config saved to /var/cache/conftool/dbconfig/20201130-093909-marostegui.json |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1089 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13465 and previous config saved to /var/cache/conftool/dbconfig/20201130-092614-root.json |
[production] |
| 09:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1089+ (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13464 and previous config saved to /var/cache/conftool/dbconfig/20201130-092154-root.json |
[production] |
| 08:51 |
<marostegui> |
Deploy schema change on db1089 |
[production] |
| 08:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1089', diff saved to https://phabricator.wikimedia.org/P13463 and previous config saved to /var/cache/conftool/dbconfig/20201130-085101-marostegui.json |
[production] |
| 08:41 |
<godog> |
swift eqiad-prod: add weight to ms-be106[0-3] - T268435 |
[production] |
| 08:36 |
<marostegui> |
Compare data between clouddb1016:3315 labsdb1012 T267090 |
[production] |
| 07:45 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 07:41 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:25 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 07:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:11 |
<marostegui> |
Deploy schema change on s1 codfw - T268004 |
[production] |
| 07:05 |
<marostegui> |
Stop mysql on db1124:3318 to clone clouddb1016:3318, lag will show up on wikireplicas on s8 T267090 |
[production] |
| 06:47 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 06:43 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 04:26 |
<kart_> |
Updated cxserver to 2020-11-23-050106-production (T262253, T268410) |
[production] |
| 04:18 |
<kartik@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 04:14 |
<kartik@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 04:11 |
<kartik@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |