|
2020-07-27
ยง
|
| 19:44 |
<jhuneidi@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 19:36 |
<jhuneidi@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 19:23 |
<jhuneidi@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'sessionstore' for release 'production' . |
[production] |
| 19:19 |
<jhuneidi@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'sessionstore' for release 'production' . |
[production] |
| 19:11 |
<jhuneidi@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 19:06 |
<jhuneidi@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'echostore' for release 'production' . |
[production] |
| 19:00 |
<jhuneidi@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'echostore' for release 'production' . |
[production] |
| 18:57 |
<urbanecm@deploy1001> |
sync-file aborted: 3833b135caf4171daa0814eba81393b6c44db619: Move footer logos to /static/images/footer (T257732) (duration: 00m 04s) |
[production] |
| 18:50 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: c6a9674366d9c8d273ce0e74dfb6a04c91d64307: Move footer logos to wmg* variables (T257732) (duration: 00m 56s) |
[production] |
| 18:50 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 57s) |
[production] |
| 18:49 |
<urbanecm@deploy1001> |
sync-file aborted: (no justification provided) (duration: 00m 01s) |
[production] |
| 18:48 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: c6a9674366d9c8d273ce0e74dfb6a04c91d64307: Move footer logos to wmg* variables (T257732) (duration: 00m 57s) |
[production] |
| 18:29 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable desktop web UI click tracking instrumentation on frwiki, hewiki, fawiki (T258058) (duration: 00m 56s) |
[production] |
| 18:13 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove WPBSkinBlacklist (T254675) (duration: 00m 57s) |
[production] |
| 17:42 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.1 |
[production] |
| 17:30 |
<liw> |
promoting train to group2 |
[production] |
| 17:14 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 17:14 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 17:14 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@f14888b]: Deploying arclamp-compress-logs (T235456) (duration: 00m 05s) |
[production] |
| 17:14 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@f14888b]: Deploying arclamp-compress-logs (T235456) |
[production] |
| 16:59 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephmon1002.eqiad.wmnet |
[production] |
| 16:58 |
<andrew@cumin1001> |
conftool action : set/pooled=no; selector: name=cloudcephmon1002.eqiad.wmnet |
[production] |
| 16:57 |
<andrew@cumin1001> |
conftool action : set/pooled=inactive; selector: name=cloudcephmon1002.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=inactive; selector: name=cloudcephosd1003.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=inactive; selector: name=cloudcephosd1002.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=inactive; selector: name=cloudcephosd1001.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephmon1003.eqiad.wmnet |
[production] |
| 16:50 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephmon1002.eqiad.wmnet |
[production] |
| 16:49 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephmon1001.eqiad.wmnet |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=no; selector: name=cloudcephosd1003.wikimedia.org |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=no; selector: name=cloudcephosd1002.wikimedia.org |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=no; selector: name=cloudcephosd1001.wikimedia.org |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephosd1001.eqiad.wmnet |
[production] |
| 16:48 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephosd1002.eqiad.wmnet |
[production] |
| 16:47 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cloudcephosd1003.eqiad.wmnet |
[production] |
| 16:44 |
<andrew@cumin1001> |
conftool action : set/pooled=yes; selector: name=cumin1001.eqiad.wmnet |
[production] |
| 16:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2087:3316, db2087:3317 after on-site maintenance T258587', diff saved to https://phabricator.wikimedia.org/P12063 and previous config saved to /var/cache/conftool/dbconfig/20200727-163311-marostegui.json |
[production] |
| 16:05 |
<marostegui> |
Will show up on labsdb hosts for s5 |
[production] |
| 16:04 |
<marostegui> |
Stop MySQL on db1082 for onsite maintenance - T258910 |
[production] |
| 15:03 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:03 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:57 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:55 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Reduce db1146:3314 weight while db1144:3314 is depooled', diff saved to https://phabricator.wikimedia.org/P12060 and previous config saved to /var/cache/conftool/dbconfig/20200727-145010-marostegui.json |
[production] |
| 14:48 |
<marostegui> |
Deploy MCR change on db1144:3314 |
[production] |
| 14:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3314 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12059 and previous config saved to /var/cache/conftool/dbconfig/20200727-144807-marostegui.json |
[production] |
| 14:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1149', diff saved to https://phabricator.wikimedia.org/P12058 and previous config saved to /var/cache/conftool/dbconfig/20200727-144034-marostegui.json |
[production] |
| 14:21 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:19 |
<XioNoX> |
standardize cr1-codfw interfaces |
[production] |
| 14:19 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |