|
2020-04-29
§
|
| 17:44 |
<elukey@cumin1001> |
START - Cookbook sre.presto.roll-restart-workers |
[production] |
| 17:24 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@6460d05]: Regular analytics weekly train [6460d05] (duration: 77m 08s) |
[production] |
| 16:55 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.30/extensions/MachineVision/src/Hooks.php: Fix hook handling for hook T251408 (duration: 01m 05s) |
[production] |
| 16:54 |
<jforrester@deploy1001> |
sync-file aborted: Fix hook handling for hook T251408 (duration: 00m 02s) |
[production] |
| 16:53 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.30/includes/EditPage.php: EditPage::showHeader - only warn editing an old revision if it exists T251404 (duration: 01m 06s) |
[production] |
| 16:45 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 16:07 |
<joal@deploy1001> |
Started deploy [analytics/refinery@6460d05]: Regular analytics weekly train [6460d05] |
[production] |
| 16:05 |
<joal@deploy1001> |
Finished deploy [analytics/aqs/deploy@c87c8e2]: Analytics regular weekly deploy (duration: 06m 59s) |
[production] |
| 15:58 |
<joal@deploy1001> |
Started deploy [analytics/aqs/deploy@c87c8e2]: Analytics regular weekly deploy |
[production] |
| 15:12 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Repooling db2087 in s6 and s7 after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11085 and previous config saved to /var/cache/conftool/dbconfig/20200429-151219-kormat.json |
[production] |
| 14:56 |
<sukhe> |
upload cescout 0.1.2-1 to apt.wm.o (buster) |
[production] |
| 14:49 |
<mdholloway> |
re-ran extension/MachineVision/maintenance/withholdImages.php on commonswiki |
[production] |
| 14:39 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: MachineVision: Update image withholding term list (duration: 01m 06s) |
[production] |
| 13:05 |
<liw@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.30 (duration: 01m 04s) |
[production] |
| 13:04 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.30 |
[production] |
| 12:50 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:47 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:26 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Depool db2087 for reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11081 and previous config saved to /var/cache/conftool/dbconfig/20200429-122602-kormat.json |
[production] |
| 11:58 |
<mutante> |
running puppet on cp-ats - switching backends of wikiworkshop.org |
[production] |
| 11:05 |
<hoo@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add new properties to wmgWBRepoPreferredPageImagesProperties (T249811) (duration: 01m 18s) |
[production] |
| 10:00 |
<kormat> |
reimaging db2087 to buster T250666 |
[production] |
| 09:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3314', diff saved to https://phabricator.wikimedia.org/P11080 and previous config saved to /var/cache/conftool/dbconfig/20200429-095629-marostegui.json |
[production] |
| 09:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11079 and previous config saved to /var/cache/conftool/dbconfig/20200429-095545-marostegui.json |
[production] |
| 09:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3314', diff saved to https://phabricator.wikimedia.org/P11078 and previous config saved to /var/cache/conftool/dbconfig/20200429-095527-marostegui.json |
[production] |
| 09:38 |
<jbond42> |
puppet enabled fleetwide |
[production] |
| 09:30 |
<jbond42> |
disable puppet fleet wide for puppetdb upgrade |
[production] |
| 09:30 |
<jbond42> |
disable puppet for puppetdb upgrade |
[production] |
| 09:10 |
<vgutierrez> |
starting rolling restart of ats-tls to enable the TLS session ID based cache - T170567 |
[production] |
| 08:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 08:49 |
<mutante> |
gerrit1002 - gzipping gerrit.log.2020-04* files in /var/log/gerrit (T243808) |
[production] |
| 08:45 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 08:32 |
<vgutierrez> |
upgrade to ATS 8.1 on cp4032 - T249335 |
[production] |
| 08:31 |
<vgutierrez> |
restart ats-tls on cp[3054,3064] |
[production] |
| 08:07 |
<moritzm> |
installing openldap security updates |
[production] |
| 08:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11075 and previous config saved to /var/cache/conftool/dbconfig/20200429-080206-marostegui.json |
[production] |
| 07:55 |
<marostegui> |
Upgrade mysql on x1 master (without restarting) in preparation for tomorrow's upgrade - T250701 |
[production] |
| 07:54 |
<_joe_> |
restarting php-fpm on mw1288 (workers die in SIGILL status) |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11074 and previous config saved to /var/cache/conftool/dbconfig/20200429-073144-marostegui.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11073 and previous config saved to /var/cache/conftool/dbconfig/20200429-071431-marostegui.json |
[production] |
| 06:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:47 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:32 |
<marostegui> |
stop mysql on db1105 for reimage |
[production] |
| 06:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311 and 3312 for reimage', diff saved to https://phabricator.wikimedia.org/P11072 and previous config saved to /var/cache/conftool/dbconfig/20200429-062254-marostegui.json |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3314', diff saved to https://phabricator.wikimedia.org/P11071 and previous config saved to /var/cache/conftool/dbconfig/20200429-061941-marostegui.json |
[production] |
| 06:17 |
<vgutierrez> |
ats-tls restart on cp[3050,3058] - T249335 |
[production] |
| 06:07 |
<vgutierrez> |
ats-tls restart on cp3064 - T249335 |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1114', diff saved to https://phabricator.wikimedia.org/P11070 and previous config saved to /var/cache/conftool/dbconfig/20200429-054733-marostegui.json |
[production] |
| 00:56 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable Growth Study QuickSurvey on enwiki (with sample size 0, for testing) (T248421) (duration: 01m 10s) |
[production] |
| 00:43 |
<catrope@deploy1001> |
Finished scap: Update WikimediaMessages with new i18n messages for T248421 (duration: 55m 23s) |
[production] |