|
2020-04-29
ยง
|
| 18:18 |
<wm-bot> |
<taavi> updating to 12f5873 |
[tools.train-blockers] |
| 17:57 |
<joal> |
Deploy refinery on HDFS |
[analytics] |
| 17:56 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@6460d05] (thin): Regular analytics weekly train THIN [6460d05] (duration: 00m 08s) |
[production] |
| 17:56 |
<joal@deploy1001> |
Started deploy [analytics/refinery@6460d05] (thin): Regular analytics weekly train THIN [6460d05] |
[production] |
| 17:54 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) |
[production] |
| 17:45 |
<elukey> |
roll restart Presto workers to pick up the new jvm settings (110G heap size) |
[analytics] |
| 17:44 |
<elukey@cumin1001> |
START - Cookbook sre.presto.roll-restart-workers |
[production] |
| 17:24 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@6460d05]: Regular analytics weekly train [6460d05] (duration: 77m 08s) |
[production] |
| 16:55 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.30/extensions/MachineVision/src/Hooks.php: Fix hook handling for hook T251408 (duration: 01m 05s) |
[production] |
| 16:54 |
<bstorm_> |
deleted the maintain-kubeusers pod to start running the new image T247455 |
[tools] |
| 16:54 |
<jforrester@deploy1001> |
sync-file aborted: Fix hook handling for hook T251408 (duration: 00m 02s) |
[production] |
| 16:53 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.30/includes/EditPage.php: EditPage::showHeader - only warn editing an old revision if it exists T251404 (duration: 01m 06s) |
[production] |
| 16:52 |
<bstorm_> |
tagged docker-registry.tools.wmflabs.org/maintain-kubeusers:beta to latest to deploy to toolforge T247455 |
[tools] |
| 16:45 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 16:14 |
<James_F> |
Zuul: CI for wikimedia/meet-accountmanager T251425 |
[releng] |
| 16:07 |
<joal@deploy1001> |
Started deploy [analytics/refinery@6460d05]: Regular analytics weekly train [6460d05] |
[production] |
| 16:06 |
<joal> |
Deploying refinery using scap |
[analytics] |
| 16:05 |
<joal@deploy1001> |
Finished deploy [analytics/aqs/deploy@c87c8e2]: Analytics regular weekly deploy (duration: 06m 59s) |
[production] |
| 16:03 |
<James_F> |
Shut down integration-slave-jessie-1002 and integration-slave-jessie-1004 in Horizon for T236576 |
[releng] |
| 16:03 |
<James_F> |
Removed integration-slave-jessie-1002 and integration-slave-jessie-1004 from Jenkins for T236576 |
[releng] |
| 15:58 |
<joal@deploy1001> |
Started deploy [analytics/aqs/deploy@c87c8e2]: Analytics regular weekly deploy |
[production] |
| 15:57 |
<joal> |
Deploying AQS using scap |
[analytics] |
| 15:12 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Repooling db2087 in s6 and s7 after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11085 and previous config saved to /var/cache/conftool/dbconfig/20200429-151219-kormat.json |
[production] |
| 14:56 |
<sukhe> |
upload cescout 0.1.2-1 to apt.wm.o (buster) |
[production] |
| 14:49 |
<mdholloway> |
re-ran extension/MachineVision/maintenance/withholdImages.php on commonswiki |
[production] |
| 14:39 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: MachineVision: Update image withholding term list (duration: 01m 06s) |
[production] |
| 14:26 |
<elukey> |
enable TLS consumer/producers for kafka main -> jumbo mirror maker - T250250 |
[analytics] |
| 13:48 |
<joal> |
Releasing refinery 0.0.123 onto archiva with Jenkins |
[analytics] |
| 13:05 |
<liw@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.30 (duration: 01m 04s) |
[production] |
| 13:04 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.30 |
[production] |
| 12:50 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:47 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:26 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Depool db2087 for reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11081 and previous config saved to /var/cache/conftool/dbconfig/20200429-122602-kormat.json |
[production] |
| 11:58 |
<mutante> |
running puppet on cp-ats - switching backends of wikiworkshop.org |
[production] |
| 11:05 |
<hoo@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add new properties to wmgWBRepoPreferredPageImagesProperties (T249811) (duration: 01m 18s) |
[production] |
| 10:00 |
<kormat> |
reimaging db2087 to buster T250666 |
[production] |
| 09:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3314', diff saved to https://phabricator.wikimedia.org/P11080 and previous config saved to /var/cache/conftool/dbconfig/20200429-095629-marostegui.json |
[production] |
| 09:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11079 and previous config saved to /var/cache/conftool/dbconfig/20200429-095545-marostegui.json |
[production] |
| 09:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3314', diff saved to https://phabricator.wikimedia.org/P11078 and previous config saved to /var/cache/conftool/dbconfig/20200429-095527-marostegui.json |
[production] |
| 09:38 |
<jbond42> |
puppet enabled fleetwide |
[production] |
| 09:30 |
<jbond42> |
disable puppet fleet wide for puppetdb upgrade |
[production] |
| 09:30 |
<jbond42> |
disable puppet for puppetdb upgrade |
[production] |
| 09:22 |
<RhinosF1> |
Prod Bot to 210s timeout refs T250861 - rolling timeout to 300s |
[tools.zppixbot-test] |
| 09:10 |
<vgutierrez> |
starting rolling restart of ats-tls to enable the TLS session ID based cache - T170567 |
[production] |
| 08:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 08:49 |
<mutante> |
gerrit1002 - gzipping gerrit.log.2020-04* files in /var/log/gerrit (T243808) |
[production] |
| 08:47 |
<elukey> |
roll restart zookeeper on an-conf* to pick up new openjdk11 updates (affects hadoop) |
[analytics] |
| 08:45 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 08:32 |
<vgutierrez> |
upgrade to ATS 8.1 on cp4032 - T249335 |
[production] |
| 08:31 |
<vgutierrez> |
restart ats-tls on cp[3054,3064] |
[production] |