|
2020-04-29
§
|
| 09:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11079 and previous config saved to /var/cache/conftool/dbconfig/20200429-095545-marostegui.json |
[production] |
| 09:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3314', diff saved to https://phabricator.wikimedia.org/P11078 and previous config saved to /var/cache/conftool/dbconfig/20200429-095527-marostegui.json |
[production] |
| 09:38 |
<jbond42> |
puppet enabled fleetwide |
[production] |
| 09:30 |
<jbond42> |
disable puppet fleet wide for puppetdb upgrade |
[production] |
| 09:30 |
<jbond42> |
disable puppet for puppetdb upgrade |
[production] |
| 09:10 |
<vgutierrez> |
starting rolling restart of ats-tls to enable the TLS session ID based cache - T170567 |
[production] |
| 08:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 08:49 |
<mutante> |
gerrit1002 - gzipping gerrit.log.2020-04* files in /var/log/gerrit (T243808) |
[production] |
| 08:45 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 08:32 |
<vgutierrez> |
upgrade to ATS 8.1 on cp4032 - T249335 |
[production] |
| 08:31 |
<vgutierrez> |
restart ats-tls on cp[3054,3064] |
[production] |
| 08:07 |
<moritzm> |
installing openldap security updates |
[production] |
| 08:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11075 and previous config saved to /var/cache/conftool/dbconfig/20200429-080206-marostegui.json |
[production] |
| 07:55 |
<marostegui> |
Upgrade mysql on x1 master (without restarting) in preparation for tomorrow's upgrade - T250701 |
[production] |
| 07:54 |
<_joe_> |
restarting php-fpm on mw1288 (workers die in SIGILL status) |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11074 and previous config saved to /var/cache/conftool/dbconfig/20200429-073144-marostegui.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 and 3312 after reimage', diff saved to https://phabricator.wikimedia.org/P11073 and previous config saved to /var/cache/conftool/dbconfig/20200429-071431-marostegui.json |
[production] |
| 06:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:47 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:32 |
<marostegui> |
stop mysql on db1105 for reimage |
[production] |
| 06:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311 and 3312 for reimage', diff saved to https://phabricator.wikimedia.org/P11072 and previous config saved to /var/cache/conftool/dbconfig/20200429-062254-marostegui.json |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3314', diff saved to https://phabricator.wikimedia.org/P11071 and previous config saved to /var/cache/conftool/dbconfig/20200429-061941-marostegui.json |
[production] |
| 06:17 |
<vgutierrez> |
ats-tls restart on cp[3050,3058] - T249335 |
[production] |
| 06:07 |
<vgutierrez> |
ats-tls restart on cp3064 - T249335 |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1114', diff saved to https://phabricator.wikimedia.org/P11070 and previous config saved to /var/cache/conftool/dbconfig/20200429-054733-marostegui.json |
[production] |
| 00:56 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable Growth Study QuickSurvey on enwiki (with sample size 0, for testing) (T248421) (duration: 01m 10s) |
[production] |
| 00:43 |
<catrope@deploy1001> |
Finished scap: Update WikimediaMessages with new i18n messages for T248421 (duration: 55m 23s) |
[production] |
|
2020-04-28
§
|
| 23:48 |
<catrope@deploy1001> |
Started scap: Update WikimediaMessages with new i18n messages for T248421 |
[production] |
| 23:40 |
<ejegg> |
updated payments-wiki from 8c896a8247 to afb84cc391 |
[production] |
| 21:55 |
<ejegg> |
updated Payments IPN listener (Standalone SmashPig) from d80e4c5abd to 8c30ed7fe5 |
[production] |
| 20:57 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 's8 weights: -db1111, +db1099,db1101', diff saved to https://phabricator.wikimedia.org/P11069 and previous config saved to /var/cache/conftool/dbconfig/20200428-205739-cdanis.json |
[production] |
| 18:07 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 0639d9f: Allow bdwikimedia bureaucrats to revoke sysop flag (T251078) (duration: 01m 05s) |
[production] |
| 18:03 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 07c28d1: GrowthExperiments: cswiki: Change manual of style to 5 pillars (T251290) (duration: 01m 05s) |
[production] |
| 17:51 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.30/extensions/OAuth: T251306 (duration: 01m 06s) |
[production] |
| 17:13 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@678fb8e]: Update mobileapps to ff88022a (duration: 03m 23s) |
[production] |
| 17:10 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@678fb8e]: Update mobileapps to ff88022a |
[production] |
| 16:37 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.30/includes/Revision/RevisionStore.php: Follow-up If770120: Fix bad combination of type cast and ?? operator (duration: 01m 06s) |
[production] |
| 16:36 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@335122b]: Deploy design/style-guide: (duration: 00m 08s) |
[production] |
| 16:36 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@335122b]: Deploy design/style-guide: |
[production] |
| 15:49 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 15:42 |
<mepps> |
updated payments-wiki from 45bf1734e0 to 8c896a8247, |
[production] |
| 15:34 |
<vgutierrez> |
rolling restart of ats-tls on cp[3050,3052,3054,3056] - T249335 |
[production] |
| 15:28 |
<ppchelko@deploy1001> |
Finished deploy [changeprop/deploy@2b87a75]: Switch off rules moved to k8s T248677 (duration: 01m 20s) |
[production] |
| 15:27 |
<ppchelko@deploy1001> |
Started deploy [changeprop/deploy@2b87a75]: Switch off rules moved to k8s T248677 |
[production] |
| 15:15 |
<hnowlan@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 15:14 |
<hnowlan@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 15:13 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 15:13 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 15:13 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 14:59 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |