|
2019-11-20
§
|
| 07:19 |
<marostegui> |
Upgrade db2078 |
[production] |
| 07:14 |
<marostegui> |
Deploy schema change on s3 (testwikidatawiki) directly on s3 primary master T237120 |
[production] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1136', diff saved to https://phabricator.wikimedia.org/P9688 and previous config saved to /var/cache/conftool/dbconfig/20191120-070511-marostegui.json |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give more traffic to db1136', diff saved to https://phabricator.wikimedia.org/P9687 and previous config saved to /var/cache/conftool/dbconfig/20191120-065718-marostegui.json |
[production] |
| 06:44 |
<marostegui> |
Upgrade db2118 (s7 codfw master) |
[production] |
| 06:41 |
<marostegui> |
Repool labsdb1011 |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1136 into s7 api', diff saved to https://phabricator.wikimedia.org/P9686 and previous config saved to /var/cache/conftool/dbconfig/20191120-064022-marostegui.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1136 after upgrade', diff saved to https://phabricator.wikimedia.org/P9685 and previous config saved to /var/cache/conftool/dbconfig/20191120-063628-marostegui.json |
[production] |
| 06:28 |
<marostegui> |
Upgrade db1136 |
[production] |
| 06:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136 for upgrade', diff saved to https://phabricator.wikimedia.org/P9684 and previous config saved to /var/cache/conftool/dbconfig/20191120-062749-marostegui.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317 after upgrade', diff saved to https://phabricator.wikimedia.org/P9683 and previous config saved to /var/cache/conftool/dbconfig/20191120-062029-marostegui.json |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3314 for compression', diff saved to https://phabricator.wikimedia.org/P9682 and previous config saved to /var/cache/conftool/dbconfig/20191120-061938-marostegui.json |
[production] |
| 05:58 |
<marostegui> |
Stop MySQL on db1101:3317, db1101:3318 for upgrade and schema change |
[production] |
| 05:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317 and db1101:3318 for upgrade and schema change', diff saved to https://phabricator.wikimedia.org/P9681 and previous config saved to /var/cache/conftool/dbconfig/20191120-055732-marostegui.json |
[production] |
| 05:55 |
<marostegui> |
Depool labsdb1011 for upgrade |
[production] |
| 05:54 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Repool db1105:3311 db1097:3314 db1098:3316 db1098:3317 after compression', diff saved to https://phabricator.wikimedia.org/P9680 and previous config saved to /var/cache/conftool/dbconfig/20191120-055426-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092 after schema change', diff saved to https://phabricator.wikimedia.org/P9679 and previous config saved to /var/cache/conftool/dbconfig/20191120-054840-marostegui.json |
[production] |
| 03:16 |
<tgr> |
T208369 ran mwscript extensions/GrowthExperiments/maintenance/deleteOldSurveys.php kowiki --cutoff 350 |
[production] |
| 02:57 |
<vgutierrez> |
restarting pybal on lvs2002 |
[production] |
| 02:54 |
<vgutierrez> |
restarting pybal on lvs2005 |
[production] |
| 02:32 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 02:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 00:10 |
<mutante> |
phab2001 - restart ssh-phab service after repooling it after buster reinstall, it wasn't listening on the IPv6 IP,causing LVS/pybal alerts |
[production] |
| 00:06 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/GrowthExperiments/: Pass token as editing_session_id for suggested edits (T238249) (duration: 00m 53s) |
[production] |
| 00:02 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/VisualEditor/: EditAttemptStep: Allow overriding session ID (T238249) (duration: 00m 52s) |
[production] |
| 00:00 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/WikiEditor/: EditAttemptStep: Allow overriding session ID (T238249) (duration: 00m 54s) |
[production] |
|
2019-11-19
§
|
| 23:58 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/MobileFrontend/: EditAttemptStep: Allow overriding session ID (T238249) (duration: 00m 53s) |
[production] |
| 23:55 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/WikimediaEvents/: EditAttemptStep: Allow other extensions to trigger oversampling (T238249) (duration: 00m 53s) |
[production] |
| 23:42 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 21:45 |
<XioNoX> |
rebooting pfw3-codfw:node1 for upgrade - T235150 |
[production] |
| 21:14 |
<XioNoX> |
rebooting pfw3-codfw for upgrade - T235150 |
[production] |
| 20:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 20:17 |
<gehel> |
completed reloading data from wdqs1007 to wdqs1004 - after failed test of merging updater - T212826 |
[production] |
| 20:14 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 20:10 |
<XioNoX> |
homer push on mgmt routers |
[production] |
| 20:09 |
<mutante> |
phab1003 after merging gerrit:551910 puppet now also stopped the actual aphlict service and removed the systemd unit file. had to manually run 'systemctl reset-failed' though to clean systemd status and avoid icinga alert (T238593) |
[production] |
| 20:07 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 19:20 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 19:18 |
<dzahn@cumin1001> |
END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) |
[production] |
| 19:18 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 19:08 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@6e6bd42]: Prevent expensive content transforms from blocking the event loop (T229286) (duration: 06m 49s) |
[production] |
| 19:01 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@6e6bd42]: Prevent expensive content transforms from blocking the event loop (T229286) |
[production] |
| 19:00 |
<elukey> |
regenerate TLS cert for yarn.wikimedia.org (containing SANs for all analytics UIs) to add datasets.w.o SAN (site was failing due to ATS not being able to contact thorium) |
[production] |
| 18:59 |
<rlazarus> |
restarted php7.2-fpm on wtp2001, wtp2002 |
[production] |
| 18:56 |
<rlazarus> |
restarted php7.2-fpm on wtp1025, wtp1026 |
[production] |
| 18:35 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/VisualEditor/: Unbreak instrumentation of init events (duration: 00m 53s) |
[production] |
| 18:34 |
<ssastry@deploy1001> |
Finished deploy [parsoid/deploy@6e7cffd]: Updating Parsoid to 1a1105a7 (duration: 02m 04s) |
[production] |
| 18:32 |
<ssastry@deploy1001> |
Started deploy [parsoid/deploy@6e7cffd]: Updating Parsoid to 1a1105a7 |
[production] |
| 18:30 |
<mutante> |
icinga config - manually added team-dcops, started icinga |
[production] |
| 18:20 |
<addshore@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikidata.org: T221774 - Wikidata.org extension (queryservice maxlag, hook) [[gerrit:551858]] (duration: 00m 53s) |
[production] |