|
2019-11-20
§
|
| 05:58 |
<marostegui> |
Stop MySQL on db1101:3317, db1101:3318 for upgrade and schema change |
[production] |
| 05:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317 and db1101:3318 for upgrade and schema change', diff saved to https://phabricator.wikimedia.org/P9681 and previous config saved to /var/cache/conftool/dbconfig/20191120-055732-marostegui.json |
[production] |
| 05:55 |
<marostegui> |
Depool labsdb1011 for upgrade |
[production] |
| 05:54 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Repool db1105:3311 db1097:3314 db1098:3316 db1098:3317 after compression', diff saved to https://phabricator.wikimedia.org/P9680 and previous config saved to /var/cache/conftool/dbconfig/20191120-055426-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092 after schema change', diff saved to https://phabricator.wikimedia.org/P9679 and previous config saved to /var/cache/conftool/dbconfig/20191120-054840-marostegui.json |
[production] |
| 03:16 |
<tgr> |
T208369 ran mwscript extensions/GrowthExperiments/maintenance/deleteOldSurveys.php kowiki --cutoff 350 |
[production] |
| 02:57 |
<vgutierrez> |
restarting pybal on lvs2002 |
[production] |
| 02:54 |
<vgutierrez> |
restarting pybal on lvs2005 |
[production] |
| 02:32 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 02:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 00:10 |
<mutante> |
phab2001 - restart ssh-phab service after repooling it after buster reinstall, it wasn't listening on the IPv6 IP,causing LVS/pybal alerts |
[production] |
| 00:06 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/GrowthExperiments/: Pass token as editing_session_id for suggested edits (T238249) (duration: 00m 53s) |
[production] |
| 00:02 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/VisualEditor/: EditAttemptStep: Allow overriding session ID (T238249) (duration: 00m 52s) |
[production] |
| 00:00 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/WikiEditor/: EditAttemptStep: Allow overriding session ID (T238249) (duration: 00m 54s) |
[production] |
|
2019-11-19
§
|
| 23:58 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/MobileFrontend/: EditAttemptStep: Allow overriding session ID (T238249) (duration: 00m 53s) |
[production] |
| 23:55 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/WikimediaEvents/: EditAttemptStep: Allow other extensions to trigger oversampling (T238249) (duration: 00m 53s) |
[production] |
| 23:42 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet |
[production] |
| 21:45 |
<XioNoX> |
rebooting pfw3-codfw:node1 for upgrade - T235150 |
[production] |
| 21:14 |
<XioNoX> |
rebooting pfw3-codfw for upgrade - T235150 |
[production] |
| 20:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 20:17 |
<gehel> |
completed reloading data from wdqs1007 to wdqs1004 - after failed test of merging updater - T212826 |
[production] |
| 20:14 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 20:10 |
<XioNoX> |
homer push on mgmt routers |
[production] |
| 20:09 |
<mutante> |
phab1003 after merging gerrit:551910 puppet now also stopped the actual aphlict service and removed the systemd unit file. had to manually run 'systemctl reset-failed' though to clean systemd status and avoid icinga alert (T238593) |
[production] |
| 20:07 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 19:20 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 19:18 |
<dzahn@cumin1001> |
END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) |
[production] |
| 19:18 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 19:08 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@6e6bd42]: Prevent expensive content transforms from blocking the event loop (T229286) (duration: 06m 49s) |
[production] |
| 19:01 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@6e6bd42]: Prevent expensive content transforms from blocking the event loop (T229286) |
[production] |
| 19:00 |
<elukey> |
regenerate TLS cert for yarn.wikimedia.org (containing SANs for all analytics UIs) to add datasets.w.o SAN (site was failing due to ATS not being able to contact thorium) |
[production] |
| 18:59 |
<rlazarus> |
restarted php7.2-fpm on wtp2001, wtp2002 |
[production] |
| 18:56 |
<rlazarus> |
restarted php7.2-fpm on wtp1025, wtp1026 |
[production] |
| 18:35 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/VisualEditor/: Unbreak instrumentation of init events (duration: 00m 53s) |
[production] |
| 18:34 |
<ssastry@deploy1001> |
Finished deploy [parsoid/deploy@6e7cffd]: Updating Parsoid to 1a1105a7 (duration: 02m 04s) |
[production] |
| 18:32 |
<ssastry@deploy1001> |
Started deploy [parsoid/deploy@6e7cffd]: Updating Parsoid to 1a1105a7 |
[production] |
| 18:30 |
<mutante> |
icinga config - manually added team-dcops, started icinga |
[production] |
| 18:20 |
<addshore@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikidata.org: T221774 - Wikidata.org extension (queryservice maxlag, hook) [[gerrit:551858]] (duration: 00m 53s) |
[production] |
| 18:12 |
<RoanKattouw> |
That was eowiktionary, not eowikisource |
[production] |
| 18:09 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Configure default search namespaces for eowikisource (T237792) (duration: 00m 52s) |
[production] |
| 17:43 |
<addshore@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikidata.org: T221774 - Wikidata.org extension (queryservice maxlag, maint script) [[gerrit:551857]] (duration: 00m 52s) |
[production] |
| 17:39 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 17:11 |
<addshore@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikidata.org: T221774 - Wikidata.org extension (queryservice maxlag) [[gerrit:551855]] [[gerrit:551856]] (duration: 00m 54s) |
[production] |
| 17:02 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@d73818a]: Deploy design/style-guide: (duration: 00m 07s) |
[production] |
| 17:02 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@d73818a]: Deploy design/style-guide: |
[production] |
| 16:58 |
<ema> |
pool cp2007 with ATS backend T227432 |
[production] |
| 16:30 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:28 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:25 |
<moritzm> |
installing glib2.0 security updates |
[production] |
| 16:21 |
<mutante> |
phab1003 - puppet restarts aphlict service even with "phabricator_aphlict_enabled: false" in Hiera. But it does properly remove the proxy config lines from apache. so service is running but not used. (T238593) |
[production] |