|
2018-05-01
§
|
| 19:27 |
<ottomata> |
rolling restart of eventbus to apply logstash tag https://phabricator.wikimedia.org/T193230 |
[production] |
| 19:17 |
<mutante> |
mw2174,mw2175,mw2176 ff - reinstalling with wmf-auto-reimage to stretch |
[production] |
| 19:00 |
<ottomata> |
rolling restart of eventlogging-service-eventbus to apply logstash logging configs - T193230 |
[production] |
| 18:53 |
<ariel@tin> |
Finished deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone (duration: 00m 04s) |
[production] |
| 18:53 |
<ariel@tin> |
Started deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone |
[production] |
| 17:40 |
<XioNoX> |
disabling eqsin<->codfw link for high packet loss on link |
[production] |
| 17:39 |
<otto@tin> |
Finished deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230 (duration: 02m 29s) |
[production] |
| 17:36 |
<otto@tin> |
Started deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230 |
[production] |
| 16:15 |
<ebernhardson> |
T192972 change eqiad elasticsearch disk watermarks from 85/85 to 80/80 to match disk space alerts |
[production] |
| 16:15 |
<herron> |
manually kicked off mirror@sodium:~$ /usr/local/sbin/update-ubuntu-mirror to clear ubuntu mirror out of sync alert |
[production] |
| 15:50 |
<urandom> |
restbase: begin culling leaked revisions, others_T_mobile__ng_lead -- T192689 |
[production] |
| 15:37 |
<ejegg> |
updated SmashPig standalone from a4de12d415 to 99585c8084 |
[production] |
| 13:35 |
<herron> |
restarted hhvm on mw1233 |
[production] |
| 02:47 |
<l10nupdate@tin> |
scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 04s) |
[production] |
| 01:20 |
<ejegg> |
re-enabled fundraising jobs |
[production] |
| 01:01 |
<ejegg> |
updated CiviCRM from 47197006d5 to 46883844a3 |
[production] |
| 00:51 |
<eileen> |
email sent advising of minor interuption |
[production] |
| 00:50 |
<ejegg> |
disabled fundraising jobs for db update |
[production] |
| 00:21 |
<ebernhardson> |
increase cluster.routing.allocation.balance.threshold from 1.0 to 1.5 for eqiad elasticsearch cluster to reduce rebalancing agressiveness |
[production] |
|
2018-04-30
§
|
| 23:38 |
<catrope@tin> |
Synchronized php-1.32.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.DiffPage.init.js: T192755 (duration: 00m 59s) |
[production] |
| 23:36 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Set $wgKartographerUsePageLanguage to false everywhere (T192955) (duration: 00m 59s) |
[production] |
| 23:33 |
<catrope@tin> |
Synchronized php-1.32.0-wmf.1/extensions/AbuseFilter/includes/AbuseFilter.php: Fix notices when disallowing edits (duration: 00m 59s) |
[production] |
| 23:21 |
<catrope@tin> |
Synchronized wmf-config/: USe internal cluster for SPARQL services (T192942) (duration: 01m 02s) |
[production] |
| 23:14 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Config cleanup patches from SWAT (duration: 01m 00s) |
[production] |
| 23:05 |
<mutante> |
ores1001: rm -rf /srv/deployment/ores/venv/ (T193422) |
[production] |
| 21:46 |
<ebernhardson> |
T192972 increase eqiad elasticsearch disk watermarks from 75/80 to 85/85 |
[production] |
| 20:27 |
<arlolra> |
Updated Parsoid to 50b0588 (T186358, T191700, T192909) |
[production] |
| 20:22 |
<awight@tin> |
Started deploy [ores/deploy@bf182e2]: ORES: Include bot edits in precaching wikidata itemquality; T187927 |
[production] |
| 20:21 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@d8d7b42]: Updating Parsoid to 50b0588 (duration: 09m 46s) |
[production] |
| 20:20 |
<awight@tin> |
Finished deploy [ores/deploy@5b27205]: Rollback ores1001 to master (duration: 02m 56s) |
[production] |
| 20:19 |
<bsitzmann@tin> |
Finished deploy [mobileapps/deploy@d3724d2]: Update mobileapps to cc00cae (T191869) (duration: 07m 32s) |
[production] |
| 20:17 |
<awight@tin> |
Started deploy [ores/deploy@5b27205]: Rollback ores1001 to master |
[production] |
| 20:11 |
<bsitzmann@tin> |
Started deploy [mobileapps/deploy@d3724d2]: Update mobileapps to cc00cae (T191869) |
[production] |
| 20:11 |
<arlolra@tin> |
Started deploy [parsoid/deploy@d8d7b42]: Updating Parsoid to 50b0588 |
[production] |
| 20:09 |
<awight@tin> |
Finished deploy [ores/deploy@4601497]: Trial LFS deployment to ORES canary; T181678 (take 2) (duration: 02m 10s) |
[production] |
| 20:06 |
<awight@tin> |
Started deploy [ores/deploy@4601497]: Trial LFS deployment to ORES canary; T181678 (take 2) |
[production] |
| 20:06 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@8cd45ed]: Don't filter bots from the ORES stream T187927 (duration: 01m 15s) |
[production] |
| 20:05 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@8cd45ed]: Don't filter bots from the ORES stream T187927 |
[production] |
| 19:10 |
<awight@tin> |
Finished deploy [ores/deploy@25579e7]: Trial LFS deployment to ORES canary; T181678 (duration: 02m 06s) |
[production] |
| 19:08 |
<awight@tin> |
Started deploy [ores/deploy@25579e7]: Trial LFS deployment to ORES canary; T181678 |
[production] |
| 19:01 |
<mutante> |
hafnium - sudo service navtiming stop; sudo service statsv stop - downtimed in icinga, decom |
[production] |
| 18:27 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: T191584 (duration: 01m 00s) |
[production] |
| 18:23 |
<awight@tin> |
Finished deploy [ores/deploy@5b27205]: Rollback ORES canary to master (duration: 00m 21s) |
[production] |
| 18:22 |
<awight@tin> |
Started deploy [ores/deploy@5b27205]: Rollback ORES canary to master |
[production] |
| 18:17 |
<catrope@tin> |
Synchronized php-1.32.0-wmf.1/extensions/CodeMirror/resources/modules/ve-cm/ve.ui.CodeMirrorAction.js: T191923 (duration: 01m 00s) |
[production] |
| 18:16 |
<ottomata> |
starting rolling reimage of kafka main-eqiad brokers kafka100[123] - T192832 |
[production] |
| 18:06 |
<awight@tin> |
Finished deploy [ores/deploy@46824bb]: Canary-only test deployment for ORES + git-lfs, T181678 (take 2) (duration: 01m 58s) |
[production] |
| 18:04 |
<awight@tin> |
Started deploy [ores/deploy@46824bb]: Canary-only test deployment for ORES + git-lfs, T181678 (take 2) |
[production] |
| 17:41 |
<awight@tin> |
Finished deploy [ores/deploy@8c586ab]: Canary-only test deployment for ORES + git-lfs, T181678 (duration: 01m 59s) |
[production] |
| 17:39 |
<ariel@tin> |
Finished deploy [dumps/dumps@8398f53]: write checksums of dump files into seperate hashfiles, reusing their contents as appropriate (duration: 00m 03s) |
[production] |