|
2020-02-17
§
|
| 15:50 |
<ayounsi@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:48 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 15:48 |
<ayounsi@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:44 |
<cdanis> |
✔️ cdanis@icinga1001.wikimedia.org ~ 🕥☕ sudo systemctl restart ircecho |
[production] |
| 14:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 after 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10422 and previous config saved to /var/cache/conftool/dbconfig/20200217-143146-marostegui.json |
[production] |
| 14:17 |
<ema> |
reprepro includedeb buster-wikimedia ~ema/cadvisor_0.35.0+ds1-4_amd64.deb T183146 |
[production] |
| 13:22 |
<arturo> |
relocating tools-sgewebgrid-lighttpd-0914 to cloudvirt1012 to spread same VMs across different hypervisors |
[tools] |
| 12:34 |
<hauskatze> |
extension-Teahouse@phab: Mirror deactivated # T176349 |
[releng] |
| 12:34 |
<hauskatze> |
extension-Teahouse@github: Mirror deleted # T176349 |
[releng] |
| 12:34 |
<XioNoX> |
add test flowspec rules to cr3-knams |
[production] |
| 12:34 |
<moritzm> |
installing postgresql-9.4 security updates |
[production] |
| 12:27 |
<vgutierrez> |
reboot acmechief instances (kernel upgrade) |
[production] |
| 10:31 |
<jynus> |
dropping all databases from db1140:3313 |
[production] |
| 10:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): ' db1107 increase API weight from 10 to 15 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10420 and previous config saved to /var/cache/conftool/dbconfig/20200217-102218-marostegui.json |
[production] |
| 10:20 |
<vgutierrez> |
rolling restart of ats-tls and varnish-fe on ulsfo to enable KA between them - T244464 |
[production] |
| 10:08 |
<arturo> |
restart webservice, because I see in the logs `uWSGI listen queue of socket ":8000" (fd: 7) full !!! (101/100)` |
[tools.copyvios] |
| 10:00 |
<moritzm> |
installing Linux 4.9.210 kernels on stretch systems |
[production] |
| 09:10 |
<godog> |
correction, +100G |
[production] |
| 09:09 |
<godog> |
+10G to prometheus/ops fs on prometheus eqiad - T245361 |
[production] |
| 09:06 |
<godog> |
+50G to prometheus/ops fs on prometheus eqiad - T245361 |
[production] |
| 07:22 |
<marostegui> |
Stop haproxy on dbproxy1002 - T245384 |
[production] |
|
2020-02-15
§
|
| 23:52 |
<legoktm> |
rebuilding tox docker image for https://gerrit.wikimedia.org/r/572453 |
[releng] |
| 22:21 |
<legoktm> |
reloading zuul for https://gerrit.wikimedia.org/r/572423 |
[releng] |
| 21:33 |
<hauskatze> |
GitHub: Activated Travis CI at https://travis-ci.org/wikimedia/texvcjs for https://github.com/wikimedia/texvcjs per https://github.com/wikimedia/texvcjs/pull/34 and T245344 |
[releng] |
| 20:25 |
<Krenair> |
T243226 Shut off deployment-puppetdb02 and deployment-puppetmaster03 - will leave for a week before deletion |
[releng] |
| 20:13 |
<bd808> |
Setup ingress objects |
[tools.iw] |
| 18:52 |
<bd808> |
Edited ingress/bash-legacy to redirect unconditionally to https://bash.toolforge.org and switched OAuth creds to matching grant |
[tools.bash] |
| 15:46 |
<andrewbogott> |
deleting project and remaining instance as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge |
[design] |
| 15:43 |
<andrewbogott> |
deleting traffic-cache-atsupload as per ema's comment in #wikimedia-sre |
[traffic] |
| 12:07 |
<elukey> |
re-run failed pageview druid hour |
[analytics] |
| 12:05 |
<elukey> |
re-run failed virtualpageview hours |
[analytics] |
| 01:01 |
<cdanis> |
✔️ cdanis@an-coord1001.eqiad.wmnet ~ 🕗🍺 sudo systemctl restart hive-server2.service ; sudo systemctl restart hive-metastore.service |
[production] |
|
2020-02-14
§
|
| 23:42 |
<XenoRyet> |
updated civicrm from cf86495d44 to 8c77e9e915 |
[production] |
| 22:02 |
<Krenair> |
Restarting webservice as everything was waiting and then returning 504 etc |
[tools.openstack-browser] |
| 21:53 |
<andrewbogott> |
moving deployment-puppetdb02 and deployment-puppetmaster03 off of cloudvirt1014 (which will be drained next week anyway) |
[releng] |
| 21:53 |
<andrewbogott> |
moving deployment-puppetdb02 and deployment-puppetmaster03 off of cloudvirt1014 (which will be drained next week anyway) |
[deployment-prep] |
| 21:31 |
<andrewbogott> |
restarting paws-puppetmaster-01 so its clients can connect |
[paws] |
| 21:01 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@1928c00]: Deploy design/style-guide: (duration: 00m 09s) |
[production] |
| 21:01 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@1928c00]: Deploy design/style-guide: |
[production] |
| 20:33 |
<brennen> |
Updating dev-images docker-pkg files on contint1001 to attempt a rebuild for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/571828 |
[releng] |
| 20:21 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Prevent some logspam T245280 (duration: 01m 05s) |
[production] |
| 20:18 |
<andrewbogott> |
adding profile::rsyslog::kafka_shipper::kafka_brokers: [] to project-wide puppet setting |
[ores] |
| 20:16 |
<andrewbogott> |
adding profile::ores::logstash_host: localhost to project-wide puppet config |
[ores] |
| 20:09 |
<andrewbogott> |
added profile::prometheus::statsd_exporter::mappings: [] setting to the thanks-be prefix to fix puppet runs |
[monitoring] |
| 20:05 |
<thcipriani> |
deployment-mwmaint01:rm lingering .~tmp~ directories from failed rsync's taking up space |
[releng] |
| 19:31 |
<thcipriani> |
deployment-snapshot01:sudo apt-get clean cleared 1.2G of space |
[releng] |