|
2019-02-14
§
|
| 23:47 |
<jynus> |
restarting labsdb1005 mysql in read only mode |
[production] |
| 23:37 |
<niharika29@deploy1001> |
Finished deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 (duration: 00m 02s) |
[production] |
| 23:37 |
<niharika29@deploy1001> |
Started deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 |
[production] |
| 22:25 |
<gtirloni> |
downtimed PAWS in Icinga |
[paws] |
| 22:16 |
<gtirloni> |
Activated maintenance page on paws-proxy-02 nginx config |
[paws] |
| 22:07 |
<andrewbogott> |
rebuilding labvirt1012 as cloudvirt1012, T216190 |
[production] |
| 21:57 |
<bd808> |
Deleted old tools-proxy-02 instance |
[tools] |
| 21:57 |
<bd808> |
Deleted old tools-proxy-01 instance |
[tools] |
| 21:56 |
<bd808> |
Deleted old tools-package-builder-01 instance |
[tools] |
| 21:28 |
<bd808> |
Updated config to point to tools-elastic-01 and restarted |
[tools.sal] |
| 20:57 |
<andrewbogott> |
rebooting tools-worker-1005 |
[tools] |
| 20:50 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528 / T216102 |
[releng] |
| 20:50 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528 / T133646 |
[releng] |
| 20:38 |
<bstorm_> |
Restarted mariadb on labsdb1005 for https://wikitech.wikimedia.org/wiki/Incident_documentation/20190214-labsdb1005 |
[production] |
| 20:34 |
<andrewbogott> |
moving tools-exec-1409, tools-exec-1410, tools-exec-1414, tools-exec-1419 |
[tools] |
| 20:18 |
<thcipriani> |
thcipriani@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.17 |
[production] |
| 20:14 |
<thcipriani@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.17 |
[production] |
| 20:09 |
<ejegg> |
updated fundraising CiviCRM from 02ea871b88 to 165fbf5894 |
[production] |
| 19:55 |
<andrewbogott> |
moving tools-webgrid-generic-1401 and tools-webgrid-lighttpd-1419 |
[tools] |
| 19:42 |
<thcipriani@deploy1001> |
Synchronized php-1.33.0-wmf.17/extensions/GrowthExperiments/modules/help: SWAT: [[gerrit:490674|Help Panel: Fix IME broken in help panel search]] T216131 (duration: 00m 54s) |
[production] |
| 19:33 |
<andrewbogott> |
moving tools-checker-01 to labvirt1003 |
[tools] |
| 19:25 |
<andrewbogott> |
moving tools-elastic-02 to labvirt1003 |
[tools] |
| 19:14 |
<thcipriani@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:487007|Stop NavPopups gadget conflict with PagePreviews on Wikivoyage]] T214878 (duration: 00m 54s) |
[production] |
| 19:11 |
<andrewbogott> |
moving tools-k8s-etcd-01 to labvirt1002 |
[tools] |
| 19:01 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev |
[production] |
| 18:58 |
<bd808> |
Stopped webservice. Implicated in ToolsDB connection overload outage. |
[tools.fountain] |
| 18:52 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev |
[production] |
| 18:37 |
<andrewbogott> |
moving tools-exec-1418, tools-exec-1424 to labvirt1003 |
[tools] |
| 18:34 |
<andrewbogott> |
moving tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1406, tools-webgrid-lighttpd-1410 to labvirt1002 |
[tools] |
| 18:34 |
<bd808> |
bd808 disabled all cron jobs by commenting them out in the Stretch grid crontab while debugging ToolsDB connection overload |
[tools.checkwiki] |
| 18:30 |
<andrewbogott> |
moving toolsbeta-puppetdb-01 to labvirt1002 |
[toolsbeta] |
| 18:26 |
<bstorm_> |
stopped update_dumps job in case that was the cause of the DB issue |
[tools.checkwiki] |
| 18:24 |
<bstorm_> |
stopping service to see if that fixes the DB |
[tools.checkwiki] |
| 18:12 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet |
[production] |
| 18:03 |
<fsero> |
upgrading tiller to 2.12.2 on eqiad |
[production] |
| 17:35 |
<arturo> |
T215154 tools-sgebastion-07 now running systemd 239 and starts enforcing user limits |
[tools] |
| 17:34 |
<godog> |
bounce rsyslog on wezen/lithium, tls listener timeout in icinga |
[production] |
| 16:59 |
<moritzm> |
restarting apertium-apy on scb1001 to pick up Python security update |
[production] |
| 16:39 |
<marostegui> |
Depool labsdb1009 - T210713 |
[production] |
| 16:26 |
<fsero> |
upgrading tiller on codfw |
[production] |
| 16:11 |
<fsero> |
updating tiller version on staging cluster |
[production] |
| 16:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2085 - T214840 (duration: 00m 52s) |
[production] |
| 15:50 |
<fsero> |
building and publishing new tiller docker image on boron |
[production] |
| 15:50 |
<END> |
(PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) (volans@cumin1001) |
[production] |
| 15:43 |
<START> |
- Cookbook sre.hosts.upgrade-and-reboot (volans@cumin1001) |
[production] |
| 15:33 |
<andrewbogott> |
moving tools-worker-1002, 1003, 1005, 1006, 1007, 1010, 1013, 1014 to different labvirts in order to move labvirt1012 to eqiad1-r |
[tools] |
| 15:28 |
<volans> |
upgraded spicerack to v0.0.15 on cumin[12]001 |
[production] |
| 15:26 |
<volans> |
uploaded spicerack_0.0.15-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 15:12 |
<marostegui> |
Clear idrac logs from db2085 - T214840 |
[production] |