|
2019-02-14
ยง
|
| 19:33 |
<andrewbogott> |
moving tools-checker-01 to labvirt1003 |
[tools] |
| 19:25 |
<andrewbogott> |
moving tools-elastic-02 to labvirt1003 |
[tools] |
| 19:14 |
<thcipriani@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:487007|Stop NavPopups gadget conflict with PagePreviews on Wikivoyage]] T214878 (duration: 00m 54s) |
[production] |
| 19:11 |
<andrewbogott> |
moving tools-k8s-etcd-01 to labvirt1002 |
[tools] |
| 19:01 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev |
[production] |
| 18:58 |
<bd808> |
Stopped webservice. Implicated in ToolsDB connection overload outage. |
[tools.fountain] |
| 18:52 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev |
[production] |
| 18:37 |
<andrewbogott> |
moving tools-exec-1418, tools-exec-1424 to labvirt1003 |
[tools] |
| 18:34 |
<andrewbogott> |
moving tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1406, tools-webgrid-lighttpd-1410 to labvirt1002 |
[tools] |
| 18:34 |
<bd808> |
bd808 disabled all cron jobs by commenting them out in the Stretch grid crontab while debugging ToolsDB connection overload |
[tools.checkwiki] |
| 18:30 |
<andrewbogott> |
moving toolsbeta-puppetdb-01 to labvirt1002 |
[toolsbeta] |
| 18:26 |
<bstorm_> |
stopped update_dumps job in case that was the cause of the DB issue |
[tools.checkwiki] |
| 18:24 |
<bstorm_> |
stopping service to see if that fixes the DB |
[tools.checkwiki] |
| 18:12 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet |
[production] |
| 18:03 |
<fsero> |
upgrading tiller to 2.12.2 on eqiad |
[production] |
| 17:35 |
<arturo> |
T215154 tools-sgebastion-07 now running systemd 239 and starts enforcing user limits |
[tools] |
| 17:34 |
<godog> |
bounce rsyslog on wezen/lithium, tls listener timeout in icinga |
[production] |
| 16:59 |
<moritzm> |
restarting apertium-apy on scb1001 to pick up Python security update |
[production] |
| 16:39 |
<marostegui> |
Depool labsdb1009 - T210713 |
[production] |
| 16:26 |
<fsero> |
upgrading tiller on codfw |
[production] |
| 16:11 |
<fsero> |
updating tiller version on staging cluster |
[production] |
| 16:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2085 - T214840 (duration: 00m 52s) |
[production] |
| 15:50 |
<fsero> |
building and publishing new tiller docker image on boron |
[production] |
| 15:50 |
<END> |
(PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) (volans@cumin1001) |
[production] |
| 15:43 |
<START> |
- Cookbook sre.hosts.upgrade-and-reboot (volans@cumin1001) |
[production] |
| 15:33 |
<andrewbogott> |
moving tools-worker-1002, 1003, 1005, 1006, 1007, 1010, 1013, 1014 to different labvirts in order to move labvirt1012 to eqiad1-r |
[tools] |
| 15:28 |
<volans> |
upgraded spicerack to v0.0.15 on cumin[12]001 |
[production] |
| 15:26 |
<volans> |
uploaded spicerack_0.0.15-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 15:12 |
<marostegui> |
Clear idrac logs from db2085 - T214840 |
[production] |
| 14:45 |
<godog> |
depool and stop logstash1009 for stretch reimage - T213898 |
[production] |
| 14:20 |
<marostegui> |
Stop MySQL on db2085 for on-site maintenance - T214840 |
[production] |
| 14:12 |
<jijiki> |
Enabling puppet on thumbor* servers - T214597 |
[production] |
| 13:39 |
<arturo> |
T215892 icinga downtime cloudvirt1024 for 2 weeks |
[production] |
| 13:24 |
<thcipriani> |
rearm keyholder on deployment-deploy01: sudo keyholder arm, passwords on https://wikitech.wikimedia.org/wiki/Keyholder |
[releng] |
| 12:22 |
<zeljkof> |
EU SWAT finished |
[production] |
| 12:21 |
<zfilipin@deploy1001> |
Synchronized php-1.33.0-wmf.17/extensions/ExternalGuidance/: SWAT: [[gerrit:490523|Fix the eventlogging schema definition as per manifest_version=2]] (duration: 00m 55s) |
[production] |
| 11:43 |
<_joe_> |
restarting hhvm on mw1338, hot tc exhausted T216084 |
[production] |
| 11:04 |
<_joe_> |
upgrading python3-etcd on stretch T209136 |
[production] |
| 11:03 |
<jbond42> |
rolling security updates for curl |
[production] |
| 11:02 |
<jijiki> |
Disabling puppet on thumbor* servers - T214597 |
[production] |
| 10:59 |
<moritzm> |
installing python3.4 security updates |
[production] |
| 10:53 |
<godog> |
bounce prometheus instances on prometheus2004 to take a snapshot |
[production] |
| 09:07 |
<joal> |
rerun mediawiki-history-wikitext-wf-2019-01 |
[analytics] |
| 09:06 |
<joal> |
Re-run webrequest-load-wf-text-2019-2-14-6 |
[analytics] |
| 08:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1106 T214840 (duration: 00m 52s) |
[production] |
| 07:57 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s) |
[production] |
| 07:36 |
<marostegui> |
Stop MySQL on db1106 for reboot - T214840 |
[production] |
| 06:10 |
<marostegui> |
Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713 |
[production] |
| 06:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1087 T210713 (duration: 00m 55s) |
[production] |
| 01:52 |
<mutante> |
scandium - removing parsoid deploy dir and letting puppet re-clone it after merging gerrit fix 484602 - replace manual clone with proper puppetization (T201366) |
[production] |