|
2019-02-14
§
|
| 23:47 |
<jynus> |
restarting labsdb1005 mysql in read only mode |
[production] |
| 23:37 |
<niharika29@deploy1001> |
Finished deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 (duration: 00m 02s) |
[production] |
| 23:37 |
<niharika29@deploy1001> |
Started deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 |
[production] |
| 22:07 |
<andrewbogott> |
rebuilding labvirt1012 as cloudvirt1012, T216190 |
[production] |
| 20:38 |
<bstorm_> |
Restarted mariadb on labsdb1005 for https://wikitech.wikimedia.org/wiki/Incident_documentation/20190214-labsdb1005 |
[production] |
| 20:18 |
<thcipriani> |
thcipriani@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.17 |
[production] |
| 20:14 |
<thcipriani@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.17 |
[production] |
| 20:09 |
<ejegg> |
updated fundraising CiviCRM from 02ea871b88 to 165fbf5894 |
[production] |
| 19:42 |
<thcipriani@deploy1001> |
Synchronized php-1.33.0-wmf.17/extensions/GrowthExperiments/modules/help: SWAT: [[gerrit:490674|Help Panel: Fix IME broken in help panel search]] T216131 (duration: 00m 54s) |
[production] |
| 19:14 |
<thcipriani@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:487007|Stop NavPopups gadget conflict with PagePreviews on Wikivoyage]] T214878 (duration: 00m 54s) |
[production] |
| 19:01 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev |
[production] |
| 18:52 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev |
[production] |
| 18:12 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet |
[production] |
| 18:03 |
<fsero> |
upgrading tiller to 2.12.2 on eqiad |
[production] |
| 17:34 |
<godog> |
bounce rsyslog on wezen/lithium, tls listener timeout in icinga |
[production] |
| 16:59 |
<moritzm> |
restarting apertium-apy on scb1001 to pick up Python security update |
[production] |
| 16:39 |
<marostegui> |
Depool labsdb1009 - T210713 |
[production] |
| 16:26 |
<fsero> |
upgrading tiller on codfw |
[production] |
| 16:11 |
<fsero> |
updating tiller version on staging cluster |
[production] |
| 16:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2085 - T214840 (duration: 00m 52s) |
[production] |
| 15:50 |
<fsero> |
building and publishing new tiller docker image on boron |
[production] |
| 15:50 |
<END> |
(PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) (volans@cumin1001) |
[production] |
| 15:43 |
<START> |
- Cookbook sre.hosts.upgrade-and-reboot (volans@cumin1001) |
[production] |
| 15:28 |
<volans> |
upgraded spicerack to v0.0.15 on cumin[12]001 |
[production] |
| 15:26 |
<volans> |
uploaded spicerack_0.0.15-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 15:12 |
<marostegui> |
Clear idrac logs from db2085 - T214840 |
[production] |
| 14:45 |
<godog> |
depool and stop logstash1009 for stretch reimage - T213898 |
[production] |
| 14:20 |
<marostegui> |
Stop MySQL on db2085 for on-site maintenance - T214840 |
[production] |
| 14:12 |
<jijiki> |
Enabling puppet on thumbor* servers - T214597 |
[production] |
| 13:39 |
<arturo> |
T215892 icinga downtime cloudvirt1024 for 2 weeks |
[production] |
| 12:22 |
<zeljkof> |
EU SWAT finished |
[production] |
| 12:21 |
<zfilipin@deploy1001> |
Synchronized php-1.33.0-wmf.17/extensions/ExternalGuidance/: SWAT: [[gerrit:490523|Fix the eventlogging schema definition as per manifest_version=2]] (duration: 00m 55s) |
[production] |
| 11:43 |
<_joe_> |
restarting hhvm on mw1338, hot tc exhausted T216084 |
[production] |
| 11:04 |
<_joe_> |
upgrading python3-etcd on stretch T209136 |
[production] |
| 11:03 |
<jbond42> |
rolling security updates for curl |
[production] |
| 11:02 |
<jijiki> |
Disabling puppet on thumbor* servers - T214597 |
[production] |
| 10:59 |
<moritzm> |
installing python3.4 security updates |
[production] |
| 10:53 |
<godog> |
bounce prometheus instances on prometheus2004 to take a snapshot |
[production] |
| 08:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1106 T214840 (duration: 00m 52s) |
[production] |
| 07:57 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s) |
[production] |
| 07:36 |
<marostegui> |
Stop MySQL on db1106 for reboot - T214840 |
[production] |
| 06:10 |
<marostegui> |
Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713 |
[production] |
| 06:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1087 T210713 (duration: 00m 55s) |
[production] |
| 01:52 |
<mutante> |
scandium - removing parsoid deploy dir and letting puppet re-clone it after merging gerrit fix 484602 - replace manual clone with proper puppetization (T201366) |
[production] |
| 01:52 |
<mutante> |
scandium - removing parsoid deploy dir and letting puppet re-clone it after merging gerrit fix 484602 - replace manual hack with proper puppet |
[production] |
| 01:15 |
<mutante> |
phab1001 - phabricator mail config converted to cluster.mailers to adjust to upstream change (T212989) |
[production] |
| 00:36 |
<bd808@deploy1001> |
Finished deploy [scholarships/scholarships@1d89fe2]: Live hack PHPMailer namespace T215302 (duration: 00m 02s) |
[production] |
| 00:36 |
<bd808@deploy1001> |
Started deploy [scholarships/scholarships@1d89fe2]: Live hack PHPMailer namespace T215302 |
[production] |
| 00:32 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable ORES (damaging only) on itwiki (T211032) (duration: 00m 53s) |
[production] |