|
2020-05-01
§
|
| 12:05 |
<mutante> |
notebook1003 - puppet was failed due to removal of jmorgan while one of his processeswas still running. "change to absent failed.. user jmorgan currently used by porcess 3288". killing 3288, running puppet T251560 |
[production] |
| 11:52 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 11:51 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:50 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 11:50 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:31 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:31 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:54 |
<_joe_> |
depooled all servers in the app pool in rack D1 |
[production] |
| 08:54 |
<oblivian@cumin1001> |
conftool action : set/pooled=no:weight=30; selector: name=mw13(49|5[0-5])\.eqiad\.wmnet |
[production] |
| 08:50 |
<oblivian@cumin1001> |
conftool action : set/weight=10; selector: name=mw13(49|5[0-5])\.eqiad\.wmnet |
[production] |
| 08:48 |
<_joe_> |
repooling mw1407 with LCStoreStaticArray, increased opcache, puppet disabled |
[production] |
| 08:45 |
<_joe_> |
repooling mw1409 |
[production] |
| 08:39 |
<_joe_> |
repool mw1352 |
[production] |
| 08:37 |
<_joe_> |
depooling mw1352 |
[production] |
| 07:44 |
<marostegui> |
Copy wikireplica dump from labsdb1009 to labsdb1011 - T249188 |
[production] |
| 01:36 |
<bmansurov@deploy1001> |
Finished deploy [recommendation-api/deploy@5f47cd7]: Update the recommendation API service (duration: 04m 33s) |
[production] |
| 01:32 |
<bmansurov@deploy1001> |
Started deploy [recommendation-api/deploy@5f47cd7]: Update the recommendation API service |
[production] |
|
2020-04-30
§
|
| 23:21 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: 9065650: Add project taglines (T249047) (duration: 01m 04s) |
[production] |
| 23:19 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 9065650: Add project taglines (T249047) (duration: 01m 04s) |
[production] |
| 23:17 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright/: SWAT: 9065650: Add project taglines (T249047) (duration: 01m 04s) |
[production] |
| 23:14 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: SWAT: 9065650: Add project taglines (T249047) (duration: 01m 05s) |
[production] |
| 23:08 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: SWAT: ae1424a: Logo wordmarks should not define fill color - opacity will be used (T251135) (duration: 01m 05s) |
[production] |
| 23:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: cf5f7ff: Assign oathauth-verify-user to stewards (T251447) (duration: 01m 05s) |
[production] |
| 20:13 |
<shdubsh> |
test mtail rc35 upgrade on logstash1007 - T251466 |
[production] |
| 20:10 |
<rzl> |
mcrouter certs re-renewed on puppetmaster1001, puppet enabled on mcrouter hosts |
[production] |
| 20:05 |
<jeh> |
cloudvirt1024 upgrade iDRAC firmware from 2.4.8 to 2.5.4 T241884 |
[production] |
| 20:04 |
<rzl> |
Disabling puppet on all mcrouter hosts for cert renewal. This isn't strictly needed, as the certs from last time are still fine -- just testing the renewal script. |
[production] |
| 19:42 |
<jeh> |
reboot cloudvirt1024 for NIC firmware updates T241884 |
[production] |
| 19:25 |
<shdubsh> |
test mtail rc35 upgrade on fermium - T251466 |
[production] |
| 17:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1091', diff saved to https://phabricator.wikimedia.org/P11104 and previous config saved to /var/cache/conftool/dbconfig/20200430-174057-marostegui.json |
[production] |
| 17:37 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group1 and group2 wikis to 1.35.0-wmf.28" |
[production] |
| 17:24 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: labs only (duration: 00m 58s) |
[production] |
| 15:28 |
<volans@deploy1001> |
Finished deploy [homer/deploy@56506db]: Release v0.2.1 (duration: 00m 21s) |
[production] |
| 15:27 |
<volans@deploy1001> |
Started deploy [homer/deploy@56506db]: Release v0.2.1 |
[production] |
| 15:11 |
<marostegui> |
Create lag on es1021 |
[production] |
| 14:53 |
<krinkle@deploy1001> |
Synchronized wmf-config/db-eqiad.php: I46d2b811f6287689 (duration: 00m 57s) |
[production] |
| 14:38 |
<krinkle@deploy1001> |
Synchronized wmf-config/db-codfw.php: I46d2b811f6287689 (duration: 00m 57s) |
[production] |
| 14:24 |
<vgutierrez> |
upgrade trafficserver to 8.0.7-1wm2 on cp[5006,5011] |
[production] |
| 14:13 |
<marostegui> |
Stop slave on es2020 for testing |
[production] |
| 14:11 |
<vgutierrez> |
rolling restart of ats-tls on text@esams - T249335 |
[production] |
| 14:01 |
<vgutierrez> |
upgrade trafficserver to version 8.0.7-1wm2 on cp4025 and cp4031 |
[production] |
| 13:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:11 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:04 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.30 |
[production] |
| 12:44 |
<arturo> |
re-enable puppet in apt1001 |
[production] |
| 12:35 |
<jbond42> |
rolling restart of php7.2-fpm on mw1* servers |
[production] |
| 12:09 |
<jbond42> |
rolling restart of thumbor service |
[production] |
| 12:02 |
<arturo> |
disable puppet in apt1001 to briefly test a reprepro pull filter before merging a proper patch |
[production] |