|
2020-05-01
§
|
| 18:57 |
<gehel> |
restart blazegraph on wdqs1006 - T242453 |
[production] |
| 14:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11110 and previous config saved to /var/cache/conftool/dbconfig/20200501-142354-marostegui.json |
[production] |
| 14:18 |
<hknust> |
holger@mwmaint1002 finished renameInvalidUsernames.php (fail) as part of T219279 |
[production] |
| 14:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11109 and previous config saved to /var/cache/conftool/dbconfig/20200501-140603-marostegui.json |
[production] |
| 13:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11108 and previous config saved to /var/cache/conftool/dbconfig/20200501-134707-marostegui.json |
[production] |
| 13:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly warm up db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11107 and previous config saved to /var/cache/conftool/dbconfig/20200501-132804-marostegui.json |
[production] |
| 13:06 |
<hknust> |
holger@mwmaint1002 Starting renameInvalidUsernames.php as part of T219279 |
[production] |
| 13:01 |
<vgutierrez> |
rolling restart of ats-tls in text@esams - T249335 |
[production] |
| 12:24 |
<mutante> |
mw230* - rolling restart of php-fpm - icinga warnings about opcache health in codfw |
[production] |
| 12:20 |
<mutante> |
mw2376 - restarting php-fpm - icinga warnings about opcache health in codfw |
[production] |
| 12:07 |
<mutante> |
notebook1004 - puppet was failed due to removal of jmorgan while one of his processes was still running. "change to absent failed.. user jmorgan currently used by process 29038". killing 29038, running puppet T251560 |
[production] |
| 12:05 |
<mutante> |
notebook1003 - puppet was failed due to removal of jmorgan while one of his processeswas still running. "change to absent failed.. user jmorgan currently used by porcess 3288". killing 3288, running puppet T251560 |
[production] |
| 11:52 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 11:51 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:50 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 11:50 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:31 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:31 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:54 |
<_joe_> |
depooled all servers in the app pool in rack D1 |
[production] |
| 08:54 |
<oblivian@cumin1001> |
conftool action : set/pooled=no:weight=30; selector: name=mw13(49|5[0-5])\.eqiad\.wmnet |
[production] |
| 08:50 |
<oblivian@cumin1001> |
conftool action : set/weight=10; selector: name=mw13(49|5[0-5])\.eqiad\.wmnet |
[production] |
| 08:48 |
<_joe_> |
repooling mw1407 with LCStoreStaticArray, increased opcache, puppet disabled |
[production] |
| 08:45 |
<_joe_> |
repooling mw1409 |
[production] |
| 08:39 |
<_joe_> |
repool mw1352 |
[production] |
| 08:37 |
<_joe_> |
depooling mw1352 |
[production] |
| 07:44 |
<marostegui> |
Copy wikireplica dump from labsdb1009 to labsdb1011 - T249188 |
[production] |
| 01:36 |
<bmansurov@deploy1001> |
Finished deploy [recommendation-api/deploy@5f47cd7]: Update the recommendation API service (duration: 04m 33s) |
[production] |
| 01:32 |
<bmansurov@deploy1001> |
Started deploy [recommendation-api/deploy@5f47cd7]: Update the recommendation API service |
[production] |
|
2020-04-30
§
|
| 23:21 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: 9065650: Add project taglines (T249047) (duration: 01m 04s) |
[production] |
| 23:19 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 9065650: Add project taglines (T249047) (duration: 01m 04s) |
[production] |
| 23:17 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright/: SWAT: 9065650: Add project taglines (T249047) (duration: 01m 04s) |
[production] |
| 23:14 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: SWAT: 9065650: Add project taglines (T249047) (duration: 01m 05s) |
[production] |
| 23:08 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: SWAT: ae1424a: Logo wordmarks should not define fill color - opacity will be used (T251135) (duration: 01m 05s) |
[production] |
| 23:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: cf5f7ff: Assign oathauth-verify-user to stewards (T251447) (duration: 01m 05s) |
[production] |
| 20:13 |
<shdubsh> |
test mtail rc35 upgrade on logstash1007 - T251466 |
[production] |
| 20:10 |
<rzl> |
mcrouter certs re-renewed on puppetmaster1001, puppet enabled on mcrouter hosts |
[production] |
| 20:05 |
<jeh> |
cloudvirt1024 upgrade iDRAC firmware from 2.4.8 to 2.5.4 T241884 |
[production] |
| 20:04 |
<rzl> |
Disabling puppet on all mcrouter hosts for cert renewal. This isn't strictly needed, as the certs from last time are still fine -- just testing the renewal script. |
[production] |
| 19:42 |
<jeh> |
reboot cloudvirt1024 for NIC firmware updates T241884 |
[production] |
| 19:25 |
<shdubsh> |
test mtail rc35 upgrade on fermium - T251466 |
[production] |
| 17:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1091', diff saved to https://phabricator.wikimedia.org/P11104 and previous config saved to /var/cache/conftool/dbconfig/20200430-174057-marostegui.json |
[production] |
| 17:37 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group1 and group2 wikis to 1.35.0-wmf.28" |
[production] |
| 17:24 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: labs only (duration: 00m 58s) |
[production] |
| 15:28 |
<volans@deploy1001> |
Finished deploy [homer/deploy@56506db]: Release v0.2.1 (duration: 00m 21s) |
[production] |
| 15:27 |
<volans@deploy1001> |
Started deploy [homer/deploy@56506db]: Release v0.2.1 |
[production] |
| 15:11 |
<marostegui> |
Create lag on es1021 |
[production] |
| 14:53 |
<krinkle@deploy1001> |
Synchronized wmf-config/db-eqiad.php: I46d2b811f6287689 (duration: 00m 57s) |
[production] |
| 14:38 |
<krinkle@deploy1001> |
Synchronized wmf-config/db-codfw.php: I46d2b811f6287689 (duration: 00m 57s) |
[production] |