|
2018-09-24
§
|
| 14:18 |
<jynus> |
stop and upgrade es1017 (es3 eqiad master)- it may create some temporary lag on es3 |
[production] |
| 13:59 |
<jynus> |
stop and upgrade es1015 (es2 eqiad master)- it may create some temporary lag on es2 |
[production] |
| 13:52 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2089 (duration: 00m 49s) |
[production] |
| 13:43 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2089 (duration: 00m 50s) |
[production] |
| 13:34 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Setup to db hosts for api-s5 (duration: 00m 50s) |
[production] |
| 13:23 |
<moritzm> |
rebooting rdb1004 for kernel security update |
[production] |
| 13:12 |
<moritzm> |
rebooting rdb1003 for kernel security update |
[production] |
| 13:03 |
<gehel> |
start isolating maps1004 for reimage to stretch - T195285 |
[production] |
| 11:39 |
<elukey> |
reboot an-master100[1,2] as part of the pre-checks before the hadoop master daemons swap - T203635 |
[production] |
| 11:21 |
<moritzm> |
installing texlive-bin security updates |
[production] |
| 10:44 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:462417|Bumping portals to master (T128546)]] (duration: 00m 49s) |
[production] |
| 10:43 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:462417|Bumping portals to master (T128546)]] (duration: 00m 50s) |
[production] |
| 09:42 |
<jynus> |
stop and upgrade db1066 (s2 eqiad master)- it may create some temporary lag on s2 |
[production] |
| 09:30 |
<godog> |
upgrade / roll restart thumbor in eqiad / codfw - T20871 T198370 |
[production] |
| 09:03 |
<jynus> |
stop and upgrade db1067 (s1 eqiad master)- it may create some temporary lag on s1 |
[production] |
| 07:55 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Mon Sep 24 07:55:18 UTC 2018 (duration 10m 50s) |
[production] |
| 07:44 |
<bawolff@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.22) (duration: 07m 44s) |
[production] |
| 07:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 46s) |
[production] |
| 07:25 |
<bawolff_> |
manually running l10n update (T205238) |
[production] |
| 07:09 |
<moritzm> |
installing libarchive-zip-perl security updates |
[production] |
| 07:05 |
<volans> |
wiping netbox DB to re-import it cleanly from racktables - T199083 |
[production] |
| 06:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2062 (duration: 00m 50s) |
[production] |
| 06:57 |
<godog> |
repair sde on ms-be2041 - T199198 |
[production] |
| 06:57 |
<godog> |
repair sdn on ms-be1041 - T199198 |
[production] |
| 06:54 |
<volans@deploy1001> |
Finished deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix (duration: 00m 05s) |
[production] |
| 06:54 |
<volans@deploy1001> |
Started deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix |
[production] |
| 05:48 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 50s) |
[production] |
| 05:28 |
<marostegui> |
Deploy schema change on db2088:3311 - T203709 |
[production] |
| 05:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 50s) |
[production] |
| 04:15 |
<kartik@deploy1001> |
Finished deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) (duration: 04m 20s) |
[production] |
| 04:10 |
<kartik@deploy1001> |
Started deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) |
[production] |
| 02:48 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Mon Sep 24 02:48:16 UTC 2018 (duration 10m 49s) |
[production] |
| 02:37 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.22) (duration: 16m 02s) |
[production] |
|
2018-09-21
§
|
| 23:47 |
<mutante> |
mwmaint1002 - rsyncing home dirs over from mwmaint1001 which synced them from terbium. still large old files from terbium here, like a NASA video :) (T201343) |
[production] |
| 22:40 |
<mutante> |
mwmaint1002 - scap pull |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (T201343) |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (201343) |
[production] |
| 22:12 |
<mutante> |
how to generate mcrouter certs, needed when you add any new mw server: https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host |
[production] |
| 19:10 |
<XioNoX> |
shutdown cr3/4-ulsfo (not in prod) |
[production] |
| 18:19 |
<bawolff> |
deploy patch related to T197279 T204825 |
[production] |
| 18:08 |
<bblack> |
upgrade multatuli to gdnsd-2.99.1729-beta |
[production] |
| 18:06 |
<bblack> |
uploaded 2.99.1729-beta-1+wmf1 to stretch-wikimedia |
[production] |
| 17:59 |
<mutante> |
*.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses |
[production] |
| 17:58 |
<mutante> |
toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty |
[production] |
| 16:37 |
<XioNoX> |
route ns2-v6 to multatuli.wikimedia.org on cr1/2-esams |
[production] |
| 16:33 |
<mutante> |
puppetmaster1001: cergen --base-path /srv/private/modules/secret/secrets/mcrouter/ --generate /etc/cergen/mcrouter.manifests.d |
[production] |
| 16:32 |
<mutante> |
puppetmaster1001: mcrouter_generate_certs --generate |
[production] |