|
2018-09-24
§
|
| 09:30 |
<godog> |
upgrade / roll restart thumbor in eqiad / codfw - T20871 T198370 |
[production] |
| 09:03 |
<jynus> |
stop and upgrade db1067 (s1 eqiad master)- it may create some temporary lag on s1 |
[production] |
| 07:55 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Mon Sep 24 07:55:18 UTC 2018 (duration 10m 50s) |
[production] |
| 07:44 |
<bawolff@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.22) (duration: 07m 44s) |
[production] |
| 07:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 46s) |
[production] |
| 07:25 |
<bawolff_> |
manually running l10n update (T205238) |
[production] |
| 07:09 |
<moritzm> |
installing libarchive-zip-perl security updates |
[production] |
| 07:05 |
<volans> |
wiping netbox DB to re-import it cleanly from racktables - T199083 |
[production] |
| 06:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2062 (duration: 00m 50s) |
[production] |
| 06:57 |
<godog> |
repair sde on ms-be2041 - T199198 |
[production] |
| 06:57 |
<godog> |
repair sdn on ms-be1041 - T199198 |
[production] |
| 06:54 |
<volans@deploy1001> |
Finished deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix (duration: 00m 05s) |
[production] |
| 06:54 |
<volans@deploy1001> |
Started deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix |
[production] |
| 05:48 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 50s) |
[production] |
| 05:28 |
<marostegui> |
Deploy schema change on db2088:3311 - T203709 |
[production] |
| 05:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 50s) |
[production] |
| 04:15 |
<kartik@deploy1001> |
Finished deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) (duration: 04m 20s) |
[production] |
| 04:10 |
<kartik@deploy1001> |
Started deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) |
[production] |
| 02:48 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Mon Sep 24 02:48:16 UTC 2018 (duration 10m 49s) |
[production] |
| 02:37 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.22) (duration: 16m 02s) |
[production] |
|
2018-09-21
§
|
| 23:47 |
<mutante> |
mwmaint1002 - rsyncing home dirs over from mwmaint1001 which synced them from terbium. still large old files from terbium here, like a NASA video :) (T201343) |
[production] |
| 22:40 |
<mutante> |
mwmaint1002 - scap pull |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (T201343) |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (201343) |
[production] |
| 22:12 |
<mutante> |
how to generate mcrouter certs, needed when you add any new mw server: https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host |
[production] |
| 19:10 |
<XioNoX> |
shutdown cr3/4-ulsfo (not in prod) |
[production] |
| 18:19 |
<bawolff> |
deploy patch related to T197279 T204825 |
[production] |
| 18:08 |
<bblack> |
upgrade multatuli to gdnsd-2.99.1729-beta |
[production] |
| 18:06 |
<bblack> |
uploaded 2.99.1729-beta-1+wmf1 to stretch-wikimedia |
[production] |
| 17:59 |
<mutante> |
*.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses |
[production] |
| 17:58 |
<mutante> |
toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty |
[production] |
| 16:37 |
<XioNoX> |
route ns2-v6 to multatuli.wikimedia.org on cr1/2-esams |
[production] |
| 16:33 |
<mutante> |
puppetmaster1001: cergen --base-path /srv/private/modules/secret/secrets/mcrouter/ --generate /etc/cergen/mcrouter.manifests.d |
[production] |
| 16:32 |
<mutante> |
puppetmaster1001: mcrouter_generate_certs --generate |
[production] |
| 16:29 |
<mutante> |
puppetmaster: running mcrouter_generate_certs to add an mcrouter cert for mwmaint1002 (T201343) https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host |
[production] |
| 15:39 |
<bblack> |
upgrading gdnsd to 2.99.42 on authdns2001 |
[production] |
| 15:16 |
<XioNoX> |
route ns2 to multatuli.wikimedia.org on cr1/2-eqiad |
[production] |
| 13:35 |
<ladsgroup@deploy1001> |
Finished deploy [ores/deploy@7b987a7]: Pass the real IP (T205087) (duration: 24m 10s) |
[production] |
| 13:11 |
<ladsgroup@deploy1001> |
Started deploy [ores/deploy@7b987a7]: Pass the real IP (T205087) |
[production] |
| 13:01 |
<volans> |
wiping netbox DB to re-import it cleanly from racktables - T199083 |
[production] |
| 12:43 |
<banyek> |
adding wmf-pt-kill_2.2.20-1+wmf1 package for jessie |
[production] |
| 12:31 |
<banyek> |
adding wmf-pt-kill_2.2.20-1+wmf1 package for strech |
[production] |
| 12:24 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool pc1006 (duration: 00m 49s) |
[production] |
| 12:13 |
<marostegui> |
Stop MySQL on pc1006 for kernel upgrade |
[production] |
| 12:13 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool pc1006 (duration: 00m 50s) |
[production] |
| 07:17 |
<moritzm> |
uploaded debdeploy 0.0.99.6 to apt.wikimedia.org |
[production] |
| 07:00 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool pc1005 (duration: 00m 50s) |
[production] |