|
2018-09-24
§
|
| 05:48 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 50s) |
[production] |
| 05:28 |
<marostegui> |
Deploy schema change on db2088:3311 - T203709 |
[production] |
| 05:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 50s) |
[production] |
| 04:15 |
<kartik@deploy1001> |
Finished deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) (duration: 04m 20s) |
[production] |
| 04:10 |
<kartik@deploy1001> |
Started deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) |
[production] |
| 03:45 |
<kart_> |
Update cxserver to d913793 |
[releng] |
| 02:48 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Mon Sep 24 02:48:16 UTC 2018 (duration 10m 49s) |
[production] |
| 02:37 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.22) (duration: 16m 02s) |
[production] |
|
2018-09-23
§
|
| 21:41 |
<krinkle@deploy1001> |
Synchronized wmf-config/profiler.php: (no justification provided) (duration: 00m 52s) |
[production] |
| 18:51 |
<legoktm> |
restarted and updated all dependencies |
[tools.extreg-wos] |
| 16:56 |
<zhuyifei1999_> |
deployed till e74f575 on -web-01, T202588 T205153 |
[quarry] |
| 14:03 |
<Krenair> |
rm stuff in deployment-deploy01:/tmp to try to clear space and stop shinken whining |
[releng] |
| 04:32 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.22/includes/user/User.php: T202149 - Ic0c25f66f23f (duration: 00m 53s) |
[production] |
| 04:08 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_wdq-mm |
[wdq-mm] |
| 04:05 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_tor |
[tor] |
| 04:04 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_reading-lists |
[reading-lists] |
| 04:02 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_mw-api-testing |
[mw-api-testing] |
| 03:58 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_jupyter |
[jupyter] |
| 01:05 |
<andrewbogott> |
rebooted deployment-maps03; OOM and also T205195 |
[releng] |
| 01:05 |
<andrewbogott> |
rebooted deployment-maps03; OOM and also T205195 |
[deployment-prep] |
|
2018-09-22
§
|
| 20:51 |
<Hauskatze> |
github: deleting several wikimedia/mediawiki-extensions-Collection-.* mirror repos for T183891 |
[releng] |
| 20:05 |
<Hauskatze> |
github: deleted mirror wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer | T183891; moving to the next one |
[releng] |
| 18:28 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.22/extensions/MultimediaViewer/resources/: I0954c42a37668b0, T205162 (duration: 00m 56s) |
[production] |
| 18:21 |
<Krenair> |
went to do the same with deployment-maps03 and accidentally broke SSH access to the server |
[releng] |
| 18:20 |
<Krenair> |
removed ferm package from deployment-snapshot01 as it appeared unmanaged by puppet and was causing problems with SSH access from the current deployment hosts (previous logs referenced T153468, this just explains why puppet hadn't purged stuff) |
[releng] |
| 18:01 |
<Krenair> |
rm deployment-maps03:/etc/ferm/conf.d/10_redis_exporter_6379 as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?) |
[releng] |
| 18:00 |
<Krenair> |
rm deployment-snapshot01:/etc/ferm/conf.d/10_prometheus-nutcracker-exporter as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?) |
[releng] |
|
2018-09-21
§
|
| 23:47 |
<mutante> |
mwmaint1002 - rsyncing home dirs over from mwmaint1001 which synced them from terbium. still large old files from terbium here, like a NASA video :) (T201343) |
[production] |
| 23:25 |
<mutante> |
deleting now unused trusty instance relic (T204564) (/etc is backed up on new instance in /root/ just in case) |
[toolserver-legacy] |
| 23:17 |
<mutante> |
deleting unused instance trusty-update (T204558) |
[puppet] |
| 23:16 |
<mutante> |
deleting unused instance puppet-mailman (T204558) |
[puppet] |
| 23:12 |
<mutante> |
migration complete. shutting down trusty instance 'relic'. (T204564) |
[toolserver-legacy] |
| 22:40 |
<mutante> |
mwmaint1002 - scap pull |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (T201343) |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (201343) |
[production] |
| 22:12 |
<mutante> |
how to generate mcrouter certs, needed when you add any new mw server: https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host |
[production] |
| 21:56 |
<mutante> |
sending all frozen messages from exim queue on old instance to clean up |
[toolserver-legacy] |
| 21:19 |
<mutante> |
update DNS zone entries and remove eqiad-r IP, revert to eqiad IP |
[toolserver-legacy] |
| 21:16 |
<mutante> |
associating floating IP (the existing one in eqiad) with relic-stretch-eqiad |
[toolserver-legacy] |
| 19:34 |
<andrewbogott> |
deleting project as per note on https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_discourse-wam |
[discourse-wam] |
| 19:29 |
<framawiki> |
deployed 4b01077 on quarry-web-01 T204964 |
[quarry] |
| 19:10 |
<XioNoX> |
shutdown cr3/4-ulsfo (not in prod) |
[production] |
| 18:19 |
<bawolff> |
deploy patch related to T197279 T204825 |
[production] |
| 18:08 |
<bblack> |
upgrade multatuli to gdnsd-2.99.1729-beta |
[production] |
| 18:06 |
<bblack> |
uploaded 2.99.1729-beta-1+wmf1 to stretch-wikimedia |
[production] |
| 18:06 |
<mutante> |
scp /etc/toolserver.aliases from relic to relic-stretch (exim legacy aliases for @toolserver.org mail address forwarding), not puppetized, contains personal email addresses |
[toolserver-legacy] |
| 17:59 |
<mutante> |
*.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses |
[production] |
| 17:58 |
<mutante> |
toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty |
[production] |
| 17:50 |
<mutante> |
scp toolserver.org Letsencrypt key and cert from relic to relic-stretch instance for migration |
[toolserver-legacy] |
| 17:26 |
<marxarelli> |
adding jenkins node integration-slave-docker-1038 with 7 executors |
[releng] |