|
2018-09-23
§
|
| 21:41 |
<krinkle@deploy1001> |
Synchronized wmf-config/profiler.php: (no justification provided) (duration: 00m 52s) |
[production] |
| 18:51 |
<legoktm> |
restarted and updated all dependencies |
[tools.extreg-wos] |
| 16:56 |
<zhuyifei1999_> |
deployed till e74f575 on -web-01, T202588 T205153 |
[quarry] |
| 14:03 |
<Krenair> |
rm stuff in deployment-deploy01:/tmp to try to clear space and stop shinken whining |
[releng] |
| 04:32 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.22/includes/user/User.php: T202149 - Ic0c25f66f23f (duration: 00m 53s) |
[production] |
| 04:08 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_wdq-mm |
[wdq-mm] |
| 04:05 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_tor |
[tor] |
| 04:04 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_reading-lists |
[reading-lists] |
| 04:02 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_mw-api-testing |
[mw-api-testing] |
| 03:58 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_jupyter |
[jupyter] |
| 01:05 |
<andrewbogott> |
rebooted deployment-maps03; OOM and also T205195 |
[releng] |
| 01:05 |
<andrewbogott> |
rebooted deployment-maps03; OOM and also T205195 |
[deployment-prep] |
|
2018-09-22
§
|
| 20:51 |
<Hauskatze> |
github: deleting several wikimedia/mediawiki-extensions-Collection-.* mirror repos for T183891 |
[releng] |
| 20:05 |
<Hauskatze> |
github: deleted mirror wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer | T183891; moving to the next one |
[releng] |
| 18:28 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.22/extensions/MultimediaViewer/resources/: I0954c42a37668b0, T205162 (duration: 00m 56s) |
[production] |
| 18:21 |
<Krenair> |
went to do the same with deployment-maps03 and accidentally broke SSH access to the server |
[releng] |
| 18:20 |
<Krenair> |
removed ferm package from deployment-snapshot01 as it appeared unmanaged by puppet and was causing problems with SSH access from the current deployment hosts (previous logs referenced T153468, this just explains why puppet hadn't purged stuff) |
[releng] |
| 18:01 |
<Krenair> |
rm deployment-maps03:/etc/ferm/conf.d/10_redis_exporter_6379 as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?) |
[releng] |
| 18:00 |
<Krenair> |
rm deployment-snapshot01:/etc/ferm/conf.d/10_prometheus-nutcracker-exporter as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?) |
[releng] |
|
2018-09-21
§
|
| 23:47 |
<mutante> |
mwmaint1002 - rsyncing home dirs over from mwmaint1001 which synced them from terbium. still large old files from terbium here, like a NASA video :) (T201343) |
[production] |
| 23:25 |
<mutante> |
deleting now unused trusty instance relic (T204564) (/etc is backed up on new instance in /root/ just in case) |
[toolserver-legacy] |
| 23:17 |
<mutante> |
deleting unused instance trusty-update (T204558) |
[puppet] |
| 23:16 |
<mutante> |
deleting unused instance puppet-mailman (T204558) |
[puppet] |
| 23:12 |
<mutante> |
migration complete. shutting down trusty instance 'relic'. (T204564) |
[toolserver-legacy] |
| 22:40 |
<mutante> |
mwmaint1002 - scap pull |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (T201343) |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (201343) |
[production] |
| 22:12 |
<mutante> |
how to generate mcrouter certs, needed when you add any new mw server: https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host |
[production] |
| 21:56 |
<mutante> |
sending all frozen messages from exim queue on old instance to clean up |
[toolserver-legacy] |
| 21:19 |
<mutante> |
update DNS zone entries and remove eqiad-r IP, revert to eqiad IP |
[toolserver-legacy] |
| 21:16 |
<mutante> |
associating floating IP (the existing one in eqiad) with relic-stretch-eqiad |
[toolserver-legacy] |
| 19:34 |
<andrewbogott> |
deleting project as per note on https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_discourse-wam |
[discourse-wam] |
| 19:29 |
<framawiki> |
deployed 4b01077 on quarry-web-01 T204964 |
[quarry] |
| 19:10 |
<XioNoX> |
shutdown cr3/4-ulsfo (not in prod) |
[production] |
| 18:19 |
<bawolff> |
deploy patch related to T197279 T204825 |
[production] |
| 18:08 |
<bblack> |
upgrade multatuli to gdnsd-2.99.1729-beta |
[production] |
| 18:06 |
<bblack> |
uploaded 2.99.1729-beta-1+wmf1 to stretch-wikimedia |
[production] |
| 18:06 |
<mutante> |
scp /etc/toolserver.aliases from relic to relic-stretch (exim legacy aliases for @toolserver.org mail address forwarding), not puppetized, contains personal email addresses |
[toolserver-legacy] |
| 17:59 |
<mutante> |
*.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses |
[production] |
| 17:58 |
<mutante> |
toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty |
[production] |
| 17:50 |
<mutante> |
scp toolserver.org Letsencrypt key and cert from relic to relic-stretch instance for migration |
[toolserver-legacy] |
| 17:26 |
<marxarelli> |
adding jenkins node integration-slave-docker-1038 with 7 executors |
[releng] |
| 17:18 |
<bd808> |
Running `sudo maintain-meta_p --all-databases --purge` across labsdb10(09|10|11) for T201890 |
[admin] |
| 17:06 |
<mutante> |
relic-stretch: a2enmod ssl; starting apache |
[toolserver-legacy] |
| 16:58 |
<wikibugs> |
Updated channels.yaml to: 59937c1b4d3c31cd15c802831117b1a4c43c6b35 Add GCI 2018 work-board to #wikimedia-dev-africa |
[tools.wikibugs] |
| 16:47 |
<marxarelli> |
added new jenkins node integration-slave-docker-1037 with 7 executors |
[releng] |
| 16:37 |
<XioNoX> |
route ns2-v6 to multatuli.wikimedia.org on cr1/2-esams |
[production] |
| 16:33 |
<mutante> |
puppetmaster1001: cergen --base-path /srv/private/modules/secret/secrets/mcrouter/ --generate /etc/cergen/mcrouter.manifests.d |
[production] |
| 16:32 |
<mutante> |
puppetmaster1001: mcrouter_generate_certs --generate |
[production] |
| 16:29 |
<mutante> |
puppetmaster: running mcrouter_generate_certs to add an mcrouter cert for mwmaint1002 (T201343) https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host |
[production] |