|
2018-09-24
§
|
| 07:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 46s) |
[production] |
| 07:25 |
<bawolff_> |
manually running l10n update (T205238) |
[production] |
| 07:09 |
<moritzm> |
installing libarchive-zip-perl security updates |
[production] |
| 07:05 |
<volans> |
wiping netbox DB to re-import it cleanly from racktables - T199083 |
[production] |
| 06:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2062 (duration: 00m 50s) |
[production] |
| 06:57 |
<godog> |
repair sde on ms-be2041 - T199198 |
[production] |
| 06:57 |
<godog> |
repair sdn on ms-be1041 - T199198 |
[production] |
| 06:54 |
<volans@deploy1001> |
Finished deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix (duration: 00m 05s) |
[production] |
| 06:54 |
<volans@deploy1001> |
Started deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix |
[production] |
| 06:41 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/462341 |
[releng] |
| 05:48 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 50s) |
[production] |
| 05:28 |
<marostegui> |
Deploy schema change on db2088:3311 - T203709 |
[production] |
| 05:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 50s) |
[production] |
| 04:15 |
<kartik@deploy1001> |
Finished deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) (duration: 04m 20s) |
[production] |
| 04:10 |
<kartik@deploy1001> |
Started deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) |
[production] |
| 03:45 |
<kart_> |
Update cxserver to d913793 |
[releng] |
| 02:48 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Mon Sep 24 02:48:16 UTC 2018 (duration 10m 49s) |
[production] |
| 02:37 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.22) (duration: 16m 02s) |
[production] |
|
2018-09-23
§
|
| 21:41 |
<krinkle@deploy1001> |
Synchronized wmf-config/profiler.php: (no justification provided) (duration: 00m 52s) |
[production] |
| 18:51 |
<legoktm> |
restarted and updated all dependencies |
[tools.extreg-wos] |
| 16:56 |
<zhuyifei1999_> |
deployed till e74f575 on -web-01, T202588 T205153 |
[quarry] |
| 14:03 |
<Krenair> |
rm stuff in deployment-deploy01:/tmp to try to clear space and stop shinken whining |
[releng] |
| 04:32 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.22/includes/user/User.php: T202149 - Ic0c25f66f23f (duration: 00m 53s) |
[production] |
| 04:08 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_wdq-mm |
[wdq-mm] |
| 04:05 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_tor |
[tor] |
| 04:04 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_reading-lists |
[reading-lists] |
| 04:02 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_mw-api-testing |
[mw-api-testing] |
| 03:58 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_jupyter |
[jupyter] |
| 01:05 |
<andrewbogott> |
rebooted deployment-maps03; OOM and also T205195 |
[releng] |
| 01:05 |
<andrewbogott> |
rebooted deployment-maps03; OOM and also T205195 |
[deployment-prep] |
|
2018-09-22
§
|
| 20:51 |
<Hauskatze> |
github: deleting several wikimedia/mediawiki-extensions-Collection-.* mirror repos for T183891 |
[releng] |
| 20:05 |
<Hauskatze> |
github: deleted mirror wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer | T183891; moving to the next one |
[releng] |
| 18:28 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.22/extensions/MultimediaViewer/resources/: I0954c42a37668b0, T205162 (duration: 00m 56s) |
[production] |
| 18:21 |
<Krenair> |
went to do the same with deployment-maps03 and accidentally broke SSH access to the server |
[releng] |
| 18:20 |
<Krenair> |
removed ferm package from deployment-snapshot01 as it appeared unmanaged by puppet and was causing problems with SSH access from the current deployment hosts (previous logs referenced T153468, this just explains why puppet hadn't purged stuff) |
[releng] |
| 18:01 |
<Krenair> |
rm deployment-maps03:/etc/ferm/conf.d/10_redis_exporter_6379 as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?) |
[releng] |
| 18:00 |
<Krenair> |
rm deployment-snapshot01:/etc/ferm/conf.d/10_prometheus-nutcracker-exporter as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?) |
[releng] |
|
2018-09-21
§
|
| 23:47 |
<mutante> |
mwmaint1002 - rsyncing home dirs over from mwmaint1001 which synced them from terbium. still large old files from terbium here, like a NASA video :) (T201343) |
[production] |
| 23:25 |
<mutante> |
deleting now unused trusty instance relic (T204564) (/etc is backed up on new instance in /root/ just in case) |
[toolserver-legacy] |
| 23:17 |
<mutante> |
deleting unused instance trusty-update (T204558) |
[puppet] |
| 23:16 |
<mutante> |
deleting unused instance puppet-mailman (T204558) |
[puppet] |
| 23:12 |
<mutante> |
migration complete. shutting down trusty instance 'relic'. (T204564) |
[toolserver-legacy] |
| 22:40 |
<mutante> |
mwmaint1002 - scap pull |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (T201343) |
[production] |
| 22:13 |
<mutante> |
mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (201343) |
[production] |
| 22:12 |
<mutante> |
how to generate mcrouter certs, needed when you add any new mw server: https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host |
[production] |
| 21:56 |
<mutante> |
sending all frozen messages from exim queue on old instance to clean up |
[toolserver-legacy] |
| 21:19 |
<mutante> |
update DNS zone entries and remove eqiad-r IP, revert to eqiad IP |
[toolserver-legacy] |
| 21:16 |
<mutante> |
associating floating IP (the existing one in eqiad) with relic-stretch-eqiad |
[toolserver-legacy] |
| 19:34 |
<andrewbogott> |
deleting project as per note on https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_discourse-wam |
[discourse-wam] |