|
2019-05-14
ยง
|
| 15:49 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@81059c6]: Deploy new reqs for reports |
[production] |
| 15:43 |
<jynus> |
reload haproxy config @ dbproxy1010, dbproxy1011 |
[production] |
| 15:38 |
<XioNoX> |
re-activate bgp to telia on cr1-codfw - T222967 |
[production] |
| 15:33 |
<XioNoX> |
deactivate bgp to telia on cr1-codfw - T222967 |
[production] |
| 15:19 |
<papaul> |
shutting down elastic2038 for memory replacement |
[production] |
| 15:14 |
<hashar> |
mw1263: scap pull |
[production] |
| 14:53 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.5 |
[production] |
| 14:50 |
<moritzm> |
rebooting mw1263 for kernel update |
[production] |
| 14:47 |
<hashar@deploy1001> |
Finished scap: testwiki to 1.34.0-wmf.5 and rebuild l10n cache (duration: 62m 47s) |
[production] |
| 14:07 |
<_joe_> |
apt-get lean on mwmaint1002 |
[production] |
| 13:44 |
<hashar@deploy1001> |
Started scap: testwiki to 1.34.0-wmf.5 and rebuild l10n cache |
[production] |
| 13:44 |
<godog> |
rearm keyholder on deploy and cumin hosts |
[production] |
| 13:27 |
<hashar@deploy1001> |
Finished scap: testwiki to 1.34.0-wmf.5 and rebuild l10n cache (duration: 14m 39s) |
[production] |
| 13:12 |
<hashar> |
train delay, I forgot to sync 1.34.0-wmf.5 |
[production] |
| 13:12 |
<hashar@deploy1001> |
Started scap: testwiki to 1.34.0-wmf.5 and rebuild l10n cache |
[production] |
| 12:37 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.4/extensions/VisualEditor/: Hot-deploy T223023 fix I1b35b28e42 for mobile VE edit section switches (duration: 00m 54s) |
[production] |
| 12:10 |
<moritzm> |
rebooting mw2164 for kernel update |
[production] |
| 11:33 |
<hashar@deploy1001> |
Pruned MediaWiki: 1.33.0-wmf.24 (duration: 03m 20s) |
[production] |
| 11:29 |
<hashar> |
Deleting 1.33.0-wmf.24 from deploy1001 # T220730 |
[production] |
| 11:28 |
<kart_> |
EU-Mid day SWAT Done. |
[production] |
| 11:25 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:508818|Decrease idwiki MT thresold for publishing]] (T222782) (duration: 00m 51s) |
[production] |
| 11:23 |
<hashar@deploy1001> |
clean aborted: Pruned MediaWiki: 1.33.0-wmf.23 (duration: 14m 31s) |
[production] |
| 11:23 |
<jbond42> |
cumin1001 ~ % sudo cumin A:all '/usr/local/sbin/run-puppet-agent --failed-only |
[production] |
| 11:18 |
<jbond42> |
enable puppet issue fixed https://gerrit.wikimedia.org/r/c/operations/puppet/+/510131 |
[production] |
| 11:12 |
<ema> |
pool cp3036 reimaged to ATS T222937 |
[production] |
| 11:09 |
<hashar> |
Deleting 1.33.0-wmf.23 from deploy1001 # T220730 |
[production] |
| 11:09 |
<jbond42> |
disable puppet |
[production] |
| 10:58 |
<hashar> |
scap prep 1.34.0-wmf.5 # T220730 |
[production] |
| 10:16 |
<hashar> |
Cutting branches for 1.34.0-wmf.5 |
[production] |
| 10:01 |
<ema> |
depool cp3036 and reimage as upload_ats T222937 |
[production] |
| 09:55 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2034 from config T219493 (duration: 00m 49s) |
[production] |
| 09:53 |
<marostegui@deploy1001> |
scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 09:51 |
<marostegui> |
Remove db2034 from tendril and zarcillo - T219493 |
[production] |
| 09:51 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2034 from config T219493 (duration: 00m 50s) |
[production] |
| 09:34 |
<jynus> |
restart apache on ununpentium |
[production] |
| 09:29 |
<marostegui> |
Parsercache deployment window FINISHED |
[production] |
| 09:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Deploy second parsercache key change everywhere after deploying it in batches first T210725 (duration: 00m 50s) |
[production] |
| 09:15 |
<godog> |
statsd_exporter 0.9 upgrade on ores - T220709 |
[production] |
| 09:02 |
<godog> |
statsd_exporter 0.9 upgrade on logstash - T220709 |
[production] |
| 08:53 |
<jynus> |
failing connections over dbproxy1006 to dbproxy1001 |
[production] |
| 07:48 |
<moritzm> |
installing bind security updates for stretch (only client-side tools/libraries in use) |
[production] |
| 06:45 |
<ema> |
cp-ats: upgrade trafficserver to 8.0.3-1wm2 |
[production] |
| 06:20 |
<ema> |
cp4021: upgrade trafficserver to 8.0.3-1wm2 |
[production] |
| 06:15 |
<ema> |
upload trafficserver 8.0.3-1wm2 to stretch-wikimedia |
[production] |
| 06:02 |
<marostegui> |
Deploy parsercache change to eqiad canaries - T210725 |
[production] |
| 06:01 |
<marostegui> |
Lock wmf-config deployment on deploy1001 to slowly change parsercache key on eqiad - T210725 |
[production] |
| 06:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Change parsercache on codfw T210725 (duration: 00m 54s) |
[production] |
| 01:55 |
<mutante> |
re-scheduled nginx / HTTP availability icinga checks |
[production] |
| 01:42 |
<mutante> |
cumin -b 6 'R:git::clone' 'run-puppet-agent -q --failed-only' |
[production] |
| 01:36 |
<mutante> |
restarting Gerrit to apply 2 config changes - disable DNS reverse lookup (gerrit:508127) & list projects from index (gerrit:508892) - removes blockers for 2.16 upgrade (T200739) |
[production] |