|
2019-05-14
ยง
|
| 17:55 |
<mutante> |
elastic2029 - enable puppet agent - was disabled without reason and nobody seems to have logged in recently |
[production] |
| 17:54 |
<mutante> |
elastic2038 - restart nagios-nrpe-server - attempt to fix "CHECK_NRPE STATE UNKNOWN" for a single check |
[production] |
| 17:32 |
<mutante> |
contint1001 - mkdir /srv/zuul-logs ; mv /var/log/zuul/debug.log* /srv/zuul-logs/ to prevent CI running out of disk again (T207707) |
[production] |
| 17:22 |
<mbsantos@deploy1001> |
Finished deploy [proton/deploy@881b22b]: Update chromium-render to 8cc96e7 make timeout handler more robust (T217724) (duration: 02m 23s) |
[production] |
| 17:20 |
<mbsantos@deploy1001> |
Started deploy [proton/deploy@881b22b]: Update chromium-render to 8cc96e7 make timeout handler more robust (T217724) |
[production] |
| 16:30 |
<jynus> |
stop replication and start table recompression on labsdb1009 T222978 |
[production] |
| 16:22 |
<godog> |
statsd_exporter 0.9 upgrade on thumbor - T220709 |
[production] |
| 16:04 |
<gilles@deploy1001> |
Finished deploy [performance/coal@5a32eb2]: T221401 (duration: 00m 06s) |
[production] |
| 16:04 |
<gilles@deploy1001> |
Started deploy [performance/coal@5a32eb2]: T221401 |
[production] |
| 15:56 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.4/extensions/VisualEditor/includes/ApiVisualEditor.php: Hot-deploy VE unset variable fix T223281 (duration: 00m 55s) |
[production] |
| 15:51 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.5/extensions/VisualEditor/includes/ApiVisualEditor.php: Hot-deploy VE unset variable fix T223281 (duration: 00m 57s) |
[production] |
| 15:49 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@81059c6]: Deploy new reqs for reports (duration: 00m 55s) |
[production] |
| 15:49 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@81059c6]: Deploy new reqs for reports |
[production] |
| 15:43 |
<jynus> |
reload haproxy config @ dbproxy1010, dbproxy1011 |
[production] |
| 15:38 |
<XioNoX> |
re-activate bgp to telia on cr1-codfw - T222967 |
[production] |
| 15:33 |
<XioNoX> |
deactivate bgp to telia on cr1-codfw - T222967 |
[production] |
| 15:19 |
<papaul> |
shutting down elastic2038 for memory replacement |
[production] |
| 15:14 |
<hashar> |
mw1263: scap pull |
[production] |
| 14:53 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.5 |
[production] |
| 14:50 |
<moritzm> |
rebooting mw1263 for kernel update |
[production] |
| 14:47 |
<hashar@deploy1001> |
Finished scap: testwiki to 1.34.0-wmf.5 and rebuild l10n cache (duration: 62m 47s) |
[production] |
| 14:07 |
<_joe_> |
apt-get lean on mwmaint1002 |
[production] |
| 13:44 |
<hashar@deploy1001> |
Started scap: testwiki to 1.34.0-wmf.5 and rebuild l10n cache |
[production] |
| 13:44 |
<godog> |
rearm keyholder on deploy and cumin hosts |
[production] |
| 13:27 |
<hashar@deploy1001> |
Finished scap: testwiki to 1.34.0-wmf.5 and rebuild l10n cache (duration: 14m 39s) |
[production] |
| 13:12 |
<hashar> |
train delay, I forgot to sync 1.34.0-wmf.5 |
[production] |
| 13:12 |
<hashar@deploy1001> |
Started scap: testwiki to 1.34.0-wmf.5 and rebuild l10n cache |
[production] |
| 12:37 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.4/extensions/VisualEditor/: Hot-deploy T223023 fix I1b35b28e42 for mobile VE edit section switches (duration: 00m 54s) |
[production] |
| 12:10 |
<moritzm> |
rebooting mw2164 for kernel update |
[production] |
| 11:33 |
<hashar@deploy1001> |
Pruned MediaWiki: 1.33.0-wmf.24 (duration: 03m 20s) |
[production] |
| 11:29 |
<hashar> |
Deleting 1.33.0-wmf.24 from deploy1001 # T220730 |
[production] |
| 11:28 |
<kart_> |
EU-Mid day SWAT Done. |
[production] |
| 11:25 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:508818|Decrease idwiki MT thresold for publishing]] (T222782) (duration: 00m 51s) |
[production] |
| 11:23 |
<hashar@deploy1001> |
clean aborted: Pruned MediaWiki: 1.33.0-wmf.23 (duration: 14m 31s) |
[production] |
| 11:23 |
<jbond42> |
cumin1001 ~ % sudo cumin A:all '/usr/local/sbin/run-puppet-agent --failed-only |
[production] |
| 11:18 |
<jbond42> |
enable puppet issue fixed https://gerrit.wikimedia.org/r/c/operations/puppet/+/510131 |
[production] |
| 11:12 |
<ema> |
pool cp3036 reimaged to ATS T222937 |
[production] |
| 11:09 |
<hashar> |
Deleting 1.33.0-wmf.23 from deploy1001 # T220730 |
[production] |
| 11:09 |
<jbond42> |
disable puppet |
[production] |
| 10:58 |
<hashar> |
scap prep 1.34.0-wmf.5 # T220730 |
[production] |
| 10:16 |
<hashar> |
Cutting branches for 1.34.0-wmf.5 |
[production] |
| 10:01 |
<ema> |
depool cp3036 and reimage as upload_ats T222937 |
[production] |
| 09:55 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2034 from config T219493 (duration: 00m 49s) |
[production] |
| 09:53 |
<marostegui@deploy1001> |
scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 09:51 |
<marostegui> |
Remove db2034 from tendril and zarcillo - T219493 |
[production] |
| 09:51 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2034 from config T219493 (duration: 00m 50s) |
[production] |
| 09:34 |
<jynus> |
restart apache on ununpentium |
[production] |
| 09:29 |
<marostegui> |
Parsercache deployment window FINISHED |
[production] |
| 09:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Deploy second parsercache key change everywhere after deploying it in batches first T210725 (duration: 00m 50s) |
[production] |
| 09:15 |
<godog> |
statsd_exporter 0.9 upgrade on ores - T220709 |
[production] |