|
2018-07-26
§
|
| 09:22 |
<marostegui> |
Deploy schema change on db1079 with replication, this will generate lag on labsdb:s7 T144010 T51190 T199368 |
[production] |
| 09:22 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 55s) |
[production] |
| 09:06 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 55s) |
[production] |
| 09:00 |
<jynus> |
restarting hhvm at mw1226, mw1276, mw1281 |
[production] |
| 08:54 |
<jynus> |
restarting hhvm at mw1230 |
[production] |
| 08:26 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2 (duration: 29m 02s) |
[production] |
| 08:20 |
<marostegui> |
Deploy schema change on db1090:3317 T144010 T51190 T199368 |
[production] |
| 08:18 |
<mobrovac@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Set readOnlyReason to false everywhere for JobQueueEventBus - T199594 (duration: 00m 55s) |
[production] |
| 08:10 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool es1015 with low load after reimage (duration: 00m 55s) |
[production] |
| 07:57 |
<joal@deploy1001> |
Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2 |
[production] |
| 07:44 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 54s) |
[production] |
| 07:32 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1086 (duration: 00m 54s) |
[production] |
| 07:24 |
<jynus> |
stop and reimage es1015 |
[production] |
| 07:19 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool es1015 (duration: 00m 55s) |
[production] |
| 06:47 |
<marostegui> |
Deploy schema change on db1086 T144010 T51190 T199368 |
[production] |
| 06:46 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 55s) |
[production] |
| 06:38 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 (duration: 00m 55s) |
[production] |
| 05:02 |
<marostegui> |
Deploy schema change on db1101:3317 T144010 T51190 T199368 |
[production] |
| 05:02 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 (duration: 01m 05s) |
[production] |
| 04:13 |
<kartik@deploy1001> |
Finished deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283) (duration: 05m 12s) |
[production] |
| 04:08 |
<kartik@deploy1001> |
Started deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283) |
[production] |
| 03:16 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Thu Jul 26 03:16:04 UTC 2018 (duration 10m 24s) |
[production] |
| 03:06 |
<twentyafterfour> |
stopped phabricator rebuild-identities migration as it's suspected of causing database connection exhaustion and replag |
[production] |
| 03:05 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 13s) |
[production] |
| 02:50 |
<Krinkle> |
phabricator.wikimedia.org is spewing HTTP 500 ("Can Not Connect to MySQL") |
[production] |
| 02:32 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 40s) |
[production] |
| 01:42 |
<mutante> |
phab1002 - starting rsync of repo data from phab1001 in a screen session after gerrit:447949 T190568 |
[production] |
| 01:32 |
<mutante> |
phab1001 - rm /usr/local/sbin/sync-srv-repos that has a reference to non-existing server iridium.eqiad.wmnet (formerly phab) (T190568) |
[production] |
| 00:16 |
<twentyafterfour> |
phabricator update complete with only momentary downtime |
[production] |
| 00:04 |
<twentyafterfour> |
deploying phabricator release/2018-07-25/1, scheduled downtime for phab1001 in icinga |
[production] |
| 00:01 |
<mutante> |
temp. disable puppet on netmon1002,netmon2001 before applying puppet ferm change out of abundance of caution. applied gerrit:439554 without issues on netmon1003 |
[production] |
|
2018-07-25
§
|
| 23:35 |
<thcipriani@deploy1001> |
Synchronized php-1.32.0-wmf.13/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWInlineImageNode.js: SWAT: [[gerrit:447653|ve.dm.MWInlineImageNode: Fix undefined data-mw in toDomElements output]] T198941 T200214 (duration: 00m 57s) |
[production] |
| 23:05 |
<mutante> |
added dchen to LDAP group "wmf" (was already in admins as shell user, so didn't have to be added in puppet repo) (T200366) |
[production] |
| 20:23 |
<bblack> |
pooling cp5006 - T187157 |
[production] |
| 20:05 |
<krinkle@deploy1001> |
Synchronized wmf-config/: Remove wgUploadThumbnailRenderHttpCustomDomain override - T200346 (duration: 00m 57s) |
[production] |
| 18:59 |
<bstorm_> |
Added new disk shelf to labstore1007 |
[production] |
| 17:53 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 54s) |
[production] |
| 17:45 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.14/extensions/Cite/includes/Cite.php: (no justification provided) (duration: 00m 55s) |
[production] |
| 17:07 |
<ejegg> |
Updated SmashPig settings: extended Ingenico Connect API timeout to 12 seconds |
[production] |
| 16:49 |
<Amir1> |
morning SWAT is done |
[production] |
| 16:49 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:447834|Enable reading from the new backend for Special:Tags in several large wikis (T199334)]] (duration: 00m 56s) |
[production] |
| 16:40 |
<akosiaris> |
remove ORES abuser blocking T200338, let's reevaluate |
[production] |
| 16:02 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts (duration: 17m 51s) |
[production] |
| 15:49 |
<gehel> |
elasticsearch cluster restart on codfw completed - T156137 |
[production] |
| 15:44 |
<joal@deploy1001> |
Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts |
[production] |
| 15:33 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 (duration: 00m 55s) |
[production] |
| 15:31 |
<anomie> |
running populateContentTables.php on testwiki for T183488 |
[production] |
| 15:26 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 55s) |
[production] |
| 15:21 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Set wgMultiContentRevisionSchemaMigrationStage to write-both read-old on testwiki (T197817) (duration: 00m 55s) |
[production] |
| 14:45 |
<marostegui> |
Deploy schema change on db1098:3317 T144010 T51190 T199368 |
[production] |