1451-1500 of 10000 results (62ms)
2018-07-26 §
09:50 <zfilipin@deploy1001> Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 53s) [production]
09:49 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14 [production]
09:45 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 55s) [production]
09:40 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool es1015 fully (duration: 00m 55s) [production]
09:27 <gehel> rolling restart of elasticsearch / cirrus / eqiad to disable G1 - T156137 [production]
09:22 <marostegui> Deploy schema change on db1079 with replication, this will generate lag on labsdb:s7 T144010 T51190 T199368 [production]
09:22 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 55s) [production]
09:06 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 55s) [production]
09:00 <jynus> restarting hhvm at mw1226, mw1276, mw1281 [production]
08:54 <jynus> restarting hhvm at mw1230 [production]
08:26 <joal@deploy1001> Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2 (duration: 29m 02s) [production]
08:20 <marostegui> Deploy schema change on db1090:3317 T144010 T51190 T199368 [production]
08:18 <mobrovac@deploy1001> Synchronized wmf-config/CommonSettings.php: Set readOnlyReason to false everywhere for JobQueueEventBus - T199594 (duration: 00m 55s) [production]
08:10 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool es1015 with low load after reimage (duration: 00m 55s) [production]
07:57 <joal@deploy1001> Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2 [production]
07:44 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 54s) [production]
07:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1086 (duration: 00m 54s) [production]
07:24 <jynus> stop and reimage es1015 [production]
07:19 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool es1015 (duration: 00m 55s) [production]
06:47 <marostegui> Deploy schema change on db1086 T144010 T51190 T199368 [production]
06:46 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 55s) [production]
06:38 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 (duration: 00m 55s) [production]
05:02 <marostegui> Deploy schema change on db1101:3317 T144010 T51190 T199368 [production]
05:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 (duration: 01m 05s) [production]
04:13 <kartik@deploy1001> Finished deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283) (duration: 05m 12s) [production]
04:08 <kartik@deploy1001> Started deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283) [production]
03:16 <l10nupdate@deploy1001> ResourceLoader cache refresh completed at Thu Jul 26 03:16:04 UTC 2018 (duration 10m 24s) [production]
03:06 <twentyafterfour> stopped phabricator rebuild-identities migration as it's suspected of causing database connection exhaustion and replag [production]
03:05 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 13s) [production]
02:50 <Krinkle> phabricator.wikimedia.org is spewing HTTP 500 ("Can Not Connect to MySQL") [production]
02:32 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 40s) [production]
01:42 <mutante> phab1002 - starting rsync of repo data from phab1001 in a screen session after gerrit:447949 T190568 [production]
01:32 <mutante> phab1001 - rm /usr/local/sbin/sync-srv-repos that has a reference to non-existing server iridium.eqiad.wmnet (formerly phab) (T190568) [production]
00:16 <twentyafterfour> phabricator update complete with only momentary downtime [production]
00:04 <twentyafterfour> deploying phabricator release/2018-07-25/1, scheduled downtime for phab1001 in icinga [production]
00:01 <mutante> temp. disable puppet on netmon1002,netmon2001 before applying puppet ferm change out of abundance of caution. applied gerrit:439554 without issues on netmon1003 [production]
2018-07-25 §
23:35 <thcipriani@deploy1001> Synchronized php-1.32.0-wmf.13/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWInlineImageNode.js: SWAT: [[gerrit:447653|ve.dm.MWInlineImageNode: Fix undefined data-mw in toDomElements output]] T198941 T200214 (duration: 00m 57s) [production]
23:05 <mutante> added dchen to LDAP group "wmf" (was already in admins as shell user, so didn't have to be added in puppet repo) (T200366) [production]
20:23 <bblack> pooling cp5006 - T187157 [production]
20:05 <krinkle@deploy1001> Synchronized wmf-config/: Remove wgUploadThumbnailRenderHttpCustomDomain override - T200346 (duration: 00m 57s) [production]
18:59 <bstorm_> Added new disk shelf to labstore1007 [production]
17:53 <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 54s) [production]
17:45 <krinkle@deploy1001> Synchronized php-1.32.0-wmf.14/extensions/Cite/includes/Cite.php: (no justification provided) (duration: 00m 55s) [production]
17:07 <ejegg> Updated SmashPig settings: extended Ingenico Connect API timeout to 12 seconds [production]
16:49 <Amir1> morning SWAT is done [production]
16:49 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:447834|Enable reading from the new backend for Special:Tags in several large wikis (T199334)]] (duration: 00m 56s) [production]
16:40 <akosiaris> remove ORES abuser blocking T200338, let's reevaluate [production]
16:02 <joal@deploy1001> Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts (duration: 17m 51s) [production]
15:49 <gehel> elasticsearch cluster restart on codfw completed - T156137 [production]
15:44 <joal@deploy1001> Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts [production]