651-700 of 10000 results (56ms)
2018-11-30 §
11:07 <banyek> deploy schema change on dbstore1002 - T85757 [production]
10:59 <marostegui> Deploy schema change on db1105:3312 T86338 T202167 [production]
10:59 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 T86338 T202167 (duration: 00m 45s) [production]
09:48 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 T86338 T202167 (duration: 00m 46s) [production]
09:00 <marostegui> Deploy schema change on db1090:3312 T86338 T202167 [production]
09:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 T86338 T202167 (duration: 00m 47s) [production]
08:55 <moritzm> removed rutherfordium from debmonitor DB (T210036) [production]
08:21 <marostegui> Deploy schema change on dbstore1002 T86338 T202167 [production]
08:12 <moritzm> installing perl security updates [production]
07:28 <marostegui> Deploy schema change on db1095:3312 T86338 T202167 [production]
07:27 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 T86338 T202167 (duration: 00m 47s) [production]
06:55 <marostegui> Purge binary logs on pc1005 [production]
06:52 <marostegui> Deploy schema change on db1103:3312 T86338 T202167 [production]
06:51 <marostegui> Deploy schema change on db1103:3312 T86338 T20216 [production]
06:51 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 T86338 T202167 (duration: 00m 48s) [production]
06:12 <marostegui> Deploy schema change on s2 codfw master with replication (db2035), this will generate lag on codfw - T86338 T202167 [production]
02:17 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:476772|update wbsearchentities ab test configuration]] T209402 (duration: 00m 47s) [production]
01:06 <thcipriani@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents: SWAT: [[gerrit:476777|wbsearchentities ab test improvements]] T209402 (duration: 00m 46s) [production]
01:04 <thcipriani@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend: SWAT: [[gerrit:476613|Change config flag for enabling Block Notice stats]] T201719 (duration: 00m 48s) [production]
00:57 <thcipriani@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/VisualEditor: SWAT: [[gerrit:476612|Rename configs for tracking block notices on visual editor]] [[gerrit:476762|ve.init.mw.ArticleTarget: Stop when we fail to load metadata]] T209542 (duration: 00m 48s) [production]
00:39 <thcipriani@deploy1001> Synchronized php-1.33.0-wmf.6/includes/api/ApiBase.php: SWAT: [[gerrit:476409|Add stats for block errors on create/edit actions]] T201717 (duration: 00m 48s) [production]
2018-11-29 §
23:26 <mutante> puppetmaster: sudo puppet cert revoke rutherfordium.eqiad.wmnet; sudo puppet node clean rutherfordium.eqiad.wmnet ; sudo puppet node deactivate rutherfordium.eqiad.wmnet ; run puppet on icinga1001.. removed host from monitoring (decom for ganeti VM) (T210036) [production]
22:21 <hashar> 1.33.0-wmf.6 is on all wikis and looks stable. [production]
22:04 <hashar> hashar@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.6 [production]
22:03 <hashar@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.6 [production]
21:54 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T210636 - I9ebbc625f98c314 (duration: 00m 55s) [production]
21:41 <tzatziki> changing email for User:Mathounette [production]
21:34 <XioNoX> removed unused vc-port on asw2-c-eqiad:fpc8 - T210788 [production]
20:50 <mutante> people - rsynced /home one last time, switched DNS people.eqiad CNAME over, varnish change merged (T210036) [production]
20:42 <mutante> people.wikimedia.org is switching backends from rutherfordium to people1001, please stand by during a short maintenance period.. data has been copied | https://wikitech.wikimedia.org/wiki/People.wikimedia.org#Backend_upgrade_November_2018 | T210036 [production]
20:01 <XioNoX> Apply Icinga:check_vcp to all VC switches - T201097 [production]
19:42 <XioNoX> remove neodymium/sarin from mgmt routers - T210612 [production]
19:16 <hashar@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/: RecordRevision::getUser() returns UserIdentity not int - T210737 (duration: 00m 55s) [production]
19:09 <ejegg> updated payments-wiki from 34250e80b5 to 7403a196b4 [production]
18:53 <XioNoX> Netbox: remove Napalm integration [production]
18:28 <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting comment migration to write-new/read-new on group 0 (T166733) (duration: 00m 52s) [production]
17:55 <XioNoX> remove test netbox user from cr3-ulsfo - T205898 [production]
17:49 <mforns@deploy1001> Finished deploy [analytics/refinery@40b1972]: deploying refinery to refinery-source version v0.0.81 (duration: 06m 01s) [production]
17:43 <mforns@deploy1001> Started deploy [analytics/refinery@40b1972]: deploying refinery to refinery-source version v0.0.81 [production]
17:41 <anomie@deploy1001> Synchronized php-1.33.0-wmf.6/includes/revisiondelete/RevisionDeleteUser.php: Fix RevisionDeleteUser rev_actor query for MySQL, for real this time (T210628) (duration: 00m 53s) [production]
17:34 <anomie@deploy1001> Synchronized php-1.33.0-wmf.6/includes/revisiondelete/RevisionDeleteUser.php: Fix RevisionDeleteUser rev_actor query for MySQL (T210628) (duration: 00m 53s) [production]
17:02 <robh> decom of labvirt101[01] continuing [production]
16:10 <anomie@mwmaint1002> Running Wikibase/populateSitesTable.php and cleanupUsersWithNoId.php on more wiktionaries, incubatorwiki, and sourceswiki for T210732 [production]
15:54 <papaul> shutting down ms-be2047 for maintenance [production]
15:35 <gtirloni> T196507 downtimed and powercycled cloudvirt1019 [production]
15:32 <anomie@mwmaint1002> Running Wikibase/populateSitesTable.php and cleanupUsersWithNoId.php on several other wiktionaries for T210732 [production]
15:29 <gehel> activating multiple elasticsearch instances on cirrus / eqiad - T207918 [production]
15:18 <$WHO> Running Wikibase populateSitesTable.php on eswiktionary for T210732 [production]
14:31 <hashar@deploy1001> rebuilt and synchronized wikiversions files: Revert all wikis to 1.33.0-wmf.6 [production]
14:29 <hashar@deploy1001> scap failed: average error rate on 11/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]