1551-1600 of 10000 results (57ms)
2019-07-30 §
05:23 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.15/includes/upload/UploadBase.php: T228929 (duration: 00m 48s) [production]
05:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove s8 ready only T227062 (duration: 00m 24s) [production]
05:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Switchover s8 master eqiad from db1071 to db1104 T227062 (duration: 00m 24s) [production]
05:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Set s8 on read-only T227062 (duration: 00m 26s) [production]
05:00 <marostegui> Starting s8 failover from db1071 to db1104 - T227062 [production]
04:48 <eileen> civicrm revision changed from 1d57aca19c to 218328b29d, config revision is 3f960c48f6 [production]
04:15 <marostegui> Start pre-steps for s8 primary master failover - T227062 [production]
02:37 <eileen> civicrm revision changed from 121feb5d53 to 1d57aca19c, config revision is 3f960c48f6 [production]
2019-07-29 §
23:37 <XioNoX> replace export policy BGP_Wikimedia_own_space with BGP_Wikimedia_no_dfz in ams [production]
23:35 <catrope@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/GrowthExperiments/: Make welcome and discovery tours fully mutually exclusive (T229044) (duration: 00m 48s) [production]
23:26 <XioNoX> replace export policy BGP_Wikimedia_own_space with BGP_Wikimedia_no_dfz in ulsfo [production]
23:22 <XioNoX> replace export policy BGP_Wikimedia_own_space with BGP_Wikimedia_no_dfz in Dallas [production]
22:33 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.15/includes/cache/MessageCache.php: T208897 - fa817b088e43975 (duration: 00m 47s) [production]
22:32 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/AbuseFilter/: T214674 - bfcaf0c26d6 (duration: 00m 48s) [production]
22:28 <XioNoX> roll out anycast DNS and syslog to all network devices - T228190 [production]
22:16 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/AbuseFilter/: T214674 - 940955ea3844721a0 (duration: 00m 48s) [production]
22:05 <XioNoX> replace ulsfo network devices' DNS target with 10.3.0.1 [production]
22:00 <Krinkle> krinkle@deploy1001: Dirty git status on extensions/AbusesFilter and extensions/CheckUser in php-1.34.0-wmf.15 [production]
21:43 <XioNoX> replace ulsfo network devices' syslog target with syslog.anycast.wmnet [production]
19:22 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@c3ffbee]: Weekly deploy (duration: 11m 42s) [production]
19:10 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@c3ffbee]: Weekly deploy [production]
18:23 <Urbanecm> Morning SWAT done [production]
18:21 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:526196|Rename Image-reviewer to image-reviewer on fawiki]] (T216406) (duration: 00m 47s) [production]
18:19 <Urbanecm> Run mwscript migrateUserGroup.php --wiki=fawiki Image-reviewer image-reviewer (T216406) [production]
18:17 <XioNoX> switch traffic to the GTT link between Ashburn and Amsterdam (set GTT metric to 820 vs. 1820 before) - T228827 [production]
18:10 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:430627|Add several rights to eliminators in fawiki]] (T176553, 2/2) (duration: 00m 47s) [production]
18:08 <urbanecm@deploy1001> Synchronized wmf-config/abusefilter.php: SWAT: [[:gerrit:430627|Add several rights to eliminators in fawiki]] (T176553, 1/2) (duration: 00m 47s) [production]
18:04 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/AbuseFilter: SWAT: [[:gerrit:525598|Initialize user-defined variables during shortcircuit]] (T214674) (duration: 00m 49s) [production]
17:37 <ejegg> updated payments-wiki config to a7dacbf8e9 [production]
17:08 <XioNoX> reprepro copy buster-wikimedia stretch-wikimedia python3-anycast-healthchecker [production]
17:05 <XioNoX> reprepro copy buster-wikimedia stretch-wikimedia python3-json-logger [production]
17:05 <XioNoX> reprepro copy buster-wikimedia stretch-wikimedia anycast-healthchecker [production]
16:47 <godog> add anycast syslog to wezen/centrallog1001 [production]
16:19 <elukey> manually stopped the sre.kafka.roll-restart-brokers cookbook after 4 brokers restarts since the sleep interval (10mins) is too tight. [production]
16:17 <elukey@cumin1001> END (ERROR) - Cookbook sre.kafka.roll-restart-brokers (exit_code=97) [production]
15:35 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Retry - Produce resource_change stream to eventgate-main - T211248 (duration: 00m 46s) [production]
15:34 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers [production]
15:30 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce resource_change stream to eventgate-main - T211248 (duration: 00m 47s) [production]
14:35 <papaul> shutting down pc2010 for maintenance [production]
13:57 <cdanis@cumin1001> dbctl commit of MediaWiki config (dc=all), diff saved to 'https://phabricator.wikimedia.org/P8816', previous config saved to /var/cache/conftool/dbconfig/20190729-135730-cdanis.json [production]
13:30 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
13:28 <marostegui> Stop MySQL on pc2010 - T227552 [production]
13:23 <arturo> T228870 reboot cloudvirt1007.eqiad.wmnet for kernel updates [production]
13:23 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:23 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:09 <arturo> T228870 reboot cloudvirt1006.eqiad.wmnet for kernel updates [production]
13:09 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:09 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:01 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
12:45 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Provision db2128 into s5 api T221533 (duration: 00m 47s) [production]