3601-3650 of 10000 results (65ms)
2019-05-10 §
05:41 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2105 db2109 into s3 T222772 (duration: 00m 49s) [production]
05:40 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2105 db2109 into s3 T222772 (duration: 00m 52s) [production]
05:40 <elukey> execute kafka preferred-replica-election on kafka-jumbo1001 as attempt to rebalance traffic (1002 seems handling way more than others since some days) [production]
05:32 <elukey> restart eventlogging daemons on eventlog1002 - kafka consumer errors in the logs, some lag built over time [production]
05:08 <marostegui> Stop MySQL on db1100 [production]
05:04 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 50s) [production]
04:56 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2112 (duration: 00m 51s) [production]
00:15 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@e13facb]: Downgrade LDF server back for T222471 (duration: 00m 37s) [production]
00:14 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@e13facb]: Downgrade LDF server back for T222471 [production]
2019-05-09 §
23:52 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T220625: Dont write to private wikis on cloudelastic (duration: 00m 50s) [production]
23:47 <ebernhardson@deploy1001> Synchronized php-1.34.0-wmf.4/extensions/CirrusSearch/: T220819 Uniquely identify connections in connection pool (duration: 00m 58s) [production]
23:43 <ebernhardson@deploy1001> Synchronized php-1.34.0-wmf.4/extensions/CirrusSearch/: T220625 Limit the clusters archive index is written to (duration: 00m 59s) [production]
23:41 <ebernhardson@deploy1001> Synchronized php-1.34.0-wmf.4/extensions/Wikibase/view/resources/jquery/wikibase/jquery.wikibase.entityselector.js: T172937 T222346 Revert Close entityselector after selecting exact match (duration: 00m 51s) [production]
23:24 <chaomodus> spicerack upgraded to 0.0.25 on cumin1001 and cumin 2001 [production]
22:58 <volans> uploaded spicerack_0.0.25-1_amd64.deb to apt.wikimedia.org stretch-wikimedia [production]
22:57 <bawolff> Manually cleared extdistributor cache T188692 [production]
22:50 <mutante> labweb1001/labweb1002 - remove "runJob" cron job from www-data's crontab, it is already also a systemd timer and puppet was meant to remove it (T222917) [production]
21:27 <foks> change user email for Melamrawy (WMF)@global [production]
21:23 <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable WikipediaAppCaptionEditCounter (T222211) (duration: 00m 52s) [production]
19:56 <thcipriani@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.4 [production]
19:28 <XioNoX> renumber mr1-esams<->cr2-knams link to 91.198.174.224/31 - T211254 [production]
19:24 <XioNoX> renumber mr1-esams<->cr1-esams link to 91.198.174.240/31 - T211254 [production]
18:22 <XioNoX> simplify filter analytics-in4 term mysql-dbstore on cr1/2-eqiad [production]
16:17 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Restore original weight on db1084 (duration: 00m 59s) [production]
16:07 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1081 (duration: 01m 13s) [production]
15:55 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1081 (duration: 01m 01s) [production]
15:40 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1081 (duration: 01m 00s) [production]
15:37 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2112 (duration: 00m 59s) [production]
15:25 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1081 (duration: 00m 56s) [production]
15:20 <marostegui> Stop mysql on db2112 for onsite work [production]
15:16 <otto@deploy1001> scap-helm eventgate-main finished [production]
15:16 <otto@deploy1001> scap-helm eventgate-main cluster eqiad completed [production]
15:16 <otto@deploy1001> scap-helm eventgate-main install -n main -f main/eqiad-values.yaml stable/eventgate [namespace: eventgate-main, clusters: eqiad] [production]
15:13 <otto@deploy1001> scap-helm eventgate-main finished [production]
15:13 <otto@deploy1001> scap-helm eventgate-main cluster codfw completed [production]
15:13 <otto@deploy1001> scap-helm eventgate-main install -n main -f main/codfw-values.yaml stable/eventgate [namespace: eventgate-main, clusters: codfw] [production]
15:12 <papaul> shurtting down db2114 for main board replacement [production]
14:53 <otto@deploy1001> scap-helm eventgate-main finished [production]
14:52 <otto@deploy1001> scap-helm eventgate-main cluster staging completed [production]
14:52 <otto@deploy1001> scap-helm eventgate-main upgrade main -f main/staging-values.yaml --reset-values stable/eventgate [namespace: eventgate-main, clusters: staging] [production]
14:48 <moritzm> removing unused uwsgi packages from scb* hosts [production]
14:13 <otto@deploy1001> scap-helm eventgate-main finished [production]
14:13 <otto@deploy1001> scap-helm eventgate-main cluster staging completed [production]
14:13 <otto@deploy1001> scap-helm eventgate-main upgrade main -f main/staging-values.yaml --reset-values stable/eventgate [namespace: eventgate-main, clusters: staging] [production]
13:34 <bblack> recdns: wiping dyna.wikimedia.org from pdns-recursors [production]
13:13 <fsero> running authdns-update for new docker-registry T221101 [production]
12:49 <fsero> switching traffic from old-registry to new registries registry[12]00[12] - T221101 [production]
12:01 <_joe_> reenabling puppet across the fleet [production]
11:57 <jbond42> all puppetmasters and puppetdbs should be restored' [production]
11:55 <jbond42> clean up old source files sudo cumin A:puppetmaster 'rm /etc/apt/sources.list.d/component-facter3.list /etc/apt/sources.list.d/component-puppet5.list' [production]