1401-1450 of 10000 results (85ms)
2019-05-13 §
09:40 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1064 T217396 (duration: 00m 49s) [production]
09:33 <hashar> Upgrading Zuul 2.5.1-wmf7 -> 2.5.1-wmf9 T105474 [production]
07:27 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully pool db1130 (s5) and db1138 (s4) T222682 (duration: 00m 50s) [production]
07:08 <elukey> slow roll restart of celery on ores* nodes to allow cores to be generated upon segfault - T222866 [production]
07:05 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic for db1130 (s5) and db1138 (s4) T222682 (duration: 00m 50s) [production]
06:53 <moritzm> installing ghostscript security updates [production]
06:44 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic for db1130 (s5) and db1138 (s4) T222682 (duration: 00m 49s) [production]
06:09 <marostegui> Compress s2, s6 and s7 on labsdb1012 - T222978 [production]
05:50 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic for db1130 (s5) and db1138 (s4) T222682 (duration: 00m 49s) [production]
05:41 <marostegui> Optimize tables on pc2007 [production]
05:18 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1130 into s5 and db1138 into s4 T222682 (duration: 00m 49s) [production]
05:17 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db1130 into s5 and db1138 into s4 T222682 (duration: 00m 51s) [production]
2019-05-12 §
15:32 <elukey> rollback python-kafka one eventlog1002 to 1.4.1-1~stretch1 - T222941 [production]
12:14 <elukey> restart eventlogging on eventlog1002 - all processors stuck due to kafka python (T222941) [production]
05:31 <marostegui> DIsable notifications for db1116:s8 Slave LAG check as this is a snapshot source [production]
2019-05-11 §
18:26 <reedy@deploy1001> Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 57s) [production]
06:37 <elukey> restart eventlogging on eventlog1002 - huge kafka consumer lag accumulated (T222941) [production]
02:01 <mutante> actinium - low disk space - apt-get clean - gzip /var/log/squid3/access.log.1 [production]
2019-05-10 §
18:58 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin -b 15 -p 95 '*' 'run-puppet-agent -q --failed-only' [production]
18:51 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin -b 15 -p 95 '*' 'run-puppet-agent -q --failed-only' [production]
18:49 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin '*' 'enable-puppet "Puppet breakages on all hosts -- cdanis"' [production]
18:39 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin '*' 'disable-puppet "Puppet breakages on all hosts -- cdanis"' [production]
16:50 <reedy@deploy1001> Synchronized dblists/: Update size related dblists (duration: 00m 49s) [production]
16:31 <ebernhardson> drop archive indices from cloudelastic [production]
16:11 <ariel@deploy1001> Finished deploy [dumps/dumps@70e8498]: look for dumpstatus json file per wiki run (duration: 00m 05s) [production]
16:11 <ariel@deploy1001> Started deploy [dumps/dumps@70e8498]: look for dumpstatus json file per wiki run [production]
16:05 <ejegg> moved adyen smashpig job runner to frdev1001 [production]
15:25 <_joe_> wiped opcache clean on all api, appservers [production]
15:05 <cdanis> cdanis@mw1239.eqiad.wmnet ~ % sudo php7adm /opcache-free [production]
15:05 <Krinkle> fix opcache krinkle@mw1268:~$ scap pull [production]
15:04 <cdanis> cdanis@mw1268.eqiad.wmnet ~ % sudo php7adm /opcache-free [production]
15:03 <Krinkle> ran 'scap pull' on mw1239.eqiad.wmnet to fix opcache corruption [production]
14:56 <jbond42> uploade zuul_2.5.10-wmf9 to jessie-wikimedia [production]
14:54 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: T99740 / d9dbecad9c7b (duration: 00m 51s) [production]
14:33 <akosiaris@deploy1001> scap-helm eventgate-analytics finished [production]
14:32 <akosiaris@deploy1001> scap-helm eventgate-analytics cluster staging completed [production]
14:32 <akosiaris@deploy1001> scap-helm eventgate-analytics upgrade -f lala.yaml staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] [production]
14:30 <akosiaris@deploy1001> scap-helm eventgate-analytics finished [production]
14:30 <akosiaris@deploy1001> scap-helm eventgate-analytics cluster eqiad completed [production]
14:30 <akosiaris@deploy1001> scap-helm eventgate-analytics upgrade -f eventgate-analytics-eqiad-values.yaml production stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad] [production]
14:30 <akosiaris@deploy1001> scap-helm eventgate-analytics finished [production]
14:30 <akosiaris@deploy1001> scap-helm eventgate-analytics cluster codfw completed [production]
14:30 <akosiaris@deploy1001> scap-helm eventgate-analytics upgrade -f eventgate-analytics-codfw-values.yaml production stable/eventgate-analytics [namespace: eventgate-analytics, clusters: codfw] [production]
13:30 <ema> pool cp3038 w/ ATS backend T222937 [production]
12:19 <ema> depool cp3038 and reimage as upload_ats T222937 [production]
11:52 <jbond42> (un)load edac kernel modules on elastic1029 to test resetting counters [production]
11:04 <jbond42> restart refinery-eventlogging-saltrotate on an-coord1001 [production]
10:30 <moritzm> installing symfony security updates [production]
09:17 <jynus> disabling replication lag alerts for backup source hosts on s1, s4, s8 T206203 [production]
07:14 <moritzm> uploaded linux-meta 1.21 for jessie-wikimedia (pointing to the new -9 ABI introduced with the 4.9.168 kernel) [production]