251-300 of 10000 results (34ms)
2017-12-18 §
06:32 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174569 (duration: 00m 57s) [production]
06:28 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 56s) [production]
06:17 <marostegui> Stop replication in sync on db1106 and db1100 - T161294 [production]
06:17 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 57s) [production]
03:02 <eileen> re-enable multiqueue_consumer process-control config revision is 1ae3778278 [production]
03:00 <eileen> civicrm revision changed from 43d3f4d739 to e0ee2d189c [production]
02:32 <l10nupdate@tin> scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 57s) [production]
01:17 <cwd> disabled fredge multiqueue consumer [production]
00:49 <eileen> civicrm revision changed from 798e24671b to 43d3f4d739 [production]
2017-12-16 §
17:27 <no_justification> gerrit: Back, might see a few transient puppet failures if git pulls happened during the d/t, but should all recover [production]
17:21 <no_justification> gerrit: halting service momentarily for account reindexing [production]
15:00 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 58s) [production]
12:29 <ariel@tin> Finished deploy [dumps/dumps@95dbfe6]: revert previous deploy (duration: 00m 02s) [production]
12:29 <ariel@tin> Started deploy [dumps/dumps@95dbfe6]: revert previous deploy [production]
12:16 <ariel@tin> Finished deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files (duration: 00m 02s) [production]
12:16 <ariel@tin> Started deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files [production]
02:04 <mutante> labweb1002 - manually downgrade to scap 3.7.4-1 (disabled puppet) [production]
01:49 <mutante> reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz | copied it from jessie-wikimedia to trusty and stretch-wikimedia. all distributions downgraded to 3.7.4-1 (T183046) [production]
01:42 <mutante> reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz [production]
01:15 <mutante> reprepro removing scap 3.7.4-2 package, attempting to reimport 3.7.4-1 package [production]
01:11 <mutante> reprepro remove trusty-wikimedia scap [production]
00:44 <demon@tin> Synchronized php-1.31.0-wmf.12/extensions/LoginNotify/includes/LoginNotify.php: T182867 (duration: 00m 57s) [production]
00:40 <demon@tin> Synchronized README: Testing again, this time with feeling (duration: 00m 56s) [production]
00:28 <mutante> re-disabled puppet on labweb1001/labweb1002 (as it was before) [production]
00:26 <mutante> re-enabling puppet on scap hosts [production]
00:25 <demon@tin> Synchronized README: Testing (duration: 00m 57s) [production]
00:10 <mutante> no more scap 3.7.4-2 found across 'R:Package = scap' (T183046) [production]
2017-12-15 §
23:55 <mutante> downgrading scap from 3.7.4-2 to 3.7.4-1 where it is installed - cumin -b 10 -s 5 'R:Package = scap' 'if dpkg -l scap | grep "3.7.4.2" && file /var/cache/apt/archives/scap_3.7.4-1_all.deb; then puppet agent --disable; apt-get remove --yes -q scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb ; fi' targeting 478 hosts (T183046) [production]
23:37 <mutante> aqs1004, analytics1003, downgraded scap to 3.7.4-1 [production]
22:55 <mutante> tin - apt-get remove scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb [production]
19:39 <chasemp> reboot labtestvirt2003 [production]
18:15 <jgleeson> rolled back to civicrm to 798e2467 to investigate prometheus bug [production]
17:32 <jynus> stop, upgrade and reboot labsdb1009 [production]
17:18 <jynus> reloading dbproxy1010 [production]
16:44 <chasemp> labtestvirt2003:~# /sbin/reboot to pickup new kernel [production]
16:23 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight (duration: 00m 57s) [production]
16:10 <elukey> re-enable piwik on bohrium after mysql backup restore [production]
15:11 <chasemp> reimage labtestvirt2003.codfw.wmnet [production]
13:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and restore original weight for db1081 (duration: 00m 56s) [production]
13:33 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and db1081 (duration: 00m 56s) [production]
13:20 <chasemp> disable puppet across eqiad lab* things to land a bit of code gracefully [production]
13:10 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 56s) [production]
12:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T180788 (duration: 00m 57s) [production]
10:59 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174569 (duration: 00m 56s) [production]
10:59 <jynus> reloading dbproxy1011 [production]
10:50 <marostegui> Stop MySQL on db1084 to clone db1111 - T180788 [production]
10:49 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1084 to clone db1111 - T180788 (duration: 00m 56s) [production]
10:31 <elukey> rolling restart of yarn nodemanagers on an103* to apply new config - T182276 [production]
10:14 <moritzm> uploaded prometheus-dns-rec-exporter 0.3 to apt.wikimedia.org [production]
09:50 <elukey> restore piwik database on bohrium after mysql corruption - piwik disabled [production]