251-300 of 10000 results (50ms)
2017-12-18 §
09:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1066 - T174569 (duration: 00m 56s) [production]
09:43 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174569 (duration: 00m 57s) [production]
09:40 <moritzm> installing openssl security updates [production]
09:39 <marostegui> Deploy schema change on db1055 (already depooled) - T174569 [production]
09:28 <gehel> removing initial import datafiles from maps[12]001 [production]
08:58 <marostegui> Stop replication in sync on db1100 and db2052 - T161294 [production]
08:57 <elukey> rolling restart of the Yarn nodemanagers (hadoop) on analytics10[456]* to pick up new settings - T182276 [production]
08:30 <Jamesofur> insert decryption key for 2017 Arb elections [production]
08:08 <volans> powercycling ganeti1005 [production]
06:57 <_joe_> reeanbling puppet across servers with scap [production]
06:44 <marostegui> Defragment s2 databases on db1102 - T172169 [production]
06:43 <_joe_> restarted hhvm on mw1283, still the same kind of lockups [production]
06:33 <marostegui> Deploy schema change on db1073 - T174569 [production]
06:32 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174569 (duration: 00m 57s) [production]
06:28 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 56s) [production]
06:17 <marostegui> Stop replication in sync on db1106 and db1100 - T161294 [production]
06:17 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 57s) [production]
03:02 <eileen> re-enable multiqueue_consumer process-control config revision is 1ae3778278 [production]
03:00 <eileen> civicrm revision changed from 43d3f4d739 to e0ee2d189c [production]
02:32 <l10nupdate@tin> scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 57s) [production]
01:17 <cwd> disabled fredge multiqueue consumer [production]
00:49 <eileen> civicrm revision changed from 798e24671b to 43d3f4d739 [production]
2017-12-16 §
17:27 <no_justification> gerrit: Back, might see a few transient puppet failures if git pulls happened during the d/t, but should all recover [production]
17:21 <no_justification> gerrit: halting service momentarily for account reindexing [production]
15:00 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 58s) [production]
12:29 <ariel@tin> Finished deploy [dumps/dumps@95dbfe6]: revert previous deploy (duration: 00m 02s) [production]
12:29 <ariel@tin> Started deploy [dumps/dumps@95dbfe6]: revert previous deploy [production]
12:16 <ariel@tin> Finished deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files (duration: 00m 02s) [production]
12:16 <ariel@tin> Started deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files [production]
02:04 <mutante> labweb1002 - manually downgrade to scap 3.7.4-1 (disabled puppet) [production]
01:49 <mutante> reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz | copied it from jessie-wikimedia to trusty and stretch-wikimedia. all distributions downgraded to 3.7.4-1 (T183046) [production]
01:42 <mutante> reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz [production]
01:15 <mutante> reprepro removing scap 3.7.4-2 package, attempting to reimport 3.7.4-1 package [production]
01:11 <mutante> reprepro remove trusty-wikimedia scap [production]
00:44 <demon@tin> Synchronized php-1.31.0-wmf.12/extensions/LoginNotify/includes/LoginNotify.php: T182867 (duration: 00m 57s) [production]
00:40 <demon@tin> Synchronized README: Testing again, this time with feeling (duration: 00m 56s) [production]
00:28 <mutante> re-disabled puppet on labweb1001/labweb1002 (as it was before) [production]
00:26 <mutante> re-enabling puppet on scap hosts [production]
00:25 <demon@tin> Synchronized README: Testing (duration: 00m 57s) [production]
00:10 <mutante> no more scap 3.7.4-2 found across 'R:Package = scap' (T183046) [production]
2017-12-15 §
23:55 <mutante> downgrading scap from 3.7.4-2 to 3.7.4-1 where it is installed - cumin -b 10 -s 5 'R:Package = scap' 'if dpkg -l scap | grep "3.7.4.2" && file /var/cache/apt/archives/scap_3.7.4-1_all.deb; then puppet agent --disable; apt-get remove --yes -q scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb ; fi' targeting 478 hosts (T183046) [production]
23:37 <mutante> aqs1004, analytics1003, downgraded scap to 3.7.4-1 [production]
22:55 <mutante> tin - apt-get remove scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb [production]
19:39 <chasemp> reboot labtestvirt2003 [production]
18:15 <jgleeson> rolled back to civicrm to 798e2467 to investigate prometheus bug [production]
17:32 <jynus> stop, upgrade and reboot labsdb1009 [production]
17:18 <jynus> reloading dbproxy1010 [production]
16:44 <chasemp> labtestvirt2003:~# /sbin/reboot to pickup new kernel [production]
16:23 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight (duration: 00m 57s) [production]
16:10 <elukey> re-enable piwik on bohrium after mysql backup restore [production]