251-300 of 10000 results (39ms)
2017-12-16 §
02:04 <mutante> labweb1002 - manually downgrade to scap 3.7.4-1 (disabled puppet) [production]
01:49 <mutante> reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz | copied it from jessie-wikimedia to trusty and stretch-wikimedia. all distributions downgraded to 3.7.4-1 (T183046) [production]
01:42 <mutante> reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz [production]
01:15 <mutante> reprepro removing scap 3.7.4-2 package, attempting to reimport 3.7.4-1 package [production]
01:11 <mutante> reprepro remove trusty-wikimedia scap [production]
00:44 <demon@tin> Synchronized php-1.31.0-wmf.12/extensions/LoginNotify/includes/LoginNotify.php: T182867 (duration: 00m 57s) [production]
00:40 <demon@tin> Synchronized README: Testing again, this time with feeling (duration: 00m 56s) [production]
00:28 <mutante> re-disabled puppet on labweb1001/labweb1002 (as it was before) [production]
00:26 <mutante> re-enabling puppet on scap hosts [production]
00:25 <demon@tin> Synchronized README: Testing (duration: 00m 57s) [production]
00:10 <mutante> no more scap 3.7.4-2 found across 'R:Package = scap' (T183046) [production]
2017-12-15 §
23:55 <mutante> downgrading scap from 3.7.4-2 to 3.7.4-1 where it is installed - cumin -b 10 -s 5 'R:Package = scap' 'if dpkg -l scap | grep "3.7.4.2" && file /var/cache/apt/archives/scap_3.7.4-1_all.deb; then puppet agent --disable; apt-get remove --yes -q scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb ; fi' targeting 478 hosts (T183046) [production]
23:37 <mutante> aqs1004, analytics1003, downgraded scap to 3.7.4-1 [production]
22:55 <mutante> tin - apt-get remove scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb [production]
19:39 <chasemp> reboot labtestvirt2003 [production]
18:15 <jgleeson> rolled back to civicrm to 798e2467 to investigate prometheus bug [production]
17:32 <jynus> stop, upgrade and reboot labsdb1009 [production]
17:18 <jynus> reloading dbproxy1010 [production]
16:44 <chasemp> labtestvirt2003:~# /sbin/reboot to pickup new kernel [production]
16:23 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight (duration: 00m 57s) [production]
16:10 <elukey> re-enable piwik on bohrium after mysql backup restore [production]
15:11 <chasemp> reimage labtestvirt2003.codfw.wmnet [production]
13:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and restore original weight for db1081 (duration: 00m 56s) [production]
13:33 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and db1081 (duration: 00m 56s) [production]
13:20 <chasemp> disable puppet across eqiad lab* things to land a bit of code gracefully [production]
13:10 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 56s) [production]
12:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T180788 (duration: 00m 57s) [production]
10:59 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174569 (duration: 00m 56s) [production]
10:59 <jynus> reloading dbproxy1011 [production]
10:50 <marostegui> Stop MySQL on db1084 to clone db1111 - T180788 [production]
10:49 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1084 to clone db1111 - T180788 (duration: 00m 56s) [production]
10:31 <elukey> rolling restart of yarn nodemanagers on an103* to apply new config - T182276 [production]
10:14 <moritzm> uploaded prometheus-dns-rec-exporter 0.3 to apt.wikimedia.org [production]
09:50 <elukey> restore piwik database on bohrium after mysql corruption - piwik disabled [production]
09:26 <marostegui> Deploy schema change on db1080 - T174569 [production]
09:26 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174569 (duration: 00m 56s) [production]
09:10 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174569 (duration: 00m 56s) [production]
08:50 <moritzm> powercycling ganeti1005 [production]
08:38 <marostegui> Stop MySQL on db1034 to decommission it - T182556 [production]
08:26 <marostegui> Remove db1034 from tendril - T182556 [production]
06:50 <marostegui> Deploy schema change on db1083 - T174569 [production]
06:49 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174569 (duration: 00m 56s) [production]
06:43 <marostegui> Deploy schema change on dbstore1001 (s1) - T174569 [production]
06:29 <marostegui> Fix dbstore1001 s5 replication [production]
06:26 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 01m 16s) [production]
03:01 <mutante> db2016 thru db2019 - had to manually kill gmond process to decom ganglia, other db codfw hosts: didnt need it | running puppet on db205* and others in codfw to remove all ganglia (T177225) [production]
01:23 <catrope@tin> Synchronized php-1.31.0-wmf.12/extensions/ORES: Fix broken join conditions (T182936) (duration: 00m 57s) [production]
01:22 <hoo> Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds [production]
01:19 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Don't discount file searches on commonswiki (duration: 00m 57s) [production]
01:11 <catrope@tin> Synchronized php-1.31.0-wmf.11: Pulling in today's cherry-picks into wmf.11 too (duration: 10m 49s) [production]