|
2017-12-18
§
|
| 06:43 |
<_joe_> |
restarted hhvm on mw1283, still the same kind of lockups |
[production] |
| 06:33 |
<marostegui> |
Deploy schema change on db1073 - T174569 |
[production] |
| 06:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174569 (duration: 00m 57s) |
[production] |
| 06:28 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 56s) |
[production] |
| 06:17 |
<marostegui> |
Stop replication in sync on db1106 and db1100 - T161294 |
[production] |
| 06:17 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 57s) |
[production] |
| 03:02 |
<eileen> |
re-enable multiqueue_consumer process-control config revision is 1ae3778278 |
[production] |
| 03:00 |
<eileen> |
civicrm revision changed from 43d3f4d739 to e0ee2d189c |
[production] |
| 02:32 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 57s) |
[production] |
| 01:17 |
<cwd> |
disabled fredge multiqueue consumer |
[production] |
| 00:49 |
<eileen> |
civicrm revision changed from 798e24671b to 43d3f4d739 |
[production] |
|
2017-12-16
§
|
| 17:27 |
<no_justification> |
gerrit: Back, might see a few transient puppet failures if git pulls happened during the d/t, but should all recover |
[production] |
| 17:21 |
<no_justification> |
gerrit: halting service momentarily for account reindexing |
[production] |
| 15:00 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 58s) |
[production] |
| 12:29 |
<ariel@tin> |
Finished deploy [dumps/dumps@95dbfe6]: revert previous deploy (duration: 00m 02s) |
[production] |
| 12:29 |
<ariel@tin> |
Started deploy [dumps/dumps@95dbfe6]: revert previous deploy |
[production] |
| 12:16 |
<ariel@tin> |
Finished deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files (duration: 00m 02s) |
[production] |
| 12:16 |
<ariel@tin> |
Started deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files |
[production] |
| 02:04 |
<mutante> |
labweb1002 - manually downgrade to scap 3.7.4-1 (disabled puppet) |
[production] |
| 01:49 |
<mutante> |
reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz | copied it from jessie-wikimedia to trusty and stretch-wikimedia. all distributions downgraded to 3.7.4-1 (T183046) |
[production] |
| 01:42 |
<mutante> |
reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz |
[production] |
| 01:15 |
<mutante> |
reprepro removing scap 3.7.4-2 package, attempting to reimport 3.7.4-1 package |
[production] |
| 01:11 |
<mutante> |
reprepro remove trusty-wikimedia scap |
[production] |
| 00:44 |
<demon@tin> |
Synchronized php-1.31.0-wmf.12/extensions/LoginNotify/includes/LoginNotify.php: T182867 (duration: 00m 57s) |
[production] |
| 00:40 |
<demon@tin> |
Synchronized README: Testing again, this time with feeling (duration: 00m 56s) |
[production] |
| 00:28 |
<mutante> |
re-disabled puppet on labweb1001/labweb1002 (as it was before) |
[production] |
| 00:26 |
<mutante> |
re-enabling puppet on scap hosts |
[production] |
| 00:25 |
<demon@tin> |
Synchronized README: Testing (duration: 00m 57s) |
[production] |
| 00:10 |
<mutante> |
no more scap 3.7.4-2 found across 'R:Package = scap' (T183046) |
[production] |
|
2017-12-15
§
|
| 23:55 |
<mutante> |
downgrading scap from 3.7.4-2 to 3.7.4-1 where it is installed - cumin -b 10 -s 5 'R:Package = scap' 'if dpkg -l scap | grep "3.7.4.2" && file /var/cache/apt/archives/scap_3.7.4-1_all.deb; then puppet agent --disable; apt-get remove --yes -q scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb ; fi' targeting 478 hosts (T183046) |
[production] |
| 23:37 |
<mutante> |
aqs1004, analytics1003, downgraded scap to 3.7.4-1 |
[production] |
| 22:55 |
<mutante> |
tin - apt-get remove scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb |
[production] |
| 19:39 |
<chasemp> |
reboot labtestvirt2003 |
[production] |
| 18:15 |
<jgleeson> |
rolled back to civicrm to 798e2467 to investigate prometheus bug |
[production] |
| 17:32 |
<jynus> |
stop, upgrade and reboot labsdb1009 |
[production] |
| 17:18 |
<jynus> |
reloading dbproxy1010 |
[production] |
| 16:44 |
<chasemp> |
labtestvirt2003:~# /sbin/reboot to pickup new kernel |
[production] |
| 16:23 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight (duration: 00m 57s) |
[production] |
| 16:10 |
<elukey> |
re-enable piwik on bohrium after mysql backup restore |
[production] |
| 15:11 |
<chasemp> |
reimage labtestvirt2003.codfw.wmnet |
[production] |
| 13:47 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and restore original weight for db1081 (duration: 00m 56s) |
[production] |
| 13:33 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and db1081 (duration: 00m 56s) |
[production] |
| 13:20 |
<chasemp> |
disable puppet across eqiad lab* things to land a bit of code gracefully |
[production] |
| 13:10 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 56s) |
[production] |
| 12:47 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T180788 (duration: 00m 57s) |
[production] |
| 10:59 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174569 (duration: 00m 56s) |
[production] |
| 10:59 |
<jynus> |
reloading dbproxy1011 |
[production] |
| 10:50 |
<marostegui> |
Stop MySQL on db1084 to clone db1111 - T180788 |
[production] |
| 10:49 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1084 to clone db1111 - T180788 (duration: 00m 56s) |
[production] |
| 10:31 |
<elukey> |
rolling restart of yarn nodemanagers on an103* to apply new config - T182276 |
[production] |