|
2018-09-19
§
|
| 12:50 |
<marostegui> |
Deploy schema change on s3 eqiad master on mediawikiwiki (this will generate some lag on s3 eqiad) - T51593 |
[production] |
| 12:18 |
<gehel> |
rolling restart of elasticsearch / logstash for new systemd unit |
[production] |
| 12:09 |
<gehel> |
rolling restart of relforge for new systemd unit |
[production] |
| 12:07 |
<gehel> |
stopping puppet on all elasticsearch servers to deploy new systemd unit - https://gerrit.wikimedia.org/r/c/operations/puppet/+/440498 |
[production] |
| 12:00 |
<moritzm> |
reimaging mw1298 (spare host) to test reimages from cumin2001 |
[production] |
| 10:26 |
<marostegui> |
Deploy schema change on x1:testwiki - T51593 |
[production] |
| 10:13 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 57s) |
[production] |
| 10:11 |
<banyek@deploy1001> |
Synchronized wmf-config/db-codfw.php: Fully pool db2041 back into production (duration: 00m 57s) |
[production] |
| 10:07 |
<banyek@deploy1001> |
Synchronized wmf-config/db-eqiad.php: T203565: Weight adjust for S1 API hosts (duration: 00m 57s) |
[production] |
| 10:03 |
<jynus> |
fixing db2033 grants |
[production] |
| 09:41 |
<marostegui> |
Upgrade MariaDB and kernel on db1110 |
[production] |
| 09:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2084:3315 db2053 db1096:3316 (duration: 00m 58s) |
[production] |
| 09:39 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db2084:3315 db2053 db1096:3316 (duration: 00m 59s) |
[production] |
| 09:26 |
<godog> |
repair sdf on ms-be2043 - T199198 |
[production] |
| 09:14 |
<moritzm> |
rebooting mc1021 for kernel security update |
[production] |
| 09:04 |
<moritzm> |
rebooting mc1020 for kernel security update |
[production] |
| 09:02 |
<moritzm> |
updating intel-microcode on Debian jessie/stretch to 3.20180807a.1 |
[production] |
| 08:39 |
<moritzm> |
rebooting mc1019 for kernel security update |
[production] |
| 08:35 |
<marostegui> |
Stop MySQL on db1110 for alert testing - T200509 |
[production] |
| 08:04 |
<moritzm> |
bounced ferm service on elastic2004 |
[production] |
| 07:53 |
<marostegui> |
Stop MySQL on db1110 for alert testing - T200509 |
[production] |
| 07:45 |
<marostegui> |
Stop MySQL on db1096:3316 for alert testing - T200509 |
[production] |
| 07:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db2084:3315 db2053 db1110 db1096:3316 db1110 (duration: 00m 57s) |
[production] |
| 07:30 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2084:3315 db2053 db1110 db1096:3316 db1110 (duration: 00m 57s) |
[production] |
| 07:26 |
<moritzm> |
reimaging mw2245 (spare host) to test reimages from cumin2001 (router policies have been updated) |
[production] |
| 07:25 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 57s) |
[production] |
| 07:12 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 58s) |
[production] |
| 07:12 |
<marostegui> |
Disable puppet on databases to test new alerts - T200509 https://phabricator.wikimedia.org/T172489 |
[production] |
| 07:10 |
<jynus> |
restart db1109 for upgrade |
[production] |
| 06:56 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2041 with low load (duration: 00m 54s) |
[production] |
| 06:00 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2036 (duration: 00m 58s) |
[production] |
| 05:53 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2036 (duration: 00m 57s) |
[production] |
| 05:49 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2050 (duration: 00m 58s) |
[production] |
| 05:41 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2050 (duration: 00m 57s) |
[production] |
| 05:37 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2074 (duration: 00m 57s) |
[production] |
| 05:31 |
<marostegui> |
Drop echo tables from s3 db2074 - T153638 |
[production] |
| 05:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2074 (duration: 00m 57s) |
[production] |
| 05:25 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2057 (duration: 00m 58s) |
[production] |
| 05:19 |
<marostegui> |
Drop echo tables from s7:kowiki - T153638 |
[production] |
| 05:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2057 (duration: 02m 13s) |
[production] |
| 02:34 |
<twentyafterfour@deploy1001> |
Finished scap: SWAT: full sync to update l10n for https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaMessages/+/461230/ (duration: 61m 20s) |
[production] |
| 01:33 |
<twentyafterfour@deploy1001> |
Started scap: SWAT: full sync to update l10n for https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaMessages/+/461230/ |
[production] |
| 01:28 |
<twentyafterfour@deploy1001> |
Synchronized php-1.32.0-wmf.22/resources/src/mediawiki.util.js: SWAT: sync https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/461227/ (duration: 01m 00s) |
[production] |
|
2018-09-18
§
|
| 23:21 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/: SWAT: tested on mwdebug1001, now syncing wmf-config settings to the whole cluster (duration: 00m 59s) |
[production] |
| 23:15 |
<twentyafterfour> |
SWAT: deploying 3 patches for ebernhardson: 77deae12f 7009fe473 and 12cc420fc |
[production] |
| 22:10 |
<bblack> |
depooled ulsfo, some uknown router issue |
[production] |
| 21:42 |
<bblack> |
authdns1001 seems stable/fine running gdnsd-2.99.9-beta so far. If issues crop up later, don't hesitate to (a) downgrade back to stretch-backports gdnsd-2.3.0-1~bpo9+1 or (b) call me! |
[production] |
| 21:25 |
<bblack> |
authdns1001: testing gdnsd version update (2.99.9-beta) |
[production] |
| 20:58 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Set ActorTableSchemaMigrationStage back to OLD on test wikis, mediawikiwiki (T188327, T204669) (duration: 00m 57s) |
[production] |
| 20:57 |
<catrope@deploy1001> |
sync-file aborted: Set ActorTableSchemaMigrationStage back to OLD on test wikis, mediawikiwiki (duration: 00m 10s) |
[production] |