|
2020-07-06
§
|
| 08:24 |
<kormat> |
restarting all mariadb instances on sanitarium hosts T256545 |
[production] |
| 08:09 |
<elukey> |
roll restart aqs on aqs100[4-9] to pick up new druid settings |
[production] |
| 08:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1076', diff saved to https://phabricator.wikimedia.org/P11742 and previous config saved to /var/cache/conftool/dbconfig/20200706-080509-marostegui.json |
[production] |
| 07:58 |
<qchris> |
Disable puppet on gerrit1002 (gerrit-test) to deploy Gerrit UI updates there to gather more feedback |
[production] |
| 07:51 |
<elukey> |
enable binlog on matomo's database on matomo1002 |
[production] |
| 07:46 |
<XioNoX> |
repool eqsin - T257154 |
[production] |
| 07:11 |
<XioNoX> |
reboot cr3-eqsin - T257154 |
[production] |
| 06:55 |
<XioNoX> |
depool eqsin for cr3-eqsin reboot/investigation - T257154 |
[production] |
| 06:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1089', diff saved to https://phabricator.wikimedia.org/P11740 and previous config saved to /var/cache/conftool/dbconfig/20200706-065437-marostegui.json |
[production] |
| 06:54 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hadoop.change-distro (exit_code=99) |
[production] |
| 06:22 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro |
[production] |
| 06:21 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) |
[production] |
| 06:14 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster |
[production] |
| 05:45 |
<kart_> |
Updated cxserver to 2020-07-01-044435-production (T254143) |
[production] |
| 05:40 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:36 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:32 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 05:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P11739 and previous config saved to /var/cache/conftool/dbconfig/20200706-051333-marostegui.json |
[production] |
| 05:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P11738 and previous config saved to /var/cache/conftool/dbconfig/20200706-050347-marostegui.json |
[production] |
| 04:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P11737 and previous config saved to /var/cache/conftool/dbconfig/20200706-044908-marostegui.json |
[production] |
|
2020-07-05
§
|
| 21:50 |
<qchris> |
Restarting gerrit on gerrit1001 to pick up new war and jars. |
[production] |
| 21:50 |
<qchris@deploy1001> |
Finished deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13, zuul plugin to master-0-g7accc67, and gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1001 (duration: 00m 07s) |
[production] |
| 21:50 |
<qchris@deploy1001> |
Started deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13, zuul plugin to master-0-g7accc67, and gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1001 |
[production] |
| 21:46 |
<qchris> |
Restarting gerrit on gerrit2001 to pick up new war and jars. |
[production] |
| 21:45 |
<qchris@deploy1001> |
Finished deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13, zuul plugin to master-0-g7accc67, and gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit2001 (duration: 00m 10s) |
[production] |
| 21:45 |
<qchris@deploy1001> |
Started deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13, zuul plugin to master-0-g7accc67, and gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit2001 |
[production] |
| 21:32 |
<qchris> |
Restarting gerrit on gerrit1002 to pick up new wars and jars. |
[production] |
| 21:32 |
<qchris@deploy1001> |
Finished deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13 and zuul plugin to master-0-g7accc67 (duration: 00m 08s) |
[production] |
| 21:32 |
<qchris@deploy1001> |
Started deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13 and zuul plugin to master-0-g7accc67 |
[production] |
| 21:20 |
<qchris> |
Enable puppet on gerrit1002 (gerrit-test) again to let it catch up again |
[production] |
| 16:01 |
<gehel> |
restart elastic-psi on elastic1052 (high GC rate) |
[production] |
| 15:56 |
<gehel> |
restart blazegraph + updater on wdqs1007 and depool to allow catching up on lag |
[production] |
|
2020-07-03
§
|
| 21:49 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.39/extensions/Score/: Sync maintenance script (duration: 00m 58s) |
[production] |
| 18:47 |
<cdanis> |
✔️ cdanis@an-coord1001.eqiad.wmnet ~ 🕒☕ sudo systemctl restart hive-server2.service |
[production] |
| 16:51 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Ifa929b2ad4 (duration: 00m 57s) |
[production] |
| 16:02 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Rename wgRestrictionMethod to wgShellRestrictionMethod (duration: 00m 58s) |
[production] |
| 15:46 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:43 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:43 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Reduce db1118 weight to spread load mode evenly', diff saved to https://phabricator.wikimedia.org/P11730 and previous config saved to /var/cache/conftool/dbconfig/20200703-154337-jynus.json |
[production] |
| 15:40 |
<jayme@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:38 |
<jayme@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:09 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) |
[production] |
| 15:02 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster |
[production] |
| 14:11 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hadoop.stop-cluster (exit_code=99) |
[production] |
| 14:11 |
<_joe_> |
restarted php-fpm on wtp1033, stuck in sigill |
[production] |