|
2021-07-12
§
|
| 09:18 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1006.eqiad.wmnet |
[production] |
| 09:12 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1183.eqiad.wmnet with reason: REIMAGE |
[production] |
| 09:10 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1183.eqiad.wmnet with reason: REIMAGE |
[production] |
| 09:07 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host rdb1006.eqiad.wmnet |
[production] |
| 09:07 |
<godog> |
repool thanos-fe2002 - T285835 |
[production] |
| 08:38 |
<godog> |
test a single frontend for thanos-swift / thanos-query to test "bad host" theory - T285835 |
[production] |
| 08:26 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.12/extensions/Wikibase/client: Backport: [[gerrit:703890|Remove subscribing to other aspect for entity usage (T286193)]] (duration: 00m 59s) |
[production] |
| 07:44 |
<jynus> |
restart db1102:x1 mariadb instance |
[production] |
| 07:01 |
<moritzm> |
installing apache2 security updates |
[production] |
| 05:14 |
<Amir1> |
start of mwscript refreshImageMetadata.php --wiki=commonswiki --mediatype=OFFICE --batch-size=10 --verbose --mime="application/pdf" --force --sleep 5 on screen - It will take days / week to finish (T275268) |
[production] |
| 05:06 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/filebackend.php: Config: [[gerrit:703951|Enable json image metadata everywhere (T275268)]] (duration: 01m 05s) |
[production] |
| 04:56 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.12/maintenance/refreshImageMetadata.php: Backport: [[gerrit:703891|Add --sleep option to refreshImageMetadata.php]] (duration: 01m 04s) |
[production] |
| 04:10 |
<Amir1> |
mwscript refreshImageMetadata.php --wiki=testcommonswiki --mediatype=OFFICE --batch-size=20 --verbose --mime="application/pdf" --force (T275268) |
[production] |
| 04:08 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/filebackend.php: Config: [[gerrit:703950|Set testcommonswiki to use json image metadata (T275268)]] (duration: 01m 10s) |
[production] |
|
2021-07-09
§
|
| 23:28 |
<legoktm@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' . |
[production] |
| 23:27 |
<legoktm@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' . |
[production] |
| 22:36 |
<legoktm> |
running benchmarking scripts again shellbox |
[production] |
| 14:49 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@cdb3fc5] (hadoop-test): Deploy for finalize event_default_test gobblin job in hadoop test - T271232 (duration: 03m 08s) |
[production] |
| 14:46 |
<otto@deploy1002> |
Started deploy [analytics/refinery@cdb3fc5] (hadoop-test): Deploy for finalize event_default_test gobblin job in hadoop test - T271232 |
[production] |
| 11:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1118', diff saved to https://phabricator.wikimedia.org/P16809 and previous config saved to /var/cache/conftool/dbconfig/20210709-115609-marostegui.json |
[production] |
| 11:40 |
<_joe_> |
deleting coredns pod in codfw, potentially causing T286360 |
[production] |
| 10:13 |
<_joe_> |
recreated all pods for zotero in codfw |
[production] |
| 00:47 |
<legoktm> |
zotero rolling restart didn't help, filed T286360 for DNS issues |
[production] |
| 00:39 |
<legoktm> |
doing a rolling restart of zotero in codfw to hopefully fix DNS ENOTFOUND issues |
[production] |
|
2021-07-08
§
|
| 22:48 |
<legoktm@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Add configuration to use Score with Shellbox (still disabled) (2/2) - T281423 (duration: 00m 57s) |
[production] |
| 22:46 |
<legoktm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Add configuration to use Score with Shellbox (still disabled) (1/2) - T281423 (duration: 00m 58s) |
[production] |
| 19:29 |
<legoktm@deploy1002> |
Synchronized php-1.37.0-wmf.12/extensions/Score/includes/Score.php: Allow setting a different path for `convert` just for Score (2/2) (duration: 00m 57s) |
[production] |
| 19:27 |
<legoktm@deploy1002> |
Synchronized php-1.37.0-wmf.12/extensions/Score/extension.json: Allow setting a different path for `convert` just for Score (1/2) (duration: 00m 58s) |
[production] |
| 18:56 |
<legoktm@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' . |
[production] |
| 18:55 |
<legoktm@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' . |
[production] |
| 18:53 |
<legoktm@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'shellbox' for release 'main' . |
[production] |
| 17:02 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@51a73f1] (hadoop-test): Analytics deploy for Gobblin replacing Camus - hadoop-test [analytics/refinery@51a73f1] (duration: 05m 38s) |
[production] |
| 16:56 |
<joal@deploy1002> |
Started deploy [analytics/refinery@51a73f1] (hadoop-test): Analytics deploy for Gobblin replacing Camus - hadoop-test [analytics/refinery@51a73f1] |
[production] |
| 16:47 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@51a73f1]: Analytics deploy for Gobblin replacing Camus - an-launcher1002 only [analytics/refinery@51a73f1] (duration: 03m 17s) |
[production] |
| 16:44 |
<joal@deploy1002> |
Started deploy [analytics/refinery@51a73f1]: Analytics deploy for Gobblin replacing Camus - an-launcher1002 only [analytics/refinery@51a73f1] |
[production] |
| 15:37 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@9883dbf] (hadoop-test): Deploy for event_default_test job in hadoop test - T271232 (duration: 03m 06s) |
[production] |
| 15:34 |
<otto@deploy1002> |
Started deploy [analytics/refinery@9883dbf] (hadoop-test): Deploy for event_default_test job in hadoop test - T271232 |
[production] |
| 15:29 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@51f4696] (hadoop-test): Deploy for eventlogging_legacy gobblin with final import path - T271232 (duration: 05m 27s) |
[production] |
| 15:23 |
<otto@deploy1002> |
Started deploy [analytics/refinery@51f4696] (hadoop-test): Deploy for eventlogging_legacy gobblin with final import path - T271232 |
[production] |
| 15:11 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@42541e6] (hadoop-test): Deploy for eventlogging_legacy gobblin migration - T271232 (duration: 05m 42s) |
[production] |
| 15:05 |
<otto@deploy1002> |
Started deploy [analytics/refinery@42541e6] (hadoop-test): Deploy for eventlogging_legacy gobblin migration - T271232 |
[production] |
| 14:52 |
<otto@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Add consumers.analytics_hadoop-ingestion stream config settings for automated gobblin imports - T271232 T273901 (duration: 01m 09s) |
[production] |
| 13:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2103 (re)pooling @ 100%: Repool after index change', diff saved to https://phabricator.wikimedia.org/P16807 and previous config saved to /var/cache/conftool/dbconfig/20210708-134421-root.json |
[production] |
| 13:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2103 (re)pooling @ 75%: Repool after index change', diff saved to https://phabricator.wikimedia.org/P16806 and previous config saved to /var/cache/conftool/dbconfig/20210708-132917-root.json |
[production] |
| 13:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2103 (re)pooling @ 50%: Repool after index change', diff saved to https://phabricator.wikimedia.org/P16805 and previous config saved to /var/cache/conftool/dbconfig/20210708-131414-root.json |
[production] |
| 13:04 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@2d4c645]: Make gobblin-netflow use production directory - T271232 (duration: 03m 22s) |
[production] |
| 13:01 |
<otto@deploy1002> |
Started deploy [analytics/refinery@2d4c645]: Make gobblin-netflow use production directory - T271232 |
[production] |
| 12:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2103 (re)pooling @ 25%: Repool after index change', diff saved to https://phabricator.wikimedia.org/P16804 and previous config saved to /var/cache/conftool/dbconfig/20210708-125910-root.json |
[production] |
| 12:52 |
<moritzm> |
installing klibc security updates on buster |
[production] |
| 12:38 |
<moritzm> |
installing openexr security updates |
[production] |