|
2025-10-30
ยง
|
| 15:37 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply |
[production] |
| 15:37 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply |
[production] |
| 15:36 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P84463 and previous config saved to /var/cache/conftool/dbconfig/20251030-153647-root.json |
[production] |
| 15:36 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply |
[production] |
| 15:36 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply |
[production] |
| 15:35 |
<dancy@deploy2002> |
Installation of scap version "4.221.0" completed for 165 hosts |
[production] |
| 15:34 |
<moritzm> |
installing imagemagick security updates |
[production] |
| 15:32 |
<moritzm> |
installing openjdk-21 security updates |
[production] |
| 15:31 |
<dancy@deploy2002> |
Installing scap version "4.221.0" for 165 host(s) |
[production] |
| 15:29 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P84462 and previous config saved to /var/cache/conftool/dbconfig/20251030-152913-marostegui.json |
[production] |
| 15:24 |
<cmooney@cumin1003> |
END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL |
[production] |
| 15:24 |
<cmooney@cumin1003> |
START - Cookbook sre.hosts.provision for host sretest1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL |
[production] |
| 15:21 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P84461 and previous config saved to /var/cache/conftool/dbconfig/20251030-152141-root.json |
[production] |
| 15:14 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P84460 and previous config saved to /var/cache/conftool/dbconfig/20251030-151405-marostegui.json |
[production] |
| 15:09 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2195 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P84459 and previous config saved to /var/cache/conftool/dbconfig/20251030-150946-root.json |
[production] |
| 15:06 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P84458 and previous config saved to /var/cache/conftool/dbconfig/20251030-150636-root.json |
[production] |
| 14:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1172 (T407997)', diff saved to https://phabricator.wikimedia.org/P84457 and previous config saved to /var/cache/conftool/dbconfig/20251030-145857-marostegui.json |
[production] |
| 14:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2170 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84456 and previous config saved to /var/cache/conftool/dbconfig/20251030-145831-marostegui.json |
[production] |
| 14:58 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2170.codfw.wmnet with reason: Maintenance |
[production] |
| 14:54 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2195 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P84455 and previous config saved to /var/cache/conftool/dbconfig/20251030-145440-root.json |
[production] |
| 14:51 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply |
[production] |
| 14:51 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply |
[production] |
| 14:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1172 (T407997)', diff saved to https://phabricator.wikimedia.org/P84454 and previous config saved to /var/cache/conftool/dbconfig/20251030-144452-marostegui.json |
[production] |
| 14:44 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:39 |
<fnegri@cumin1003> |
END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database pcmwikiquote (T408354) |
[production] |
| 14:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2195 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P84453 and previous config saved to /var/cache/conftool/dbconfig/20251030-143934-root.json |
[production] |
| 14:37 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dborch1001.wikimedia.org |
[production] |
| 14:33 |
<stevemunene@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply |
[production] |
| 14:32 |
<eevans@cumin1003> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Apply JVM upgrade to 11.0.29 - eevans@cumin1003 |
[production] |
| 14:32 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:32 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1167 (T407997)', diff saved to https://phabricator.wikimedia.org/P84452 and previous config saved to /var/cache/conftool/dbconfig/20251030-143204-marostegui.json |
[production] |
| 14:28 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host dborch1001.wikimedia.org |
[production] |
| 14:27 |
<klausman@deploy2002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:27 |
<klausman@deploy2002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 14:26 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:26 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 14:24 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2195 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P84451 and previous config saved to /var/cache/conftool/dbconfig/20251030-142428-root.json |
[production] |
| 14:23 |
<stevemunene@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply |
[production] |
| 14:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P84450 and previous config saved to /var/cache/conftool/dbconfig/20251030-141657-marostegui.json |
[production] |
| 14:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2195 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84449 and previous config saved to /var/cache/conftool/dbconfig/20251030-141638-marostegui.json |
[production] |
| 14:16 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2195.codfw.wmnet with reason: Maintenance |
[production] |
| 14:15 |
<aqu@deploy2002> |
Finished deploy [analytics/refinery@39e92e9] (thin): Update pageview allowlist THIN [analytics/refinery@39e92e9f] (duration: 01m 16s) |
[production] |
| 14:13 |
<aqu@deploy2002> |
Started deploy [analytics/refinery@39e92e9] (thin): Update pageview allowlist THIN [analytics/refinery@39e92e9f] |
[production] |
| 14:12 |
<aqu@deploy2002> |
Finished deploy [analytics/refinery@39e92e9]: Update pageview allowlist [analytics/refinery@39e92e9f] (duration: 03m 52s) |
[production] |
| 14:11 |
<mfossati@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1193703|Add feature flag for pilot wikis about visual changes coming from Wikibase having an icon. (T397258)]], [[gerrit:1199246|Disable default user-agent collection. (T384964)]], [[gerrit:1199725|[huwiki] Set $wgUploadNavigationUrl (T408298)]], [[gerrit:1199727|[ruwiki] Enable WikiLove extension (T408514)]], [[gerrit:1198626|core-Namespaces: Add R: |
[production] |
| 14:08 |
<aqu@deploy2002> |
Started deploy [analytics/refinery@39e92e9]: Update pageview allowlist [analytics/refinery@39e92e9f] |
[production] |
| 14:07 |
<aqu@deploy2002> |
Finished deploy [analytics/refinery@39e92e9] (hadoop-test): Update pageview allowlist TEST [analytics/refinery@39e92e9f] (duration: 01m 04s) |
[production] |
| 14:06 |
<aqu@deploy2002> |
Started deploy [analytics/refinery@39e92e9] (hadoop-test): Update pageview allowlist TEST [analytics/refinery@39e92e9f] |
[production] |
| 14:05 |
<mfossati@deploy2002> |
superpes, bunnypranav, javiermonton, mfossati, seanleong-wmde: Continuing with sync |
[production] |
| 14:01 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P84448 and previous config saved to /var/cache/conftool/dbconfig/20251030-140147-marostegui.json |
[production] |