|
2025-06-20
§
|
| 06:48 |
<jmm@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2023.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" |
[production] |
| 06:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P78507 and previous config saved to /var/cache/conftool/dbconfig/20250620-064652-marostegui.json |
[production] |
| 06:44 |
<jmm@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 06:40 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.decommission for hosts ganeti2023.codfw.wmnet |
[production] |
| 06:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2172 (T396130)', diff saved to https://phabricator.wikimedia.org/P78506 and previous config saved to /var/cache/conftool/dbconfig/20250620-063145-marostegui.json |
[production] |
| 06:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2172 (T396130)', diff saved to https://phabricator.wikimedia.org/P78504 and previous config saved to /var/cache/conftool/dbconfig/20250620-062307-marostegui.json |
[production] |
| 06:23 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2172.codfw.wmnet with reason: Maintenance |
[production] |
| 06:17 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 06:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T396130)', diff saved to https://phabricator.wikimedia.org/P78503 and previous config saved to /var/cache/conftool/dbconfig/20250620-061659-marostegui.json |
[production] |
| 06:02 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-staging-etcd2002.codfw.wmnet to plain |
[production] |
| 06:02 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.changedisk for changing disk type of ml-staging-etcd2002.codfw.wmnet to plain |
[production] |
| 06:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P78502 and previous config saved to /var/cache/conftool/dbconfig/20250620-060151-marostegui.json |
[production] |
| 05:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P78501 and previous config saved to /var/cache/conftool/dbconfig/20250620-054644-marostegui.json |
[production] |
| 05:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T396130)', diff saved to https://phabricator.wikimedia.org/P78500 and previous config saved to /var/cache/conftool/dbconfig/20250620-053137-marostegui.json |
[production] |
| 05:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2147 (T396130)', diff saved to https://phabricator.wikimedia.org/P78499 and previous config saved to /var/cache/conftool/dbconfig/20250620-052300-marostegui.json |
[production] |
| 05:22 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2147.codfw.wmnet with reason: Maintenance |
[production] |
| 05:21 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Upgrade |
[production] |
| 05:18 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1236.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:10 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc1012.eqiad.wmnet with reason: Maintenance |
[production] |
|
2025-06-19
§
|
| 23:40 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1253 (T396130)', diff saved to https://phabricator.wikimedia.org/P78498 and previous config saved to /var/cache/conftool/dbconfig/20250619-234009-marostegui.json |
[production] |
| 23:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P78497 and previous config saved to /var/cache/conftool/dbconfig/20250619-232502-marostegui.json |
[production] |
| 23:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P78496 and previous config saved to /var/cache/conftool/dbconfig/20250619-230955-marostegui.json |
[production] |
| 22:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1253 (T396130)', diff saved to https://phabricator.wikimedia.org/P78495 and previous config saved to /var/cache/conftool/dbconfig/20250619-225448-marostegui.json |
[production] |
| 22:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1253 (T396130)', diff saved to https://phabricator.wikimedia.org/P78494 and previous config saved to /var/cache/conftool/dbconfig/20250619-224901-marostegui.json |
[production] |
| 22:48 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1253.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1227 (T396130)', diff saved to https://phabricator.wikimedia.org/P78493 and previous config saved to /var/cache/conftool/dbconfig/20250619-224839-marostegui.json |
[production] |
| 22:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P78492 and previous config saved to /var/cache/conftool/dbconfig/20250619-223332-marostegui.json |
[production] |
| 22:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P78491 and previous config saved to /var/cache/conftool/dbconfig/20250619-221824-marostegui.json |
[production] |
| 22:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1227 (T396130)', diff saved to https://phabricator.wikimedia.org/P78490 and previous config saved to /var/cache/conftool/dbconfig/20250619-220317-marostegui.json |
[production] |
| 21:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1227 (T396130)', diff saved to https://phabricator.wikimedia.org/P78489 and previous config saved to /var/cache/conftool/dbconfig/20250619-214540-marostegui.json |
[production] |
| 21:45 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1227.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1202 (T396130)', diff saved to https://phabricator.wikimedia.org/P78488 and previous config saved to /var/cache/conftool/dbconfig/20250619-214517-marostegui.json |
[production] |
| 21:39 |
<kostajh> |
mwscript-k8s -f --comment="T397483" -- updateCollation.php --wiki=plwikiquote --previous-collation=uppercase |
[production] |
| 21:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P78487 and previous config saved to /var/cache/conftool/dbconfig/20250619-213010-marostegui.json |
[production] |
| 21:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P78486 and previous config saved to /var/cache/conftool/dbconfig/20250619-211502-marostegui.json |
[production] |
| 20:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1202 (T396130)', diff saved to https://phabricator.wikimedia.org/P78485 and previous config saved to /var/cache/conftool/dbconfig/20250619-205955-marostegui.json |
[production] |
| 20:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1202 (T396130)', diff saved to https://phabricator.wikimedia.org/P78484 and previous config saved to /var/cache/conftool/dbconfig/20250619-205505-marostegui.json |
[production] |
| 20:54 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T396130)', diff saved to https://phabricator.wikimedia.org/P78483 and previous config saved to /var/cache/conftool/dbconfig/20250619-205443-marostegui.json |
[production] |
| 20:40 |
<kostajh> |
UTC late deploys done |
[production] |
| 20:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P78482 and previous config saved to /var/cache/conftool/dbconfig/20250619-203935-marostegui.json |
[production] |
| 20:36 |
<kharlan@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1161588|PageChangeEmissionTest: order move events by kind. (T397087)]], [[gerrit:1161589|DomainEvents: Constant repeating notifications (T397103)]] (duration: 10m 08s) |
[production] |
| 20:29 |
<kharlan@deploy1003> |
kharlan, matmarex: Continuing with sync |
[production] |
| 20:28 |
<kharlan@deploy1003> |
kharlan, matmarex: Backport for [[gerrit:1161588|PageChangeEmissionTest: order move events by kind. (T397087)]], [[gerrit:1161589|DomainEvents: Constant repeating notifications (T397103)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 20:26 |
<kharlan@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1161588|PageChangeEmissionTest: order move events by kind. (T397087)]], [[gerrit:1161589|DomainEvents: Constant repeating notifications (T397103)]] |
[production] |
| 20:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P78481 and previous config saved to /var/cache/conftool/dbconfig/20250619-202428-marostegui.json |
[production] |
| 20:15 |
<kharlan@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1159626|Configure instrument for CheckUser - UserInfoCard (T386440)]], [[gerrit:1161581|Set category collation to `uca-pl-u-kn` for plwikiquote (T397466)]] (duration: 10m 30s) |
[production] |
| 20:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T396130)', diff saved to https://phabricator.wikimedia.org/P78480 and previous config saved to /var/cache/conftool/dbconfig/20250619-200921-marostegui.json |
[production] |
| 20:08 |
<kharlan@deploy1003> |
kharlan, msz2001, mimurawil: Continuing with sync |
[production] |