|
2024-01-31
§
|
| 07:08 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet |
[production] |
| 07:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 75%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P55917 and previous config saved to /var/cache/conftool/dbconfig/20240131-070624-root.json |
[production] |
| 07:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2114 (re)pooling @ 1%: After Bookworm upgrade T354506', diff saved to https://phabricator.wikimedia.org/P55916 and previous config saved to /var/cache/conftool/dbconfig/20240131-070111-root.json |
[production] |
| 06:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2125 (T355609)', diff saved to https://phabricator.wikimedia.org/P55915 and previous config saved to /var/cache/conftool/dbconfig/20240131-065922-marostegui.json |
[production] |
| 06:59 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance |
[production] |
| 06:59 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance |
[production] |
| 06:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2107 (T355609)', diff saved to https://phabricator.wikimedia.org/P55914 and previous config saved to /var/cache/conftool/dbconfig/20240131-065901-marostegui.json |
[production] |
| 06:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2142.codfw.wmnet with reason: host reimage |
[production] |
| 06:54 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2114.codfw.wmnet with OS bookworm |
[production] |
| 06:53 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2142.codfw.wmnet with reason: host reimage |
[production] |
| 06:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 50%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P55913 and previous config saved to /var/cache/conftool/dbconfig/20240131-065118-root.json |
[production] |
| 06:47 |
<moritzm> |
installing glibc security updates on bookworm |
[production] |
| 06:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P55912 and previous config saved to /var/cache/conftool/dbconfig/20240131-064353-marostegui.json |
[production] |
| 06:39 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2114.codfw.wmnet with reason: host reimage |
[production] |
| 06:36 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2114.codfw.wmnet with reason: host reimage |
[production] |
| 06:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 25%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P55911 and previous config saved to /var/cache/conftool/dbconfig/20240131-063613-root.json |
[production] |
| 06:35 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db2142.codfw.wmnet with OS bookworm |
[production] |
| 06:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P55910 and previous config saved to /var/cache/conftool/dbconfig/20240131-062846-marostegui.json |
[production] |
| 06:22 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db2114.codfw.wmnet with OS bookworm |
[production] |
| 06:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 10%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P55909 and previous config saved to /var/cache/conftool/dbconfig/20240131-062109-root.json |
[production] |
| 06:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2114 T354506', diff saved to https://phabricator.wikimedia.org/P55908 and previous config saved to /var/cache/conftool/dbconfig/20240131-061932-root.json |
[production] |
| 06:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2107 (T355609)', diff saved to https://phabricator.wikimedia.org/P55907 and previous config saved to /var/cache/conftool/dbconfig/20240131-061340-marostegui.json |
[production] |
| 06:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 5%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P55906 and previous config saved to /var/cache/conftool/dbconfig/20240131-060602-root.json |
[production] |
| 06:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2107 (T355609)', diff saved to https://phabricator.wikimedia.org/P55905 and previous config saved to /var/cache/conftool/dbconfig/20240131-060337-marostegui.json |
[production] |
| 06:03 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2107.codfw.wmnet with reason: Maintenance |
[production] |
| 06:03 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2107.codfw.wmnet with reason: Maintenance |
[production] |
| 05:53 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance |
[production] |
| 05:53 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance |
[production] |
| 05:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 1%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P55904 and previous config saved to /var/cache/conftool/dbconfig/20240131-055057-root.json |
[production] |
| 05:41 |
<eileen> |
civicrm upgraded from 6de61520 to 520337a0 |
[production] |
| 05:30 |
<fab@deploy2002> |
Finished deploy [airflow-dags/research@97c6a4e]: (no justification provided) (duration: 00m 14s) |
[production] |
| 05:30 |
<fab@deploy2002> |
Started deploy [airflow-dags/research@97c6a4e]: (no justification provided) |
[production] |
| 03:29 |
<eileen> |
tools upgraded from 02281338 to c823e692 |
[production] |
| 03:05 |
<fab@deploy2002> |
Finished deploy [airflow-dags/research@6a97a34]: (no justification provided) (duration: 00m 23s) |
[production] |
| 03:05 |
<fab@deploy2002> |
Started deploy [airflow-dags/research@6a97a34]: (no justification provided) |
[production] |
|
2024-01-30
§
|
| 23:54 |
<mutante> |
LDAP - added aklapper to group releng T356043 |
[production] |
| 23:07 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for sessionstore1006.eqiad.wmnet |
[production] |
| 23:07 |
<eevans@cumin1002> |
START - Cookbook sre.hosts.remove-downtime for sessionstore1006.eqiad.wmnet |
[production] |
| 22:49 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on sessionstore1006.eqiad.wmnet with reason: Bootstrapping — T353402 |
[production] |
| 22:48 |
<eevans@cumin1002> |
START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on sessionstore1006.eqiad.wmnet with reason: Bootstrapping — T353402 |
[production] |
| 22:40 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate first private IP host config - bking@cumin2002 - T355617 |
[production] |
| 22:20 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for sessionstore1005.eqiad.wmnet |
[production] |
| 22:20 |
<eevans@cumin1002> |
START - Cookbook sre.hosts.remove-downtime for sessionstore1005.eqiad.wmnet |
[production] |
| 22:10 |
<cjming> |
end of UTC late backport window |
[production] |
| 22:09 |
<cjming@deploy2002> |
Finished scap: Backport for [[gerrit:994254|[eswiki] Add 13 namespaces to $wgExemptFromUserRobotsControl (T355033)]] (duration: 08m 24s) |
[production] |
| 22:02 |
<cjming@deploy2002> |
cjming and superpes: Continuing with sync |
[production] |
| 22:02 |
<cjming@deploy2002> |
cjming and superpes: Backport for [[gerrit:994254|[eswiki] Add 13 namespaces to $wgExemptFromUserRobotsControl (T355033)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 22:00 |
<cjming@deploy2002> |
Started scap: Backport for [[gerrit:994254|[eswiki] Add 13 namespaces to $wgExemptFromUserRobotsControl (T355033)]] |
[production] |
| 21:59 |
<cjming@deploy2002> |
Finished scap: Backport for [[gerrit:994211|[ukwiki] Change autoconfirmed setting (T355972)]], [[gerrit:994214|[ganwiki] Add 'suppressredirect' to transwiki usergroup and change assignment and revocation methods (T354850)]], [[gerrit:994220|[ganwiki] Add new namespace aliases (T355854)]] (duration: 09m 32s) |
[production] |
| 21:53 |
<cjming@deploy2002> |
superpes and cjming: Continuing with sync |
[production] |