|
2023-02-08
ยง
|
| 17:09 |
<jynus> |
disable bacula job backup1002.eqiad.wmnet-Weekly-Thu-EsRwCodfw-mysql-srv-backups-dumps-latest |
[production] |
| 17:08 |
<oblivian@cumin2002> |
END (FAIL) - Cookbook sre.discovery.datacenter (exit_code=93) status all services in codfw: maintenance |
[production] |
| 17:08 |
<oblivian@cumin2002> |
START - Cookbook sre.discovery.datacenter status all services in codfw: maintenance |
[production] |
| 16:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P43844 and previous config saved to /var/cache/conftool/dbconfig/20230208-165500-marostegui.json |
[production] |
| 16:47 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:47 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:45 |
<ladsgroup@cumin1001> |
END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:45 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<sukhe> |
[done] rolling restart of haproxy and trafficserver in A:cp |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:43 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P43842 and previous config saved to /var/cache/conftool/dbconfig/20230208-163954-marostegui.json |
[production] |
| 16:27 |
<sukhe> |
rolling restart of haproxy and trafficserver in A:cp |
[production] |
| 16:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43841 and previous config saved to /var/cache/conftool/dbconfig/20230208-162447-marostegui.json |
[production] |
| 16:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2138:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43840 and previous config saved to /var/cache/conftool/dbconfig/20230208-161828-marostegui.json |
[production] |
| 16:18 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance |
[production] |
| 16:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance |
[production] |
| 16:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2126 (T328817)', diff saved to https://phabricator.wikimedia.org/P43839 and previous config saved to /var/cache/conftool/dbconfig/20230208-161807-marostegui.json |
[production] |
| 16:14 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on rpki1001.eqiad.wmnet with reason: Restarting to increase VM RAM allocation |
[production] |
| 16:14 |
<cmooney@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:20:00 on rpki1001.eqiad.wmnet with reason: Restarting to increase VM RAM allocation |
[production] |
| 16:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P43838 and previous config saved to /var/cache/conftool/dbconfig/20230208-160301-marostegui.json |
[production] |
| 15:50 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mw2420.codfw.wmnet with OS buster |
[production] |
| 15:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P43837 and previous config saved to /var/cache/conftool/dbconfig/20230208-154754-marostegui.json |
[production] |
| 15:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T328817)', diff saved to https://phabricator.wikimedia.org/P43836 and previous config saved to /var/cache/conftool/dbconfig/20230208-154420-marostegui.json |
[production] |
| 15:38 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new test VM in drmrs - jmm@cumin2002" |
[production] |
| 15:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2126 (T328817)', diff saved to https://phabricator.wikimedia.org/P43835 and previous config saved to /var/cache/conftool/dbconfig/20230208-153248-marostegui.json |
[production] |
| 15:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2126 (T328817)', diff saved to https://phabricator.wikimedia.org/P43834 and previous config saved to /var/cache/conftool/dbconfig/20230208-153022-marostegui.json |
[production] |
| 15:30 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance |
[production] |
| 15:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance |
[production] |
| 15:30 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance |
[production] |
| 15:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2125 (T328817)', diff saved to https://phabricator.wikimedia.org/P43833 and previous config saved to /var/cache/conftool/dbconfig/20230208-152956-marostegui.json |
[production] |
| 15:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P43832 and previous config saved to /var/cache/conftool/dbconfig/20230208-152913-marostegui.json |
[production] |
| 15:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P43831 and previous config saved to /var/cache/conftool/dbconfig/20230208-151450-marostegui.json |
[production] |
| 15:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P43830 and previous config saved to /var/cache/conftool/dbconfig/20230208-151407-marostegui.json |
[production] |
| 15:11 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:10 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:02 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2420.codfw.wmnet with OS buster |
[production] |
| 14:59 |
<jforrester@deploy1002> |
Finished scap: Backport for [[gerrit:887752|Add a wordmark to itwiktionary (T329168)]] (duration: 08m 26s) |
[production] |
| 14:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P43829 and previous config saved to /var/cache/conftool/dbconfig/20230208-145944-marostegui.json |
[production] |
| 14:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T328817)', diff saved to https://phabricator.wikimedia.org/P43828 and previous config saved to /var/cache/conftool/dbconfig/20230208-145901-marostegui.json |
[production] |
| 14:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1168 (T328817)', diff saved to https://phabricator.wikimedia.org/P43826 and previous config saved to /var/cache/conftool/dbconfig/20230208-145651-marostegui.json |
[production] |
| 14:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:56 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:56 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T328817)', diff saved to https://phabricator.wikimedia.org/P43825 and previous config saved to /var/cache/conftool/dbconfig/20230208-145625-marostegui.json |
[production] |
| 14:53 |
<jforrester@deploy1002> |
superpes and jforrester: Backport for [[gerrit:887752|Add a wordmark to itwiktionary (T329168)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 14:51 |
<jforrester@deploy1002> |
Started scap: Backport for [[gerrit:887752|Add a wordmark to itwiktionary (T329168)]] |
[production] |
| 14:50 |
<jforrester@deploy1002> |
Finished scap: Backport for [[gerrit:887751|Replace trwiki temporary legacy logo with one including the wordmark (T329047)]] (duration: 07m 52s) |
[production] |