|
2024-04-15
§
|
| 08:54 |
<btullis@cumin1002> |
START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop test cluster: Restart of jvm daemons. |
[production] |
| 08:53 |
<jynus> |
restart dbprov2005 |
[production] |
| 08:46 |
<godog> |
logstash.w.o now uses sso - T246998 |
[production] |
| 08:45 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid test cluster: Roll restart of Druid jvm daemons. |
[production] |
| 08:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P60490 and previous config saved to /var/cache/conftool/dbconfig/20240415-084130-marostegui.json |
[production] |
| 08:36 |
<btullis> |
roll-restarting druid on test cluster for T356382 |
[analytics] |
| 08:35 |
<btullis@cumin1002> |
START - Cookbook sre.druid.roll-restart-workers for Druid test cluster: Roll restart of Druid jvm daemons. |
[production] |
| 08:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1170 (T356166)', diff saved to https://phabricator.wikimedia.org/P60488 and previous config saved to /var/cache/conftool/dbconfig/20240415-082623-marostegui.json |
[production] |
| 08:24 |
<wmbot~dcaro@urcuchillay> |
END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api |
[toolsbeta] |
| 08:24 |
<wmbot~dcaro@urcuchillay> |
START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api |
[toolsbeta] |
| 08:01 |
<Emperor> |
depool wdqs in codfw T362508 |
[production] |
| 08:01 |
<mvernon@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=wdqs,name=codfw |
[production] |
| 07:55 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: reboot multiinstance replica |
[production] |
| 07:54 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db1217.eqiad.wmnet with reason: reboot multiinstance replica |
[production] |
| 07:48 |
<jayme> |
restarting k8s-mlstaging and k8s-staging prometheus instances - T343529 |
[production] |
| 07:11 |
<dcausse> |
restarting blazegraph on wdqs1020 (BlazegraphFreeAllocatorsDecreasingRapidly) |
[production] |
| 06:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1170 (T356166)', diff saved to https://phabricator.wikimedia.org/P60487 and previous config saved to /var/cache/conftool/dbconfig/20240415-065659-marostegui.json |
[production] |
| 06:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:56 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60486 and previous config saved to /var/cache/conftool/dbconfig/20240415-065636-marostegui.json |
[production] |
| 06:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P60485 and previous config saved to /var/cache/conftool/dbconfig/20240415-064129-marostegui.json |
[production] |
| 06:30 |
<wmbot~bsadowski1@tools-sgebastion-10> |
Restarted StewardBot/SULWatcher because of a connection loss |
[tools.stewardbots] |
| 06:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P60484 and previous config saved to /var/cache/conftool/dbconfig/20240415-062621-marostegui.json |
[production] |
| 06:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60483 and previous config saved to /var/cache/conftool/dbconfig/20240415-061114-marostegui.json |
[production] |
| 05:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60482 and previous config saved to /var/cache/conftool/dbconfig/20240415-053001-marostegui.json |
[production] |
| 05:29 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:29 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:29 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:29 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:06 |
<wmbot~deltaquad@tools-sgebastion-10> |
./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC |
[tools.stewardbots] |
|
2024-04-14
§
|
| 17:07 |
<wmbot~anticomposite@tools-sgebastion-10> |
SULWatcher/manage.sh restart # SULWatchers disconnected |
[tools.stewardbots] |
| 16:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db2142 as x2 codfw master', diff saved to https://phabricator.wikimedia.org/P60481 and previous config saved to /var/cache/conftool/dbconfig/20240414-160016-marostegui.json |
[production] |
| 15:06 |
<wmbot~lucaswerkmeister@tools-sgebastion-10> |
Double IRC messages to other bridges |
[tools.bridgebot] |
| 13:51 |
<wmbot~melos@tools-sgebastion-10> |
./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC |
[tools.stewardbots] |
| 11:22 |
<marostegui> |
Restart x2 codfw master |
[production] |
| 11:17 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: Investigating |
[production] |
| 11:17 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 6 hosts with reason: Investigating |
[production] |
|
2024-04-13
§
|
| 23:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60479 and previous config saved to /var/cache/conftool/dbconfig/20240413-233953-marostegui.json |
[production] |
| 23:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60478 and previous config saved to /var/cache/conftool/dbconfig/20240413-232443-marostegui.json |
[production] |
| 23:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60477 and previous config saved to /var/cache/conftool/dbconfig/20240413-230935-marostegui.json |
[production] |
| 22:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60476 and previous config saved to /var/cache/conftool/dbconfig/20240413-225428-marostegui.json |
[production] |
| 15:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60475 and previous config saved to /var/cache/conftool/dbconfig/20240413-154240-marostegui.json |
[production] |
| 15:42 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:42 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60474 and previous config saved to /var/cache/conftool/dbconfig/20240413-154217-marostegui.json |
[production] |
| 15:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60473 and previous config saved to /var/cache/conftool/dbconfig/20240413-152709-marostegui.json |
[production] |
| 15:12 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60472 and previous config saved to /var/cache/conftool/dbconfig/20240413-151201-marostegui.json |
[production] |
| 14:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60471 and previous config saved to /var/cache/conftool/dbconfig/20240413-145653-marostegui.json |
[production] |
| 13:14 |
<wmbot~lucaswerkmeister@tools-sgebastion-10> |
Double IRC messages to other bridges |
[tools.bridgebot] |
| 13:14 |
<wmbot~melos@tools-sgebastion-10> |
SULWatcher/manage.sh restart # SULWatchers disconnected |
[tools.stewardbots] |