|
2022-02-05
§
|
| 17:53 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt2001-dev.codfw.wmnet with OS bullseye |
[production] |
| 16:54 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2001-dev.codfw.wmnet with OS bullseye |
[production] |
| 06:11 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt2001-dev.codfw.wmnet with OS bullseye |
[production] |
| 06:09 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt2001-dev.codfw.wmnet with OS bullseye |
[production] |
| 05:41 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt2001-dev.codfw.wmnet with OS bullseye |
[production] |
| 00:09 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[community-labs-monitoring] |
| 00:09 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[antiharassment] |
| 00:08 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[annotation] |
| 00:07 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[wikidata-realtime-dumps] |
| 00:06 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[wikidata-autodesc] |
| 00:06 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[thumbor] |
| 00:05 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[sentry] |
| 00:02 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[ircwebchat] |
| 00:01 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[graphql] |
| 00:01 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[globalcu] |
| 00:00 |
<andrewbogott> |
stopping all VMs as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[glampipe] |
|
2022-02-04
§
|
| 23:59 |
<andrewbogott> |
accidentally restarted all VMs due to misreading the project purge page. sorry! |
[cvn] |
| 23:43 |
<jhathaway@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel |
[production] |
| 23:43 |
<jhathaway@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel |
[production] |
| 23:02 |
<inflatador> |
bking@deployment-puppetmaster04 local commit to public/private repo, see T299797 for more details |
[production] |
| 22:50 |
<wm-bot> |
<root> Cleaned up old log files |
[tools.ru_monuments] |
| 22:37 |
<jhathaway@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel |
[production] |
| 22:36 |
<jhathaway@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel |
[production] |
| 22:33 |
<taavi> |
`root@tools-sgebastion-10:/data/project/ru_monuments/.kube# mv config old_config` # experimenting with T301015 |
[tools] |
| 21:36 |
<taavi> |
clear error state from some webgrid nodes |
[tools] |
| 20:28 |
<wm-bot> |
<bd808> Updated to php7.4 runtime |
[tools.bash] |
| 19:44 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices2002-dev.wikimedia.org with OS bullseye |
[production] |
| 18:53 |
<taavi> |
upgrading to kubernetes 1.21 T282942 |
[toolsbeta] |
| 18:52 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudservices2002-dev.wikimedia.org with OS bullseye |
[production] |
| 17:00 |
<arturo> |
add mcrouter 2022.01.31.00-1 to bullseye-wikimedia (T300578) |
[production] |
| 16:48 |
<jbond> |
update add new ferm package ferm_2.5.1-1+wmf11u2 |
[production] |
| 16:38 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:35 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:05 |
<elukey> |
unmask prometheus-mysqld-exporter.service and clean up the old @analytics + wmf_auto_restart units (service+timer) not used anymore on an-coord100[12] |
[production] |
| 16:05 |
<elukey> |
unmask prometheus-mysqld-exporter.service and clean up the old @analytics + wmf_auto_restart units (service+timer) not used anymore on an-coord100[12] |
[analytics] |
| 14:25 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. |
[production] |
| 14:18 |
<btullis@cumin1001> |
START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. |
[production] |
| 12:55 |
<joal> |
Rerun cassandra-daily-wf-local_group_default_T_pageviews_per_article_flat-2022-2-3 |
[analytics] |
| 12:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1020.eqiad.wmnet with OS buster |
[production] |
| 11:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 100%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P20174 and previous config saved to /var/cache/conftool/dbconfig/20220204-114117-root.json |
[production] |
| 11:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 75%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P20173 and previous config saved to /var/cache/conftool/dbconfig/20220204-112613-root.json |
[production] |
| 11:14 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti1020.eqiad.wmnet with OS buster |
[production] |
| 11:13 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 11:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 50%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P20172 and previous config saved to /var/cache/conftool/dbconfig/20220204-111110-root.json |
[production] |
| 11:07 |
<akosiaris@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 11:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove all special groups from s1 codfw T263127', diff saved to https://phabricator.wikimedia.org/P20171 and previous config saved to /var/cache/conftool/dbconfig/20220204-110427-marostegui.json |
[production] |
| 10:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 25%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P20170 and previous config saved to /var/cache/conftool/dbconfig/20220204-105606-root.json |
[production] |
| 10:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 10%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P20165 and previous config saved to /var/cache/conftool/dbconfig/20220204-104102-root.json |
[production] |
| 10:40 |
<moritzm> |
rebalancing row A in ganeti/eqiad, all nodes of that row are now running Buster T296721 |
[production] |
| 10:12 |
<arturo> |
restart backup_vms service in cloudvirt1024 (T300956) |
[admin] |