|
2023-04-11
ยง
|
| 12:16 |
<ladsgroup@cumin1001> |
END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:09 |
<wm-bot2> |
build & push docker image docker-registry.tools.wmflabs.org/volume-admission:40bd3b3 from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (40bd3b3) - cookbook ran by dcaro@vulcanus |
[tools] |
| 11:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1122 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P46312 and previous config saved to /var/cache/conftool/dbconfig/20230411-115137-root.json |
[production] |
| 11:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1122 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P46311 and previous config saved to /var/cache/conftool/dbconfig/20230411-113631-root.json |
[production] |
| 11:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1122 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P46310 and previous config saved to /var/cache/conftool/dbconfig/20230411-112126-root.json |
[production] |
| 11:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 100%: Pooling', diff saved to https://phabricator.wikimedia.org/P46309 and previous config saved to /var/cache/conftool/dbconfig/20230411-111854-root.json |
[production] |
| 11:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1122 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P46308 and previous config saved to /var/cache/conftool/dbconfig/20230411-110621-root.json |
[production] |
| 11:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 75%: Pooling', diff saved to https://phabricator.wikimedia.org/P46307 and previous config saved to /var/cache/conftool/dbconfig/20230411-110349-root.json |
[production] |
| 10:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1122 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P46306 and previous config saved to /var/cache/conftool/dbconfig/20230411-105116-root.json |
[production] |
| 10:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 T334447', diff saved to https://phabricator.wikimedia.org/P46305 and previous config saved to /var/cache/conftool/dbconfig/20230411-105100-marostegui.json |
[production] |
| 10:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 50%: Pooling', diff saved to https://phabricator.wikimedia.org/P46304 and previous config saved to /var/cache/conftool/dbconfig/20230411-104844-root.json |
[production] |
| 10:36 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=5; selector: service=thumbor,name=kubernetes201[0123].codfw.wmnet |
[production] |
| 10:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1122 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P46303 and previous config saved to /var/cache/conftool/dbconfig/20230411-103611-root.json |
[production] |
| 10:36 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=5; selector: service=thumbor,name=kubernetes101[0123].eqiad.wmnet |
[production] |
| 10:34 |
<wm-bot2> |
deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (9aed7e5) - cookbook ran by taavi@runko |
[tools] |
| 10:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 25%: Pooling', diff saved to https://phabricator.wikimedia.org/P46302 and previous config saved to /var/cache/conftool/dbconfig/20230411-103339-root.json |
[production] |
| 10:27 |
<wm-bot2> |
deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (8f0bfcd) - cookbook ran by taavi@runko |
[toolsbeta] |
| 10:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1122 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P46301 and previous config saved to /var/cache/conftool/dbconfig/20230411-102106-root.json |
[production] |
| 10:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 10%: Pooling', diff saved to https://phabricator.wikimedia.org/P46300 and previous config saved to /var/cache/conftool/dbconfig/20230411-101835-root.json |
[production] |
| 10:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 5%: Pooling', diff saved to https://phabricator.wikimedia.org/P46298 and previous config saved to /var/cache/conftool/dbconfig/20230411-100330-root.json |
[production] |
| 09:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 4%: Pooling', diff saved to https://phabricator.wikimedia.org/P46297 and previous config saved to /var/cache/conftool/dbconfig/20230411-094825-root.json |
[production] |
| 09:44 |
<jelto@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gitlab2003.wikimedia.org with OS bullseye |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 3%: Pooling', diff saved to https://phabricator.wikimedia.org/P46296 and previous config saved to /var/cache/conftool/dbconfig/20230411-093320-root.json |
[production] |
| 09:27 |
<Amir1> |
start of watchlist clean up of a user in wikidatawiki (T328501) |
[production] |
| 09:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 100%: Pooling', diff saved to https://phabricator.wikimedia.org/P46295 and previous config saved to /var/cache/conftool/dbconfig/20230411-092224-root.json |
[production] |
| 09:20 |
<moritzm> |
installing nodejs security updates on buster |
[production] |
| 09:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 2%: Pooling', diff saved to https://phabricator.wikimedia.org/P46294 and previous config saved to /var/cache/conftool/dbconfig/20230411-091815-root.json |
[production] |
| 09:15 |
<wm-bot2> |
deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico (c6a3e29) (T329677) - cookbook ran by taavi@runko |
[tools] |
| 09:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 75%: Pooling', diff saved to https://phabricator.wikimedia.org/P46293 and previous config saved to /var/cache/conftool/dbconfig/20230411-090720-root.json |
[production] |
| 09:04 |
<moritzm> |
installing pcre2 security updates |
[production] |
| 09:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 1%: Pooling', diff saved to https://phabricator.wikimedia.org/P46292 and previous config saved to /var/cache/conftool/dbconfig/20230411-090310-root.json |
[production] |
| 08:59 |
<wm-bot2> |
Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko |
[toolsbeta] |
| 08:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1122 to clone db1222 T326669', diff saved to https://phabricator.wikimedia.org/P46290 and previous config saved to /var/cache/conftool/dbconfig/20230411-085654-marostegui.json |
[production] |
| 08:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 50%: Pooling', diff saved to https://phabricator.wikimedia.org/P46289 and previous config saved to /var/cache/conftool/dbconfig/20230411-085215-root.json |
[production] |
| 08:50 |
<jelto@cumin2002> |
START - Cookbook sre.hosts.reimage for host gitlab2003.wikimedia.org with OS bullseye |
[production] |
| 08:46 |
<wm-bot2> |
Adding a new k8s worker node - cookbook ran by taavi@runko |
[toolsbeta] |
| 08:45 |
<wm-bot2> |
Adding a new k8s worker node - cookbook ran by taavi@runko |
[tools] |
| 08:43 |
<wm-bot2> |
deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico (c6a3e29) - cookbook ran by taavi@runko |
[toolsbeta] |
| 08:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 25%: Pooling', diff saved to https://phabricator.wikimedia.org/P46288 and previous config saved to /var/cache/conftool/dbconfig/20230411-083710-root.json |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 100%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46287 and previous config saved to /var/cache/conftool/dbconfig/20230411-083339-root.json |
[production] |
| 08:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P46286 and previous config saved to /var/cache/conftool/dbconfig/20230411-083106-root.json |
[production] |
| 08:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 100%: Pooling', diff saved to https://phabricator.wikimedia.org/P46285 and previous config saved to /var/cache/conftool/dbconfig/20230411-082521-root.json |
[production] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 10%: Pooling', diff saved to https://phabricator.wikimedia.org/P46284 and previous config saved to /var/cache/conftool/dbconfig/20230411-082205-root.json |
[production] |
| 08:19 |
<aqu@deploy2002> |
Finished deploy [analytics/refinery@bed78f6] (hadoop-test): Deploy analytics_refinery including last webrequest load scripts in TEST 3nd try [analytics/refinery@bed78f6] (duration: 01m 25s) |
[production] |
| 08:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 75%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46283 and previous config saved to /var/cache/conftool/dbconfig/20230411-081834-root.json |
[production] |
| 08:18 |
<aqu@deploy2002> |
Started deploy [analytics/refinery@bed78f6] (hadoop-test): Deploy analytics_refinery including last webrequest load scripts in TEST 3nd try [analytics/refinery@bed78f6] |
[production] |
| 08:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P46282 and previous config saved to /var/cache/conftool/dbconfig/20230411-081601-root.json |
[production] |
| 08:15 |
<aqu> |
About to deploy analytics/refinery (To migrate webrequest load from Oozie to Airflow) |
[production] |
| 08:14 |
<aqu> |
About to deploy analytics/refinery (To migrate webrequest load from Oozie to Airflow) |
[analytics] |