|
2021-11-24
§
|
| 09:13 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.downtime for 3:00:00 on blubberoid.svc.codfw.wmnet with reason: helm3 de-deploy T251305 |
[production] |
| 09:13 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on apple-search.svc.codfw.wmnet with reason: helm3 de-deploy T251305 |
[production] |
| 09:13 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.downtime for 3:00:00 on apple-search.svc.codfw.wmnet with reason: helm3 de-deploy T251305 |
[production] |
| 09:13 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on api-gateway.svc.codfw.wmnet with reason: helm3 de-deploy T251305 |
[production] |
| 09:13 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.downtime for 3:00:00 on api-gateway.svc.codfw.wmnet with reason: helm3 de-deploy T251305 |
[production] |
| 09:11 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on apertium.svc.codfw.wmnet with reason: helm3 de-deploy T251305 |
[production] |
| 09:11 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.downtime for 3:00:00 on apertium.svc.codfw.wmnet with reason: helm3 de-deploy T251305 |
[production] |
| 09:10 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM deneb.codfw.wmnet |
[production] |
| 09:08 |
<_joe_> |
switching search.wikimedia.org to be served by the apple-search servcie |
[production] |
| 09:04 |
<jelto> |
start re-deploy procedure in codfw Kubernetes T251305 |
[production] |
| 09:01 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM deneb.codfw.wmnet |
[production] |
| 08:59 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:56 |
<_joe_> |
repooling cp2027 |
[production] |
| 08:55 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:55 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'apple-search' for release 'main' . |
[production] |
| 08:51 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:741082|Set actor migration to write both on all wikis (T275246)]] (duration: 00m 57s) |
[production] |
| 08:51 |
<oblivian@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'apple-search' for release 'main' . |
[production] |
| 08:41 |
<vgutierrez> |
depool cp2027 |
[production] |
| 08:05 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1125.eqiad.wmnet with OS bullseye |
[production] |
| 07:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye |
[production] |
| 07:23 |
<elukey> |
reboot kubernetes1018 (role::insetup) to verify negotiated speed of eth interface |
[production] |
| 07:12 |
<elukey> |
drop /tmp/blockmgr-20fe4b2b-31fb-4a85-b5b1-bebe254120f8 and other blockmgr-* dirs on stat1006 to free space on the root partition |
[production] |
| 06:47 |
<Amir1> |
running optimize table with replication on db1155:3314 (T296143) |
[production] |
| 06:45 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance (T296143) |
[production] |
| 06:45 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance (T296143) |
[production] |
| 06:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17807 and previous config saved to /var/cache/conftool/dbconfig/20211124-063228-root.json |
[production] |
| 06:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17806 and previous config saved to /var/cache/conftool/dbconfig/20211124-061725-root.json |
[production] |
| 06:05 |
<marostegui> |
Upgrade db1128's kernel T288720 |
[production] |
| 06:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17805 and previous config saved to /var/cache/conftool/dbconfig/20211124-060221-root.json |
[production] |
| 05:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 10%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17804 and previous config saved to /var/cache/conftool/dbconfig/20211124-054718-root.json |
[production] |
| 00:25 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS buster |
[production] |
|
2021-11-23
§
|
| 23:53 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2012.codfw.wmnet with OS buster |
[production] |
| 23:43 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2011.codfw.wmnet with OS buster |
[production] |
| 23:12 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2011.codfw.wmnet with OS buster |
[production] |
| 23:11 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2010.codfw.wmnet with OS buster |
[production] |
| 22:40 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2010.codfw.wmnet with OS buster |
[production] |
| 22:28 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2009.codfw.wmnet with OS buster |
[production] |
| 21:58 |
<tgr> |
UTC evening deploys done |
[production] |
| 21:57 |
<tgr@deploy1002> |
Finished scap: (no justification provided) (duration: 10m 03s) |
[production] |
| 21:57 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS buster |
[production] |
| 21:56 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2009.codfw.wmnet with OS buster |
[production] |
| 21:53 |
<krinkle@deploy1002> |
Finished deploy [integration/docroot@a3435a7]: (no justification provided) (duration: 00m 07s) |
[production] |
| 21:53 |
<krinkle@deploy1002> |
Started deploy [integration/docroot@a3435a7]: (no justification provided) |
[production] |
| 21:47 |
<tgr@deploy1002> |
Started scap: (no justification provided) |
[production] |
| 21:47 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:740777|Add Image: Validate GEInfoboxTemplates size (T294518)]] (duration: 00m 56s) |
[production] |
| 21:39 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes/Api/ApiQueryGrowthTasks.php: Backport: [[gerrit:740776|Structured task caching/filtering cherry-picks step 3]] (duration: 00m 55s) |
[production] |
| 21:35 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:740775|Structured task caching/filtering cherry-picks step 2]] (duration: 00m 57s) |
[production] |
| 21:28 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS buster |
[production] |
| 20:57 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:50 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |