|
2021-11-24
§
|
| 08:41 |
<vgutierrez> |
depool cp2027 |
[production] |
| 08:05 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1125.eqiad.wmnet with OS bullseye |
[production] |
| 07:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye |
[production] |
| 07:23 |
<elukey> |
reboot kubernetes1018 (role::insetup) to verify negotiated speed of eth interface |
[production] |
| 07:12 |
<elukey> |
drop /tmp/blockmgr-20fe4b2b-31fb-4a85-b5b1-bebe254120f8 and other blockmgr-* dirs on stat1006 to free space on the root partition |
[production] |
| 06:47 |
<Amir1> |
running optimize table with replication on db1155:3314 (T296143) |
[production] |
| 06:45 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance (T296143) |
[production] |
| 06:45 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance (T296143) |
[production] |
| 06:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17807 and previous config saved to /var/cache/conftool/dbconfig/20211124-063228-root.json |
[production] |
| 06:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17806 and previous config saved to /var/cache/conftool/dbconfig/20211124-061725-root.json |
[production] |
| 06:05 |
<marostegui> |
Upgrade db1128's kernel T288720 |
[production] |
| 06:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17805 and previous config saved to /var/cache/conftool/dbconfig/20211124-060221-root.json |
[production] |
| 05:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 10%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17804 and previous config saved to /var/cache/conftool/dbconfig/20211124-054718-root.json |
[production] |
| 00:25 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS buster |
[production] |
|
2021-11-23
§
|
| 23:53 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2012.codfw.wmnet with OS buster |
[production] |
| 23:43 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2011.codfw.wmnet with OS buster |
[production] |
| 23:12 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2011.codfw.wmnet with OS buster |
[production] |
| 23:11 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2010.codfw.wmnet with OS buster |
[production] |
| 22:40 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2010.codfw.wmnet with OS buster |
[production] |
| 22:28 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2009.codfw.wmnet with OS buster |
[production] |
| 21:58 |
<tgr> |
UTC evening deploys done |
[production] |
| 21:57 |
<tgr@deploy1002> |
Finished scap: (no justification provided) (duration: 10m 03s) |
[production] |
| 21:57 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS buster |
[production] |
| 21:56 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2009.codfw.wmnet with OS buster |
[production] |
| 21:53 |
<krinkle@deploy1002> |
Finished deploy [integration/docroot@a3435a7]: (no justification provided) (duration: 00m 07s) |
[production] |
| 21:53 |
<krinkle@deploy1002> |
Started deploy [integration/docroot@a3435a7]: (no justification provided) |
[production] |
| 21:47 |
<tgr@deploy1002> |
Started scap: (no justification provided) |
[production] |
| 21:47 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:740777|Add Image: Validate GEInfoboxTemplates size (T294518)]] (duration: 00m 56s) |
[production] |
| 21:39 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes/Api/ApiQueryGrowthTasks.php: Backport: [[gerrit:740776|Structured task caching/filtering cherry-picks step 3]] (duration: 00m 55s) |
[production] |
| 21:35 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:740775|Structured task caching/filtering cherry-picks step 2]] (duration: 00m 57s) |
[production] |
| 21:28 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS buster |
[production] |
| 20:57 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:50 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:04 |
<legoktm@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/Echo/: re-enable cross-wiki notifications by default (T296270) (duration: 00m 57s) |
[production] |
| 19:58 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:57 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:52 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:51 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/: 7d5f779a73594bb11f359bda055f2c7af8e92feb: Structured task caching/filtering cherry-picks, step 1 (duration: 00m 56s) |
[production] |
| 19:42 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/: c26e407118e1cd8e1e3fea6e2f4e3e43a609ea62: GrowthExperiments backports (duration: 01m 03s) |
[production] |
| 19:27 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:23 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:18 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: bf82bfb3ddcaff04a1e90abc435ccb26f792780c: Add new icons, wordmarks & taglines for several wikis (T290091; 2/2) (duration: 00m 56s) |
[production] |
| 19:17 |
<urbanecm@deploy1002> |
Synchronized static/images/mobile/copyright/: bf82bfb3ddcaff04a1e90abc435ccb26f792780c: Add new icons, wordmarks & taglines for several wikis (T290091; 1/2) (duration: 00m 56s) |
[production] |
| 19:12 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:08 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 3993aacbfdbbfb6cdcc198ce369bf08b32ace865: Increase reading depth sampling rate to .1% (T294777) (duration: 00m 57s) |
[production] |
| 19:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:29 |
<mbsantos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |