651-700 of 10000 results (37ms)
2021-11-24 §
06:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance (T296143) [production]
06:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17807 and previous config saved to /var/cache/conftool/dbconfig/20211124-063228-root.json [production]
06:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17806 and previous config saved to /var/cache/conftool/dbconfig/20211124-061725-root.json [production]
06:05 <marostegui> Upgrade db1128's kernel T288720 [production]
06:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17805 and previous config saved to /var/cache/conftool/dbconfig/20211124-060221-root.json [production]
05:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 10%: After optimize table (T296143)', diff saved to https://phabricator.wikimedia.org/P17804 and previous config saved to /var/cache/conftool/dbconfig/20211124-054718-root.json [production]
00:25 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS buster [production]
2021-11-23 §
23:53 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2012.codfw.wmnet with OS buster [production]
23:43 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2011.codfw.wmnet with OS buster [production]
23:12 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2011.codfw.wmnet with OS buster [production]
23:11 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2010.codfw.wmnet with OS buster [production]
22:40 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2010.codfw.wmnet with OS buster [production]
22:28 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2009.codfw.wmnet with OS buster [production]
21:58 <tgr> UTC evening deploys done [production]
21:57 <tgr@deploy1002> Finished scap: (no justification provided) (duration: 10m 03s) [production]
21:57 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS buster [production]
21:56 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2009.codfw.wmnet with OS buster [production]
21:53 <krinkle@deploy1002> Finished deploy [integration/docroot@a3435a7]: (no justification provided) (duration: 00m 07s) [production]
21:53 <krinkle@deploy1002> Started deploy [integration/docroot@a3435a7]: (no justification provided) [production]
21:47 <tgr@deploy1002> Started scap: (no justification provided) [production]
21:47 <tgr@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:740777|Add Image: Validate GEInfoboxTemplates size (T294518)]] (duration: 00m 56s) [production]
21:39 <tgr@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes/Api/ApiQueryGrowthTasks.php: Backport: [[gerrit:740776|Structured task caching/filtering cherry-picks step 3]] (duration: 00m 55s) [production]
21:35 <tgr@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:740775|Structured task caching/filtering cherry-picks step 2]] (duration: 00m 57s) [production]
21:28 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS buster [production]
20:57 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:50 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:25 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:07 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:04 <legoktm@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/Echo/: re-enable cross-wiki notifications by default (T296270) (duration: 00m 57s) [production]
19:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:57 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:52 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:51 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/: 7d5f779a73594bb11f359bda055f2c7af8e92feb: Structured task caching/filtering cherry-picks, step 1 (duration: 00m 56s) [production]
19:42 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/: c26e407118e1cd8e1e3fea6e2f4e3e43a609ea62: GrowthExperiments backports (duration: 01m 03s) [production]
19:27 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:18 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: bf82bfb3ddcaff04a1e90abc435ccb26f792780c: Add new icons, wordmarks & taglines for several wikis (T290091; 2/2) (duration: 00m 56s) [production]
19:17 <urbanecm@deploy1002> Synchronized static/images/mobile/copyright/: bf82bfb3ddcaff04a1e90abc435ccb26f792780c: Add new icons, wordmarks & taglines for several wikis (T290091; 1/2) (duration: 00m 56s) [production]
19:12 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:08 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 3993aacbfdbbfb6cdcc198ce369bf08b32ace865: Increase reading depth sampling rate to .1% (T294777) (duration: 00m 57s) [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:29 <mbsantos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
18:25 <ejegg> updated SmashPig standalone (IPN listener) from be68299b -> 211f8e65 [production]
18:18 <mbsantos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
18:18 <cmjohnson1> upgrading msw-c1-eqiad T259758 [production]
18:04 <mbsantos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
18:01 <mbsantos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
18:00 <moritzm> systemctl reset-failed ifup@ens5.service on durum2001 T273026 [production]
17:59 <mbsantos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]