{"id":6629,"date":"2025-05-01T05:57:00","date_gmt":"2025-04-30T20:57:00","guid":{"rendered":"https:\/\/devneko.jp\/wordpress\/?p=6629"},"modified":"2025-05-01T05:57:00","modified_gmt":"2025-04-30T20:57:00","slug":"knowledge-distillation-and-dataset-distillation-of-large-language-models-emerging-trends-challenges-and-future-directions","status":"publish","type":"post","link":"https:\/\/devneko.jp\/wordpress\/?p=6629","title":{"rendered":"Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions\u00a0"},"content":{"rendered":"\n<ul class=\"wp-block-list\">\n<li><strong>Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions\u00a0<\/strong>[35.8]<br>LLM(Large Language Models)\u306e\u6307\u6570\u95a2\u6570\u7684\u6210\u9577\u306f\u3001\u7d76\u3048\u9593\u306a\u304f\u62e1\u5927\u3059\u308b\u8a08\u7b97\u304a\u3088\u3073\u30c7\u30fc\u30bf\u8981\u6c42\u3092\u6e80\u305f\u3059\u305f\u3081\u306e\u52b9\u7387\u7684\u306a\u6226\u7565\u306e\u5fc5\u8981\u6027\u3092\u5f37\u8abf\u3057\u7d9a\u3051\u3066\u3044\u308b\u3002 \u672c\u8abf\u67fb\u306f\u3001\u77e5\u8b58\u84b8\u7559(KD)\u3068\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u84b8\u7559(DD)\u306e2\u3064\u306e\u76f8\u88dc\u7684\u30d1\u30e9\u30c0\u30a4\u30e0\u3092\u5305\u62ec\u7684\u306b\u5206\u6790\u3059\u308b\u3002<br><a href=\"http:\/\/arxiv.org\/abs\/2504.14772v1\">\u8ad6\u6587<\/a>\u00a0\u00a0<a href=\"https:\/\/fugumt.com\/fugumt\/paper_check\/2504.14772v1\">\u53c2\u8003\u8a33\uff08\u30e1\u30bf\u30c7\u30fc\u30bf\uff09<\/a>\u00a0 \u00a0(Sun, 20 Apr 2025 23:50:23 GMT)<\/li>\n\n\n\n<li>\u84b8\u7559\u306b\u95a2\u3059\u308b\u30b5\u30fc\u30d9\u30a4<\/li>\n\n\n\n<li>\u300cCrucially, the success of KD in LLMs hinges on DD techniques, which enable the creation of compact, informationrich synthetic datasets that encapsulate the diverse and complex knowledge of the teacher LLMs.\u300d\u3068Knowledge distillation\u3068Dataset distillation\u3092\u5bfe\u3068\u3057\u3066\u30b5\u30fc\u30d9\u30a4\u3059\u308b\u3082\u306e\u306f\u73cd\u3057\u3044\u304b\u3082\u3057\u308c\u306a\u3044<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[387,635],"class_list":["post-6629","post","type-post","status-publish","format-standard","hentry","category-arxiv","tag-survey","tag-635"],"_links":{"self":[{"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/6629","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6629"}],"version-history":[{"count":0,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/6629\/revisions"}],"wp:attachment":[{"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6629"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6629"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6629"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}