{"id":6018,"date":"2025-01-06T04:56:00","date_gmt":"2025-01-05T19:56:00","guid":{"rendered":"https:\/\/devneko.jp\/wordpress\/?p=6018"},"modified":"2025-01-06T04:56:00","modified_gmt":"2025-01-05T19:56:00","slug":"large-concept-models-language-modeling-in-a-sentence-representation-space","status":"publish","type":"post","link":"https:\/\/devneko.jp\/wordpress\/?p=6018","title":{"rendered":"Large Concept Models: Language Modeling in a Sentence Representation Space\u00a0"},"content":{"rendered":"\n<ul class=\"wp-block-list\">\n<li><strong>Large Concept Models: Language Modeling in a Sentence Representation Space\u00a0<\/strong>[62.7]<br>\u672c\u7a3f\u3067\u306f,\u6982\u5ff5\u3092\u547d\u540d\u3057\u305f\u660e\u793a\u7684\u306a\u9ad8\u30ec\u30d9\u30eb\u306a\u610f\u5473\u8868\u73fe\u306b\u57fa\u3065\u304f\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u8a66\u307f\u3092\u884c\u3046\u3002 \u6982\u5ff5\u306f\u8a00\u8a9e\u3068\u30e2\u30c0\u30ea\u30c6\u30a3\u306b\u4f9d\u5b58\u3057\u306a\u3044\u3082\u306e\u3067\u3042\u308a\u3001\u30d5\u30ed\u30fc\u306b\u304a\u3051\u308b\u3088\u308a\u9ad8\u3044\u30ec\u30d9\u30eb\u306e\u8003\u3048\u3084\u884c\u52d5\u3092\u8868\u3057\u3066\u3044\u308b\u3002 \u672c\u30e2\u30c7\u30eb\u3067\u306f,\u591a\u304f\u306e\u8a00\u8a9e\u306b\u5bfe\u3057\u3066,\u30bc\u30ed\u30b7\u30e7\u30c3\u30c8\u306e\u4e00\u822c\u5316\u6027\u80fd\u304c\u9855\u8457\u3067\u3042\u308b\u3053\u3068\u3092\u793a\u3059\u3002<br><a href=\"http:\/\/arxiv.org\/abs\/2412.08821v2\">\u8ad6\u6587<\/a>\u00a0\u00a0<a href=\"https:\/\/fugumt.com\/fugumt\/paper_check\/2412.08821v2\">\u53c2\u8003\u8a33\uff08\u30e1\u30bf\u30c7\u30fc\u30bf\uff09<\/a>\u00a0 \u00a0(Sun, 15 Dec 2024 21:20:12 GMT)<\/li>\n\n\n\n<li>\u30c8\u30fc\u30af\u30f3\u5358\u4f4d\u3067\u306f\u306a\u304f\u30b3\u30f3\u30bb\u30d7\u30c8\u5358\u4f4d\u306b\u8a00\u8a9e\u3092\u6271\u3063\u305f\u30e2\u30c7\u30eb\u306e\u63d0\u6848\u3001\u300cIn this study, as proof of feasibility, we assume that a concept corresponds to a sentence, and use an existing sentence embedding space, SONAR, which supports up to 200 languages in both text and speech modalities. The Large Concept Model is trained to perform autoregressive sentence prediction in an embedding space.\u300d\u3068\u3044\u3046\u8a2d\u5b9a\u3067\u300cThe LCM outperforms Llama-3.1-8B-IT on English and on the average over foreign languages officially supported by the LLM.\u300d\u3068\u306e\u8208\u5473\u6df1\u3044\u7d50\u679c\u3002\u4e00\u65b9\u3067\u300cWe acknowledge that there is still a long path to reach the performance of current flagship LLMs.\u300d\u3068\u306e\u8a18\u8f09\u3082\u3002<\/li>\n\n\n\n<li>\u30ea\u30dd\u30b8\u30c8\u30ea\u306f<a href=\"https:\/\/github.com\/facebookresearch\/large_concept_model\">GitHub &#8211; facebookresearch\/large_concept_model: Large Concept Models: Language modeling in a sentence representation space<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[223],"class_list":["post-6018","post","type-post","status-publish","format-standard","hentry","category-arxiv","tag-llm"],"_links":{"self":[{"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/6018","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6018"}],"version-history":[{"count":0,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/6018\/revisions"}],"wp:attachment":[{"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6018"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6018"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devneko.jp\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6018"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}