基于LLaVA多模态大模型的冬虫夏草分类系统
史明上 赵婧（通讯作者） 吕静怡 解江

西京学院电子信息学院，陕西西安，710123；

摘要：针对冬虫夏草（虫草）传统鉴别依赖专家经验、主观性强且难以应对复杂采集场景的问题，提出一种基于多模态大语言模型LLaVA结合LoRA参数高效微调的虫草细粒度分类方法。自主构建了涵盖6类虫草共29,445张图像的Cordyceps29K多模态数据集；设计包含模糊化、旋转、亮度增强、翻转与缩放颜色变换共5种操作的自适应数据增强策略；基于LLaMAFactory框架采用LoRA方法完成LLaVA的虫草领域指令微调；并通过vLLM引擎完成高效推理部署。实验结果表明，微调后模型各类别准确率均达97%以上，BLEU-4达95.8%，验证了所提方法的有效性，为虫草质量检测与市场监管提供了智能化技术支撑。

关键词：冬虫夏草；多模态大语言模型；LoRA微调；数据增强；细粒度分类

参考文献

[1] TULI H S, SANDHU S S, SHARMA A K. Pharmacological and therapeutic potential of Cordyceps with special reference to cordycepin[J]. 3 Biotech, 2014, 4(1): 1-12.

[2] LIU H, LI C, WU Q, et al. Visual instruction tuning[C]// Advances in Neural Information Processing Systems. 2024, 36.

[3] ZHENG Y, ZHANG R, ZHANG J, et al. LlamaFactory: Unified efficient fine-tuning of 100+ language models[C]// Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024.

[4] FU J, ZHENG H, MEI T. Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 4438-4446.

[5] HU T, QI H, HUANG Q, et al. See better before looking closer: Weakly supervised data augmentation network for fine-grained visual classification[EB/OL]. 2019[2025-01-10]. https://arxiv.org/abs/1901.09891.

[6] HU E J, SHEN Y L, WALLIS P, et al. LoRA: low-rank adaptation of large language models[EB/OL]. 2021[2025-01-10]. https://doi.org/10.48550/arXiv.2106.09685.

[7] KWON W, LI Z, ZHUANG S, et al. Efficient memory management for large language model serving with PagedAttention[C]// Proceedings of the 29th Symposium on Operating Systems Principles. 2023: 611-626.