Bangguo Ye (叶邦国)

Fine-tuning large pre-trained models under resource constraints remains challenging due to the massive number of parameters involved. Existing parameter-efficient tuning methods, such as low-rank adaptation (LoRA) and its variants, rely heavily on matrix factorization and often struggle in extremely low-parameter regimes. In this work, we propose TRAC, a novel fine-tuning framework that leverages Tensor-Train decomposition with Across-layer Compression. Specifically, TRAC represents each adaptation module as a compact sequence of tensor-train cores and allows certain cores to be frozen or shared across layers, thereby exploiting the inherent similarity and redundancy among layer weight matrices. To retain layer-specific flexibility, lightweight controllers are introduced, enabling shared tensor cores to adaptively modulate representations. We evaluate TRAC on diverse architectures, including Qwen, LLaMA, GPT, BERT, and ViT, across benchmarks covering text classification, text generation, and image classification. Experimental results demonstrate that TRAC achieves performance comparable to or better than LoRA and its variants, while substantially reducing trainable parameters and storage requirements.

The diverse tasks of X-ray bone image analysis necessitate training and maintaining numerous task-specific models. Recent advancements in foundation models have attracted extensive attention across various fields, promising significant improvements in medical applications. These models depend on large-scale datasets and utilize the self-supervised learning framework for their training. However, the study for the specific scenario when data acquisition is challenging still remains limited, such as the modality of bone X-ray images. This limitation is partly due to the limited availability of large-scale and curated bone X-ray datasets for research and model development. In this paper, we aim to fill this gap by applying DINO and DINOv2 frameworks for self-supervised learning across two bone X-ray image datasets. Our comprehensive analysis covers a range of downstream tasks, including supervised regression for accurate bone age assessment, unsupervised clustering through a Gaussian mixture model, and supervised regions of interest (ROI) localization. Our experiments demonstrate that pretrained models can significantly improve the downstream task performance and the extracted features conform to the property of bone age. The study shows the effectiveness of self-supervised learning on the bone X-ray images, providing insights into their capabilities to improve diagnostic precision and generalizability in medical imaging.