Aligning large language models with human values is a critical challenge for ensuring public safety. This alignment is commonly achieved through methods such as Direct Preference Optimization and RLAIF/RLHF. Despite extensive research, prior studies show that models often fail to maintain consistent alignment performance across languages, resulting in unequal treatment of users. To address this issue, we introduce a regularization term in the Direct Preference Optimization fine-tuning process that penalizes discrepancies in model behavior between languages using a bilingual corpus. Our results demonstrate that this approach effectively reduces the alignment gap across languages following fine-tuning.