BUILDING AN IMAGE-BASED MEDICATION IDENTIFICATION AND REMINDER APPLICATION

Abstract

The thesis addresses the problem of misinformation that can occur when users look up manually or inaccurate tracking of daily drug prescription intake. To overcome these challenges, we develop a mobile application “MediApp” that integrates Optical Character Recognition (OCR) based on the EasyOCR framework with a LLM for drug labels and drug prescriptions image-based recognition. We also perform a fine-tuned experiment on drug label datasets to improve the performance of CRNN, a text recognition model. The experiment results shows that fine-tuning with the combined drug label and general image dataset has achieved an accuracy of 80.392% and a NED score of 0.955 in the drug label domain, while still maintaining strong general text recognition capability. Developed using Flutter and Pocketbase, the mobile application provides drug information retrieval, drug prescriptions management and daily reminders to help users keep track of their medication schedules.