Abstract Quantum computing systems harness the power of quantum mechanics to execute computationally demanding tasks more effectively than their classical counterparts. This has led to the emergence of Quantum Software Engineering (QSE), which focuses on unlocking the full potential of quantum computing systems. As QSE gains prominence, it seeks to address the evolving challenges of quantum software development by offering comprehensive concepts, principles, and guidelines. This paper aims to identify, prioritize, and develop a systematic decision-making framework of the challenging factors associated with QSE process execution. We conducted a literature survey to identify the challenging factors associated with QSE process and mapped them into 7 core categories. Additionally, we used a questionnaire survey to collect insights from practitioners regarding these challenges. To examine the relationships between core categories of challenging factors, we applied Interpretive Structure Modeling (ISM). Lastly, we applied fuzzy TOPSIS to rank the identified challenging factors concerning to their criticality for QSE process. We have identified 22 challenging factors of QSE process and mapped them to 7 core categories. The ISM results indicate that the ‘resources’ category has the most decisive influence on the other six core categories of the identified challenging factors. Moreover, the fuzzy TOPSIS indicates that ‘complex programming’, ‘limited software libraries’, ‘maintenance complexity’, ‘lack of training and workshops’, and ‘data encoding issues’ are the highest priority challenging factor for QSE process execution. Organizations using QSE could consider the identified challenging factors and their prioritization to improve their QSE process.