Patient-specific modeling of the cardiovascular system offers a powerful framework for advancing treatment strategies and understanding disease mechanisms. These models rely heavily on clinical data, particularly imaging techniques that provide structural and functional insights. However, in-vivo imaging captures loaded states, i.e., deformed configurations under specific internal pressures. This is a problem for obtaining accurate biomechanical modeling, which requires a reference or unloaded configuration, especially for strain and stress estimation. Various methods have been proposed to estimate this reference configuration or capture its influence on forward simulations. Among these, inverse mechanics stands out as a promising approach, allowing the determination of the reference configuration using the observed physical domain, its mechanical properties, and the external forces acting on it. However, studies that have used either inverse mechanics or other methods to estimate the reference configuration have generally overlooked interactions with surrounding tissues. To address this limitation, we developed a method to estimate these external forces by penalizing unrealistic displacements in both heart and aorta models. For the heart, we demonstrate the critical role of external forces in determining the reference configuration from end-diastolic imaging data using inverse mechanics, showing how our method effectively recovers it. For the aorta, a predominantly passive tissue, we extend this approach further. Our method estimates not only the reference configuration and external forces but also the regional material stiffness using two states pressurized at different values. Notably, both cases are solved using a single finite element simulation, eliminating the need for optimization schemes. By integrating imaging data and biomechanics knowledge, our proposed methods can enhance patient-specific cardiovascular modeling, offering more physiological simulations that can provide better insights for clinical and research applications.