Recently proposed control scheme for a stable teleoperation, which was based on two-port time-domain passivity approach[21], has been successful for a contact with high stiffness environments. However, we found several conservatisms during the contact with deformable environments and unconstrained motion. The two-port time-domain passivity controller was excessively dissipating energy even though it was not necessary for some cases of an unconstrained motion and soft contact. The main reason of those conservatisms was on the fact that the two-port time-domain passivity controller was activated without considering the amount of energy dissipation at the master and slave manipulators. Especially, the exclusion of the slave manipulator from the two-port was the dominant reason of the conservatisms. In this paper, we consider the amount of energy dissipation at slave manipulator for designing the time-domain passivity observer and controller. The measured interaction force between slave manipulator and environment allow the time-domain passivity observer to include the amount of energy dissipation at the slave manipulator. Based on the modified passivity observer, reference energy following method[24] is applied to satisfy the passivity condition in real-time. The feasibility of the developed methods is proved with experiments. Improved performance is obtained for an interaction with deformable environments and an unconstrained motion.