Bit error rate (BER) monitoring is the ultimate goal of performance monitoring in all digital transmission systems as well as optical fiber transmission systems. To achieve this goal, optimization of the decision threshold must also be considered because BER is dependent on the level of decision threshold. In this paper, we analyze a pseudo-error counting scheme and propose an algorithm to achieve both BER monitoring and adaptive decision threshold optimization in optical fiber transmission systems. To verify the effectiveness of the proposed algorithm, we conduct computer simulations in both Gaussian and non-Gaussian distribution cases. According to the simulation results, BER and the optimum decision threshold can be estimated with the errors of < 20% and < 10 mV, respectively, within 0.1-s processing time in > 40-Gb/s transmission systems.