The finite element method (FEM) is suitable for the analysis of a complicated region that includes nonlinear materials, whereas the boundary element method (BEM) is naturally effective for analyzing a very large region with linear characteristics. Therefore, considering the advantages in both methods, a novel algorithm for the alternate application of the FEM and BEM to magnetic field problems with the open boundary is presented. This approach avoids the disadvantages of the typical numerical methods with the open boundary problem such as a great number of unknown values for the FEM and non-symmetric matrix for the Hybrid FE-BE method. The solution of the overall problems is obtained by iterative calculations accompanied with the new acceleration method.