We introduce a new class of graphs, called quasi $m$-Cayley graphs, having good symmetry properties, in the sense that they admit a group of automorphisms G that fixes a vertex of the graph and acts semiregularly on the other vertices. We determine when these graphs are strongly regular, and this leads us to define a new algebro-combinatorial structure, called quasi-partial difference family, or QPDF for short. We give several infinite families and sporadic examples of QPDFs. We also study several properties of QPDFs and determine, under several conditions, the form of the parameters of QPDFs when the group G is cyclic.