Publications on the structure of the physical phase space (PS) of dynamical systems with gauge symmetry are reviewed. The recently discovered phenomenon of reduction of the phase space of the physical degrees of freedom is studied systematically on mechanical models with a finite number of dynamical variables. In the simplest case of one degree of freedom this phenomenon consists of replacement of the phase space by a cone that is unfoldable into a half-plane. In the general case the reduction of the phase space is related with the existence of a residual discrete gauge group, acting in the physical space after the unphysical variables are eliminated. In ``natural'' gauges for the adjoint representation this group is isomorphic to Weyl's group. A wide class of modes with both the normal and Grassmann (anticommuting) variables and with arbitrary compact gauge groups is studied; the classical analysis and the quantum analysis are performed in parallel. It is shown that the reduction of the phase space radically changes the physical characteristics of the system, in particular its energy spectrum. A significant part of the review is devoted to a description of such systems on the basis of the method of Hamiltonian path integrals (HPIs). It is shown how the HPI is modified in the case of an arbitrary gauge group. The main attention is devoted to the correct formulation of the HPI with a poor choice of gauge. The analysis performed can serve as an elementary illustration of the well-known problem of copies in the theory of Yang--Mills fields. The dependence of the quasiclassical description on the structure of the phase space is demonstrated on a model with quantum-mechanical instantons.