Radiative transfer, the propagation of electromagnetic radiation through a medium, is pervasive in nature and necessary in many human endeavors, from astrophysics and climate prediction to nuclear technology and medical imaging. Numerical solutions to the time-independent radiative transfer equation are often computationally expensive and memory intensive, as the radiation field is six dimensional: three in space, two in angle and one in wavelength. In large-domain applications such as the atmosphere, the radiation field is often parametrized, or some dimensions are omitted altogether. As models of such application areas are better able to leverage exascale computing resources and increase in resolution, the error introduced by these simplifications grows and becomes more pronounced. Recently, a spatial hp-adaptive discontinuous Galerkin method was introduced to solve the radiative transfer equation which was shown to converge exponentially for problems featuring steep gradients and its computational cost and memory requirements scaled linearly with the number of spatial degrees of freedom. However, spatial adaptivity does not ameliorate challenges that arise from the angular discretization of the domain, such as the ray effect and false scattering. Here we present an extension of this method to include angular hp-adaptivity and its application to several test problems featuring steep gradients in both space and angle.