We study the endomorphism algebra of the motiveattached to a non-CM elliptic modular cusp form. We prove that thisalgebra has the structure of a crossed product algebra over anumber field. As a consequence we obtain the Tate conjecture for themotive. We then investigate the Brauer class of this algebra.We show that in many cases it is locally at $p$ determined by the$p$-adic valuations of the Fourier coefficients of the form.This is joint work with Alexander Brown.