Rate of exothermic/endothermic reactions
A reaction being endothermic or exothermic depends on thermochemistry while the rate of a reaction depends on its chemical kinetics. Hence, there is no relationship between the activation energy of a reaction and its enthalpy change. Also there is no relationship between the change enthalpy and the rate of a reaction. A rate of reaction is dependent on the concentration of the reactants which are to a certain power which is multiplied by a certain constant, k, which is called the reaction rate constant. For example, for the general reaction: $$\ce{A + B -&gt; C}$$ the rate of the reaction is given by the following equation: $$ rate = k[A]^a[B]^b$$
The reaction rate constant is not dependent on whether a reaction is exothermic or endothermic. However is does depend on the temperature and and also the activation energy of the reaction, given by the equation: $$k = Ae^\frac{-E_a}{RT}$$ So you are right to say that a reaction will be slower if it has a higher activation energy, but you are wrong to say that endothermic reactions have a higher activation energy than exothermic reactions.

There are actually several endothermic reactions that are fast, for example, dissolving common table salt (NaCl) occurs pretty quickly, but it is actually a endothermic reaction. Meanwhile, the spontaneously decomposition of diamond to graphite is actually a exothermic reaction, however it is extremely slow. It is so slow that it not even observed after several decades.