In this paper, we consider the problem of improving the convergence speed of an average consensus gossip algorithm by sparsifying a sufficiently dense network graph. Thus, instead of adding links, as usually proposed in the literature, or globally optimizing the mixing matrix of the gossip algorithm for a given network, which requires global knowledge at every node, we find a sparser network that has better spectral properties and faster convergence than the original denser one. This allows to reduce simultaneously both the convergence time and the communication cost involved in the execution of the gossip algorithm. We first show why it is possible to sparsify a network while increasing its convergence rate and also that there exists an optimal fraction of links to be removed. As a benchmark, we devise a centralized method that selects in an optimal way the set of links to be removed from the original network. Then, we propose a low complexity and scalable decentralized protocol requiring only local information at each node, which also generates a sparser network having a substantially better convergence rate. Simulation results are presented to verify and show clearly the efficiency of our approach.