Genomic selection is based on breeding values that are estimated using genome‐wide dense marker maps. The objective of this paper was to investigate the effect of including or ignoring the polygenic effect on the accuracy of total genomic breeding values, when there is coverage of the genome with approximately one SNP per cM. The importance of the polygenic effect might differ for high and low heritability traits, and might depend on the design of the reference dataset. Hence, different scenarios were evaluated using stochastic simulation. Accuracies of the total breeding value of juvenile selection candidates depended on the number of animals included in the reference data. When excluding polygenic effects, those accuracies ranged from 0.38 to 0.55 and from 0.73 to 0.79 for traits with heritabilities of 10 and 50%, respectively. Accuracies were improved by including a polygenic effect in the model for the low heritability trait, when the LD‐measure r² between adjacent markers became smaller than ∼0.10, while for the high heritability trait there was already a small improvement at r² between adjacent markers of 0.14. In all situations, the estimated total genetic variance was underestimated, particularly when the polygenic effect was excluded from the model. The haplotype variance was less underestimated when more animals were added in the reference dataset.