Cancer is a primary cause of morbidity and mortality worldwide. The radiotherapy plays a more and more important role in cancer treatment. In the radiotherapy, the dose distribution maps in patient need to be calculated and evaluated for the purpose of killing tumor and protecting healthy tissue. Monte Carlo (MC) radiation transport calculation is able to account for all aspects of radiological physics within 3D heterogeneous media such as the human body and generate the dose distribution maps accurately. However, an MC calculation for doses in radiotherapy usually takes a great mass of time to achieve acceptable statistical uncertainty, impeding the MC methods from wider clinic applications. Here we introduce a convolutional neural network (CNN), termed as Monte Carlo Denoising Net (MCDNet), to achieve the acceleration of the MC dose calculations in radiotherapy, which is trained to directly predict the high-photon (noise-free) dose maps from the low-photon (noise-much) dose maps. Thirty patients with postoperative rectal cancer who accepted intensity-modulated radiation therapy (IMRT) were enrolled in this study. 3D Gamma Index Passing Rate (GIPR) is used to evaluate the performance of predicted dose maps. The experimental results demonstrate that the MCDNet can improve the GIPR of dose maps of 1×10⁷ photons over that of 1×10⁸ photons, yielding over 10× speed-up in terms of photon numbers used in the MC simulations of IMRT. It is of great potential to investigate the performance of this method on the other tumor sites and treatment modalities.