A pedestrian-vehicle detection network (PVDNet) is presented for pedestrian and vehicle detection in driving environment based on deep learning. First, on the low layers, an improved skip connection called Multi-Level Skip Connection (MLSC) is proposed to accelerate the convergence speed and the accuracy of the model. Second, on the top layers, a Multi-Layer Features Fusion (MLFF) method is designed to improve the detection accuracy by combining the low-level features with the high-level features. Finally, on the output layer, an One-Dimensional Convolution (ODC) Method is proposed to reduce the model parameters and improve the detection speed by replacing the fully connection layer. Experiments of the proposed PVDNet were carried out on the PascalVOC2007, PascalVOC2012, MS COCO, KITTI datasets. results show that, compared with the original Faster R-CNN, the mean average detection accuracies on the PascalVOC2007, PascalVOC2012, MS COCO, KITTI datasets are promoted 3.7%, 6.1%, 5.6%, 9.62% respectively by using PVDNet.