The use of technology in cricket has seen a significant increase in recent years, leading to overlapping computer vision-based research efforts. This study aims to extract front pitch view shots in cricket broadcasts by utilizing deep learning. The front pitch view (FPV) shots include ball delivery by the bowler and the stroke played by the batter. FPV shots are valuable for highlight generation, automatic commentary generation and bowling and batting techniques analysis. We classify each broadcast video frame as FPV and non-FPV using deep-learning models. We also detect pitch and umpire using recent object detectors. If a frame is classified FPV and pitch and umpire are detectable, it is considered as part of the FPV shot. In each FPV shot, we then track the cricket ball for further analysis using deep-learning-based fast-moving object trackers. Using the ball tracking, we first detect the ball bounce position on the pitch and using that position we classify the FPV video shots as full-pitched, good-length, or short-pitched deliveries. For training and testing purposes, we manually annotated four different datasets: (i) Front Pitch View (FPV) classification dataset consisting of 10k frames including 5k positive and 5k negative frames, (ii) Pitch and Umpire Detection (PUD) dataset consisting of 5k frames with manually annotated pitches and umpires, (iii) Ball Tracking  Dataset (BTD) consisting of 3k frames with balls annotated and 6k frames with no visible balls, (iv) Ball Delivery Classification (BDC) dataset consisting of 300 video clips annotated as full-pitched, good-length and short-pitched deliveries. For the FPV and PUD datasets, training and testing splits are marked as 50% each. For the BTD and BDC datasets training and testing splits are marked as 80% and 20%. The results of the proposed framework demonstrates its ability to accurately classify cricket shots into FPV and non-FPV, creating the foundation for large datasets of video shots of ball deliveries and strokes that can be used for further processing and semantic extraction.  Our work has the potential to be of great benefit to cricket coaches and players for analyzing ball deliveries in a match. Our trained models and four datasets are online available at our Github page https://www.github.com/KumailAbb/FPVShotDetection.