Targeted fertilization of grass clover leys shows high financial and environmental potentials leading to higher yields of increased quality, while reducing nitrate leaching. To realize the gains, an accurate fertilization map is required, which is closely related to the local composition of plant species in the biomass. In our setup, we utilize a top-down canopy view of the grass clover ley to estimate the composition of the vegetation, and predict the composition of the dry matter of the forage. Using a deep learning approach, the canopy image is automatically pixel wise segmented and classified into white clover, red clover, grass and weeds. While robust grass and clover segmentation has proven to be a difficult task to automate, red and white clover discrimination in images is challenging, even for human experts, due to many visual similarities between the two clover species. Using high-resolution color images with a ground sampling distance of 4 to 6 pixels per mm and data simulation of hierarchical labels, a cascaded convolutional neural network was trained for segmentation and classification. Clover, grass and weeds was automatically segmented and classified with a pixel wise accuracy of 87.3 percent, while red clovers and white clovers could be distinguished automatically with 89.6 percent accuracy. Utilizing the image analysis on 179 images of mixed crop plots of ryegrass, white clover and red clover, demonstrated a linear correlation between the detected clover and clover species fractions in the canopy, and the corresponding compositions in harvested dry matter.