The amount of raw information available for basketball analytics has been given a great boost with the availability of player tracking data. This facilitates detailed analyses of player movement patterns. In this paper, we focus on the difficult problem of offensive playcall classification. While outstanding individual players are crucial for the success of a team, the strategies that a team can execute and their understanding of the opposing team’s strategies also greatly influence game outcomes. These strategies often involve complex interactions between players. We apply techniques from machine learning to directly process SportVU tracking data, specifically variants of neural networks. Our system can label as many sequences with the correct playcall given roughly double the data a human expert needs with high precision, but takes only a fraction of the time. We also show that the system can achieve good recognition rates when trained on one season and tested on the next.