In this paper, dynamic capacity management refers to the process of dynamically changing the capacity allocation (reservation) of a pseudo-wire established between two network end points. This process is based on certain criteria including instantaneous traffic load for the pseudo-wire, network utilization, time of day, or day of week. Frequent adjustment of the capacity yields a scalability issue in the form of a significant amount of message processing in the network elements involved in the capacity update process. On the other hand, if the capacity is adjusted once and for the worst possible traffic conditions, a significant amount of bandwidth may be wasted depending on the actual traffic load. There is then a need for dynamic capacity management that takes into account the tradeoff between scalability and bandwidth efficiency. This problem is motivated by voice over packet networks in which end-to-end reservation requests are initiated by PSTN voice calls and these reservations are aggregated into one single reservation in the core packet network for scalability. In this paper, we introduce a Markov decision framework for an optimal reservation aggregation scheme for voice over packet networks. Moreover, for problems with large sizes, we provide a suboptimal scheme using reinforcement learning. We show a significant improvement in bandwidth efficiency in voice over packet networks using aggregate reservations.