Reinforcement learning (RL) is a promising new approach for automatically developing effective policies for real-time self-* management. RL has the potential to achieve superior performance to traditional methods while requiring less built-in domain knowledge. Several case studies from real and simulated systems-management applications demonstrate RL's promises and challenges. These studies show that standard online RL can learn effective policies in feasible training times. Moreover, a Hybrid RL approach can profit from any knowledge contained in an existing policy by training on the policy's observable behavior without needing to interface directly to such knowledge.