The computing power of microprocessors has exponentially increased in the past few decades, so the support to computation intensive multimedia applications has increased too. With such improved computing power, memory subsystem deficiency becomes the major barrier to support video encoder on Digital Signal Processor (DSP). H.264/AVC becomes the next generation video codec for embedded systems. In this paper, our focus is to enhance the encoding performance of H.264/AVC through memory optimization for DSP. The experimental results show that the proposed solutions can improve the H.264/AVC encoding speed by almost 30%. Also, the memory optimization method presented in this paper has been integrated into the developed embedded H.264/AVC video encoder. The encoder can perform real time encoding under four channels Common Intermediate Format (CIF) frame size, which is the typical requirement of networked video surveillance applications nowadays.