Predictive multiplexed switching is a new approach for building interconnection switches for high performance parallel systems. This approach advocates sacrificing some link bandwidth in return for more efficient network control and simpler connection management. The main idea is to depart from the traditional packet and wormhole switching in favor of row data communication over established communication pipes (connections). The overhead of this circuit switching approach can be justified when established connections are repeatedly used before they are torn down. For this, we use multiplexing to allow multiple connections to share the same resources (links and switches), thus avoiding tearing down connections prematurely. The connection establishment overhead is further reduced by exploring communication locality and predictability in applications that exhibit these properties. We present the design of an interconnection system which is based on multiplexed switching and which establishes connections either reactively, in response to dynamically generated requests, or proactively, in response to compiler or application directives. A communication prediction component may be supported to reduce the network control overhead in applications that exhibit communication locality and predictability. The design is evaluated using hardware design, synthesis, and cycle-accurate simulation. Comparison with more traditional switching paradigms shows the potential of our predictive multiplexed switching approach.