An ad hoc agent environment is a way for users to interact with an ambient intelligent environment. Agents are associated with every device, service or content. The user interacts with his environment as a whole, instead of interacting with individual applications on individual devices. Devices and services in the environment have to be more or less independent, which fits well with the notion that agents are autonomous. A research application that demonstrates some of the user agent interaction issues involved when employing agents has been developed. It shows the tension between the user being in control and the autonomy of agents. The notion of cooperating groups is introduced as a way for users to gain control over which agents collaborate. Users can then establish connections between devices and content that are meaningful to them, in the context of their task.