The heart of any physics engine is a collision detection system that provides information needed for calculations of the constraint forces. This forces must be calculated to resolve equations that describe the motion of rigid bodies. So, first of all, we need a system that can handle a collision detection in a system of rigid bodies and be able to generate contact information.
Contact detection system has to deal with bodies penetrations and time-zero intersections, because of finite time step and numerical errors. This system must provide information about the contact time and contact position. There would be a set of intersection points, so there is no a contact point, but there is a contact manifold. Let us omit for a while problems of finding exact contact information, assume we need only information if collision happened or not.
How can we detect collisions in a system of interacting objects? If we do not make any specific assumptions, we have to consider that all possible bodies pairs potentially can collide. So, in the most naive solution, we have to check all possible pairs of interacting objects. Obviously, this is a highly expensive approach.