The ferry was Stena Superfast VII, which operates between Northern Ireland and Scotland.
It has a capacity for 1,300 passengers and 660 cars.
The submarine was submerged at the depth needed to extend its periscope above the surface of the water.
The Royal Navy would not confirm which of its 10 submarines was involved. They are all nuclear-powered but only four carry Trident nuclear missiles.
A spokesman for the Marine Accident Investigation Branch (MAIB) said: "In November, we were notified of a close-quarters incident between the ro-ro (roll-on/roll-off) ferry Stena Superfast VII and a submarine operating at periscope depth.
"We have carried out a preliminary assessment of the evidence in this case and the Chief Inspector of Marine Accidents has decided to open a safety investigation.
"The investigation is being conducted with the full co-operation of the Royal Navy. A report will be published when our investigation has concluded."