Richmond Council leader, Councillor Gareth Roberts, said the earlier kick-off time of 14:00 BST saw "no noticeable improvement in behaviour" with fans seeing the match "primarily as a drunken, al fresco fancy dress party".
He called on the RFU and the Armed Forces to consider whether they want to allow the fixture to continue.
Councillor Roberts said if the fixture is to remain, it should be rotated at other stadiums, confirming that they 'will be putting that case to the RFU'.
He added the council have requested an "urgent meeting" with RFU officials and organisers to seek assurances that residents will not experience another day of "drunken, loutish behaviour which brings the RFU and our armed services personnel into disrepute".
The council leader said if the assurances are "not satisfactorily forthcoming", then it is time the "fixture is moved out of our borough".
An RFU spokesperson told Forces News: "We took significant steps to improve this event including reduced licensing hours, earlier kick off and bar closures plus no games before the main match.
"While there have been some complaints received, behaviour at this year’s match was better than previous years.
"We will have a full debrief of this event with the police and the council as we do every year and will look at every complaint individually."