I have understood the concept of summary vs detailed transfer concept but I didn't get what GL references table is doing?
The GL interface table is a meduim to pass data from subledgers to GL as subledgers cannot send data directly to GL.
Secondly,I didn't get what do you mean by that GL references table is populated when import references is enabled.Please explain.

The role of GL References table is to store the links between the data in the source system and that in GL.
One can choose to do a summary transfer, meaning multiple lines (e.g. invoice distribution lines) in the source system would be summarized as one journal line in GL. If you choose to drill down to the source system from Account Inquiry screen then the system should know which lines from the source system resulted into that journal line (many to one relationship). The role to maintain the mapping is played by GL References table.