When it comes to family grandparents for example, that language-wise the mother's side of the family is always considered to be outside, as 外 (wài) means "outside".
The same principle may also help you remember the different words for cousins. The word 堂 (táng) means 'hall', all cousins in your own hall will be belonging to your father's family. All cousins on your mother's side do not belong to your hall. So a male cousin older than you on your father's side, will be a 堂哥 (táng gē), while on your mother's side it would be a 表哥 (biǎo gē). Compare with "older brother".
