Welcome to the forum!
The 'sbps' category contains the squads. Open that category at 'Categories' in the menu and select your squad. In there should be a 'squad_requirement_ext', otherwise add it by right clicking on 'extensions' > 'add' at the top of the file. This 'requirement_ext' contains the requirements to build the squad. Add 'required_cap' to the requirement_table and set the 'max_cap' to your preferred limit.
The 'ui_name' is the text which is shown ingame after the limit is reached.
The 'reason' can be set to 'usage', this keeps the squad visible in the building, but locks the build option.
'usage_and_display' makes the whole squad button invisible (not recommended).
'display' should never be used, since hotkeys and the AI can still build the squad, but it is not visible in the building.
I hope this helps, just ask if something is not clear