There is not a 'Location Default Group' associated with Locations unless you specifically synced one on the Group level. The automated process will typically add the machines to the 'All Agents', 'All Clients.XYZ', and the 'Agent Types - Windows Server/Workstation/Laptop' and some others based on internal searches. Looking at the 'Effective Policy', you can see the Groups -> Auto Joined by default and the Client group which is automatically synced. You can go through each of these group searches and add an exclusion, but some of these searches are locked by default and will not allow changes.
A majority of the groups are added based on the Location settings by enabling Onboarding, assigning a Service Plan, and placing the machines 'Under MSP Contract' and 'Patching Covered Under Contract'. What might work for you is to create a new Location, do not enable any features and add a machine. You do not have to assign and default actions and you would get a pretty clean server.
Be careful if you do want to build an Exclusion EDF, as certain terms like 'FROM', 'AND', and 'OR' will cause the search to fail, along with certain characters. An example, an EDF named 'Exclude from MSE' would cause issues. Instead use just 'Exclude MSE'.