I wanted to point out that there is another approach to consider that might bypass the pixelation (which might be as simple as anti-alias not being turned on)
Since you have provided the PSD file, here is an approach that you can take
- Turn off the Final 2 Layer and turn on and expand the Raw 2 group
- Turn the Background Layer into a Regular Layer and turn off its visibility
- On the compass Layer, convert the Layers effects to Layers by right click int he Layer panel panel on the Layers effect icon and choose "Create Layers"
- One of the Layers created is the "Compass's Outer Stroke". Highlight this Layer and turn the transparency into a Layer Mask with Layer > Layer Mask > From Transparency
- With that mask selected, use Cmd + I (Ctrl + I for PCs) to invert the mask
- Drag that mask down to the Anchor Layer
- Turn off visibility of Compass's Outer Stroke Layer
I think this achieves your objective. No selections required as the masking was achieved with already existing information from the stroke of the Layer Effects
Hope this alternate approach is worth considering
John Wheeler