Welcome to Photoshop Gurus forum. Register a free account today to become a member! It's completely free. Once signed in, you'll enjoy an ad-free experience and be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!
I wish I could help..................I'm simply out of ideas. I really don't think this is a Photoshop issue anymore.
Make a simple PSD file with a transparent BG, and pull a black stroked white ellipse shape. Then upload the file here.
What happens when you start with a very large document? Do you have the same problem?
You can always transform the ellipse down.
Not sure what your seeing but there is no pixelation at all in the second two shots I posted. These are screen shots and they won't look the same as the original PSD file on your computer. In the first image, the point was to show you that at the size your working, even my Ps shows pixelation just like yours.All three of your images looked like a pixelated stroke to me as well the last two images were less but it was still there.
I think I found your solution. My own version of Photoshop started doing what you described. My lines were all looking jagged. So when I double-clicked to open Photoshop, I hit Control+ALT+SHIT. That brings up a box that asks if you want to delete your file presets. Click Yes. It worked like a dream for me.