What's new
By:
Photoshop Gurus Forum

Welcome to Photoshop Gurus forum. Register a free account today to become a member! It's completely free. Once signed in, you'll enjoy an ad-free experience and be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

Layer Mask Technique


T

TonyCooper

Well-Known Member
Messages
93
Likes
42
Using PSCC(2014) and a Wacom tablet, when I use a layer mask to cut out an
area with the intent of allowing something else to show in the cut out area, I
can get nice, precise lines. However, I don't always want nice, precise edges.

For example, this image is of a model train layout. Using a layer mask, I'll remove
the wall at the top of the image and add a layer of one of my stock sky shots
behind the scene. Even though I work with highly magnified areas of the image,
and a soft brush, the edges of the pine trees will be too smooth and even.

Is there a technique that will result in an edge that is less sharp? Jagged, even,
since the edge of a tree is somewhat jagged.

I don't photograph model trains very often, but this is used as an example.

mask.jpg
 
Two features of an image enormously complicate any selection attempt. The first is if the both the background and subject are both complicated (ie, with lots of different colors, textures, shadows, etc.). The second is if the demarcation between foreground and background is soft because of focusing errors, camera movement (too low a shutter speed to hand hold) or film grain / digital noise.

The edges of the trees in the photo you posted have only a moderately complicated background and essentially no noise, but, unfortunately, are quite soft because of what looks appears to be camera movement and low contrast, omni-directional ambient lighting.

Unless there is something seriously wrong with your camera/lens (eg, it was once dropped), there is absolutely no reason that your d300 and lens shouldn't produce a *much* sharper image. Given the 6 second exposure (at ISO = 200 and f/22) that you used, you must have used a tripod or some other form of other camera support, but since everything in this image (ie, all distances) are all slightly blurry, my guess is that your camera mounting is not sturdy enough and is allowing some camera movement. BTW, did you use a cable release or the self-timer so that your finger did not have to be on the shutter button at the moment of exposure?

FWIW, I'm pretty sure that the blur is not from diffraction (even at f/22), but it's easy enough to check this.

Also, if you had someone hold a white sheet in back of the problem area and had increased the lighting contrast by using your own strobe lights instead of relying on the overhead lights in the room, this would have eliminated camera movement, increased the lighting contrast, and likely made the selection task much easier.

However, if you are stuck using this particular photo, Sam's suggestions are excellent. Specifically, "Refine Edge" is probably the first thing I would reach for in such cases. It can work wonders on tree lines.

You should also be aware that there are also commercial tools/plugins that one can bring to bear on the toughest cases. These include OnOne Software's "Mask Pro", Vertus' "Fluid Mask", Topaz's "ReMask", and even simpler-to-use (but excellent) color-based selection tools such as Asiva's "Selection" that allow one to select very highly specific ranges in hue, in saturation, and in luminosity. Asiva's tool can be thought of as channels-based selection method on steroids.

That being said, realize that the more sophisticated of these tools take lots of practice to use effectively, so give yourself plenty of time to learn and perfect each technique.

HTH,

Tom M
 
Thanks for the replies.

First, the image provided is not one that I intend to use for anything. I just grabbed
one with trees and a multi-patterned. multi-colored background as an example because
I'm interested in learning how to improve my technique using a Layer Mask.

It's soft for two reasons: I shot on a tripod and forgot about turning off the VR and because I didn't
use a timer delay and just pressed the shutter release. The D300, unfortunately,
doesn't have wireless remote release. The lighting and background was not something that
could be controlled under the circumstances of the shoot.

Regarding "refine selection", I sometimes use other tools to make a selection - including
the Quick Selection Tool, Quick Mask, and the Lassos - but my question is about using
a Layer Mask. As far as I know, "refine selection" isn't available using a Layer Mask.

My choice of tools to make a selection or to cut out a background depends on the image
and what the unwanted background is like. There is no "one way" to do it.

I watched the tutorials Sam provided, and did pick up something useful. It never occurred
to me to create a brush and use it with the clone tool. I can add some jagged edges
with that. I tried this on a practice image and it does work.

In response to Tom, I do have OnOne and have worked with that to drop a background,
but haven't found it particularly useful unless there's a clear differentiation between the
background colors and the subject colors. PS's Quick Mask or a Layer Mask with my
Wacom tablet is much easier for me.

Again, thanks for the suggestions and comments.
 
I have found that Topaz remask will give a softer less precise edge than Refine Edge .....BEFORE using the "Magic Brush" Make the "blue" mask in remask wide then look at your mask
 
TonyCooper said:
As far as I know, "refine selection" isn't available using a Layer Mask.
Click to expand...
Cmd/Ctrl + click the layer mask, this will make a selection of the mask. Then go to SELECT > REFINE EDGE. Or, just right click on your canvas and select Refine Edge.

edit: Here is another technique that demonstrates the use of the Dodge & Burn Tool in channel selection.

http://www.youtube.com/watch?v=TJR20e7JcBg

Any selection can be made into a layer mask, how you arrive at the selection is what matters.
 
Last edited:
IamSam said:
Cmd/Ctrl + click the layer mask, this will make a selection of the mask. Then go to SELECT > REFINE EDGE. Or, just right click on your canvas and select Refine Edge.

edit: Here is another technique that demonstrates the use of the Dodge & Burn Tool in channel selection.

http://www.youtube.com/watch?v=TJR20e7JcBg

Any selection can be made into a layer mask, how you arrive at the selection is what matters.
Click to expand...


That's when you use "Refine mask"
 
Yes, mike is also right. Thanks Mike, I forget it's there and results are the same.

Highlight your layer mask and go to SELECT > REFINE MASK (Alt/Opt + Cmd/Ctrl + R)or right click over the highlighted mask.



EDIT: Actually Mike is more correct. After playing with the two, REFINE MASK works much better on the layer mask.
 
Last edited:

Back
Top