Real-time Coastal Hatching

You've probably seen old maps with little lines along the coast, right?  Back in the days of ink and engravings those were tedious to draw.  I've tried a lot of different methods to add this kind of hatching to maps in Photoshop, but they've all been slow.

What I'm going to show you here is applied in real time.  Once you've set up your layer style, you can just draw and the hatching is applied automatically.

These instructions are for Photoshop, so if you're using a different drawing program, let me know what you do differently.

Mask a fill layer

Start off with a simple fill layer.  This is a layer that's already full of a single color, with a mask that lets you show or hide it as you like.  (Go to the menu: Layer > New Fill Layer > Solid Color.)

Draw the shape of the land in the mask to reveal the color.  (By default the mask will show color everywhere, which is the opposite of what you want.  There are many ways to switch the mask to hide everything: one is to select the mask and hit Ctrl-I to invert it.)

Noisy inner shadow

Open up the layer style.  (Right-click on the name of the layer and go to Blending Options, or just double-click on the layer.)

Add an inner shadow to show the edge of the coastline.  The default settings make a soft-edged shadow, which isn't what we want here.  Turn the Choke most of the way up (so the shadow is mostly opaque) and set the Size to just a few pixels.  Add a little bit of Noise (I'm using 10% for this demo).

Notice how the white area is hidden here?  Put the layer in a folder, then set that folder's blending mode to Multiply.  That means anything in white will be transparent — this will be important later.

Stroke hatching

Next we're going to add the little lines that stick out into the water.  You'll need to draw a hatching pattern to do this, or you can use mine if you like.  (Non-commercial purposes only.)

If you draw one yourself, make sure the left and right edges match (and the top and bottom edges) so that it can tile seamlessly.  (I could do a tutorial on making seamless textures if you guys want.)

Open up your hatching tile as a separate image, make it into a pattern, then head back to the map.  (Edit > Define Pattern.)

In the layer style, add a Stroke.  For the Fill Type, choose Pattern, and select the hatching pattern you just defined.  Set the Blend Mode to Difference.  This will draw a band of that pattern around the edge of the coastline.

Click the Overprint check box for this stroke, so the hatching will be drawn on top of the other effects we'll add soon.

This is ok, but it's not great.  Ideally we'd like the hatching to be bolder near the land and less intense further out in the water.  To do that, we'll need a few more copies of the pattern.

Add another Stroke just like the previous one, but increase the size.

Then add a third Stroke the same way.  (You might need to tinker with the size and opacity a bit to get an effect you like.)

Reversed outer glow to hide hatching

We're getting closer, but the outer edge is still too regular.  We need something further out from the coast that's fuzzy and noisy to cover up the edges of the hatching.

An outer glow or a drop shadow would be perfect, but those usually are stronger near the object, fading off further away.  Thankfully, we can invert the direction.

(Outer Glow and Drop Shadow both have a similar effect here.  I ended up using both on my own image for finer control, but you can go it with just one.)

Add an Outer Glow, then reverse the contour.

If you don't already have a reversed contour in that little menu, just click on the contour itself and drag the handles to reverse it, like in this image.

When we're done the outer glow will need to be white, but for now, make it red.

Adjust the size and spread until the outer glow is strongest out at the far edge of the hatching, fading off as the glow approaches the land.  Add some noise for roughness, then it should look something like this:

Notice how that red band is covering up the outer part of the hatching?  That's exactly what we want, if only the red band were invisible.

Change its color from red to white, and it'll disappear.  (It disappears because this layer is inside a folder set to Multiply.  Using the Multiply blending mode, white manifests as transparency.)

That's it for the coastline.  Draw the rest of the owl and you get something like this:

Summary

1. Make a hatching pattern from a separate image.
   
2. Draw the land shape using a fill layer.
3. Put the fill layer in a folder that has Blend (Multiply).
   
4. Add a layer style: Inner Shadow: high Choke, some Noise.
5. Add a Stroke: Fill (Pattern), Blend (Difference), Overprint.
6. Add two more of that Stroke, each one with a larger Size.
7. Add a red Outer Glow: Contour (reversed), some Noise, moderate Spread, enough Size so it's out at the edge of the hatching.
8. Change the Outer Glow to white.
   

Once you've made your layer style, you can save it or just copy it from one map to another.  You only have to do those steps once; from then on, all your coastal hatching can be done in real time.

Are there any steps you'd like to see in more detail?  Let me know, either here or on the subreddit, and I'll add them in.