How to Smooth the Edges of a Shape or Mask in After Effects

Sometimes when editing videos in After Effects, you will want to enclose part of your videos inside some custom shape and leave the rest invisible. However, you may not always want the edges of such a selection to be as sharp, but you may like the edges to look soft and blurred. 

The Pen Tool is ideal for this purpose. If you repeatedly click G on your keyboard, you will find a different option from those used to create or modify custom shapes. Of course, we are talking about the Mask Feather Tool, which you can use to smooth the edges of a shape or mask in After Effects. Here’s how to use it.

Creating a solid background before using the Mask Feather Tool

Masks work similarly to shapes, but instead of drawing an object from scratch, masks select a part of your background to create a shape that includes and excludes specific parts of a layer in your compositions.​​ 

If you have followed our previous tutorials, you will remember that available options for making shapes also serve to mask out backdrops. The Mask Feather tool is one of them, so you must create a solid background by clicking Ctrl + Y or Cmd + Y to use it.

How to use the Mask Feather Tool in After Effects to soften the edges of a mask 

Once you have selected your background color and successfully created it, hit the Pen Tool. You’ll see how instead of the cursor having a star (which would indicate that you are about to create an object), it has a square with a dot inside. That icon means that the tool will create a mask.

• Start putting dots on your composition in the same way you would make a Shape with the Pen Tool.

• If, when placing the points, you did it by holding the clicks to make a curved mask, you can refine its sides by right-clicking, selecting Mask and Shape Path, and hitting RotoBezier.

• Click G repeatedly until the Pen Tool icon turns into a feather. That's the Mask Feather Tool. (Image 38_5)

• Place yourself on the edge of your shape until you see a plus sign appear next to the cursor. At that point, hold the click and increase or decrease its size. This way, you will see your edges soften.

Move the handle that appears while you are smoothing your shape to further define or blur your mask to your liking.

This is an interesting effect to mask your videos naturally and leave out the composition all you don’t need. Play with its proportions to see all the handy design possibilities that After Effects offers.