DeepFaceLab 2.0 XSeg Tutorial - DeepfakeVFX.com (2024)

What is XSeg?

If you’ve watched my previous tutorial on faceset extraction, you’ll already know that each face will include a default mask generated during extraction. DeepFaceLab includes a masking tool called XSeg, which allows you to specify which areas of the image is the face or the background. This mask is used during model training and later on for merging the final image.

Why Use XSeg?

Using XSeg will result in a better composition and increased likeness to the source faceset, more realistic eye and mouth movement, skin detail, and color. XSeg also allows you to exclude obstructions such as hands, hair, glasses, piercings, tattoos, and so on. While the default mask may be useful for smaller face types, larger face types such as Whole Face and Head do require a custom mask for the best result. Additionally, a custom mask is recommended when using model training style powers and color transfer modes.

Generic XSeg Pretrained Mask

The fastest way to get started with XSeg is to apply a pretrained mask to your faceset. DeepFaceLab includes a generic whole face XSeg mask. You can also pretrain your own or download a pretrained mask. The generic mask files can be found in the _internal/model_generic_xseg folder. To apply the mask to your images run the file ‘5.XSeg Generic) data_dst whole face mask apply’ and ‘5.XSeg Generic) data_src whole face mask apply’. Remember this is a Whole Face type mask and may not work the same on other face types. This generic mask is a good starting point for many projects, however there may be difficulty with extreme angles and dark, blurry, or heavily obstructed faces.

XSeg Terminology

Before getting into XSeg it’s important that you understand some basic terminology. The process of drawing masks is called labelling, as in labeling data. The lines and points that define the mask are called polygons. We can edit, fetch, or remove these polygons. The trained mask is a model produced by the XSeg trainer. We can apply or remove the trained mask. The applied mask is embedded in the faceset image and will be used during model training. The learned mask is created as part of the deepfake model and can be used during merging.

Launch the XSeg Editor

To create your own XSeg mask you will need to label some faces with mask polygons, then train the XSeg model and apply the mask to your facesets. First, open the XSeg editor by running the file ‘5.XSeg) data_dst mask edit’. You will see the XSeg splash screen and once the destination faceset images have loaded you will be able to use the interface.

XSeg Editor User Interface

First let’s go over the XSeg UI.

In the center of course is the current image canvas where you will label the polygons. Below that is the current filename and an image carousel with buttons for navigating to the previous and next files. Over to the right of the image filename, you will see the count of labeled images. In the bottom right corner, there is a trash you can click to delete the current image.

The main polygon tools you’ll be using are listed along the left-hand side. There are 2 drawing modes: poly include, and poly exclude. Poly include means drawing the area that includes part of the image, creating an inclusion mask around the face. Poly exclude means excluding part of the image, which I’ll come back to later on. You can easily undo and redo points, delete the entire polygon, and switch the cursor point mode to add or delete points.

On the right side are the mask preview tools. You can view the XSeg trained mask which, if applied first, will appear as a grey overlay on the background with a transparent area over the face. You can choose a red, green or blue color for the polygons to suit your preference. You can also view the baked in aka drawn polygon mask or the applied XSeg mask as simple alpha channel images. Finally, we have the cursor lock, which will keep the cursor centered, moving the image around instead.

Labeling Mask Polygons

Let’s start by labeling some of the easier faces that are clear, fully exposed, and not obstructed. Use the A and D keys or the on-screen arrows to scroll through the images. You can hold down the shift key to scroll faster. Holding the control key will let you skip to images that have already been labeled.

Choose a comfortable starting point and direction to work around the face. Use the left mouse button to place points along the edge of the face. The mouse wheel will zoom in and out, re-centering the image on the cursor. Use Ctrl + Z to undo previous points. When you are finished labeling the face you can complete the polygon by clicking on the first point.

You can then modify the polygon by moving, adding, or deleting points. As you work through the faceset try to keep the mask shapes somewhat consistent by following a similar path around the jaw and hairline in each image. With smaller face types you should follow the jawline and across the forehead just above the eyebrows. When using a Head type faceset you will need to include the entire face, ears, hair, and optionally part of the neck. Thin or moving hair will be difficult to deepfake so you may want to refine them in post-processing.

There is no need to focus on making a precise mask with hundreds of points; a few dozen will do. Instead, spend your time by labeling a variety of faces throughout the faceset yaw, pitch, and color ranges. You should also label a variety of facial expressions, such as with open and closed mouth, and with eyes looking in different directions. You can close the editor and use the sorting tool to reorder the images, making sure you have a good spread of labeled faces. It is recommended that you label at least a few dozen faces in both the source and destination facesets.

Masking Obstructions

Obstructions in front of the face can be excluded from the mask area. There are 2 methods which will produce the same result; choosing one is simply a matter of preference. The first and faster method is to draw the polygon around the face and follow the edge of the obstruction. The second method is to draw a polygon around the face, then draw a polygon around the object in exclusion mode. Press the W key or click the icon to switch to exclusion mode and draw a polygon around the object. This method will remove any part of the obstruction that intersects with or is inside the face polygon.

It is important to remember that when using exclusion mode, you must also draw an inclusion mask around the face. Do not label a face with only an exclusion mask. Press the Q key or use the icon to switch back to include mode. You should also label the obstruction on a variety of frames if possible.

Keep in mind that any changes you make to the shape of the mask may affect the training and merging of the deepfake face. You may need to take extra steps during training to ensure that the excluded obstruction does not morph the face. During merging you will have to account for blending the edge of the obstruction. For the best result you should handle these obstructions with post-processing software such as Adobe After Effects.

Fetch a Backup & Remove Labels

After you have labeled your images, you can close the XSeg editor. Now you can fetch a backup of your labeled faces by running ‘5.XSeg) data_dst mask_fetch’. This will copy all the XSeg labeled files to data_dst/aligned_xseg and you will be asked if you want to delete the original files.

The file ‘5.XSeg) data_dst mask remove’ will completely remove all XSeg labels you have created and restore the default mask. Warning: this will delete the polygons you have drawn and cannot be undone.

XSeg Training

The next step is to train the XSeg model so that it can create a mask based on the labels you provided. Run the file ‘5.XSeg) train’ and choose a hardware device from the list. Set the face type to be the same as your faceset. It is likely that you can use the highest batch size, however if the training fails to run you can lower the batch size.

The command window will show the training progress as numerical values while the preview window will show the images and masks being trained. Use the spacebar to cycle through the different previews, and the ‘P’ key to generate the current preview. You can see the transformations being applied to the images as the trainer attempts to match the labels to the faceset. At first the masks will be misshapen and choppy. After some time, you will notice the mask take on a consistent shape with defined edges. The model will be saved every (25 minutes), however you can also press the ‘S’ key to save the training, or the ‘Enter’ key to save and exit.

Applying the XSeg Mask

The XSeg mask has been trained but to use the mask you must first apply it to the faceset images. Run ‘5.XSeg) data_dst trained mask – apply’ and select the device to use. If you have more than one GPU, you can use it to apply the source faceset mask at the same time. Otherwise apply the source mask afterward by running ‘5.XSeg) data_src trained mask – apply’.

Once the masks are applied you can begin or continue training the deepfake model. However, I suggest you check the applied XSeg mask first. Open the XSeg editor then press the backtick or tilde key to toggle the applied mask. Use the ‘A’ and ‘D’ keys to scroll through the images and create new face labels where needed. Run the XSeg trainer again, apply the mask, and repeat the process until the applied mask is relatively clean.

You can remove the applied mask and return to the default mask using the files ‘5.XSeg) data_dst trained mask – remove.bat’ and ‘5.XSeg) data_scr trained mask – remove.bat’. This will not affect the polygon labels you have drawn.