Forced Async operation in a (event like) method
Instead of WatchDirectories.FirstOrDefault you can make use of Any() which is more meaningful.  

Also if you are checking Any() you can return early and save horizontal spacing.  

public Task AddDirectoryWatcher(string directory) {
    return Task.Run(async () =&gt; {
        if (!Directory.Exists(directory)) return;

        using (var context = _contextFactory.CreateContext()) {
            if (context.WatchDirectories.Any(wd =&gt; wd.Directory == directory)) { return; };

            var childWatchers = context.WatchDirectories.Where(wd =&gt; wd.Directory.StartsWith(directory));




This AddDirectoryWatcher() is using a context to delete and insert items. Doesn't the RemoveDirectoryWatcher() method already uses a context too ? If yes, the call to context.Delete(watchDirectory); is redundant.  



Instead of checking if the passed directory exists and silently returning , I would check this before calling the method and throw a DirectoryNotFoundException.