Symlinked directories don't work for routes inside the app folder

[snivels]

Summary

I am creating a directory outside of the project, let's call it vendors and creating a symlink from app/vendor to one of these vendors in this external directory. However, when navigating to a page.tsx inside one of these directories it just results in 404 being shown in the browser.

I've tried to trace the code and it seems to be resolving the symlinks correctly in the next\dist\lib\recursive-readdir.js file inside the recursiveReadDir function. When testing whether a link is a directory or a pathname it correctly (seemingly) ends up in the directories array.

I've created a minimal reproduction of the issue and linked the repo to this discussion.

Steps to reproduce:

1. Clone the repo attached
2. npm i
3. Navigate to src\apps in the command line
4. Create a symlink in Windows by running mklink /D vendor ..\..\vendors\test-vendor-1 or in Unix with ln -s ../../vendors/test-vendor-1 vendor
5. npm run dev
6. Navigate to http://localhost:3000/vendor/posts

Observe the following:

If you remove the symlink and paste the vendor files directly into the app/vendor folder you observe:

Link to repo with reproduction.

Additional information

No response

Example

https://github.com/snivels/symlinks-not-working

[snivels]

It seems like watchpack is the issue, it's using fs.lstat instead of fs.stat and is not recognizing symlinks.

Next.js itself has correct support for symlinks.

If anyone wants to use symlinks in their project, you can change the calls from .lstat( to .stat( in the following file:

node_modules\next\dist\compiled\watchpack

Then use patch-package to create a patch for your version of Next.js:

npx patch-package next

And then update your postinstall to patch packages after install:

Closing the discussion with a resolution in case anyone wants a temporary solution until watchpack updates this.

Also updated the reproduction branch to now include the patch.

[bgw]

The scope of this bug is relatively small: You can symlink files and non-route directories, just not route directories. The bug is in the development server's route manifest generation.

I don't have the time to prioritize and fix this myself, but hopefully this helps explain things if somebody does want to try to fix it.

---

I looked into this. Unfortunately, a proper fix would be a lot more complicated than modifying watchpack:

• The issue is in packages/next/src/server/lib/router-utils/setup-dev-bundler.ts where watchpack is called (so you're very close!).
• Watchpack intentionally watches the symlink and not the target. Watchpack needs to know if the symlink changes to point to a new target. If you just replace fs.lstat with fs.stat, it'll break that behavior.

Watchpack has a followSymlinks: true option, but it doesn't do what we want either. It ensures that both the symlink and the target are in the watched set, but it won't return the "contents" of a symlinked directory. E.g. for a directory structure like this:

/foo             (directory)
/foo/buzz.txt    (regular file)
/foo/bar.txt     (symlink to /bar.txt)
/foo/fizz        (symlink to /fizz)
/bar.txt         (regular file)
/fizz            (directory)
/fizz/nested.txt (regular file)

if you watch /foo with followSymlinks: false, your watched set will be:

/foo
/foo/buzz.txt
/foo/bar.txt
/foo/fizz

If you use followSymlinks: true, it'll follow those symlinks outside of the /foo directory:

/foo
/foo/buzz.txt
/foo/bar.txt
/foo/fizz
/bar.txt
/fizz

However, it won't watch the contents of /fizz. For that, we'd need some logic to simulate expanding the contents of the symlinked directory.

We can simplify things a bit by not allowing symlinks for route directories that point outside of the watched project directory/repository, but a proper solution would still be rather complicated:

• Use readlink on every file to try to read the contents of a symlink (if it errors with EINVAL, we know it's not a symlink).
• Cache the link resolution, so we only re-run it on changed files.
• If a symlink points outside of the watched directory, log a warning and don't try to follow it.
• Use the cached set of all symlinks inside the watched directory to follow all symlink chains and expand their paths. Bail out if you detect cycles.
• For expanded paths pointing to directories, copy all of their children. Rewrite the path's parent directory to be the symlink, so that the symlink "contains" these files.

If fixed, this may reveal other issues with directory traversal in other parts of Next.js.