snowpack build - Error: EMFILE: too many open files

[N8-B]

We are currently doing some tests with snowpack on our very large code base and are getting the EMFILE: too many open files error in our Bamboo builds. I suspect that the issue has to do with the ulimit -n of the machines used in these deploys.

The maximum limit for these machines is 4096 and, unfortunately, we cannot set a higher limit. We believe the error is the result of the reads being done by es-module-lexer during the import scanning process.

Is this something that anyone else has encountered?

I think a potential workaround for this error would be to use graceful-fs for those reads.

This is a real bummer for us, as this blocks us from being able to use snowpack for our production builds.

[FredKSchott]

If your theory is right, then the scanner would be causing trouble in the scanImports() function, where we call fs.promises.readFile() on all scanned files at once. You appear to be the first to hit this limit in a breaking fashion! (others at your size are probably just seeing degraded performance).

Mind sharing the output when you run snowpack build --verbose, or at least the part of the output around where the EMFILE error is occurring? That would help us confirm, although regardless this feels like a valuable improvement to make for everyone.

I'll spin up a prioritized issue to resolve, would love your help on a fix if you're interested!

[FredKSchott]

Issue created: #1245

[waynevanson]

@FredKSchott I feel I've hit the same issue.

The project I'm using has react & material-ui, which is huge and probably a problem.

I'm on voidlinux, so I'm unsure if you have ulimit -n. If you do, what value do you get?
If yours is default higher, I can increase it but It'll take a while as doing ulimit -n ##### won't work because it's above 1024.

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30181
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096 # the issue, according to the internet
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30181
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[FredKSchott]

if it's a problem in your node_modules folder, then that would actually be due to Rollup. Mind sharing your snowpack build --verbose output?

[waynevanson]

Here are both outputs for build and dev.

# yarn snowpack build --verbose
yarn run v1.22.10
warning package.json: No license field
$ /home/wayne/Documents/code/nodejs/codeeq/node_modules/.bin/snowpack build --verbose
[snowpack] run command: build
[snowpack] config loaded: {
  plugins: [
    {
      name: '@snowpack/plugin-esbuild',
      resolve: [Object],
      load: [AsyncFunction: load],
      cleanup: [Function: cleanup]
    }
  ],
  alias: {},
  scripts: {},
  exclude: [
    '**/node_modules/**/*',
    '**/web_modules/**/*',
    '**/.types/**/*',
    '/home/wayne/Documents/code/nodejs/codeeq/build/**/*'
  ],
  installOptions: { packageLookupFields: [], rollup: { plugins: [] } },
  devOptions: {
    secure: false,
    hostname: 'localhost',
    port: 8080,
    open: 'default',
    output: 'dashboard',
    fallback: 'index.html',
    hmrDelay: 0,
    hmrPort: undefined,
    hmrErrorOverlay: true
  },
  buildOptions: {
    out: '/home/wayne/Documents/code/nodejs/codeeq/build',
    baseUrl: '/',
    webModulesUrl: '/web_modules',
    clean: false,
    metaDir: '__snowpack__',
    minify: false,
    sourceMaps: false,
    watch: false,
    htmlFragments: false
  },
  testOptions: { files: [ '__tests__/**/*', '**/*.@(spec|test).*' ] },
  experiments: { source: 'local', routes: [], ssr: false },
  webDependencies: undefined,
  knownEntrypoints: [],
  proxy: [],
  mount: {
    '/home/wayne/Documents/code/nodejs/codeeq': { url: '/', static: false, resolve: true }
  },
  _extensionMap: { '.mjs': '.js', '.jsx': '.js', '.ts': '.js', '.tsx': '.js' }
}
[snowpack] lockfile not loaded
[snowpack] generating meta files
[snowpack] ! building source files...

<--- Last few GCs --->

[2064:0x5560257c2df0]     8843 ms: Mark-sweep (reduce) 2110.6 (2113.1) -> 2110.6 (2114.1) MB, 28.6 / 0.0 ms  (average mu = 0.475, current mu = 0.001) last resort GC in old space requested
[2064:0x5560257c2df0]     8867 ms: Mark-sweep (reduce) 2110.6 (2113.1) -> 2110.6 (2114.1) MB, 24.3 / 0.0 ms  (average mu = 0.312, current mu = 0.001) last resort GC in old space requested


<--- JS stacktrace --->

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x5560234a4b70 node::Abort() [/usr/bin/node]
 2: 0x5560233e046c node::FatalError(char const*, char const*) [/usr/bin/node]
 3: 0x55602362a322 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
 4: 0x55602362a59e v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
 5: 0x5560237d7485  [/usr/bin/node]
 6: 0x5560237ed651 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
 7: 0x5560237b44a5 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/bin/node]
 8: 0x5560237ae4d4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [/usr/bin/node]
 9: 0x5560237b08c0 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [/usr/bin/node]
10: 0x5560237b6714 v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const> const&, v8::internal::AllocationType) [/usr/bin/node]
11: 0x556023646f78 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [/usr/bin/node]
12: 0x5560235906f2 node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [/usr/bin/node]
13: 0x55602347a1d8  [/usr/bin/node]
14: 0x55602369343f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/usr/bin/node]
15: 0x5560236937f0  [/usr/bin/node]
16: 0x55602369408a  [/usr/bin/node]
17: 0x556023694976 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
18: 0x556023e7d379  [/usr/bin/node]
Aborted
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

# yarn snowpack dev --verbose


[snowpack] run command: dev
[snowpack] loaded plugin: @snowpack/plugin-typescript
[snowpack] config loaded: {
  plugins: [
    {
      name: '@snowpack/plugin-esbuild',
      resolve: [Object],
      load: [AsyncFunction: load],
      cleanup: [Function: cleanup]
    },
    {
      name: '@snowpack/plugin-typescript',
      run: [AsyncFunction: run],
      markChanged: [Function (anonymous)]
    }
  ],
  alias: {},
  scripts: {},
  exclude: [
    '**/node_modules/**/*',
    '**/web_modules/**/*',
    '**/.types/**/*',
    '/home/wayne/Documents/code/nodejs/codeeq/build/**/*'
  ],
  installOptions: { packageLookupFields: [], rollup: { plugins: [] } },
  devOptions: {
    secure: false,
    hostname: 'localhost',
    port: 8080,
    open: 'default',
    output: 'dashboard',
    fallback: 'index.html',
    hmrDelay: 0,
    hmrPort: undefined,
    hmrErrorOverlay: true
  },
  buildOptions: {
    out: '/home/wayne/Documents/code/nodejs/codeeq/build',
    baseUrl: '/',
    webModulesUrl: '/web_modules',
    clean: false,
    metaDir: '__snowpack__',
    minify: false,
    sourceMaps: false,
    watch: false,
    htmlFragments: false
  },
  testOptions: { files: [ '__tests__/**/*', '**/*.@(spec|test).*' ] },
  experiments: { source: 'local', routes: [], ssr: false },
  webDependencies: undefined,
  knownEntrypoints: [],
  proxy: [],
  mount: {
    '/home/wayne/Documents/code/nodejs/codeeq': { url: '/', static: false, resolve: true }
  },
  _extensionMap: { '.mjs': '.js', '.jsx': '.js', '.ts': '.js', '.tsx': '.js' }
}
[snowpack] lockfile not loaded
[snowpack] attaching console.log listeners
[snowpack] dashboard started
[snowpack] Using in-memory cache.
[snowpack] Mounting directories:
[snowpack]  -> '/home/wayne/Documents/code/nodejs/codeeq' as URL '/'
[snowpack] Using cache folder: node_modules/.cache/snowpack/development
[snowpack] Cache out of date or missing. Updating...
[snowpack] Skip scanning netlify.toml for imports (unknown file extension .toml)
[snowpack] Skip scanning package.json for imports (unknown file extension .json)
[snowpack] Scanning parcel-bundle-reports/default.html for imports as HTML
[snowpack] Scanning public/index.css for imports as CSS
[snowpack] Scanning public/index.html for imports as HTML
[snowpack] Scanning public/index.tsx for imports as JS
[snowpack] Skip scanning readme.md for imports (unknown file extension .md)
[snowpack] Scanning snowpack.config.js for imports as JS
[snowpack] Scanning src/app.tsx for imports as JS
[snowpack] Scanning src/components/code-block.tsx for imports as JS
[snowpack] Scanning src/components/page-template/index.tsx for imports as JS
[snowpack] Scanning src/components/page-template/nav-drawer.tsx for imports as JS
[snowpack] Scanning src/components/page-template/top-bar.tsx for imports as JS
[snowpack] Scanning src/components/page-template/use-styles.tsx for imports as JS
[snowpack] Scanning src/components/pattern.tsx for imports as JS
[snowpack] Scanning src/components/statement.tsx for imports as JS
[snowpack] Scanning src/data.ts for imports as JS
[snowpack] Scanning src/domain.ts for imports as JS
[snowpack] Scanning src/fp-ts-react.ts for imports as JS
[snowpack] Scanning src/lib.ts for imports as JS
[snowpack] Scanning src/pages/home.tsx for imports as JS
[snowpack] Scanning src/pages/statement.tsx for imports as JS
[snowpack] Scanning src/pages/statements.tsx for imports as JS
[snowpack] Scanning src/routes.ts for imports as JS
[snowpack] Scanning src/state.tsx for imports as JS
[snowpack] Scanning tailwind.config.js for imports as JS
[snowpack] Skip scanning tsconfig.json for imports (unknown file extension .json)
[snowpack] Skip scanning yarn-error.log for imports (unknown file extension .log)
[snowpack] Skip scanning yarn.lock for imports (unknown file extension .lock)
[snowpack] ! installing dependencies...
[snowpack] running installer with options: {
  input: {
    '@material-ui/core': '/home/wayne/Documents/code/nodejs/codeeq/node_modules/@material-ui/core/esm/index.js',
    '@material-ui/core/styles': '/home/wayne/Documents/code/nodejs/codeeq/node_modules/@material-ui/core/esm/styles/index.js',
    '@material-ui/icons': '/home/wayne/Documents/code/nodejs/codeeq/node_modules/@material-ui/icons/esm/index.js',
    clsx: '/home/wayne/Documents/code/nodejs/codeeq/node_modules/clsx/dist/clsx.m.js',
    'fp-ts': '/home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/index.js',
    'fp-ts/lib/function': '/home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/lib/function.js',
    'fp-ts/lib/pipeable': '/home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/lib/pipeable.js',
    highlightjs: '/home/wayne/Documents/code/nodejs/codeeq/node_modules/highlight.js/lib/index.js',
    react: '/home/wayne/Documents/code/nodejs/codeeq/node_modules/react/index.js',
    'react-dom': '/home/wayne/Documents/code/nodejs/codeeq/node_modules/react-dom/index.js',
    'react-router-dom': '/home/wayne/Documents/code/nodejs/codeeq/node_modules/react-router-dom/esm/react-router-dom.js'
  },
  context: undefined,
  external: [Function: external],
  treeshake: { moduleSideEffects: 'no-external' },
  plugins: [
    { name: 'alias', resolveId: [Function: noop] },
    {
      name: 'snowpack:fetch-handler',
      resolveId: [Function: resolveId],
      load: [Function: load]
    },
    {
      name: 'node-resolve',
      buildStart: [Function: buildStart],
      generateBundle: [Function: generateBundle],
      resolveId: [AsyncFunction: resolveId],
      load: [Function: load],
      getPackageInfoForId: [Function: getPackageInfoForId]
    },
    { name: 'json', transform: [Function: transform] },
    {
      name: 'snowpack:rollup-plugin-css',
      transform: [AsyncFunction: transform]
    },
    {
      name: 'replace',
      renderChunk: [Function: renderChunk],
      transform: [Function: transform]
    },
    {
      name: 'commonjs',
      buildStart: [Function: buildStart],
      resolveId: [Function: resolveId],
      load: [Function: load],
      transform: [Function: transform],
      moduleParsed: [Function: moduleParsed]
    },
    {
      name: 'snowpack:wrap-install-targets',
      buildStart: [Function: buildStart],
      resolveId: [Function: resolveId],
      load: [Function: load]
    },
    {
      name: 'snowpack:rollup-plugin-stats',
      generateBundle: [Function: generateBundle],
      writeBundle: [Function: writeBundle]
    },
    {
      name: 'snowpack:rollup-plugin-node-process-polyfill',
      transform: [Function: transform],
      resolveId: [Function: resolveId],
      load: [Function: load]
    },
    {
      name: 'snowpack:rollup-plugin-catch-unresolved',
      resolveId: [Function: resolveId]
    },
    {
      name: 'snowpack:rollup-plugin-strip-source-mapping',
      transform: [Function: transform]
    }
  ],
  onwarn: [Function: onwarn]
}
[snowpack] cjsAutoDetectExportsUntrusted error: Unexpected import statement in CJS module.
  at @:7:8
[snowpack] cjsAutoDetectExportsUntrusted error: Unexpected export statement in CJS module.
  at @:1:8
[snowpack] cjsAutoDetectExportsUntrusted error: Unexpected export statement in CJS module.
  at @:6:8
[snowpack] cjsAutoDetectExportsUntrusted error: Unexpected export statement in CJS module.
  at @:29:8
[snowpack] cjsAutoDetectExportsUntrusted error: Unexpected import statement in CJS module.
  at @:4:8
[snowpack] cjsAutoDetectExportsUntrusted error: require is not defined
[snowpack] cjsAutoDetectExportsUntrusted error: Unexpected import statement in CJS module.
  at @:1:8
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/ReaderTaskEither.js:1:17 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/ReaderTaskEither.js:1:25 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/ReaderTaskEither.js:10:19 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/ReaderTaskEither.js:10:27 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/StateReaderTaskEither.js:1:17 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/StateReaderTaskEither.js:1:25 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/StateReaderTaskEither.js:10:19 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/StateReaderTaskEither.js:10:27 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/Task.js:1:17 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/Task.js:1:25 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/Task.js:10:19 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/Task.js:10:27 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/TaskEither.js:1:17 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/TaskEither.js:1:25 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/TaskEither.js:10:19 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] /home/wayne/Documents/code/nodejs/codeeq/node_modules/fp-ts/es6/TaskEither.js:10:27 The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
[snowpack] Could not load /home/wayne/Documents/code/nodejs/codeeq/node_modules/@material-ui/icons/esm/RestaurantMenuTwoTone.js (imported by node_modules/@material-ui/icons/esm/index.js): EMFILE: too many open files, open '/home/wayne/Documents/code/nodejs/codeeq/node_modules/@material-ui/icons/esm/RestaurantMenuTwoTone.js'
[snowpack] Error: Could not load /home/wayne/Documents/code/nodejs/codeeq/node_modules/@material-ui/icons/esm/RestaurantMenuTwoTone.js (imported by node_modules/@material-ui/icons/esm/index.js): EMFILE: too many open files, open '/home/wayne/Documents/code/nodejs/codeeq/node_modules/@material-ui/icons/esm/RestaurantMenuTwoTone.js'

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[waynevanson]

@FredKSchott I figured out a way to change the amount of file descriptors available from this reddit thread

I'm having other issues relating to config, which is a separate concern for me.

thank you for prompt response :)