Installing JupyterHub extensions

[infogulch]

The JupyterHub app allows you to install extensions, such as jupyterlab-git.

When I select to install the extension and trigger a rebuild (required apparently), the rebuild fails, and suggests to run it manually with jupyter lab build.

jovyan@5a04ce7f0bf2:~$ jupyter lab build
[LabBuildApp] JupyterLab 3.0.15
[LabBuildApp] Building in /opt/conda/share/jupyter/lab
[LabBuildApp] Building jupyterlab assets (production, minimized)
Build failed.
Troubleshooting: If the build failed due to an out-of-memory error, you
may be able to fix it by disabling the `dev_build` and/or `minimize` options.

If you are building via the `jupyter lab build` command, you can disable
these options like so:

jupyter lab build --dev-build=False --minimize=False

You can also disable these options for all JupyterLab builds by adding these
lines to a Jupyter config file named `jupyter_config.py`:

c.LabBuildApp.minimize = False
c.LabBuildApp.dev_build = False

If you don't already have a `jupyter_config.py` file, you can create one by
adding a blank file of that name to any of the Jupyter config directories.
The config directories can be listed by running:

jupyter --paths

Explanation:

- `dev-build`: This option controls whether a `dev` or a more streamlined
`production` build is used. This option will default to `False` (i.e., the
`production` build) for most users. However, if you have any labextensions
installed from local files, this option will instead default to `True`.
Explicitly setting `dev-build` to `False` will ensure that the `production`
build is used in all circumstances.

- `minimize`: This option controls whether your JS bundle is minified
during the Webpack build, which helps to improve JupyterLab's overall
performance. However, the minifier plugin used by Webpack is very memory
intensive, so turning it off may help the build finish successfully in
low-memory environments.

An error occured.
RuntimeError: JupyterLab failed to build
See the log file for details:  /tmp/jupyterlab-debug-dk5alq3x.log

Printing the logs shows:

jovyan@5a04ce7f0bf2:~$ cat /tmp/jupyterlab-debug-dk5alq3x.log
[LabBuildApp] Building in /opt/conda/share/jupyter/lab
[LabBuildApp] Node v15.14.0

[LabBuildApp] Yarn configuration loaded.
[LabBuildApp] Building jupyterlab assets (production, minimized)
[LabBuildApp] > node /opt/conda/lib/python3.9/site-packages/jupyterlab/staging/yarn.js install --non-interactive
[LabBuildApp] yarn install v1.21.1
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
Done in 1.18s.

[LabBuildApp] > node /opt/conda/lib/python3.9/site-packages/jupyterlab/staging/yarn.js yarn-deduplicate -s fewer --fail
[LabBuildApp] yarn run v1.21.1
$ /opt/conda/share/jupyter/lab/staging/node_modules/.bin/yarn-deduplicate -s fewer --fail
Done in 0.81s.

[LabBuildApp] > node /opt/conda/lib/python3.9/site-packages/jupyterlab/staging/yarn.js run build:prod:minimize
[LabBuildApp] yarn run v1.21.1
$ webpack --config webpack.prod.minimize.config.js

<--- Last few GCs --->

[281:0x56120a659a50]    43888 ms: Scavenge 248.8 (257.5) -> 248.4 (257.5) MB, 1.9 / 0.0 ms  (average mu = 0.501, current mu = 0.333) allocation failure 
[281:0x56120a659a50]    43897 ms: Scavenge 248.8 (257.5) -> 248.6 (258.0) MB, 2.8 / 0.0 ms  (average mu = 0.501, current mu = 0.333) allocation failure 
[281:0x56120a659a50]    44272 ms: Mark-sweep 249.5 (258.0) -> 248.8 (259.0) MB, 361.5 / 0.0 ms  (average mu = 0.363, current mu = 0.115) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 0x7f6679728ed9 node::Abort() [/opt/conda/bin/../lib/libnode.so.88]
 2: 0x7f6679619e37 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > node::SPrintFImpl<int&, char const*>(char const*, int&, char const*&&) [/opt/conda/bin/../lib/libnode.so.88]
 3: 0x7f6679ae9442 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/conda/bin/../lib/libnode.so.88]
 4: 0x7f6679ae971b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/conda/bin/../lib/libnode.so.88]
 5: 0x7f6679caef36  [/opt/conda/bin/../lib/libnode.so.88]
 6: 0x7f6679ce37a4  [/opt/conda/bin/../lib/libnode.so.88]
 7: 0x7f6679ced83d void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/opt/conda/bin/../lib/libnode.so.88]
 8: 0x7f6679cf4b6d v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/opt/conda/bin/../lib/libnode.so.88]
 9: 0x7f6679cdc334 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/opt/conda/bin/../lib/libnode.so.88]
10: 0x7f6679cdc60a  [/opt/conda/bin/../lib/libnode.so.88]
11: 0x7f6679cccf35 v8::internal::ItemParallelJob::Task::RunInternal() [/opt/conda/bin/../lib/libnode.so.88]
12: 0x7f6679ccd321 v8::internal::ItemParallelJob::Run() [/opt/conda/bin/../lib/libnode.so.88]
13: 0x7f6679ceea8f void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/opt/conda/bin/../lib/libnode.so.88]
14: 0x7f6679cf35b5 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/opt/conda/bin/../lib/libnode.so.88]
15: 0x7f6679cf38dd v8::internal::MarkCompactCollector::Evacuate() [/opt/conda/bin/../lib/libnode.so.88]
16: 0x7f6679d106ce v8::internal::MarkCompactCollector::CollectGarbage() [/opt/conda/bin/../lib/libnode.so.88]
17: 0x7f6679cc1ae5 v8::internal::Heap::MarkCompact() [/opt/conda/bin/../lib/libnode.so.88]
18: 0x7f6679cc22e9 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/conda/bin/../lib/libnode.so.88]
19: 0x7f6679cc2a5a v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/conda/bin/../lib/libnode.so.88]
20: 0x7f6679cc5d2d v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/conda/bin/../lib/libnode.so.88]
21: 0x7f6679cc5d95 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/conda/bin/../lib/libnode.so.88]
22: 0x7f6679c8af6b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/conda/bin/../lib/libnode.so.88]
23: 0x7f6679fed92c v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/conda/bin/../lib/libnode.so.88]
24: 0x7f667995f3f9  [/opt/conda/bin/../lib/libnode.so.88]
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[LabBuildApp] JupyterLab failed to build
[LabBuildApp] Traceback (most recent call last):

[LabBuildApp]   File "/opt/conda/lib/python3.9/site-packages/jupyterlab/debuglog.py", line 47, in debug_logging
    yield

[LabBuildApp]   File "/opt/conda/lib/python3.9/site-packages/jupyterlab/labapp.py", line 166, in start
    raise e

[LabBuildApp]   File "/opt/conda/lib/python3.9/site-packages/jupyterlab/labapp.py", line 162, in start
    build(name=self.name, version=self.version,

[LabBuildApp]   File "/opt/conda/lib/python3.9/site-packages/jupyterlab/commands.py", line 469, in build
    return handler.build(name=name, version=version, static_url=static_url,

[LabBuildApp]   File "/opt/conda/lib/python3.9/site-packages/jupyterlab/commands.py", line 678, in build
    raise RuntimeError(msg)

[LabBuildApp] RuntimeError: JupyterLab failed to build

[LabBuildApp] Exiting application: JupyterLab

This seems to indicate that it runs out of memory during the rebuild. I increased the app memory usage in cloudron to 1G and also set c.Spawner.mem_limit = '1G', then restarted both the app and the notebook 'server' from the control panel, but neither of those actions changed the outcome.

Thoughts?

[infogulch]

I just noticed that git is available from the terminal, which solves my immediate problem, though I am still interested in how to use extensions.

[girish]

@infogulch said in Installing JupyterHub extensions:

This seems to indicate that it runs out of memory during the rebuild. I increased the app memory usage in cloudron to 1G and also set c.Spawner.mem_limit = '1G', then restarted both the app and the notebook 'server' from the control panel, but neither of those actions changed the outcome.

Did you also remove the existing notebook containers? Changing just the memory limit does not change the memory limit of already created notebook containers. This is because re-creating container is "lossy" (as in the conda/pip packages that got installed have to be re-installed). Please see https://docs.cloudron.io/apps/jupyterhub/#notebook-memory-limit . You have to run /app/code/remove_notebook_containers.py

[girish]

@infogulch OK, I tried this quickly now and it looks like 1GB is not enough for those webpack builds. I just put it to 4GB and that worked out. Maybe 2GB will suffice too...

jovyan@195c02caae19:~$ jupyter lab build
[LabBuildApp] JupyterLab 3.0.15
[LabBuildApp] Building in /opt/conda/share/jupyter/lab
[LabBuildApp] Building jupyterlab assets (production, minimized)
jovyan@195c02caae19:~$ 

[bitjungle]

@girish said in Installing JupyterHub extensions:

@infogulch OK, I tried this quickly now and it looks like 1GB is not enough for those webpack builds. I just put it to 4GB and that worked out. Maybe 2GB will suffice too...

I tried several different memory settings. In my use case 4G was the minimum needed for rebuilding.