RTS linker: unloadObj() does not actually unload the code

[simonmar]

We've known about this for a long time, but it hasn't been a pressing issue in GHCi. The problem with actually unloading code is that there might be pointers into it from the heap or other RTS data structures, so we need to do a full heap traversal to discover whether it is safe to unload code or not.

This is an issue for a project at Facebook that needs to have long running processes that regularly unload and load code using the RTS linker, and right now the process grows over time.

This ticket is to track the issue, and I'm also working on a fix. The same problem affects the dynamic linker, although I'm not planning to fix that (yet).

[ezyang]

Excellent! I've run into this problem, and I'm curious what the fix's approach is.

[simonmar]

preview patch

[simonmar]

I've attached a patch that sort of works. There's at least one problem still to fix (it doesn't get the static objects correctly when the GC is running in parallel).

It seems to work with some simple testing in GHCi, but I still need to make some automated test cases.

[simonmar]

fixed version of the patch

[igloo]

By "in GHCi", I assumed you mean in 7.6, or HEAD with DYNAMIC_GHC_PROGRAMS=NO?

[ezyang]

OK, so essentially, do a heap census. Sounds good to me. (I think I'd like it to work for the dynamic linker to but we want to eject that code, don't we?)

[simonmar]

@igloo: yes, I have DYNAMIC_GHC_PROGRAMS=NO in my mk/build.mk. I don't know how to do this with the system linker yet, because there doesn't seem to be an easy way to get hold of the start and end address of the image in memory - there's a fairly complex API in rtld-audit, but even that only seems to give you the start address.

Anyway, for the application I need this for, we're using the RTS linker and not shared libraries. So please don't make the RTS linker go away :-)

[ezyang]

OK, so maybe the comments here are out of date? ​http://hackage.haskell.org/trac/ghc/wiki/DynamicByDefault

[heisenbug]

Great! This is a dearly needed feature for ​http://hackage.haskell.org/package/dynamic-loader too...

[igloo]

Replying to simonmar:

Anyway, for the application I need this for, we're using the RTS linker and not shared libraries. So please don't make the RTS linker go away :-)

My plan was to leave it in 7.8, but not used by default, and then to remove it from HEAD shortly after 7.8.1 is released. The goal is to stop maintaining the ghci linker, after all!

[simonmar]

Replying to igloo:

My plan was to leave it in 7.8, but not used by default, and then to remove it from HEAD shortly after 7.8.1 is released. The goal is to stop maintaining the ghci linker, after all!

I thought we had some platforms that didn't have dynamic linking support, e.g. ARM/Linux?

I personally don't care if we get rid of the MACH-O support, and maybe even the COFF support from the linker. The ELF support is fairly solid though, and for the use case I'm interested in we don't run into any of the known limitations.

Maybe I could use shared libraries. But (a) this is a scenario where dynamic linking is generally shunned because it creates more problems than it solves (deploying to large numbers of machines), (b) we really care about performance, and dynamic linking adds unnecessary overhead, and (c) we need to unload objects, and I don't know how to do that with shared libraries yet.

I still buy the arguments for moving to dynamic linking for GHCi. But applications with plugin support is a use case we weren't really thinking about during that discussion, and I think the tradeoffs are different.

[igloo]

For platforms that don't support either the GHCi linker or dynlibs currently, then I think it would be easier and more useful to add dynlib support. There may be one or two (I'd guess that Arm Linux and PPC Linux are the only likely candidates) that currently support the GHCi linker and not dynlibs, but I think that it would be better for people who care about such platforms to add dynlib support rather than continuing to maintain the GHCi linker.

[Simon Marlow <marlowsd@…>]

In bdfefb3b72a71cd0afca6e7766456c0d97c47c86/ghc:

Really unload object code when it is safe to do so (#8039)

The next major GC after an unloadObj() will do a traversal of the heap
to determine whether the object code can be removed from memory or
not.  We'll keep doing these until it is safe to remove the object
code.

In my experiments with GHCi, the objects get unloaded immediately,
which is a good sign: we're not accidentally holding on to any
references anywhere in the GHC data structures.

Changes relative to the patch earlier posted on the ticket:
 - fix two memory leaks discovered with Valgrind, after
   testing with tests/rts/linker_unload.c

[Simon Marlow <marlowsd@…>]

In 269c89062dd5b6d2b8bc4d41e5dca0eca2308d02/testsuite:

Add a test for unloading object files in the linker (#8039)

[simonmar]

Not ready to close yet: the linker doesn't work when DYNAMIC_GHC_PROGRAMS is on. I can't think of a reason for this.

[simonmar]

Now fixed: 0717462d7724bfe904cffda5706dd74d0b31b883/ghc, 4b94f0c757dd924ba104172e9323b521e689b837/testsuite

[heisenbug]

I have pushed a new branch on my github ghc fork. It ​contains just one commit, about unloading dlopened shared object files.

It is actually untested, and non-windows. But it may be of use for doing what Simon did with unloading .o files in this ticket. Might be interesting to folks.

Of course the GC must be wired up in a similar fashion, etc. Be careful.

[simonmar]

The big missing piece with unloading of .so's is knowing their boundaries in memory so that the GC can detect whether there are any references left. I don't know of a way to do that with the system linker, if anyone knows of a way then I'd be happy to add support for unloading .so's.

[heisenbug]

Replying to simonmar:

The big missing piece with unloading of .so's is knowing their boundaries in memory

​Eli Bendersky's article suggests to use the ​dl_iterate_phdr function for finding information about loaded libraries. Seems to be linux only. There is a ​workaround on OSX, though, on StackOverflow.

And here is how Böhm-Demers-Weiser's GC implement a ​callback function for dl_iterate_phdr.

[simonmar]

Created #8238 to track unloading of shared libraries.