Use ghc-6.12's new Unicode I/O

[judah]

We should use the new Unicode I/O library on ghc-6.12.1 and later.

Things which should be changed:

• POSIX interaction when stdin is a terminal
• POSIX/Windows i/o when stdin is not a terminal
• POSIX/Windows custom getDirectoryContents for filename completion.

[judah]

Punting this task until ghc-6.14; the new I/O library doesn't supply a few bits yet:

• Full Windows codepage support
• Decode POSIX filenames to Unicode
• Transliterate encoding errors

[guest]

Replying to judah:

ghc-7.2.1 improved Unicode Support. http://www.haskell.org/ghc/docs/7.2.1/html/users_guide/release-7-2-1.html#id567292

Okay, I know that some problems are left now. http://hackage.haskell.org/trac/ghc/wiki/Status/Encoding-Tickets But I want know what is current problem to switch GHC's unicode text I/O.

- Full Windows codepage support

Is ghc-7.2.1's problem just it? Or does ghc-7.2.1 have other prolems, too?

[guest]

I heard that utf8-string's encode/decode are slower than GHC's.

http://www.serpentine.com/blog/2010/10/15/unicode-text-performance-improvements/

How about use GHC's unicode I/O's utf8 encode/decode instead of utf8-string's one, if GHC's unicode I/O isn't good enough?

[judah]

Replying to guest:

Replying to judah: ghc-7.2.1 improved Unicode Support. http://www.haskell.org/ghc/docs/7.2.1/html/users_guide/release-7-2-1.html#id567292 Okay, I know that some problems are left now. http://hackage.haskell.org/trac/ghc/wiki/Status/Encoding-Tickets But I want know what is current problem to switch GHC's unicode text I/O.

- Full Windows codepage support

Is ghc-7.2.1's problem just it? Or does ghc-7.2.1 have other prolems, too?

On POSIX, from a quick glance it looks like we have everything necessary to switch over. The tricky part will be making sure it still works on older versions (i.e., by choosing between the old or new code using CPP).

On Windows, it's harder because you need to use the Win32 API to see console user events such as arrow keys. That, plus the codepage issue, means I probably won't touch it for now.

I'll update this ticket as I make progress on the issue. (You can add yourself to the CC field to get emails when that happens)

The full codepage support is a problem, but

[judah]

Clarification: I mean I'll work on switching over the POSIX backend, but probably not the Windows backend. This will still noticeably reduce the codebase and dependencies; in particular, utf8-string shouldn't be necessary anymore.

[guest]

Replying to judah:

Okay, I understand current status. Thank you for your information and clarification!

[judah]

We now use the base library's encoding/decoding on ghc>=7.4.1. As mentioned, this removes the dependency on utf8-string for newer versions of GHC.

(For reference, we require ghc-7.4.1 due to ghc bug #5436 which was first fixed in that version.)