Changes between Version 6 and Version 7 of StaticData


Ignore:
Timestamp:
Feb 7, 2018 11:04:20 AM (21 months ago)
Author:
hsyl20
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • StaticData

    v6 v7  
    11= Static Data
    22
    3 WIP proposal allowing to store static data/objects into programs (see #5218).
     3WIP proposal allowing to store static data/objects into programs (see #5218). It would allow large resources data (images, sounds, etc.) to be embedded (see #14741).
    44
    5 == Desugaring static strings
     5== Step 1: support dependent object files
    66
    7 With OverloadedStrings, when we have:
     7https://phabricator.haskell.org/D4217
     8
     9We need to be able to link with arbitrary object files. Object files can contain arbitrary data in their data sections (data, rodata, bss, etc.).
     10
     11== Step 2: provide helpers to generate object files
     12
     13Provide helpers (in GHC and in TH) to generate object files (ELF, etc.) containing a data section with specified properties: read-only or mutable, section alignment, allow section merging, etc.
     14
     15Provide helper to generate unique data symbols and to add symbols referring to the data in the symbol table of the object file.
     16
     17At this step, we can already use TH to embed data. We can retrieve the data address from the symbol with:
    818
    919{{{#!hs
    10    "My text" :: String
    11    "My text" :: Text
    12    "123456"  :: ByteString
     20foreign import ccall "&" myDataSymbol :: Ptr ()
    1321}}}
    1422
    15 it is desugared to:
     23== Step 3: add GHC primitive
    1624
    17 {{{#!hs
    18    fromString "My text" :: String
    19    fromString "My text" :: Text
    20    fromString "123456"  :: ByteString
    21 }}}
     25To make the interface easier to use even when TH is not supported, let's add a `StaticData#` primitive type.
    2226
    23 Let's add a StaticStrings extension that desugars to this instead:
    24 
    25 {{{#!hs
    26    [string|My text|] :: String
    27    [text|My text|]   :: Text
    28    [byte|123456|]    :: ByteString
    29 }}}
    30 
    31 The quasiquoter is selected from the type. I.e.,
    32 
    33 {{{#!hs
    34    class StringQuote a where
    35       fromStringQuote :: QuasiQuoter
    36 
    37       -- or directly
    38       fromStringQuote :: String -> Q Exp
    39 }}}
    40 
    41 It makes string literals quasiquoters in disguise. It's only for syntactic convenience (the type can be inferred, hence the quasiquoter can be inferred).
    42 
    43 === Builtin StringQuote
    44 
    45 Not all GHC support Template Haskell. Hence some StringQuote instances could be builtin or provided by compiler plugins. E.g.,
    46 * String (current string storage)
    47 * Text (UTF-16 encoding)
    48 
    49 
    50 == Storing raw bytes
    51 
    52 We want to be able to store raw bytes ("static data") into programs (e.g., to
    53 embed resources like images, arrays, etc.).
    54 
    55 Let's add a `StaticData#` primitive type. `StaticData#` have a size in bytes, an
    56 alignment constraint and can be:
     27`StaticData#` have a size in bytes, an alignment constraint and can be:
    5728   * initialized (we know their contents) and immutable
    5829   * initialized and mutable
     
    7748      , staticDataAlignment :: Word    -- ^ Alignment constraint (1 if none)
    7849      , staticDataMutable   :: Bool    -- ^ Is the data mutable?
    79       , staticDataContents  :: Maybe ByteString
     50      , staticDataContents  :: Maybe ByteString -- ^ Initialized or not? (maybe support 0-initialized data too?)
     51      , staticDataSymbol    :: String
    8052      }
    8153}}}
    8254
    83 Static data end up in appropriate sections of the program (.bss, .data, .rodata, etc.).
    84 Immutable data should be shared (with least common multiple alignment) to reduce
    85 binary size.
     55Static data end up in appropriate sections (.bss, .data, .rodata, etc.) in dependent object files.
     56
     57Identical immutable data should be shared (with least common multiple alignment) to reduce binary size.
    8658
    8759Note that only the contents of the static data is stored in programs: data size
     
    8961
    9062
    91 == Storing Haskell objects
     63== Step 4: Storing Haskell objects
    9264
    9365Some packages would like to store static Haskell object (e.g., `ByteArray#`) and not raw bytes (which are retrieved from an
    9466`Addr#`).
    9567
    96 We could use a static compact region embedded in the program. Let's add a
     68We could use a static compact region embedded in object files (reusing the `StaticData#` machinery). Let's add a
    9769`StaticObject#` primitive type which is a reference to an object in the static
    9870compact region.
     
    12294(in order to support mutable unboxed vectors for instance).
    12395
     96== Step 5: revamp string literals
    12497
    125 == Desugaring usual strings
     98We would like string literals to be `StaticData#` like other data.
    12699
    127 Instead of using a specific string-literal, we can use generic static data.
     100Currently with OverloadedStrings, the following:
    128101
    129 Desugaring becomes:
     102{{{#!hs
     103   "My text" :: String
     104   "My text" :: Text
     105   "123456"  :: ByteString
     106}}}
     107
     108is desugared into:
     109
     110{{{#!hs
     111   fromString "My text" :: String
     112   fromString "My text" :: Text
     113   fromString "123456"  :: ByteString
     114}}}
     115
     116Let's add a `StaticStrings` extension that desugars to this instead:
     117
     118{{{#!hs
     119   [string|My text|] :: String
     120   [text|My text|]   :: Text
     121   [byte|123456|]    :: ByteString
     122}}}
     123
     124Note that the *quasiquoter is selected from the type*. I.e., we have:
     125
     126{{{#!hs
     127   class StringQuote a where
     128      fromStringQuote :: QuasiQuoter
     129
     130   instance StringQuote ByteString where ...
     131   instance StringQuote String where ...
     132   instance StringQuote Text where ...
     133   instance StringQuote (Ptr a) where ...
     134}}}
     135
     136It makes string literals quasiquoters in disguise. It's only for syntactic convenience: the type can be inferred, hence the quasiquoter can be inferred.
     137
     138=== Builtin StringQuote
     139
     140Not all GHC support Template Haskell. Hence some StringQuote instances could be builtin or provided by compiler plugins. E.g.,
     141* String (current string storage)
     142* Text (UTF-16 encoding)
     143
     144
     145=== Desugaring usual strings
     146
     147When the StaticStrings extension is not in use, instead of desugaring into a specific string-literal, we can desugar into the following:
     148
    130149{{{#!hs
    131150   unpackCString# (staticDataAddr# d)
    132151}}}
    133152
    134 Most rules should still match. Some would need a minor refactoring (e.g.,
    135 builtin rules).
     153Most rules should still match. Some would need a minor refactoring (e.g., builtin rules).