Opened 6 years ago

Last modified 20 months ago

#8684 new bug

hWaitForInput cannot be interrupted by async exceptions on unix

Reported by: nh2 Owned by:
Priority: normal Milestone:
Component: Core Libraries Version: 7.6.3
Keywords: Cc: mail@…, hvr, ekmett, core-libraries-committee@…
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: #13497, #13525 Blocking:
Related Tickets: #12912, #13525, #7353 Differential Rev(s): Phab:D42
Wiki Page:

Description (last modified by nh2)

http://hackage.haskell.org/package/base-4.6.0.1/docs/System-Timeout.html

claims that timeout can interrupt hWaitForInput, but in fact that's false (e.g. mentioned in https://ghc.haskell.org/trac/ghc/ticket/7353#comment:4).

-- import Control.Concurrent
import System.IO
import System.Timeout

main = timeout (1 * 1000000) $ hWaitForInput stdin (5 * 1000)

will not be killed after 1 second, but instead wait for the full 5 seconds timeout passed to hWaitForInput.

The implementation is ready at https://downloads.haskell.org/~ghc/7.6.3/docs/html/libraries/base/src/GHC-IO-FD.html, where we have two foreign calls: safe fdReady and unsafe unsafe_fdReady.

The actual C implementation is at https://github.com/ghc/packages-base/blob/52c0b09036c36f1ed928663abb2f295fd36a88bb/cbits/inputReady.c#L16. It uses select on Unix, and does check for EINTR, so I believe that according to http://www.haskell.org/ghc/docs/7.6.3/html/users_guide/ffi.html#ffi-interruptible both foreign calls can be replaced by a single interruptible one.

Is that true?

If not, it's a documentation bug in timeout at least.

Also, does interruptible, apart from allowing the function to be interrupted, behave more like safe or unsafe?

Change History (33)

comment:1 in reply to:  description ; Changed 6 years ago by ezyang

Replying to nh2:

Is that true?

This would work fine for Unix. It would be good to test if it does the right thing with CancelSynchronousIO as well.

Also, does interruptible, apart from allowing the function to be interrupted, behave more like safe or unsafe?

Interruptible acts like safe, except for the extra signal throwing behavior.

comment:2 in reply to:  1 Changed 6 years ago by nh2

Replying to ezyang:

This would work fine for Unix. It would be good to test if it does the right thing with CancelSynchronousIO as well.

Ah, great.

I guess we should update the documentation until this is actually implemented.

comment:3 Changed 6 years ago by lnandor

I have tried to fix the bug by replacing select with pselect to ignore the SIGVTALRM signal sent by the runtime, but to properly terminate when SIGPIPE is received. https://github.com/nandor/packages-base/compare/fix-8684?expand=1

diff --git a/GHC/IO/FD.hs b/GHC/IO/FD.hs
index 2023526..0b0b1de 100644
--- a/GHC/IO/FD.hs
+++ b/GHC/IO/FD.hs
@@ -3,6 +3,7 @@
            , NoImplicitPrelude
            , BangPatterns
            , DeriveDataTypeable
+           , InterruptibleFFI
   #-}
 {-# OPTIONS_GHC -fno-warn-identities #-}
 -- Whether there are identities depends on the platform
@@ -395,7 +396,7 @@ setNonBlockingMode fd set = do
 
 ready :: FD -> Bool -> Int -> IO Bool
 ready fd write msecs = do
-  r <- throwErrnoIfMinus1Retry "GHC.IO.FD.ready" $
+  r <- throwErrnoIfMinus1 "GHC.IO.FD.ready" $
           fdReady (fdFD fd) (fromIntegral $ fromEnum $ write)
                             (fromIntegral msecs)
 #if defined(mingw32_HOST_OS)
@@ -405,7 +406,7 @@ ready fd write msecs = do
 #endif
   return (toEnum (fromIntegral r))
 
-foreign import ccall safe "fdReady"
+foreign import ccall interruptible "fdReady"
   fdReady :: CInt -> CInt -> CInt -> CInt -> IO CInt
 
 -- ---------------------------------------------------------------------------
@@ -502,7 +503,7 @@ indicates that there's no data, we call threadWaitRead.
 readRawBufferPtr :: String -> FD -> Ptr Word8 -> Int -> CSize -> IO Int
 readRawBufferPtr loc !fd buf off len
   | isNonBlocking fd = unsafe_read -- unsafe is ok, it can't block
-  | otherwise    = do r <- throwErrnoIfMinus1 loc 
+  | otherwise    = do r <- throwErrnoIfMinus1Retry loc
                                 (unsafe_fdReady (fdFD fd) 0 0 0)
                       if r /= 0
                         then read
diff --git a/GHC/IO/Handle/Text.hs b/GHC/IO/Handle/Text.hs
index f182e7f..31f2cac 100644
--- a/GHC/IO/Handle/Text.hs
+++ b/GHC/IO/Handle/Text.hs
@@ -106,7 +106,6 @@ hWaitForInput h msecs = do
                writeIORef haCharBuffer cbuf'
 
                if not (isEmptyBuffer cbuf') then return True else do
-
                 r <- IODevice.ready haDevice False{-read-} msecs
                 if r then do -- Call hLookAhead' to throw an EOF
                              -- exception if appropriate
diff --git a/cbits/inputReady.c b/cbits/inputReady.c
index 51f278f..9d51750 100644
--- a/cbits/inputReady.c
+++ b/cbits/inputReady.c
@@ -22,9 +22,10 @@ fdReady(int fd, int write, int msecs, int isSock)
 #else
     ( 1 ) {
 #endif
-	int maxfd, ready;
+    int maxfd;
     fd_set rfd, wfd;
-	struct timeval tv;
+    struct timespec tv;
+    sigset_t set;
 
     FD_ZERO(&rfd);
     FD_ZERO(&wfd);
@@ -39,16 +40,14 @@ fdReady(int fd, int write, int msecs, int isSock)
      */
     maxfd = fd + 1;
     tv.tv_sec  = msecs / 1000;
-	tv.tv_usec = (msecs % 1000) * 1000;
+    tv.tv_nsec = (msecs % 1000) * 1000000;
 
-	while ((ready = select(maxfd, &rfd, &wfd, NULL, &tv)) < 0 ) {
-	    if (errno != EINTR ) {
-		return -1;
-	    }
-	}
+    /* Block SIGVTALRM */
+    sigprocmask(SIG_BLOCK, NULL, &set);
+    sigaddset(&set, SIGVTALRM);
 
     /* 1 => Input ready, 0 => not ready, -1 => error */
-	return (ready);
+    return pselect(maxfd, &rfd, &wfd, NULL, &tv, &set);
     }
 #if defined(_MSC_VER) || defined(__MINGW32__) || defined(_WIN32)
     else {

comment:4 Changed 6 years ago by nh2

Status: newpatch

comment:5 Changed 5 years ago by thoughtpolice

Cc: hvr ekmett added
Differential Rev(s): Phab:D42

comment:6 Changed 5 years ago by thoughtpolice

Component: libraries/baseCore Libraries

Moving over to new owning component 'Core Libraries'.

comment:7 Changed 5 years ago by snoyberg

Cc: core-libraries-committee@… added
Owner: set to snoyberg

comment:8 Changed 2 years ago by nh2

Ai, it seems that ghc 8.0.2 changed some behaviour here:

import System.IO
import System.Timeout

main = hWaitForInput stdin (5 * 1000)

On ghc 8.0.2, this crashes with fdReady: msecs != 0, this shouldn't happen.

On ghc 8.0.1, this works as expected (doing nothing, terminating after 5 seconds).

Probably this commit:

https://github.com/ghc/ghc/commit/f46369b8a1bf90a3bdc30f2b566c3a7e03672518#diff-f727d72230bd33b0e218d47df4738565R28

comment:9 Changed 2 years ago by nh2

Description: modified (diff)

comment:10 Changed 2 years ago by Ben Gamari <ben@…>

In 37d7c15/ghc:

base: Add test for #8684

Reviewers: austin, hvr

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D3420

comment:11 in reply to:  8 Changed 2 years ago by nh2

Replying to nh2:

On ghc 8.0.2, this crashes with fdReady: msecs != 0, this shouldn't happen.

This is fixed for GHC 8.2 per #13525 (https://phabricator.haskell.org/rGHCe5732d2a28dfb8a754ee73e124e3558222a543bb)

comment:12 Changed 2 years ago by bgamari

Owner: snoyberg deleted
Status: patchnew

Sorry, it looks like I was mistaken; this is in fact not fixed. The timeout does not, in fact, interrupt the hWaitForInput. It looks like Phab:D42 was never actually finished.

comment:13 Changed 2 years ago by nh2

Blocked By: 13525 added

comment:14 Changed 2 years ago by nh2

Blocked By: 13497 added

Marking this as blocked by #13497 because that one is about getting the timing of fdReady() right.

I believe that should be completed before making sure fdReady() can be interrupted.

comment:15 Changed 2 years ago by nh2

I've implemented a work-in-progress fix for GHC 8.2 in this commit. It's partly based on my older commit b23420378f6.

It makes my example from the issue description work on the -threaded runtime on Linux (other platforms yet to be tested), but for unknown reason it doesn't fix it for the nonthreaded runtime.

Maybe somehow in the nonthreaded runtime timeout doesn't actually throw the other thread the kill?

comment:16 Changed 2 years ago by nh2

OK new info with the power of -debug and +RTS -Ds:

In non-threaded, it does raise the exception, but _after_ the entire hWaitForInput is over.

I can see it in cap 0: raising exception in thread 2., and also the reason:

timeout's throwTo also only happens after hWaitForInput is over (it prints cap 0: throwTo: from thread 1 to thread 2 only at the very end).

So that means it's not the exception handling that's not working, it's the throwing.

My C code returns back into Haskell land, but there is no exception there at the time so it continues.


On my debug commit https://github.com/nh2/ghc/blob/49a5e9ce7062da7594bfd86ca7af92796b84b52a/libraries/base/cbits/inputReady.c#L53-L68 that looks like this (relevant output without -debug in this gist):

cap 0: running thread 1 (ThreadRunGHC)
cap 0: thread 1 stopped (suspended while making a foreign call)
fdReady called with msecs = 20
fdReady res = -1
cap 0: running thread 1 (ThreadRunGHC)
cap 0: thread 1 stopped (suspended while making a foreign call)
fdReady called with msecs = 10
fdReady res = -1
cap 0: running thread 1 (ThreadRunGHC)
cap 0: thread 1 stopped (suspended while making a foreign call)
fdReady called with msecs = 0
fdReady res = 0
cap 0: running thread 1 (ThreadRunGHC)
cap 0: throwTo: from thread 1 to thread 2
	thread    2 @ 0x42001059d0 is blocked until 558048911098409 (TSO_DIRTY)
cap 0: raising exception in thread 2.
cap 0: thread 1 stopped (finished)
bound thread (1) finished
task exiting

Observe here how cap 0: throwTo: from thread 1 to thread 2 is after the fdReady called with msecs = 0 (hWaitForInput ran its entire 5 seconds).


So now we just need to find out why timeout doesn't result in a prompt cap 0: throwTo: from thread 1 to thread 2 after 1 second.

comment:17 Changed 2 years ago by nh2

OK, looking at the definition of timeout here, and making an instrumented copy of it into my test case file, it seems that the threadDelay n >> throwTo pid ex is never executed.

The cap 0: throwTo: from thread 1 to thread 2 I observe at the very end is from killThread.

Changing it into this

        handleJust (\e -> if e == ex then Just () else Nothing)
                   (\_ -> return Nothing)
                   (bracket (forkIOWithUnmask $ \unmask -> do
                                 putStrLn "before unmask"
                                 unmask $ putStrLn "before delay" >> threadDelay n >> putStrLn "delay over" >> throwTo pid ex)
                            (\x -> putStrLn "before killThread" >> uninterruptibleMask_ (killThread x))
                            (\_ -> fmap Just f))

yields the output at the very end:

fdReady called with msecs = 0
fdReady res = 1
before killThread
fdReady called with msecs = 0
fdReady res = 1
before unmask

and then the program terminates.

This suggests that nothing inside unmask is ever executed in my non-threaded case, and that the thread started with forkIOWithUnmask doesn't actually run until hWaitForInput is over.

comment:18 Changed 2 years ago by nh2

Looking at this with +RTS -Ds again, we get:

created capset 0 of type 2
created capset 1 of type 3
cap 0: initialised
assigned cap 0 to capset 0
assigned cap 0 to capset 1
new task (taskCount: 1)
cap 0: created thread 1
new bound thread (1)
cap 0: schedule()
cap 0: running thread 1 (ThreadRunGHC)
fdReady called with msecs = 0
fdReady res = 1
setup
cap 0: created thread 2
cap 0: thread 1 stopped (suspended while making a foreign call)
fdReady called with msecs = 5000
fdReady res = -1
...
[lots of output running only thread 1]
...
cap 0: running thread 1 (ThreadRunGHC)
cap 0: thread 1 stopped (suspended while making a foreign call)
fdReady called with msecs = 4690
fdReady res = -1
cap 0: running thread 1 (ThreadRunGHC)
cap 0: thread 1 stopped (suspended while making a foreign call)
fdReady called with msecs = 4680
fdReady res = 0
cap 0: running thread 1 (ThreadRunGHC)
cap 0: throwTo: from thread 1 to thread 2
  thread    2 @ 0x4200105aa0 is not blocked (TSO_DIRTY)
cap 0: throwTo: blocking on thread 2
cap 0: thread 1 stopped (blocked on throwTo)
  thread    1 @ 0x4200105388 is blocked on a throwto message (TSO_DIRTY)
cap 0: running thread 2 (ThreadRunGHC)
cap 0: raising exception in thread 2.
cap 0: waking up thread 1 on cap 0
cap 0: thread 2 stopped (yielding)
cap 0: running thread 1 (ThreadRunGHC)

So the new question here is: Why is thread 2 (the one that contains unmask $ threadDelay n >> throwTo pid ex) never run, and thread 1 is run all the time?

comment:19 Changed 2 years ago by nh2

The above seems to be because in rts/Schedule.c only resumeThread() is called, and schedule() (or whatever function's task it is to choose functions to run on our single capability in the non-threaded runtime) is not called.

Last edited 2 years ago by nh2 (previous) (diff)

comment:20 Changed 2 years ago by nh2

I think I've got it!

Inserting a yield here after fdReady() returns back into Haskell makes delay over appear and my example code terminate after the 1 second timeout.

EDIT: It needs interruptible yield, either of those functions alone doesn't cut it.

Last edited 2 years ago by nh2 (previous) (diff)

comment:21 Changed 2 years ago by nh2

Hmm, interruptible yield would be a workaround, but I think it wouldn't fix the key problem.

It would fix only hWaitForInput/ready.

But it looks like other safe foreign calls that are called in a tight loop that does not allocate cannot be timeouted.

It seems a proper solution would be to insert a yield right after /any/ foreign call returns (at least in the non-threaded RTS).

comment:22 Changed 2 years ago by nh2

In other words, I believe we have a scheduling unfairness problem in the non-threaded RTS:

A thread that loops tightly around a foreign call will never give other threads the chance to run.

I suspect this is because of the run-queue logic mentioned in the Scheduler commentary:


In more detail, threads are put in front (pushOnRunQueue) if: [...]

  • In the non-threaded runtime, when a thread waiting on IO unblocks.

comment:23 Changed 2 years ago by Ben Gamari <ben@…>

In 13758c6c/ghc:

Added a test for 'timeout' to be accurate.

This is the first in a series of regression tests prompted by
https://ghc.haskell.org/trac/ghc/ticket/8684 and D4011, D4012, D4041

Test Plan: This _is_ a test.

Reviewers: nh2, austin, hvr, bgamari

Reviewed By: bgamari

Subscribers: rwbarton, thomie

GHC Trac Issues: #8684

Differential Revision: https://phabricator.haskell.org/D4074

comment:24 Changed 23 months ago by Tamar Christina <tamar@…>

In 3825b7e/ghc:

Remove the 'legroom' part of the timeout-accurate-pure test.

Summary:
This removes the part of the test that checks whether the timeout happened in
a 'reasonable' amount of time, because it is flaky.
In subsequent work, we can turn this into a benchmark.

Test Plan: This _is_ a test

Reviewers: nh2, bgamari, Phyx, austin, hvr

Reviewed By: Phyx

Subscribers: rwbarton, thomie

GHC Trac Issues: #8684

Differential Revision: https://phabricator.haskell.org/D4120

comment:25 Changed 22 months ago by nh2

Some more trouble on Windows (also for -threaded):

If the FD is a FILE_TYPE_CHAR, then the implementation of ccall interruptible (see here) via CancelSynchronousIo doesn't seem to kill the WaitForSingleObject() invocation.

Unfortunately, the call to CancelSynchronousIo(), here named pCSIO(), does not actually check the return value of the function, which returns a BOOL success.

If this function cannot find a request to cancel, the return value is 0 (zero), and GetLastError returns ERROR_NOT_FOUND.

In my case, when logging it, I see it returns error code 1168, which is ERROR_NOT_FOUND, so apparently nothing was cancelled.

As a result, my example program from the issue description runs for 5 seconds instead of one.

With a bit more instrumentation, I get this:

fdReady called with msecs = 5000
calling WaitForSingleObject
calling pCSIO
pCSIO ret 0
pCSIO error: 1168
WaitForSingleObject rc 258 (WAIT_TIMEOUT)

After 1 second, as expected, it's calling pCSIO, but it doesn't cancel anything due to the error, so after 5 seconds WaitForSingleObject rc 258 (WAIT_TIMEOUT) happens.

comment:26 Changed 22 months ago by nh2

I've augmented my debugging output by the Windows thread ID here and with GetCurrentThreadId():

fdReady called with msecs = 5000
calling WaitForSingleObject in windows thread id 0000000000000b14
calling pCSIO(thread id 0000000000000b14)
pCSIO ret 0
pCSIO error: 1168
WaitForSingleObject rc 258 (WAIT_TIMEOUT)

Looks like the thread IDs are the same, so I currently have no idea why pCSIO() running wouldn't switftly terminate the WaitForSingleObject() on this FILE_TYPE_CHAR FD.

comment:27 Changed 22 months ago by nh2

Hmm, in this book Windows via C/C++ it says

Note that the thread calling CancelSynchronousIo doesn't really know where the thread that called the synchronous operation is. The thread could have been pre-empted and it has yet to actually communicate with the device; it could be suspended, waiting for the device to respond; or the device could have just responded, and the thread is in the process of returning from its call. If CancelSynchronousIo is called when the specified thread is not actually suspended waiting for the device to respond, CancelSynchronousIo returns FALSE and GetLastError returns error not found.

comment:28 Changed 22 months ago by nh2

I managed to make it work on Windows for FILE_TYPE_CHAR, by using WaitForMultipleObjects() and passing it 2 objects: The HANDLE we're interested in, an Event object that we signal when we want to interrupt the thread.

After searching for a very long time, I ended up with the above solution.

Most important links:

Here's the backstory of me finding it out via the #ghc and #winapi IRC channels:

In #ghc:

<nh2[m]>      JaffaCake_: may I steal your attention for a minute? `ccall interruptible` doesn't seem to work for me on Windows, it claims that there's nothing to cancel, do you have an idea what this could be? https://ghc.haskell.org/trac/ghc/ticket/8684#comment:25
<JaffaCake_>  no idea, sorry
<JaffaCake_>  there was a bug reported recently in process that also looks like foreign ccall interruptible not working on windows
<nh2[m]>      JaffaCake_: yes, I'm aware of it and have talked a bit with Neil about it, I'll try to make that interruptible after I've made hWaitForInput interruptible (already works on Linux -threaded for me), but currently it seems interruptible doesn't work at all for me on Windows
<JaffaCake_>  TBH I don't remember whether it ever worked
<nh2[m]>      JaffaCake_: there's another thing that I've already looked into a couple hours but haven't found a way yet:
<nh2[m]>      My non-Windows fix works nicely on -threaded, but for nonthreaded it doesn't, because the scheduler makes it immediately re-enter the foreign call, so the `throwTo` in `timeout` doesn't get a chance to run.
<nh2[m]>      Would it be possible to say, in the scheduler: Whenever a thread returns from a foreign call, yield, so that other Haskell threads have a chance to run?
<JaffaCake_>  I think that's probably a bad idea
<JaffaCake_>  if the timeout has fired, then there should be a blocked exception
<JaffaCake_>  and that should get thrown immediately the FFI call returns
<nh2[m]>      hWaitForInput/fdReady() gets woken up by the SIGVTALRM timer signal, and then the scheduler immediately runs fdReady() again
<nh2[m]>      so `timeout` never actually gets a chance to throwTo the exception
<nh2[m]>      so in other words, the timeout can never fire
<JaffaCake_>  nh2[m], you're not running with -threaded?
<nh2[m]>      JaffaCake_: no, that's what I meant, it's working fine with -threaded on Linux, but not with non-threaded. This problem (and my question on whether we could yield after FFI return) is only for non-threaded
<JaffaCake_>  oh, I'm not worried about non-threaded
<nh2[m]>      JaffaCake_: but I am :D I'm trying to make it work equally well across both threaded and non-threaded; in general I care a lot about non-threaded as for some applications it is significantly faster, and it is also easier to debug
<JaffaCake_>  but you'll need to add hacks to make it work, hacks that could add overhead for -threaded
<nh2[m]>      JaffaCake_: my hope is that such overhead would be negligible, as it would occur only for `interruptible` syscalls (which by nature are expensive, so a bit of bookkeeping around them should not matter much). For example, all calls to `fdReady()` that are performance-sensitive, nonblocking (timeout=0), already use `unsafe` and thus wouldn't suffer any overhead
<nh2[m]>      having non-threaded not working would be quite a drag for us, as we found it to perform way better for sequential-IO-heavy (e.g. network) code
<nh2[m]>      and with things like `timeout` not working for `hWaitForInput`, we can't put timeouts on reads or writes in high-level the code (would have to handle such logic at the very low level of hWaitForInput itself)
<nh2[m]>      e.g. `timeout N $ seriesOfVariousNetworkProtocolActions`

In #winapi:

nh2
18:54 hi, I'm trying to WaitForSingleObject(GetStdHandle(STD_INPUT_HANDLE), ...) on stdin, and to cancel this waiting using CancelSynchronousIo(), but the cancellation does nothing (returns 0 and GetLastError() is ERROR_NOT_FOUND). Any idea what I could be doing wrong?

SleepyISIS (IRC)
19:29 nh2[m]: If this function cannot find a request to cancel, the return value is 0 (zero), and GetLastError returns ERROR_NOT_FOUND.
19:30 Are you sure that you want to cancel IO, not the wait itself?

nh2
19:31 SSleepyISIS: can you elaborate a bit on that distinction? To me those sound like the same things
19:33 what I need to do is use a function that blocks until input is available on the given HANDLE to a FILE_TYPE_CHAR, up to a maximum amount of time (e.g. in milliseconds), but if some other event occurs, I need to cancel that blocking wait

John___ (IRC)
19:33 nh2[m], there's SleepEx()
19:34 which does what you want.
19:35 also WaitForMultiple/SingleObject()
19:35 depends what your goal is.

nh2
19:35 JJohn___: that one seems to just wait a given amount of time, independent from input being available on some HANDLE. I want to wait until either the specified time has passed, or there's input available on the HANDLE, or the waiting has been cancelled (e.g. using CancelSynchronousIO)

SleepyISIS (IRC)
19:35 nh2[m]: Is terminating a thread an option?

John___ (IRC)
19:36 nh2[m], then WaitForSingleObject()

nh2
19:37 JJohn___: yes, that is what I am using, and what doesn't seem to work (as I wrote above)

SleepyISIS (IRC)
19:37 Or stick to asynch solution.

John___ (IRC)
19:37 it waits for a timeout or when input is available on a handle.
19:37 I was not here to see what the problem was. mind pasting it again?

nh2
19:37 SSleepyISIS: regarding terminating a thread, likely not, it should ideally work in a single-threaded use case (I'm doing this for an improvement in the GHC Haskell compiler), but I would still very much like to hear what you have in mind, maybe it is possible

SleepyISIS (IRC)
19:38 John___: If I got nh2[m] correctly, (s)he wants to abort synch IO before the wait timeouts.

nh2
19:38 JJohn___: I've just written up the problem more clearly on https://stackoverflow.com/questions/47336755/how-to-cancelsynchronousio-on-waitforsingleobject-waiting-on-stdin
How to CancelSynchronousIo() on WaitForSingleObject() waiting on stdin?
On Windows 10, I'm waiting for input from the console using WaitForSingleObject( GetStdHandle(STD_INPUT_HANDLE), ... ) and to cancel this waiting using CancelSynchronousIo(). But the cancellatio...

SleepyISIS (IRC)
19:39 nh2[m]: https://stackoverflow.com/questions/34372611/how-to-signal-file-handle-waiting-with-waitforsingleobject
How to signal file HANDLE waiting with WaitForSingleObject
This code, which I have no control over, reads a file using overlapped I/O: // Read file asynchronously HANDLE hFile = CreateFile(..., FILE_FLAG_OVERLAPPED, ...); BYTE buffer[10]; OVERLAPPED oRead...
19:39 Hmm, what value do you get from GetStdHandle?

John___ (IRC)
19:40 nh2[m], synchronous solutions cannot be "unblocked"
19:40 that's why you use async.
19:40 let me read the stack post first though before I reply differently.

nh2
19:42 JJohn___: it was my understanding so far that CancelSynchronousIo()'s purpose is to cancel specifically functions like WaitForSingleObject(), so I'm surprised that it doesn't work

John___ (IRC)
19:42 I don't think it's for those purposes.

nh2
19:42 SSleepyISIS: the HANDLE returned from GetStdHandle() is 0 for the stdin

SleepyISIS (IRC)
19:43 >My colleague suggested WaitForMultipleObjects() as a workaround. So we can wait on either the console/stdin handle or an event which signals termination.
19:43 Good solution.
19:43 0 doesn't seem to be a valif handle.
19:44 Yeah: If an application does not have associated standard handles, such as a service running on an interactive desktop, and has not redirected them, the return value is NULL.
19:44 https://docs.microsoft.com/en-us/windows/console/getstdhandle

John___ (IRC)
19:44 yeah, that ^
19:45 using that in combination with SetEvent()
19:45 allows you to either make WaitForMultipleObjects() to catch on WAIT_SIGNALED or WAIT_TIMEOUT

nh2
19:45 SSleepyISIS: oh sorry, I typoed my printf. The HANDLE returned from GetStdHandle() is 84 for the stdin

John___ (IRC)
19:45 or whatevet
19:45 so you can make it either trigger when the handle input is available

nh2
19:48 SSleepyISIS: JJohn___ OK, so I would get rid of the CancelSynchronousIo() completely, and always make sure that any WaitForSingleObject() on stdin is instead a WaitForMultipleObjects() on stdin AND the special separate event handle I use for "early termination" of the wait?


Mysoft (IRC)
19:50 nh2[m]
19:50 you tried to use CancelIoEx
19:50 isntead of CancelIoSynchronous?

nh2
19:55 MMysoft: I will try that now; what would you pass as the lpOverlapped argument, NULL?

Mysoft (IRC)
19:56 yeah
19:56 and which OS you are trying that?
19:56 and you're running from console?
19:56 or inside an IDE?

nh2
19:57 MMysoft: that call fails with GetLastError() == 6 (ERROR_INVALID_HANDLE). I'm on Windows 10, running from cmd.exe

Mysoft (IRC)
19:58 ok
19:58 so this was a behavior change
19:58 because here it works
19:58 i think it changes in some update on win7
19:58 (bug?)
19:59 there's a OldNewThing about the subject
19:59 https://blogs.msdn.microsoft.com/oldnewthing/20150323-00/?p=44413
20:00 but anyway using WaitForMultipleObjects is the way to go
20:00 as i think using CancelIoEx... looks as bad as "ungetc" :P

nh2
20:04 MMysoft: ah wait, I am using CancelIoEx() wrong. For the CancelSynchronousIo() I had to provide a thread handle as an argument, but for CancelIoEx() I have to provide the actual file handle as the argument. Let me retry
20:05 

nh2
20:06 MMysoft: OK, with that fixed, CancelIoEx() gives me GetLastError() == ERROR_NOT_FOUND, like for CancelSynchronousIo()
20:07 I'll try the oldnewthing example code on my system

[Note by nh2: I didn't actually manage to make that compile immediately with msys, and it didn't turn out to be necessary to solve the issue]

John___ (IRC)
20:46 nh2[m], yes, you pass 2 handles and you check which got signaled.
20:46 the result is WAIT_OBJECT_0 to (WAIT_OBJECT_0 + nCount– 1)

nh2
00:11 JJohn__: SSleepyISIS MMysoft : thanks for your help, I just got the first working implementation for GHC that fixes its behaviour on Windows, using your proposed extra event + WaitForMultipleObjects() approach!

Back in #ghc:

<nh2[m]>      JaffaCake: I think I have finally figured out how to fix the Windows problem with CancelSynchronousIo() having no effect. Some people on #winapi helped me, saying that this function might not work in practice and instead we could use WaitForMultipleObjects(), with 2 objects, one for the actual FD HANDLE and one Event we'd signal if we want WaitForMultipleObjects() to return early. I've coded that up and it works.
<nh2[m]>      But for efficiency, I need one Event per thread, so that I don't wake up all threads in `interruptWorkerTask()`, but only the target one.
<nh2[m]>      My plan is thus to have a `HANDLE interruptOSThreadEvent;` in the `struct Task`
<nh2[m]>      But I need to access that in `inputReady.c`, which is in base, where I cannot get at the currently running task with `myTask()`
<nh2[m]>      is there a way I can ask for the currently running Haskell task from that location?
<JaffaCake>   I'm pretty sure we do that in integer-gmp
<nh2[m]>      JaffaCake: I don't know how thread-local storage actually works internally, but I suppose it's done by symbol names somehow, and if I just declare `extern __thread Task *my_task;` in `inputReady.c`, it would refer to the right thing technically? (though I'd still have to bring `Task` into scope)
<JaffaCake>   just expose a function from the RTS to return the thread-local Event
<JaffaCake>   I was thinking of rts_unsafeGetMyCapability()
<JaffaCake>   which is a similar kind of thing
<nh2[m]>      JaffaCake: which would be the best header file to put it in? Probably somewhere under `includes/rts`?
<JaffaCake>   next to rts_unsafeGetMyCapability()
<nh2[m]>      JaffaCake: OK, and it would have to go somewhere in RtsSymbols.c, probably `RTS_MINGW_ONLY_SYMBOLS`, is that right? 

comment:29 Changed 21 months ago by nh2

As part of the review of my patch in https://phabricator.haskell.org/D42, I have gathered some information of how the timer signal is implemented. Since that may be a useful by itself, I post it here. It is as of commit a1950e6, and, since not much of this has changed since the last release, also of GHC 8.2.

# How the timer signal is implemented

In general, the tick callbacks go like this to do context switching:

handle_tick()
  contextSwitchAllCapabilities()
    for all capabilities:
      contextSwitchCapability()
        stopCapability()
          cap->r.rHpLim = NULL; // makes the heap check fail
        also sets `cap->interrupt = 1;`

Methods used on the various platforms:

EDIT: Wrong table, corrected table below:

- POSIX (method selected in `posix/Itimer.c`)
  - Linux, threaded RTS            -> timer_create() if it exists, otherwise setitimer()
  - Linux, non-threaded, >= 2.6.25 -> pthread with timerfd
  - Linux, non-threaded, <  2.6.25 -> pthread without timerfd
  - Darwin                         -> pthread without timerfd
  - iOS                            -> pthread without timerfd
- Windows (`win32/Ticker.c`)
  - Windows                        -> CreateTimerQueueTimer()

EDIT: Corrected table, as per simonmar from comment 31:

- POSIX (method selected in `posix/Itimer.c`)
  - Linux, threaded, >= 2.6.25     -> pthread with timerfd
  - Linux, threaded, <  2.6.25     -> timer_create() if it exists, otherwise setitimer()
  - Linux, non-threaded RTS        -> timer_create() if it exists, otherwise setitimer()
  - Darwin                         -> pthread without timerfd
  - iOS                            -> pthread without timerfd
- Windows (`win32/Ticker.c`)
  - Windows                        -> CreateTimerQueueTimer()

Notably the Darwin and iOS implementations use a pthread even for the non-threaded RTS!

Relevant trac issues about the above methods:

Method implementation locations:

- pthread with timerfd                 -> `itimer/Pthread.c`
- pthread without timerfd (sleep loop) -> `itimer/Pthread.c`
- timer_create()                       -> `itimer/TimerCreate.c`
- setitimer()                          -> `itimer/Setitimer.c`

How the implementations work:

  • pthread with timerfd
    • A pthread is started that runs a loop reading from the timerfd. No SIGVTALRM is used. When the timerfd ticks, that thread wakes up and calls handle_tick().
  • pthread without timerfd
    • A pthread is started that runs a loop running sleep(itimer_interval). No SIGVTALRM is used. When that thread finishes the sleep, it calls handle_tick().
  • timer_create()
    • A SIGVTALRM signal handler is set up that handle_tick(). Then timer_create() is called to set up a SIGVTALRM signal occurring regularly, using the ITIMER_REAL real-time clock. The SIGVTALRM signal occurring will EINTR all system calls of all threads of the process.
  • setitimer()
    • A SIGVTALRM signal handler is set up that handle_tick(). Then setitimer() is called to set up a SIGVTALRM signal occurring regularly, using the CLOCK_ID clock, which is CLOCK_MONOTONIC if available and CLOCK_REALTIME otherwise. The SIGVTALRM signal occurring will EINTR all system calls of all threads of the process.
  • CreateTimerQueueTimer()
    • CreateTimerQueueTimer() is set up to call tick_callback() which calls tick_proc = handle_tick() regularly. The option WT_EXECUTEINTIMERTHREAD is passed which results in "callback function is invoked by the timer thread itself". There are a couple issues with it:
      1. The period is set to TimeToUS(tick_interval) / 1000 milliseconds, which becomes 0 if less than a millisecond is chosen. CreateTimerQueueTimer() does not document what happens if a 0-period is given. It might busy-poll, but it's not documented, so who knows?
      2. A comment in the code remarks that this timer has a maximum accuracy of 15ms on Windows 7, and even worse on older platforms.
Last edited 20 months ago by nh2 (previous) (diff)

comment:30 Changed 20 months ago by nh2

OK, due to the reasons at https://phabricator.haskell.org/D42#119714 I found that this bug is also present in non-threaded on Darwin and iOS:

ghc-bug-8684-test.hs

import Control.Concurrent
import System.IO
import System.Timeout

main :: IO ()
main = do
    forkIO $ do
        threadDelay (5 * 1000000)
        -- The timeout should terminate before we ever make it here
        putStrLn "t=5 seconds: we shouldn't be here"

    timeout (1 * 1000000) $ do
        hWaitForInput stdin (10 * 1000)
        putStrLn "we shouldn't be here"

    return ()

I just confirmed that this prints t=5 seconds: we shouldn't be here on OSX.

comment:31 Changed 20 months ago by simonmar

I think the table above is a bit wrong. The relevant bit of code is

#if defined(linux_HOST_OS) && defined(THREADED_RTS) && HAVE_SYS_TIMERFD_H
#define USE_PTHREAD_FOR_ITIMER
#endif

which means we should have (I re-ordered the lines a bit)

- POSIX (method selected in `posix/Itimer.c`)
  - Linux, threaded, >= 2.6.25     -> pthread with timerfd
  - Linux, threaded, <  2.6.25     -> timer_create() if it exists, otherwise setitimer()
  - Linux, non-threaded RTS        -> timer_create() if it exists, otherwise setitimer()
  - Darwin                         -> pthread without timerfd
  - iOS                            -> pthread without timerfd
- Windows (`win32/Ticker.c`)
  - Windows                        -> CreateTimerQueueTimer()

comment:32 Changed 20 months ago by nh2

@simonmar Oops, you are right. I swapped threaded and non-threaded when typing down my table, thus writing we use pthreads in non-threaded, which is exactly the wrong way around. I'll edit to reflect that.

comment:33 Changed 20 months ago by nh2

Note: See TracTickets for help on using tickets.