Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#10345 closed bug (fixed)

Testsuite timeout_multiplier setting does not work as expected

Reported by: jstolarek Owned by:
Priority: normal Milestone: 8.0.1
Component: Test Suite Version: 7.11
Keywords: Cc:
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Other Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D982
Wiki Page:

Description

I have a test case that - if failed - sends GHC into an infinite loop. The test case is small so I don't want to wait for default timeout (currently that's 5 minutes). The testsuite provides timeout_multiplier function to modify the default timeout but that setting seems to be ignored and the test is killed only after the default timeout time. I added my test to all.T file like this:

test('tc265', timeout_multiplier(0.01), compile, [''])

I also tried putting timeout setting into an array ([timeout_multiplier(0.01)]) but with no result.

Merijn reports having same problems and bypassing them by writing a test that forks itself and kills the child process after a given timeout - see libraries/base/tests/T8089.hs.

I am adding a note on Building/RunningTests/Adding that this field does not work currently. Once this bug is fixed we need to remove that note.

Change History (12)

comment:1 Changed 4 years ago by thomie

I can not reproduce this issue yet. Can you try a few things. I am assuming you are on Linux.

When you run the following, does it stop after 3 seconds?

  • python testsuite/timeout/timeout.py 3 'sleep 100'
  • In testsuite/tests/ghc-api/annotations/all.T, make the following change:
    -test('annotations', normal, run_command, ['$MAKE -s --no-print-directory annotations'])
    +test('annotations', timeout_multiplier(0.01), run_command, ['$MAKE -s --no-print-directory annotations'])
    

When running make TEST=annotations, does it stop after 3 seconds?

I guess it doesn't, otherwise you wouldn't have opened this ticket, but I want to be sure it's not something with the test you were writing in particular.

It must be something specific to your platform, as timeout works for me. The issue only I seem to be having is that running python testsuite/timeout/timeout.py 10 'echo ok' doesn't return immediately, but only after 10 seconds. Can you try that one as well? For some reason producing output is not allowed.

comment:2 in reply to:  1 ; Changed 4 years ago by jstolarek

Replying to thomie:

I am assuming you are on Linux.

Yes, that's correct

When you run the following, does it stop after 3 seconds?

  • python testsuite/timeout/timeout.py 3 'sleep 100'

Yes.

  • In testsuite/tests/ghc-api/annotations/all.T, make the following change:
    -test('annotations', normal, run_command, ['$MAKE -s --no-print-directory annotations'])
    +test('annotations', timeout_multiplier(0.01), run_command, ['$MAKE -s --no-print-directory annotations'])
    

When running make TEST=annotations, does it stop after 3 seconds?

Yes.

I guess it doesn't, otherwise you wouldn't have opened this ticket

So the above tests work, but my tests still doesn't time out. An obvious explanation is that I've configured the test incorrectly. Another is that the timeout does not work if GHC becomes unresponsive (ie. hits a black hole). (In the latter case: why would a 5 minute timeout work?)

The issue only I seem to be having is that running python testsuite/timeout/timeout.py 10 'echo ok' doesn't return immediately, but only after 10 seconds. Can you try that one as well? For some reason producing output is not allowed.

That works for me as it should - finishes immediately.

Last edited 4 years ago by jstolarek (previous) (diff)

comment:3 in reply to:  2 Changed 4 years ago by thomie

There are 2 timeout programs. On Linux we use the one written in Python by default, on Windows the one in written in Haskell.

Can you try the following:

$ cd testsuite/timeout
$ rm -r install-inplace
$ make WINDOWS=YES
$ cd ../
$ make TEST='<your test>'

comment:4 Changed 4 years ago by thomie

Sorry, the one in Haskell doesn't compile at the moment, I forgot. You need this patch:

commit c5759a04c73de36f94bb7239f9ead91f8741e6af
Author: Thomas Miedema <thomasmiedema@gmail.com>
Date:   Fri May 29 20:37:29 2015 +0200

    Testsuite: fix timeout.hs

diff --git a/testsuite/timeout/timeout.hs b/testsuite/timeout/timeout.hs
index f78baa1..d75e13c 100644
--- a/testsuite/timeout/timeout.hs
+++ b/testsuite/timeout/timeout.hs
@@ -65,7 +65,7 @@ run secs cmd = do
                         killProcess pid
                         exitWith (ExitFailure 99)
                   Just (Exited r) -> exitWith r
-                  Just (Terminated s) -> raiseSignal s
+                  Just (Terminated s _) -> raiseSignal s
                   Just _ -> exitWith (ExitFailure 1)

comment:5 Changed 4 years ago by jstolarek

Still the same.

comment:6 Changed 4 years ago by thomie

Should I try building Phab:D202 or do you have a smaller testcase?

comment:7 Changed 4 years ago by jstolarek

I'm afraid I don't have a smaller one :-/ If you want to build D202 please note that it changes format of interface files.

comment:8 Changed 4 years ago by thomie

Oh, I see it now. Fix coming.

comment:9 Changed 4 years ago by Thomas Miedema <thomasmiedema@…>

In 6e542a62e070b113f95908315c81d01c300d8803/ghc:

Testsuite: add function compile_timeout_multiplier (#10345)

And rename timeout_multiplier to run_timeout_multiplier.

timeout_multiplier was added in commit
a00389794b839971c7d52ead9e8570bfaa25ac55. The name suggested that it
would affect any test, but it actually only affected tests that had a
run component, and only that run component (as needed by test T367).

Differential Revision: https://phabricator.haskell.org/D982

comment:10 Changed 4 years ago by thomie

Differential Rev(s): Phab:D982
Milestone: 7.12.1
Resolution: fixed
Status: newclosed

comment:11 Changed 4 years ago by thoughtpolice

Milestone: 7.12.18.0.1

Milestone renamed

comment:12 in reply to:  1 Changed 4 years ago by thomie

Replying to thomie:

The issue only I seem to be having is that running python testsuite/timeout/timeout.py 10 'echo ok' doesn't return immediately, but only after 10 seconds.

I finally figured this one out. I had stty tostop set in ~/.bashrc. When stty tostop is set, a background process that tries to write to the terminal receives a SIGTTOU signal, which suspends it.

In timeout.py, the first thing the child process does is call os.setpgrp(). This creates a new process group, with the child process itself as process group leader. Because only one process group per session can be the foreground process group, it will be a background process group.

I'm not sure yet why timeout.py turns the child process that it creates into a background process. One implication is that CTRL-C doesn't kill it.

References:

Note: See TracTickets for help on using tickets.