Opened 4 years ago

Closed 3 years ago

#2893 closed user story (wontfix)

Support Boost 1.64 and 1.65

Reported by: GaryM Owned by:
Priority: normal Milestone: Iteration I4
Component: infrastructure Keywords: boost make_array serialization
Cc: fcooper, jmpf@…, GaryM Estimated pair-hours remaining: 0
Pair-hours expended on ticket: 6 Initial estimate of effort: 3
Editable by public: yes

Description (last modified by fcooper)

New boosts released.

http://www.boost.org/users/history/version_1_64_0.html http://www.boost.org/users/history/version_1_65_0.html

Boost 1.64 changed exponential random number generation, causing a number of test suites to fail.

The appropriate course of action would seem to be the same fix as was applied to the older changes to the Normal distribution.

Change History (21)

comment:1 Changed 4 years ago by fcooper

  • Cc fcooper jmpf@… added
  • Keywords boost make_array added

This one looks like it might be a bit of a nightmare!

I have:

/home/robert/boost_1_64/include/boost/numeric/ublas/storage.hpp: In member function ‘void boost::numeric::ublas::unbounded_array<T, ALLOC>::serialize(Archive&, unsigned int)’:
/home/robert/boost_1_64/include/boost/numeric/ublas/storage.hpp:299:18: error: ‘make_array’ is not a member of ‘boost::serialization’
             ar & serialization::make_array(data_, s);
                  ^

I'm yet to track down precisely what the problem is, but it appears to be some form of new incompatibility between ublas and serialization, although there are no changes from either mentioned in the release notes!

It might have something to do with: https://lists.boost.org/Archives/boost/2016/11/231762.php

or: https://svn.boost.org/trac/boost/ticket/12516

comment:2 Changed 4 years ago by GaryM

Yeah I think that first link is the right idea:

[from the lead developer of Boost serialization]:
>> So I think the best would be for ublas / storage to replace
>>
>> #include boost/serialization/array.hpp
>>
>> with
>>
>> #include boost/serialization/array_wrapper.hpp 

comment:3 Changed 3 years ago by fcooper

  • Pair-hours expended on ticket changed from 0 to 2

An update on this:

This is the correct link with information about why the problem exists: https://svn.boost.org/trac10/ticket/12516

Furthermore, the commit here: https://github.com/boostorg/serialization/commit/1d86261581230e2dc5d617a9b16287d326f3e229

does indeed appear to resolve the problem. I added that include to an otherwise vanilla boost 1.64 on lofty, and recompiled (in /home/robert/boost_1_64_p1), and the continuous test pack compiles and passes with no problems.

Unfortunately, because serialization ships as a compiled library, we can't just tell people using that version to simply patch that header. Do we:

  • Mark 1.64 is as unsupported
  • Add the patched version to the loft rotations, mark it as tested, but leave a footnote explaining it needs patching and re-compiling

comment:4 Changed 3 years ago by fcooper

Spoke too soon: a number of tests fail with 1.64:

TestArchivingHelperClasses and TestRandomNumberGenerator seem to be the main two, and then a number of tests that all seem to depend on random numbers.

comment:5 Changed 3 years ago by fcooper

Here's the output of TestRandomNumberGenerator:

    Start 28: TestRandomNumberGenerator

28: Test command: /home/fcooper/Chaste/chaste_debug/global/test/TestRandomNumberGenerator
28: Test timeout computed to be: 9.99988e+06
28: Running 13 tests
28: 
28:  ***** TestRandomNumberGenerator.hpp *****
28: 
28: 
28: Entering TestRandomNumbersCreation
28: Passed
28: Entering TestNewMethodSeed
28: Passed
28: Entering TestOtherRandomStuffDoesNotDestroyRandomSequence
28: Passed
28: Entering TestDifferentRandomSeed
28: Passed
28: Entering TestArchiveRandomNumberGenerator
28: Passed
28: Entering TestPermutationShuffle
28: Passed
28: Entering TestGenericShuffle
28: Passed
28: Entering TestDistributionReproducibilityAcrossPlatforms
28: 
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:373: Error: Expected (p_gen->ExponentialRandomDeviate(4.0) == 0.1173) up to 1e-4 (0.0001), found (0.1881 != 0.1173)
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:377: Error: Expected (p_gen->GammaRandomDeviate(1.0, 2.0) == 0.9385) up to 1e-4 (0.0001), found (1.5054 != 0.9385)
28: Failed
28: Entering TestExponentialMethodsInsideBoost
28: 
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:396: Error: Expected (var_exponential() == 0.7958) up to 1e-4 (0.0001), found (0.8647 != 0.7958)
28: Failed
28: Entering TestReproducibilityAcrossPlatforms
28: 
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:435: Error: Expected (p_gen->StandardNormalRandomDeviate() == 0.2132) up to 1e-4 (0.0001), found (-0.7667 != 0.2132)
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:436: Error: Expected (p_gen->NormalRandomDeviate(256.0, 0.5) == 255.6166) up to 1e-4 (0.0001), found (256.2037 != 255.6166)
28: Failed
28: Entering TestReseedingWorksProperly
28: Passed
28: Entering TestGammaRandomDeviate
28: 
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:490: Error: Expected (p_gen->GammaRandomDeviate(1.0, 1.0) == 0.2510) up to 1e-4 (0.0001), found (0.9120 != 0.2509)
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:491: Error: Expected (p_gen->GammaRandomDeviate(2.0, 1.0) == 1.3033) up to 1e-4 (0.0001), found (2.3151 != 1.3032)
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:492: Error: Expected (p_gen->GammaRandomDeviate(1.0, 2.0) == 3.5595) up to 1e-4 (0.0001), found (2.4686 != 3.5595)
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:493: Error: Expected (p_gen->GammaRandomDeviate(3.5, 2.9) == 12.6437) up to 1e-4 (0.0001), found (21.6512 != 12.6437)
28: Failed
28: Entering TestExponentialRandomDeviate
28: 
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:508: Error: Expected (p_gen->ExponentialRandomDeviate(1.0) == 0.0499) up to 1e-4 (0.0001), found (0.3363 != 0.0499)
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:509: Error: Expected (p_gen->ExponentialRandomDeviate(2.0) == 0.8029) up to 1e-4 (0.0001), found (0.6298 != 0.8028)
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:510: Error: Expected (p_gen->ExponentialRandomDeviate(3.0) == 0.1137) up to 1e-4 (0.0001), found (0.2967 != 0.1137)
28: /home/fcooper/Chaste/chaste/global/test/TestRandomNumberGenerator.hpp:511: Error: Expected (p_gen->ExponentialRandomDeviate(4.0) == 0.2847) up to 1e-4 (0.0001), found (0.2715 != 0.2847)
28: Failed
28: 
28: Failed 5 of 13 tests
28: Success rate: 61%
1/1 Test #28: TestRandomNumberGenerator ........***Failed    0.94 sec

0% tests passed, 1 tests failed out of 1

comment:6 Changed 3 years ago by fcooper

Here's the change to the exponential random number generation:

https://github.com/boostorg/random/commit/c7d1b4f3516098b3e2fc8f8531d716881ab5834e

and the associated ticket:

https://github.com/boostorg/random/pull/23

comment:7 Changed 3 years ago by GaryM

This quotation from that discussion page seems to suggest the changes we are seeing are expected:

"... will be a slight change in behaviour in normal_distribution as a result of these changes, because normal_distribution depends on exponential_distribution for draws from the tail, and exponential_distribution now advances the RNG engine.

Comparing a sequence of, say, 10 normal values with 1.60.0, before we would get a sequence such as:

A B C D E F G H I J

If one of those, let's say C, required a tail draw, the value of C changes, and we skip D and E (because the engine advanced twice in the two exponential_distribution calls we use to generate the C replacement). So now we could get:

A B C' F G H I J L M"

How many tests fail? For now can we simply add in some [not ideal, admittedly] #if BOOST_VER < ... #else ... #endif around the results we test, since they are 'right' but just different. Becomes a bit fiddly when people commit new random-number dependent results that then fail in portability and these guards would need adding in.

See also #2918.

comment:8 Changed 3 years ago by fcooper

From Continuous:

The following tests FAILED:
	  1 - TestArchivingHelperClasses (Failed)
	 29 - TestRandomNumberGenerator (Failed)
	197 - TestSimpleCellCycleModels (Failed)
	303 - TestOffLatticeCryptProjectionSimulation (Failed)
	309 - TestCryptProjectionStatistics (Failed)

comment:9 Changed 3 years ago by fcooper

  • Cc GaryM added
  • Description modified (diff)
  • Summary changed from Support Boost 1.64 to Support Boost 1.64 and 1.65

comment:10 Changed 3 years ago by GaryM

Tested Boost 1.65 on lofty.

It doesn't have the problem with archiving helper classes, or need the patch, and in Continuous it compiles everything OK and just fails the same four tests above:

cell_based-test-cell-TestSimpleCellCycleModels.2_24.0
crypt-test-simulation-TestOffLatticeCryptProjectionSimulation.1_2.2
crypt-test-statistics-TestCryptProjectionStatistics.1_1.2
global-test-TestRandomNumberGenerator.5_13.0

so probably just the same random number issue to deal with as 1.64.

comment:11 Changed 3 years ago by GaryM

  • Owner set to GaryM
  • Status changed from new to assigned

having a go at this...

comment:12 follow-up: Changed 3 years ago by GaryM

A series of commits from 1a1fa64 to 4f161fb have a go at this.

  • First, thankfully, most of the specialisation we had to do last time disappeared as boost < 1.47 that required special treatment is now red on InstallGuides/DependencyVersions. So I took some of that out of the Normal distribution implementation, moved it to its own folder, and checked it still worked.
  • Then I made a similar copy of the exponential distribution for 1.65 to the one we made for Normal back in boost 1.56.
  • Then got that passing with changes to tests for exponential distributions.
  • Then I noticed the Gamma distribution uses the exponential, so we needed a copy of that too!
  • Then finally updated the Normal distribution to latest algorithm and got that to use new exponential too, and got tests passing again.

As before, if you have boost >= 1.64 it will use the boost implementation, if not it uses these new copies for all three distributions now. When we finally get past boost 1.63 these classes can be deleted!

Tested on boost 1.48 (the oldest purple on InstallGuides/DependencyVersions) boost 1.58, boost 1.63 and boost 1.64 and 1.65 on lofty.

We should probably add 1.65 to rotation. I think 1.64 should go orange given it needs patching to compile.

comment:13 Changed 3 years ago by GaryM

  • Estimated pair-hours remaining changed from 3 to 0.5
  • Milestone changed from Future to Iteration I4
  • Pair-hours expended on ticket changed from 2 to 6

comment:14 in reply to: ↑ 12 Changed 3 years ago by fcooper

Replying to GaryM:

We should probably add 1.65 to rotation. I think 1.64 should go orange given it needs patching to compile.

I have swapped Sunday from 1.63 to 1.65, and swapped the colours on the dependency page accordingly.

comment:15 Changed 3 years ago by fcooper

1.65 seems to have broken the build:

Possibly an archiving thing?

I have temporarily taken 1.65 out of the rotation - 1.63 is back in for now.

Last edited 3 years ago by fcooper (previous) (diff)

comment:16 Changed 3 years ago by GaryM

Seems to have fixed itself now, or did you take 1.65 out of the rotation!?

comment:17 Changed 3 years ago by GaryM

Ah, you did!

comment:18 Changed 3 years ago by GaryM

This looks like it might be fun.

Boost 1.65 all runs and passes on scons BUT it does the old boost 1-46 trick of seg faulting AFTER all the tests pass and we have seen "OK!" in the std::out which might suggest an old bug has been reintroduced into boost destructors. On cmake this breaks the build!

This doesn't happen on the patched 1.64, which exits OK.

Might have to contact the boost serialization people...

comment:19 Changed 3 years ago by GaryM

I've opened a ticket on boost's trac: https://svn.boost.org/trac10/ticket/13186

comment:20 Changed 3 years ago by GaryM

  • Keywords serialization added
  • Owner GaryM deleted

comment:21 Changed 3 years ago by GaryM

  • Estimated pair-hours remaining changed from 0.5 to 0
  • Resolution set to wontfix
  • Status changed from assigned to closed

Their bug has been fixed in boost 1.66 - see #2945. So closing this as a 'wontfix' for now.

Note: See TracTickets for help on using tickets.