Opened 12 months ago

Closed 7 weeks ago

#3021 closed user story (fixed)

Support for Ubuntu 19.10 (inc. Petsc 3.11.3)

Reported by: GaryM Owned by:
Priority: high Milestone: Iteration I4
Component: infrastructure Keywords:
Cc: jmpf@… Estimated pair-hours remaining: 3
Pair-hours expended on ticket: 3 Initial estimate of effort: 6
Editable by public: yes

Description (last modified by GaryM)

Codenamed eoan ermine and released on Oct '17.

A summary:

  • PETSc 3.11.1 worked (#2997)
  • A citation file needed changing for 3.11.2 up (done below)
  • There's a problem with 3.11.3 (and maybe 3.11.2?) being investigated below.

Change History (17)

comment:1 Changed 12 months ago by bdevans

After building a new Eoan Ermine based image for chaste-docker, running the continuous test set produces the following failures:

The following tests FAILED:
	  3 - TestCitations (Failed)
	 26 - TestPetscTools (Failed)
	 53 - TestChebyshevIteration (Failed)
	 55 - TestLinearSystem (Failed)
	 59 - TestPCBlockDiagonal (Failed)
	 60 - TestPCLDUFactorisation (Failed)
	 61 - TestPCTwoLevelsBlockDiagonal (Failed)
Errors while running CTest

This is the contents of /home/chaste/lib/CMakeFiles/CMakeError.log:

Determining if the function sgemm_ exists failed with the following output:
Change Dir: /home/chaste/lib/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_47719/fast"
/usr/bin/make -f CMakeFiles/cmTC_47719.dir/build.make CMakeFiles/cmTC_47719.dir/build
make[1]: Entering directory '/home/chaste/lib/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_47719.dir/CheckFunctionExists.c.o
/usr/bin/cc   -Wall  -Wextra -Wno-unused-parameter -Wvla  -DCHECK_FUNCTION_EXISTS=sgemm_   -o CMakeFiles/cmTC_47719.dir/CheckFunctionExists.c.o   -c /usr/share/cmake-3.13/Modules/CheckFunctionExists.c
Linking C executable cmTC_47719
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_47719.dir/link.txt --verbose=1
/usr/bin/cc  -Wall  -Wextra -Wno-unused-parameter -Wvla  -DCHECK_FUNCTION_EXISTS=sgemm_   -pthread  CMakeFiles/cmTC_47719.dir/CheckFunctionExists.c.o  -o cmTC_47719 
/usr/bin/ld: CMakeFiles/cmTC_47719.dir/CheckFunctionExists.c.o: in function `main':
CheckFunctionExists.c:(.text+0x14): undefined reference to `sgemm_'
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_47719.dir/build.make:87: cmTC_47719] Error 1
make[1]: Leaving directory '/home/chaste/lib/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_47719/fast] Error 2

The tests all pass when the image is based on Disco Dingo. As far as I can tell, the changes in relevant dependencies between the two Ubuntu releases (in the Docker image) are as follows:

CMake (cmake) 3.12.1-1 --> 3.13.4-1build1
GCC (g++) g++: 8.3.0-6ubuntu1 --> 9.2.1-9ubuntu2
PETSc (libpetsc-real3.11-dbg) 3.10.5+dfsg1-1 --> 3.11.3+dfsg1-2
parMETIS (libparmetis-dev) 4.0.3-5 --> 4.0.3-5build1
HDF5 (libhdf5-openmpi-dev, hdf5-tools) 1.10.0-patch1 --> 1.10.4+repack-10
Xerces (libxerces-c-dev) 3.2.1 --> 3.2.2+debian-1build
SUNDIALS CVODE (libsundials-dev) 3.1.1 --> 3.1.2+dfsg-3build3

Possibly relevant: I tried updating the image to include VTK7 (instead of 6) but this appears to need Python 3 which caused problems due to the current dependence on Python 2.

comment:2 Changed 12 months ago by GaryM

Got the same failing tests on an eoan VM (without docker).

TestCitations might be trivial (they are asking us to cite a newer paper I think).

Appears there must be some PETSc change that is upsetting quite a few of those tests. The first error message is this on all of them:

55: [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
55: [0]PETSC ERROR: Object is in wrong state
55: [0]PETSC ERROR: Not for unassembled matrix
55: [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
55: [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
55: [0]PETSC ERROR: /home/pmzgm/chaste-build/linalg/test/TestLinearSystem on a  named ubuntu by pmzgm Mon Nov  4 04:10:18 2019
55: [0]PETSC ERROR: Configure options --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --with-silent-rules=0 --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --with-maintainer-mode=0 --with-dependency-tracking=0 --with-debugging=0 --shared-library-extension=_real --with-shared-libraries --with-pic=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-cxx-dialect=C++11 --useThreads=0 --with-fortran-interfaces=1 --with-blas-lib=-lblas --with-lapack-lib=-llapack --with-scalapack=1 --with-scalapack-lib=-lscalapack-openmpi --with-mumps=1 --with-mumps-include="[]" --with-mumps-lib="-ldmumps -lzmumps -lsmumps -lcmumps -lmumps_common -lpord" --with-suitesparse=1 --with-suitesparse-include=/usr/include/suitesparse --with-suitesparse-lib="-lumfpack -lamd -lcholmod -lklu" --with-ptscotch=1 --with-ptscotch-include=/usr/include/scotch --with-ptscotch-lib="-lptesmumps -lptscotch -lptscotcherr" --with-fftw=1 --with-fftw-include="[]" --with-fftw-lib="-lfftw3 -lfftw3_mpi" --with-superlu=1 --with-superlu-include=/usr/include/superlu --with-superlu-lib=-lsuperlu --with-superlu_dist=1 --with-superlu_dist-include=/usr/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-hdf5-include=/usr/include/hdf5/openmpi --with-hdf5-lib="-L/usr/lib/x86_64-linux-gnu/hdf5/openmpi -L/usr/lib/openmpi/lib -lhdf5 -lmpi" --CXX_LINKER_FLAGS=-Wl,--no-as-needed --with-hypre=1 --with-hypre-include=/usr/include/hypre --with-hypre-lib=-lHYPRE_core --prefix=/usr/lib/petscdir/petsc3.11/x86_64-linux-gnu-real --PETSC_ARCH=x86_64-linux-gnu-real CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC" CXXFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC" FCFLAGS="-g -O2 -fstack-protector-strong -fPIC -ffree-line-length-0" FFLAGS="-g -O2 -fstack-protector-strong -fPIC -ffree-line-length-0" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro -fPIC" MAKEFLAGS=w
55: [0]PETSC ERROR: #1 MatScale() line 5232 in /build/petsc-cwbe4b/petsc-3.11.3+dfsg1/src/mat/interface/matrix.c
55: [0]PETSC ERROR: #2 MatAYPX() line 347 in /build/petsc-cwbe4b/petsc-3.11.3+dfsg1/src/mat/utils/axpy.c
55: [0]PETSC ERROR: #3 MatCopy_Basic() line 4044 in /build/petsc-cwbe4b/petsc-3.11.3+dfsg1/src/mat/interface/matrix.c
55: [0]PETSC ERROR: #4 MatCopy_MPIAIJ() line 2162 in /build/petsc-cwbe4b/petsc-3.11.3+dfsg1/src/mat/impls/aij/mpi/mpiaij.c
55: [0]PETSC ERROR: #5 MatCopy() line 4098 in /build/petsc-cwbe4b/petsc-3.11.3+dfsg1/src/mat/interface/matrix.c

Not sure why the portability builds haven't picked this up for PETSc 3.11, are they definitely using 3.11 Fergus?!

I didn't have any cmake error though?

comment:3 follow-up: Changed 12 months ago by GaryM

However, scons is complaining that a few extra libraries that have been present in our ubuntu install before are now missing:

/usr/bin/ld: cannot find -lspooles
/usr/bin/ld: cannot find -lHYPRE_utilities
/usr/bin/ld: cannot find -lHYPRE_struct_mv
/usr/bin/ld: cannot find -lHYPRE_struct_ls
/usr/bin/ld: cannot find -lHYPRE_sstruct_mv
/usr/bin/ld: cannot find -lHYPRE_sstruct_ls
/usr/bin/ld: cannot find -lHYPRE_IJ_mv
/usr/bin/ld: cannot find -lHYPRE_parcsr_ls

don't know if it is something to do with that. Did I read something about 'slimming down' the deb?!

comment:4 Changed 12 months ago by GaryM

Citations is just a matter of them updating the user guide reference - fixed in 5620052. However this shouldn't have been needed if 3.11 is already being tested! Unless PETSc changed the citation between 3.11.0 and 3.11.3 which eoan is on. In which case detective work is needed to refine the #ifdef, I'll check this fails in a sensible way on portability, if it doesn't we need to worry that 3.11 isn't being tested properly.

comment:5 Changed 12 months ago by GaryM

Failed portability on 3.11.1, so looks as though they changed it between that and 3.11.3 which is eoan's version. Not a heinous crime if it is just citation, but other test failures suggest something fundamental altered too! To be investigated...

comment:6 Changed 12 months ago by GaryM

  • Pair-hours expended on ticket changed from 2 to 3

Test did fail on server. Detective work showed it was this change from PETSc team: https://github.com/petsc/petsc/commit/a1601671872b9ed707a37701216bb19149d10110 which github tells us (helpfully) is in the 3.11.2 tag.

So I've updated the logic to agree with that in 1b63220.

So either they introduced other breaking changes between 3.11.1 and 3.11.3 or the ubuntu petsc is dodgy. To be investigated...

comment:7 Changed 12 months ago by bdevans

Top sleuthing there Gary! ;)

comment:8 Changed 12 months ago by GaryM

  • Cc jmpf@… added

It's this MatCopy(rMat, temp_mat, DIFFERENT_NONZERO_PATTERN); line that seems to be failing: https://chaste.cs.ox.ac.uk/trac/browser/git_repo/global/src/parallel/PetscTools.cpp#L405

in the method PetscTools::ReadPetscObject(Mat,...) to account for most of the failing tests. So maybe the previous line PetscTools::SetupMat is leaving things a bit unhappy for 3.11.3, very strange that they'd change behaviour in a minor patch though...

comment:9 Changed 7 weeks ago by fcooper

Following up on this...

I have compiled PETSc 3.11.1 and 3.11.3 on my machine, and can confirm there are errors with the latter that don't happen with the former.

For instance, TestLinearSystem::TestConsecutiveSolvesDifferentPreconditioner() which emits the following:

Entering TestConsecutiveSolvesDifferentPreconditioner

/home/fergus/GitRepos/Chaste/chaste/linalg/test/TestLinearSystem.hpp:1272: Error: Test failed: 
Chaste error: ./global/src/parallel/PetscException.cpp:72: Object is in wrong state in function 'Solve' on line 1099 of file ./linalg/src/LinearSystem.cpp
Failed
Entering TestDifferentMatrixForPreconditioning

/home/fergus/GitRepos/Chaste/chaste/linalg/test/TestLinearSystem.hpp:1335: Error: Test failed: 
Chaste error: ./global/src/parallel/PetscException.cpp:72: Object is in wrong state in function 'Solve' on line 1099 of file ./linalg/src/LinearSystem.cpp
Failed
Entering TestFixedNumberOfIterations

/home/fergus/GitRepos/Chaste/chaste/linalg/test/TestLinearSystem.hpp:1416: Error: Test failed: 
Chaste error: ./global/src/parallel/PetscException.cpp:72: Object is in wrong state in function 'Solve' on line 1099 of file ./linalg/src/LinearSystem.cpp
Failed
Entering TestFixedNumberOfIterationsRelativeToleranceCoverage

/home/fergus/GitRepos/Chaste/chaste/linalg/test/TestLinearSystem.hpp:1536: Error: Test failed: 
Chaste error: ./global/src/parallel/PetscException.cpp:72: Object is in wrong state in function 'Solve' on line 1099 of file ./linalg/src/LinearSystem.cpp
Failed
Entering TestSolveZerosInitialGuessForSmallRhs
Passed
Entering TestSetFromOptions
Passed

Failed 4 of 26 tests
Success rate: 84%
Chaste warning: in file /home/fergus/GitRepos/Chaste/chaste/linalg/src/LinearSystem.cpp at line 991: Using zero initial guess due to small right hand side vector
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
[0]PETSC ERROR: /home/fergus/GitRepos/Chaste/chaste/cmake-build-debug-petsc-3113/linalg/test/TestLinearSystem on a linux-gnu named fergus by fergus Tue Sep  8 22:42:19 2020
[0]PETSC ERROR: Configure options --with-make-np=10 --with-cc=gcc --with-cxx=g++ --with-fc=0 --COPTFLAGS=-Og --CXXOPTFLAGS=-Og --with-x=false --with-ssl=false --download-f2cblaslapack=1 --download-mpich=https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz --download-hdf5=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries
[0]PETSC ERROR: #1 MatScale() line 5232 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 MatAYPX() line 347 in /software/petsc-3.11.3_h5-1_10_5/src/mat/utils/axpy.c
[0]PETSC ERROR: #3 MatCopy_Basic() line 4044 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 MatCopy_MPIAIJ() line 2162 in /software/petsc-3.11.3_h5-1_10_5/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #5 MatCopy() line 4098 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
[0]PETSC ERROR: /home/fergus/GitRepos/Chaste/chaste/cmake-build-debug-petsc-3113/linalg/test/TestLinearSystem on a linux-gnu named fergus by fergus Tue Sep  8 22:42:19 2020
[0]PETSC ERROR: Configure options --with-make-np=10 --with-cc=gcc --with-cxx=g++ --with-fc=0 --COPTFLAGS=-Og --CXXOPTFLAGS=-Og --with-x=false --with-ssl=false --download-f2cblaslapack=1 --download-mpich=https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz --download-hdf5=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries
[0]PETSC ERROR: #6 MatGetOrdering() line 180 in /software/petsc-3.11.3_h5-1_10_5/src/mat/order/sorder.c
[0]PETSC ERROR: #7 PCSetUp_ILU() line 134 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: #8 PCSetUp() line 932 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #9 KSPSetUp() line 391 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #10 PCSetUpOnBlocks_BJacobi_Singleblock() line 625 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/bjacobi/bjacobi.c
[0]PETSC ERROR: #11 PCSetUpOnBlocks() line 963 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #12 KSPSetUpOnBlocks() line 223 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #13 KSPSolve() line 726 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
[0]PETSC ERROR: /home/fergus/GitRepos/Chaste/chaste/cmake-build-debug-petsc-3113/linalg/test/TestLinearSystem on a linux-gnu named fergus by fergus Tue Sep  8 22:42:19 2020
[0]PETSC ERROR: Configure options --with-make-np=10 --with-cc=gcc --with-cxx=g++ --with-fc=0 --COPTFLAGS=-Og --CXXOPTFLAGS=-Og --with-x=false --with-ssl=false --download-f2cblaslapack=1 --download-mpich=https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz --download-hdf5=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries
[0]PETSC ERROR: #14 MatScale() line 5232 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #15 MatAYPX() line 347 in /software/petsc-3.11.3_h5-1_10_5/src/mat/utils/axpy.c
[0]PETSC ERROR: #16 MatCopy_Basic() line 4044 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #17 MatCopy_MPIAIJ() line 2162 in /software/petsc-3.11.3_h5-1_10_5/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #18 MatCopy() line 4098 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
The event associated with the counter for 'Ksp' had already begun when BeginEvent was called.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
[0]PETSC ERROR: /home/fergus/GitRepos/Chaste/chaste/cmake-build-debug-petsc-3113/linalg/test/TestLinearSystem on a linux-gnu named fergus by fergus Tue Sep  8 22:42:19 2020
[0]PETSC ERROR: Configure options --with-make-np=10 --with-cc=gcc --with-cxx=g++ --with-fc=0 --COPTFLAGS=-Og --CXXOPTFLAGS=-Og --with-x=false --with-ssl=false --download-f2cblaslapack=1 --download-mpich=https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz --download-hdf5=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries
[0]PETSC ERROR: #19 MatGetDiagonal() line 4685 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #20 PCSetUp_Jacobi() line 170 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #21 PCSetUp_Jacobi_NonSymmetric() line 247 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #22 PCApply_Jacobi() line 270 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #23 PCApply() line 462 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #24 KSP_PCApply() line 281 in /software/petsc-3.11.3_h5-1_10_5/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #25 KSPSolve_CG() line 133 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/impls/cg/cg.c
[0]PETSC ERROR: #26 KSPSolve() line 782 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
[0]PETSC ERROR: /home/fergus/GitRepos/Chaste/chaste/cmake-build-debug-petsc-3113/linalg/test/TestLinearSystem on a linux-gnu named fergus by fergus Tue Sep  8 22:42:19 2020
[0]PETSC ERROR: Configure options --with-make-np=10 --with-cc=gcc --with-cxx=g++ --with-fc=0 --COPTFLAGS=-Og --CXXOPTFLAGS=-Og --with-x=false --with-ssl=false --download-f2cblaslapack=1 --download-mpich=https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz --download-hdf5=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries
[0]PETSC ERROR: #27 MatScale() line 5232 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #28 MatAYPX() line 347 in /software/petsc-3.11.3_h5-1_10_5/src/mat/utils/axpy.c
[0]PETSC ERROR: #29 MatCopy_Basic() line 4044 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #30 MatCopy_MPIAIJ() line 2162 in /software/petsc-3.11.3_h5-1_10_5/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #31 MatCopy() line 4098 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
[0]PETSC ERROR: /home/fergus/GitRepos/Chaste/chaste/cmake-build-debug-petsc-3113/linalg/test/TestLinearSystem on a linux-gnu named fergus by fergus Tue Sep  8 22:42:19 2020
[0]PETSC ERROR: Configure options --with-make-np=10 --with-cc=gcc --with-cxx=g++ --with-fc=0 --COPTFLAGS=-Og --CXXOPTFLAGS=-Og --with-x=false --with-ssl=false --download-f2cblaslapack=1 --download-mpich=https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz --download-hdf5=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries
[0]PETSC ERROR: #32 MatGetDiagonal() line 4685 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #33 PCSetUp_Jacobi() line 170 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #34 PCSetUp_Jacobi_NonSymmetric() line 247 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #35 PCApply_Jacobi() line 270 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #36 PCApply() line 462 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #37 KSP_PCApply() line 281 in /software/petsc-3.11.3_h5-1_10_5/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #38 KSPSolve_CG() line 133 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/impls/cg/cg.c
[0]PETSC ERROR: #39 KSPSolve() line 782 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
[0]PETSC ERROR: /home/fergus/GitRepos/Chaste/chaste/cmake-build-debug-petsc-3113/linalg/test/TestLinearSystem on a linux-gnu named fergus by fergus Tue Sep  8 22:42:19 2020
[0]PETSC ERROR: Configure options --with-make-np=10 --with-cc=gcc --with-cxx=g++ --with-fc=0 --COPTFLAGS=-Og --CXXOPTFLAGS=-Og --with-x=false --with-ssl=false --download-f2cblaslapack=1 --download-mpich=https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz --download-hdf5=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries
[0]PETSC ERROR: #40 MatScale() line 5232 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #41 MatAYPX() line 347 in /software/petsc-3.11.3_h5-1_10_5/src/mat/utils/axpy.c
[0]PETSC ERROR: #42 MatCopy_Basic() line 4044 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #43 MatCopy_MPIAIJ() line 2162 in /software/petsc-3.11.3_h5-1_10_5/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #44 MatCopy() line 4098 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 
[0]PETSC ERROR: /home/fergus/GitRepos/Chaste/chaste/cmake-build-debug-petsc-3113/linalg/test/TestLinearSystem on a linux-gnu named fergus by fergus Tue Sep  8 22:42:19 2020
[0]PETSC ERROR: Configure options --with-make-np=10 --with-cc=gcc --with-cxx=g++ --with-fc=0 --COPTFLAGS=-Og --CXXOPTFLAGS=-Og --with-x=false --with-ssl=false --download-f2cblaslapack=1 --download-mpich=https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz --download-hdf5=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries
[0]PETSC ERROR: #45 MatGetDiagonal() line 4685 in /software/petsc-3.11.3_h5-1_10_5/src/mat/interface/matrix.c
[0]PETSC ERROR: #46 PCSetUp_Jacobi() line 170 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #47 PCSetUp_Jacobi_NonSymmetric() line 247 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #48 PCApply_Jacobi() line 270 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/impls/jacobi/jacobi.c
[0]PETSC ERROR: #49 PCApply() line 462 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #50 KSP_PCApply() line 281 in /software/petsc-3.11.3_h5-1_10_5/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #51 KSPSolve_CG() line 133 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/impls/cg/cg.c
[0]PETSC ERROR: #52 KSPSolve() line 782 in /software/petsc-3.11.3_h5-1_10_5/src/ksp/ksp/interface/itfunc.c

comment:10 Changed 7 weeks ago by fcooper

While investigating this I came across some PETSc methods that were deprecated in 3.11:

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecLockPush.html

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecLockPop.html

They were straightforward replacements with VecLockReadPush and VecLockReadPop, and Chaste is updated in a164a74/git_repo.

These deprecations were strikingly absent from any release notes(!) but the warning caused a compile error on my machine.

comment:11 Changed 7 weeks ago by fcooper

Having investigated as far as I can, it looks as though the problem *might* be related to this ticket: #1082, and this revision: https://chaste.cs.ox.ac.uk/trac/changeset/9796/chaste

At any rate, I think it's something in the function:

void PetscTools::ReadPetscObject(Mat& rMat, const std::string& rOutputFileFullPath, Vec rParallelLayout)

Over to someone who knows anything about PETSc..!

comment:12 Changed 7 weeks ago by GaryM

  • Description modified (diff)
  • Summary changed from Support for Ubuntu 19.10 to Support for Ubuntu 19.10 (inc. Petsc 3.11.3)

comment:13 Changed 7 weeks ago by GaryM

  • Description modified (diff)

comment:14 Changed 7 weeks ago by fcooper

0b963dc/git_repo introduces a temporary fix by replicating old PETSc behaviour.

This solution is not ideal and we definitely need to get to the bottom of the change - but this may at least let us get support out for Ubuntu 19.10 and 20.04.

comment:15 in reply to: ↑ 3 Changed 7 weeks ago by fcooper

Replying to GaryM:

However, scons is complaining that a few extra libraries that have been present in our ubuntu install before are now missing:

/usr/bin/ld: cannot find -lspooles
/usr/bin/ld: cannot find -lHYPRE_utilities
/usr/bin/ld: cannot find -lHYPRE_struct_mv
/usr/bin/ld: cannot find -lHYPRE_struct_ls
/usr/bin/ld: cannot find -lHYPRE_sstruct_mv
/usr/bin/ld: cannot find -lHYPRE_sstruct_ls
/usr/bin/ld: cannot find -lHYPRE_IJ_mv
/usr/bin/ld: cannot find -lHYPRE_parcsr_ls

don't know if it is something to do with that. Did I read something about 'slimming down' the deb?!

You may get those with the Ubuntu deb package, but by default the PETSc we compile on Lofty aren't built with hypre.

comment:16 Changed 7 weeks ago by fcooper

PETSc 3.11.3 has now been tested on Lofty, and works:

https://chaste.cs.ox.ac.uk/buildbot/builders/Portability%206_Sat/builds/309/

comment:17 Changed 7 weeks ago by fcooper

  • Resolution set to fixed
  • Status changed from new to closed

Future work on the PETSc issue has moved to #3040.

Note: See TracTickets for help on using tickets.