Jean Michel D Sellier @ 03:38 PM on 22 Mar, 2012

Dear Jonathan,

The first thing I can say is that your input deck does not specify any parallelization scheme. You will have to add a new section right after the Region section in the Geometry section. You can do it in 2 ways. The first one is the following (attention, the numbers there are just examples):

  Partitioning
  {
   x_extension = (-0.5, 5.5)
   y_extension = (-0.5, 5.5)
   z_extension = (-0.5, 5.5)
   num_geom_CPUs = 24
  }

In this specific example, the user wants to parallelize his device between 120 CPUs. The keyword x_extension defines the range of the system to be parallelized in the x direction. The code will determine automatically what spatial parallelization scheme to use.

If you want to specify the partition scheme then you will have to use a syntax like the following one:

  Partitioning
  {
   x_partition_nodes = (-0.5, 1.5, 2.5, 5.5)
   y_partition_nodes = (-0.5, 2.5, 5.5)
   z_partition_nodes = (-0.5, 1.5, 2.5, 3.5, 5.5)

   num_geom_CPUs = 24
  }

The keyword x_partition_nodes specify the boundary of the slices of the spatial parallelization.

Concerning the parameters in the Schroedinger solver, try what you have now but if they don’t work I would suggest to use the following ones:

  max_number_iterations = 20000
  convergence_limit = 1e-8
  monitor_convergence = true
  preconditioner = mumps
  ncv = 42

  shift = 1.10
  solver_transformation_type = sinvert

  job_list       =  (passivate_H,calculate_band_structure)
  eigen_values_solver   = krylovschur
  number_of_eigenvalues = 20

  eps_orthog_refinement = never

  output         = (energies,eigenfunctions_Point3D,eigenfunctions_Silo,eigenfunctions_xyz)

  tb_basis = sp3d5sstar_SO

I hope this helps,

JM and Jim

P.S.: next time you post a input deck please add { { { and close them with } } } without the spaces.

Jonathan DuBois @ 03:53 PM on 22 Mar, 2012

Ok thanks, I thought the parallelization would be done automagically if I didnt specify the partition based on the available cpus.

now I see that our quing system isnt playing nice with the static nemo binary. we use srun here. so I typically submit jobs like this:

srun -p pdebug -n 60 ../../nemostatic structure.in > out

but nemo doesnt seem to be aware of the other cpus. as I now get errors like this after adding a partition section

 Partitioning
  {
   x_extension = (-0.5, 5.5)
   y_extension = (-0.5, 5.5)
   z_extension = (-0.5, 5.5)
   num_geom_CPUs = 24
  }

SimpleShapes: number of CPU is less than the number of partitions: (1

James Fonseca @ 05:28 PM on 22 Mar, 2012

Hi Jonathan, Just to eliminate a potential issue I would make sure the # of processors you’re requesting with srun (60) equals the num_geom_CPUs. ( I realize the error is basically saying the opposite).

I’m not really sure how transferable the static executable is going to be to different systems. I think it may depend on how mpiexec is built, but I’m just guessing.

Ideally, tomorrow, you will have the capability to submit jobs from nanohub onto the RCAC clusters here at Purdue. Thanks, Jim

Jonathan DuBois @ 06:18 PM on 22 Mar, 2012

I tried ncpu = num_geom_CPU but I got the same error. I think the problem is, as you suggest, with how mpiexec was built. I would attempt build from source but the resource seems to be not present at https://nanohub.org/resources/13244

Thanks for all your help.

James Fonseca @ 10:53 AM on 23 Mar, 2012

Hi Jonathan, Sorry about that. I’m not sure why it wasn’t there. I’ve put up the latest revision

Jim