HTCondor

You can use HTCondor to run the nextnano software on your local computer infrastructure (“on-premise”). Essentially, nextnanomat submits the job either locally or on the “HTCondor” cluster. In both cases, the results of the calculations are located on your local computer.

This feature is only supported with our new license system.

HTCondor on nextnanomat

The following shows a screenshot from nextnanomat. 6 computers are connected to the HTCondor pool called e25nn. 120 slots are configured, 44 are currently available. Computers 2, 3, 4 and 6 are selected to accept jobs. Computers 2 and 6 are currently not available as they are in use.

HTCondor screenshot

Figure 8 Screenshot taken from nextnanomat with integrated HTCondor feature.

Submitting jobs to HTCondor pool with nextnanomat

Submit job

  1. Add a job to the Batch list in the Run tab.

  2. Click on the Run in HTCondor Cluster button (button with triangle and network).

Show information on HTCondor cluster

  1. Click on Show Additional Info for Cluster Simulation.

  2. Press the Refresh button on the right.

  3. The results of the condor_status command are shown, i.e. the number of compute slots are displayed.

  4. You can select another HTCondor command such as condor_q to show the status of your submitted jobs, i.e. select condor_q, and then press the Refresh button.

  • You can type in any command in the line System command:, e.g. dir.

  • The button Open Documentation opens the online documentation (this website).

Results of HTCondor simulations

  • Once your HTCondor jobs are finished, the results are automatically copied back to your simulation output folder <nextnano simulation output folder\<name of input file>\.

  • For debugging purposes regarding the HTCondor job, you can analyze the generated log file, <input file name>.log.

Useful HTCondor commands for the Command Prompt

  • condor_submit <filename>.sub Submit a job to the pool.

  • condor_q Shows current state of own jobs in the queue.

    • condor_q -nobatch -global -allusers Shows state of all jobs in the cluster. Of all users.

    • condor_q -goodput -global -allusers Shows state and occupied CPU of all jobs in the cluster.

    • condor_q -allusers -global -analyze Detailed information for every job in the cluster.

    • condor_q -global -allusers -hold Shows why jobs are in hold state.

  • condor_status Shows state of all available resources.

  • condor_status -long Shows state of all available resources and many other information.

  • condor_status -debug Shows state of all available resources and some additional information, e.g. WARNING: Saw slow DNS query, which may impact entire system: getaddrinfo(<Computername>) took 11.083566 seconds.

  • condor_rm Remove jobs from a queue:

    • condor_rm -all Removes all jobs from a queue.

    • condor_rm <cluster>.<id> Removes jobs on cluster <cluster> with id <id> (It seems <cluster>. can be omitted, and id is the JOB_IDS number.)

  • condor_release -all If any jobs are in state hold, use this command to restart them.

  • condor_restart Restart all HTCondor daemons/services after changes in config file.

  • condor_version Returns the version number of HTCondor

  • condor_store_cred query Returns info about the credentials stored for HTCondor jobs

  • condor_history Lists the recently submitted jobs. If for a specific job ID the status has the value ST=C, then this job has been completed (C) successfully.

  • condor_status -master: returns Name, HTCondor Version, CPU and Memory of central manager

  • Open Command Prompt cmd.exe as Administrator. Type in: net start condor. This has the same effect as restarting your computer, i.e. the networking service condor is started. This is useful if you have changed your local condor_config file.

HTCondor Pool - Managing Slots

  • Each PC runs one condor_startd daemon. By default, the condor_startd will automatically divide the machine into slots, placing one core in each slot. E.g. a 6-core computer with hyperthreading has 12 logical processors. Alternatively, the number of cores (or logical processors) can be distributed to the slots as follows.

SLOT_TYPE_1 = cpus=4
SLOT_TYPE_2 = cpus=4
SLOT_TYPE_3 = cpus=2
SLOT_TYPE_4 = cpus=1
SLOT_TYPE_5 = cpus=1
SLOT_TYPE_1_PARTITIONABLE = TRUE
SLOT_TYPE_2_PARTITIONABLE = TRUE
SLOT_TYPE_3_PARTITIONABLE = TRUE
SLOT_TYPE_4_PARTITIONABLE = TRUE
SLOT_TYPE_5_PARTITIONABLE = TRUE
NUM_SLOTS_TYPE_1 = 1
NUM_SLOTS_TYPE_2 = 1
NUM_SLOTS_TYPE_3 = 1
NUM_SLOTS_TYPE_4 = 1
NUM_SLOTS_TYPE_5 = 1

  • PartitionableSlot: For SMP (symmetric multiprocessing) machines, a boolean value identifying that this slot may be partitioned.

  • DynamicSlot: For SMP machines that allow dynamic partitioning of a slot, this boolean value identifies that this dynamic slot may be partitioned.

  • SlotID: For SMP machines, the integer that identifies the slot.

  • A useful command might be: condor_status -af Name TotalCpus DynamicSlot PartitionableSlot SlotID. It returns the requested properties of each slot:

    • Name

    • TotalCpus

    • DynamicSlot

    • PartitionableSlot

    • SlotID

Dynamic slots

In our pool we have chosen dynamic partitioning which gives full flexibility. For instance, a quad-core CPU that is dynamically partitioned can accept

  • 4 single-threaded jobs (request_cpus = 1)

  • 2 jobs with 2 threads each (request_cpus = 2)

  • 2 jobs of which one is single-threaded (request_cpus = 1) and the other uses 3 threads (request_cpus = 3)

  • 1 job with 4 threads (request_cpus = 4).

####################################################
# Dynamic partitioning
# We use HTCondors dynamic partitioning mechanism.
# Each PC has one partitionable whole machine slot.
# (It seems that hyperthreading is not taken into account.)
####################################################
NUM_SLOTS = 1
NUM_SLOTS_TYPE_1 = 1
SLOT_TYPE_1 = 100%
SLOT_TYPE_1_PARTITIONABLE = true
SlotWeight = Cpus

Machine states

A machine is in any of the following 6 states. The most important one are Owner, Unclaimed, Claimed.

  • Backfill: (not relevant for us)

  • Owner: The machine is being used by the machine owner, and/or is not available to run HTCondor jobs. When the machine first starts up, it begins in this state.

  • Unclaimed: The machine is available to run HTCondor jobs, but it is not currently doing so.

  • Matched: The machine is available to run jobs, and it has been matched by the negotiator with a specific schedd. That schedd just has not yet claimed this machine. In this state, the machine is unavailable for further matches.

  • Claimed: The machine has been claimed by a schedd.

  • Preempting: The machine was claimed by a schedd, but is now preempting that claim for one of the following reasons.

- The owner of the machine came back
- Another user with higher priority has jobs waiting to run.
- Another request that this resource would rather serve was found.

Machine activities

Each machine state can have different activities. The machine state Claimed can have one out of these four activities.

  • Idle:

  • Busy:

  • Suspended:

  • Retiring:

Configuration options for the Central Manager computer

With this option in the condor.config file on the central manager, one can set a policy that the jobs are spread out over several machines rather than filling all slots of one computer before filling the slots of the other computers.

##------nn: SPREAD JOBS BREADTH-FIRST OVER SERVERS
##-- Jobs are "spread out" as much as possible,
##   so that each machine is running the fewest number of jobs.
NEGOTIATOR_PRE_JOB_RANK = isUndefined(RemoteOwner) * (- SlotId)

FAQ

Q: I submitted a job to HTCondor, but nothing happens. The nextnanomat GUI says “transmitted”.

A: It could be that nextnanomat does not have read in all required settings. You can try to type in the command line condor_restart. Please make sure that you entered your credentials using condor_store_cred add -debug. You should then start nextnanomat again.

Q: I submitted a job to HTCondor, but the Batch line of nextnanomat is stuck with preparing. What is wrong?

A1: Did you store your credentials after the installation of HTCondor? If not, enter condor_store_cred add into the command prompt to add your password, see above (Recommended Installation Process).

A2: Did you change your password recently? If yes you have to reenter your credentials for HTCondor. Enter condor_store_cred add into the command prompt to add your password, see above (Recommended Installation Process). If this does not work, try to enter condor_store_cred add -debug for more output information on the error.

Q: I specified target machines in Tools - Options. Afterwards every submitted job to HTCondor is stuck with transmitting. What is wrong?

A: The value for UID_DOMAIN within the condor_config file needs to be the same for every computer of your cluster. (You can easily test it in a command prompt with condor_status -af uiddomain) If it’s not the same value, no matching computer will be found and the job will not be transmitted successfully.

Problems with HTCondor

Error: communication error

If you receive the following error when you type in condor_status

C:\Users\"<your user name>">condor_status
Error: communication error
CEDAR:6001:Failed to connect to <123.456.789.123>

you can check whether the computer associated with this IP address is your HTCondor computer using the following command.

nslookup 123.456.789.123

It is also a good idea to type in

nslookup

This will return the name of the Default Server that resolves DNS names. If it is not the expected computer, you can open a Command Prompt as Administrator and type in ipconfig /flushdns to flush the DNS Resolver Cache.

C:\Users\"<your user name>">ipconfig /flushdns

If the DNS address cannot be resolved correctly it could be related to a VPN connection that has configured a different default server for Domain Name to IP address mapping. E.g. if your Windows Domain is called contoso.com (which is only visible within your own network and your own HTCondor pool) but your DNS is resolved to www.contoso.com (which might be outside your local HTCondor pool).

Error: condor_store_cred add failed with Operation failed. Make sure your ALLOW_WRITE setting include this host.

Solution: Edit condor_config file and add host, i.e. local computer name (here: nn-delta).

    ALLOW_WRITE = $(CONDOR_HOST), $(IP_ADDRESS)
==> ALLOW_WRITE = $(CONDOR_HOST), $(IP_ADDRESS), nn-delta

Error? Check the Log files

If you encounter any strange errors, you can find some hints in the history or Log files generated by HTCondor. You can find them here:

C:\condor\spool

  • history

C:\condor\log

  • CollectorLog

  • MasterLog

  • MatchLog

  • NegotiatorLog

  • ProcLog

  • SchedLog

  • ShadowLog

  • SharedPortLog

  • StarterLog

  • StartLog

More details can be found here: Logging in HTCondor

Known bugs

  • HTCondor < 8.9.5 works with all nextnano executables

  • HTCondor >= 8.9.5 works with nextnano executables newer than 2020-Jan

Run your custom executable on HTCondor with nextnanomat

You can even run your own executable with nextnanomat locally or on HTCondor! We tested the following programs:

Input file identifier

An input file identifier is a special string in the input file that signals to nextnanomat whether the input file is an input file for the nextnano++, nextnano³, nextnano.NEGF or nextnano.MSB tools, or for a custom executable.

Settings for Hello World (HW)

In nextnanomat, we need the following settings:

  • Path to executable file: e.g. D:\HW\HelloWorld.exe

  • Input file identifier: e.g. HelloWorld

  • Working directory: Select ‘Simulation output folder’

  • HTCondor: Output folder and files (transfer_output_files = …): .

Open input file input_file_for_HelloWorld.in (or any other input file that contains the string HelloWorld) and run the simulation either locally or on HTCondor.

Settings for Quantum ESPRESSO (QE)

Our folder structure is

  • D:\QE\inputfile\My_QE_inputfile.in (QE input file)

  • D:\QE\input\pseudo\C.UPF (pseudopotential file for atom species ‘C’ as specified in input file)

  • D:\QE\exe\pw.exe (QE executable file)

  • D:\QE\exe\*.dll (all dll files needed by pw.exe)

  • D:\QE\working_directory\QE_nextnanomat_HTCondor.bat (batch file)

In nextnanomat, we need the following settings:

  • Path to executable file: e.g. D:\QE\working_directory\QE_nextnanomat_HTCondor.bat

  • Path to folder with additional files: D:\QE\

  • Input file identifier: e.g. &control

  • Working directory: Select ‘Simulation output folder’

  • HTCondor: Output folder and files (transfer_output_files = …): .

  • (Additional arguments passed to the executable: $INPUTFILE)

The batch file (*.bat) contains the following content:

.\exe\pw.exe -in .\inputfile\My_QE_inputfile.in

This means that relative to the working directory, pw.exe is started, and the specified input file is read in. In this input file, the following quantities are specified:

  • C.UPF: name of pseudopotential file

  • ./input/pseudo/: path to pseudopotential file C.UPF

Open input file My_QE_inputfile.in and run the simulation either locally or on HTCondor.

Things that could be improved:

  • Write all files into output folder created by nextnanomat. In particular, the folder output/ should be moved.

  • condor_exec.exe is deleted (better: do not copy it back)

  • all *.dll files should be deleted (better: do not copy them back)

  • Do not copy back *.exe and *.dll files (both HTCondor and local)

Settings for ABINIT

Our folder structure is

  • D:\abinit\inputfile\t30.in (ABINIT input file)

  • D:\abinit\input\* (input files needed by ABINIT)

  • D:\abinit\exe\abinit.exe (ABINIT executable file)

  • D:\abinit\exe\*.dll (all dll files needed by abinit.exe)

  • D:\abinit\working_directory\abinit_nextnanomat.bat (batch file)

  • Path to executable file: e.g. D:\abinit\working_directory\abinit_nextnanomat.bat

  • Path to folder with additional files: D:\abinit\

  • Input file identifier: e.g. acell

  • Working directory: Select ‘Simulation output folder’

  • HTCondor: Output folder and files (transfer_output_files = …): .

  • Additional arguments passed to the executable: (empty)

The batch file (*.bat) contains the following content:

.\exe\abinit.exe < .\input\ab_nextnanomat_HTCondor.files

This means that relative to the working directory, abinit.exe is started, and the specified input file is read in. In this input file, the following quantities are specified:

  • .\inputfile\t30.in: name of input file

  • .\input\14si.pspnc:

Open input file t30.in and run the simulation either locally or on HTCondor.

Notes

  • condor_exec.exe is deleted (better: do not copy it back)

  • all *.dll files should be deleted (better: do not copy them back)

  • Do not copy back *.exe and *.dll files (both HTCondor and local)