<div class="sright">
  <div class="sub-content-head">
    Maui Scheduler  </div>
  <div id="sub-content-rpt" class="sub-content-rpt" >
 <div class="tab-container docs" id="tab-container">
<div class="topNav">

<div class="docsSearch">
</div>

<div class="navIcons topIcons">
<a href="index.php"><img src="/resources/docs/images/home.png" title="Home" alt="Home" border="0"></a>
<a href="13.0rmandinterfaces.php"><img src="/resources/docs/images/upArrow.png" title="Up" alt="Up" border="0"></a>
<a href="13.0rmandinterfaces.php"><img src="/resources/docs/images/prevArrow.png" title="Previous" alt="Previous" border="0"></a>
<a href="13.2rmconfiguration.php"><img src="/resources/docs/images/nextArrow.png" title="Next" alt="Next" border="0"></a>
</div>

<h1>13.1 Resource Manager Overview</h1>

<P> Maui requires the services of a resource manager in order
to properly function. This resource manager provides information
about the state of compute resources (nodes) and workload (jobs).
Maui also depends on the resource manager to manage jobs, instructing it
when to start and/or cancel jobs.
<p> Maui can be configured to manage one or more resource
managers simultaneously, even resource managers of different types.
However, migration of jobs from one resource manager to another is not
currently allowed. Or, in other words, jobs submitted onto one resource manager cannot run on the resources of another.

<ul>
   <li><a href="#scheduler">13.1.1 Scheduler/Resource Manager Interactions</a>
   <ul>
      <li><a href="#commands">13.1.1.1 Resource Manager Commands</a>
      <li><a href="#flow">13.1.1.2 Resource Manager Flow</a>
   </ul>
   <li><a href="#details">13.1.2 Resource Manager Specific Details (Limitations/Special Features)</a>
</ul>
   
<h2><a name=scheduler></a>13.1.1 Scheduler/Resource Manager Interactions</h2>
 Maui interacts with all resource managers in the same
basic format. Interfaces are created to translate Maui concepts regarding
workload and resources into native resource manager objects, attributes,
and commands.
<p> Information on creation a new scheduler resource
manager interface can be found in the <a href="13.4addingrminterfaces.php">Adding
 New Resource Manager Interfaces</a> section.

<h3><a name=commands></a>13.1.1.1 Resource Manager Commands</h3>

<P> In the simplest configuration, Maui interacts with the
resource manager using the four primary functions listed below:
<p><b>GETJOBINFO</b>
<p> Collect detailed state and requirement information
about idle, running, and recently completed jobs.
<p><b>GETNODEINFO</b>
<p> Collect detailed state information about idle, busy,
and defined nodes.
<p><b>STARTJOB</b>
<p> Immediately start a specific job on a particular
set of nodes.
<p><b>CANCELJOB</b>
<p> Immediately cancel a specific job regardless of job
state.
<p> Using these four simple commands, Maui enables nearly
its entire suite of scheduling functions. More detailed information
about resource manager specific requirements and semantics for each of
these commands can be found in the specific resource manager overviews.
(LL, PBS, or <a href="wiki">WIKI</a>).
<p> In addition to these base commands, other commands
are required to support advanced features such a dynamic job support, suspend/resume,
gang scheduling, and scheduler initiated checkpoint/restart.

<h3><a name=flow></a>13.1.1.2 Resource Manager Flow</h3>
<P> Early versions of Maui (i.e., Maui 3.0.x) interacted
with resource managers in a very basic manner stepping through a serial
sequence of steps each scheduling iteration. These steps are outlined
below:
<ol>
<li>
load global resource information</li>

<li>
load node specific information (optional)</li>

<li>
load job information</li>

<li>
load queue information (optional)</li>

<li>
cancel jobs which violate policies</li>

<li>
start jobs in accordance with available resources and policy constraints</li>

<li>
handle user commands</li>

<li>
repeat</li>
</ol>
 Each step would complete before the next step started.
As systems continued to grow in size and complexity however, it became
apparent that the serial model described above would not work. Three
primary motivations drove the effort to replace the serial model with a
concurrent threaded approach. These motivations were reliability,
concurrency, and responsiveness.
<p><b>Reliability</b>
<p> A number of the resource managers Maui interfaces
to were unreliable to some extent. This resulted in calls to resource
management API's with exited or crashed taking the entire scheduler with
them. Use of a threaded approach would cause only the calling thread
to fail allowing the master scheduling thread to recover. Additionally,
a number of resource manager calls would hang indefinitely, locking up
the scheduler. These hangs could likewise be detected by the master
scheduling thread and handled appropriately in a threaded environment.
<p><b>Concurrency</b>
<p> As resource managers grew in size, the duration of
each API global query call grew proportionally. Particularly, queries
which required contact with each node individually became excessive as
systems grew into the thousands of nodes. A threaded interface allowed
the scheduler to concurrently issue multiple node queries resulting in
much quicker aggregate RM query times.
<p><b>Responsiveness</b>
<p> Finally, in the non-threaded serial approach, the
user interface was blocked while the scheduler updated various aspects
of its workload, resource, and queue state. In a threaded model,
the scheduler could continue to respond to queries and other commands even
while fresh resource manager state information was being loaded resulting
in much shorter average response times for user commands.
<p> Under the threaded interface, all resource manager
information is loaded and processed while the user interface is still active.
Average aggregate resource manager API query times are tracked and new
RM updates are launched so that the RM query will complete before the next
scheduling iteration should start. Where needed, the loading process
uses a <i>pool</i> of worker threads to issue large numbers of node specific
information queries concurrently to accelerate this process.
The master thread continues to respond to user commands until all needed
resource manager information is loaded <b>and</b> either a scheduling relevant
<i>event</i> has occurred <b>or</b> the scheduling iteration time has arrived.
At this point, the updated information is integrated into Maui's state
information and scheduling is performed.
<h2><a name=details></a>13.1.2Resource Manager Specific Details (Limitations/Special
Features)</h2>
<p> (Under Construction)
<p> LL/LL2
<br> PBS
<br> <a href="wikiinterface.php">Wiki</a>
<p> Synchronizing Conflicting Information
<br> Maui does not trust resource manager. All
node and job information is reloaded
<br> on each iteration. Discrepancies
are logged and handled where possible.
<br> NodeSyncDeadline/JobSyncDeadline overview.
<p> Purging Stale Information
<p> Thread

<p><b>See Also:</b>
<p> <a href="13.2rmconfiguration.php">Resource Manager
Configuration</a>, <a href="13.3rmextensions.php">Resource Manager Extensions</a>

<div class="navIcons bottomIcons">
<a href="index.php"><img src="/resources/docs/images/home.png" title="Home" alt="Home" border="0"></a>
<a href="13.0rmandinterfaces.php"><img src="/resources/docs/images/upArrow.png" title="Up" alt="Up" border="0"></a>
<a href="13.0rmandinterfaces.php"><img src="/resources/docs/images/prevArrow.png" title="Previous" alt="Previous" border="0"></a>
<a href="13.2rmconfiguration.php"><img src="/resources/docs/images/nextArrow.png" title="Next" alt="Next" border="0"></a>
</div>

 </div>
 </div>
  </div>
  <div class="sub-content-btm"></div>
</div>
</div>