Why does setting the hard-limit for maxfiles to "unlimited" using `launchctl limit` result in a hard-limit slightly above the soft-limit?
A fresh system install, with default parameters is configured with the following system wide file limits:


Kernel Max Files: 12288
Max Files per Process: 10240


Current kernel sysctl parameters can be viewed with sysctl command:

$ sysctl -a |grep kern.maxf

kern.maxfiles: 12288
kern.maxfilesperproc: 10240


Now let's check limits values, using ulimit command:

For SOFT limits: [we can see that: open files = 256]

$ ulimit -aS


core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 266
virtual memory          (kbytes, -v) unlimited


For HARD limits: [we can see that: open files = Unlimited]

$ ulimit -aH

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) unlimited
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 65532
cpu time               (seconds, -t) unlimited
max user processes              (-u) 532
virtual memory          (kbytes, -v) unlimited





Using launchctl

launchctl has the capacity to configure limit values, or show current values.

Using launchctl limit it shows all the current Soft and Hard limits, and we can see that soft limit is 256 and hard limit is 'unlimited' for the item maxfiles. (the same information we got using ulimit command above)

$ uname -a 
18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64


$ launchctl limit
    cpu         unlimited      unlimited      
    filesize    unlimited      unlimited      
    data        unlimited      unlimited      
    stack       8388608        67104768       
    core        0              unlimited      
    rss         unlimited      unlimited      
    memlock     unlimited      unlimited      
    maxproc     266            532            
    maxfiles    256            unlimited      



To filter, we can just use launchctl limit maxfiles

$ launchctl limit maxfiles
    maxfiles    256            unlimited




Now let's try to change those values #

Using your command line as example:

sudo launchctl limit maxfiles 10000 unlimited

But after it, when I execute again the command to display the limits, we can see that:


the Soft limit was adjusted from 256 to 10000, 
but the Hard limit was reduced from 'unlimited' to '10240' 


sudo launchctl limit          
    maxfiles    10000          10240          



And from now, it is impossible to get it back to unlimited!

We can see that after executing it, it also changed the kernel sysctl parameters

$ sysctl -A |grep max

kern.maxfiles: 10240
kern.maxfilesperproc: 10000



The reason is


  The word 'unlimited' when used as a parameter for setting maxfiles, 
  is the same as using value '10240'. 
  
  Because launchctl do not accept the string 'unlimited' and convert it 
  to the value 10240! 
  
  [I will prove it below, with the source code of launchclt and source of MacOS Mojave kernel]


But first, let see if we can use a bigger value like 2 million:

sudo launchctl limit maxfiles 10000 2000000 

$ launchctl limit maxfiles
    maxfiles    10000          2000000 


Yes, we can, it accepted the bigger value. 



Why this happens

It is hardcoded on the source code to not accept 'unlimited' as a value, so once we changed it, we can never put it back. 

But what is the value that corresponds to  'unlimited' ?


R: The value of 'unlimited' is INT_MAX, which is '2147483647' (the maximum value of the Integer Type)




Let's go inside the source code #


  Source code analysis of launchclt.c and Kernel routines  of MacOS
  10.14 Mojave about why is not possible to set  maxfiles to 'unlimited' via launchctl command line.
  
  All information below is my analysis, using the source  code from
  Apple, that can be obtained via: https://opensource.apple.com
  
  Rafael Prado - 2019/08/09


/* START OF CODE ANALYSIS */

// --------------- Analisys Comment ----------------------------------
// Looking at MacOS system core header file syslimits.h there is 
// a definition of OPEN_MAX with value 10240. 
// 
// This is the base value that will be used by default for 
// maxfiles value during system startup. 
//
// This comment you are reading here is mine.
//

/* FILE: sys/syslimits.h */
// *****************************************************************

#define OPEN_MAX                10240   /* max open files per process - todo, make a config option? */

// *****************************************************************



//
// --------------- Analisys Comment ---------------------------------
// Now we can see inside kernel config, that maxfiles parameter
// is a subprodut of OPEN_MAX, by adding 2048 to its value, 
// which give us the number 12288. 
// 
// This will became the default value for sysctl kern.maxfiles: 12288
//


/* FILE: conf/param.c */
// *****************************************************************

#define MAXFILES (OPEN_MAX + 2048)
int     maxfiles = MAXFILES;

// *****************************************************************



//
// --------------- Analisys Comment ---------------------------------
// Now here is why it is impossible to set 'unlimited' value using 
// the command line of launchctl:
//


/* FILE: launchd/support/launchctl.c */
// *****************************************************************
//

// (:..) Function lim2str() on line 2810
const char *
lim2str(rlim_t val, char *buf)
{
    if (val == RLIM_INFINITY)
        strcpy(buf, "unlimited");  
    else
        sprintf(buf, "%lld", val);
    return buf;
}


// (:..) Function srt2lim() on line 2821
bool
str2lim(const char *buf, rlim_t *res)
{
        char *endptr;
        *res = strtoll(buf, &amp;endptr, 10);
        if (!strcmp(buf, "unlimited")) {
                *res = RLIM_INFINITY;
                return false;
        } else if (*endptr == '\0') {
                 return false;
        }
        return true;
}



// (:..) Function limit_cmd() found on line 2835
int
limit_cmd(int argc, char *const argv[])
{



    // ------------------------------------------------------------
    // This comment is not part of the source file, it is 
    // my analisys comment about what this function does.
    //
    // Here it is reading the parameters of 'launchctl limit argv2 argv3'  
    // and will call function srt2lim() to convert the args to limit values
    // 
    // Function srt2lim() is printed above.
    // ------------------------------------------------------------
    //
    // (:....) skipping to line 2850 inside this function


    if (argc &gt;= 3 &amp;&amp; str2lim(argv[2], &amp;slim))
            badargs = true;
    else
            hlim = slim;

    if (argc == 4 &amp;&amp; str2lim(argv[3], &amp;hlim))
            badargs = true;


    // ------------------------------------------------------------
    // But before settin the values, there is a verification that 
    // prohibits using the word 'unlimited' as a parameter (only 
    // if used for 'maxfiles') and this check returns error, exiting
    // the limit_cmd() function [the function we are now inside] and
    // the system kernel paramter is never adjusted if we use 
    // the word 'unlimited' as a value.
    //
    // This comment was written by me 
    // and is not part of the source file.
    // ------------------------------------------------------------
    //
    // (:....) skiping to line 2906 inside this function


    bool maxfiles_exceeded = false;

    if (strncmp(argv[1], "maxfiles", sizeof("maxfiles")) == 0) {
        if (argc &gt; 2) {
            maxfiles_exceeded = (strncmp(argv[2], "unlimited", sizeof("unlimited")) == 0);
        }

        if (argc &gt; 3) {
            maxfiles_exceeded = (maxfiles_exceeded || strncmp(argv[3], "unlimited", sizeof("unlimited")) == 0);
        }

        if (maxfiles_exceeded) {
            launchctl_log(LOG_ERR, "Neither the hard nor soft limit for \"maxfiles\" can be unlimited. Please use a numeric parameter for both.");
            return 1;
        }
    }

    // -------------------------------------------------------------
    // We can see above, that it is prohibited to use the 'unlimited' value
    // And when we use it, the system assumes the 
    // default value of OPEN_MAX which is 10240 !
    // 
    // This explains and prove why setting 'unlimited' gives us 10240
    // --------------------------------------------------------------

// *****************************************************************



//
// --------------- Analisys Comment ---------------------------------
// Looking at another file from Kernel source, we can find another 
// place where it checks for 'unlimited' value, and also there is
// an explanation by Apple on it.
// 
// The comment BELOW IS Original comment from Apple Developers and 
// it is inside the file kern_resource.c from MacOS Kernel Source.
//


/* FILE: kern/kern_resource.c  */ 
// *****************************************************************

case RLIMIT_NOFILE:
        /*
         * Only root can set the maxfiles limits, as it is
         * systemwide resource.  If we are expecting POSIX behavior,
         * instead of clamping the value, return EINVAL.  We do this
         * because historically, people have been able to attempt to
         * set RLIM_INFINITY to get "whatever the maximum is".
        */
        if ( kauth_cred_issuser(kauth_cred_get()) ) {
                if (limp-&gt;rlim_cur != alimp-&gt;rlim_cur &amp;&amp;
                    limp-&gt;rlim_cur &gt; (rlim_t)maxfiles) {
                        if (posix) {
                                error =  EINVAL;
                                goto out;
                        }
                        limp-&gt;rlim_cur = maxfiles;
                }
                if (limp-&gt;rlim_max != alimp-&gt;rlim_max &amp;&amp;
                    limp-&gt;rlim_max &gt; (rlim_t)maxfiles)
                        limp-&gt;rlim_max = maxfiles;
        }
        else {
                if (limp-&gt;rlim_cur != alimp-&gt;rlim_cur &amp;&amp;
                    limp-&gt;rlim_cur &gt; (rlim_t)maxfilesperproc) {
                        if (posix) {
                                error =  EINVAL;
                                goto out;
                        }
                        limp-&gt;rlim_cur = maxfilesperproc;
                }
                if (limp-&gt;rlim_max != alimp-&gt;rlim_max &amp;&amp;
                    limp-&gt;rlim_max &gt; (rlim_t)maxfilesperproc)
                        limp-&gt;rlim_max = maxfilesperproc;
        }
        break;

// *****************************************************************

/* END OF CODE ANALYSIS */




This explains why it is impossible to set 'unlimited' again, after changing it for some other value.



How to do it? #

RLIM_INFINITY is defined on line 416 of &lt;sys/resources.h&gt;

/*
 * Symbolic constants for resource limits; since all limits are representable
 * as a type rlim_t, we are permitted to define RLIM_SAVED_* in terms of
 * RLIM_INFINITY.
 */
#define RLIM_INFINITY   (((__uint64_t)1 &lt;&lt; 63) - 1)     /* no limit */
#define RLIM_SAVED_MAX  RLIM_INFINITY   /* Unrepresentable hard limit */
#define RLIM_SAVED_CUR  RLIM_INFINITY   /* Unrepresentable soft limit */



In practice, for the current platform, the unlimited INFINITY maximum value is limited to INT_MAX (which is the maximum value of the Integer Type)

Considering that, it means that 'unlimited' means INT_MAX, which is the value 2147483647.

A possible workaround may be set the limit value of 2147483647:

sudo launchctl limit maxfiles 2147483647 2147483647

Because this is the maximum possible value. 

Since this is a possible workaround. I would have to go deeper into source to check if this really means 'the same thing' as 'unlimited'. 



Warning:

PS: Please, don't set it bigger than 2147483647, because it is SIGNED INT, and if you put 2147483648 (notice the number eight, I just add +1 to that maximum value), the SIGNED INT will be interpreted as negative, and you will instantly almost crash your macOS, because it will not be able to open any other file descriptor, including the one you will need to revert this value back, if trying to adjust it again by executing launchctl with different parameters. 

The only solution will be to hard reset your system if you try this. And you may loose all your unsaved things. Try it on some MacOS virtual machine for testing purposes



Source code References:

&lt;sys/resource.h&gt;  https://opensource.apple.com/source/xnu/xnu-4903.221.2/bsd/sys/resource.h.auto.html

&lt;dev/unix_startup.c&gt; https://opensource.apple.com/source/xnu/xnu-4903.221.2/bsd/dev/unix_startup.c.auto.html

&lt;kern/kern_resource.c&gt; https://opensource.apple.com/source/xnu/xnu-4903.221.2/bsd/kern/kern_resource.c.auto.html

&lt;conf/param.c&gt; https://opensource.apple.com/source/xnu/xnu-4903.221.2/bsd/conf/param.c.auto.html

&lt;sys/syslimits.h&gt; https://opensource.apple.com/source/xnu/xnu-4903.221.2/bsd/sys/syslimits.h.auto.html

launchctl.c https://opensource.apple.com/source/launchd/launchd-442.21/support/launchctl.c.auto.html