def make_fakelc_collection(lclist,
                           simbasedir,
                           magrmsfrom,
                           magrms_interpolate='quadratic',
                           magrms_fillvalue='extrapolate',
                           maxlcs=25000,
                           maxvars=2000,
                           randomizemags=True,
                           randomizecoords=False,
                           vartypes=('EB','RRab','RRc','cepheid',
                                     'rotator','flare','HADS',
                                     'planet','LPV'),
                           lcformat='hat-sql',
                           lcformatdir=None,
                           timecols=None,
                           magcols=None,
                           errcols=None):

    '''This prepares light curves for the recovery sim.

    Collects light curves from `lclist` using a uniform sampling among
    them. Copies them to the `simbasedir`, zeroes out their mags and errs but
    keeps their time bases, also keeps their RMS and median mags for later
    use. Calculates the mag-rms relation for the entire collection and writes
    that to the `simbasedir` as well.

    The purpose of this function is to copy over the time base and mag-rms
    relation of an existing light curve collection to use it as the basis for a
    variability recovery simulation.

    This returns a pickle written to the `simbasedir` that contains all the
    information for the chosen ensemble of fake light curves and writes all
    generated light curves to the `simbasedir/lightcurves` directory. Run the
    `add_variability_to_fakelc_collection` function after this function to add
    variability of the specified type to these generated light curves.

    Parameters
    ----------

    lclist : list of str
        This is a list of existing project light curves. This can be generated
        from :py:func:`astrobase.lcproc.catalogs.make_lclist` or similar.

    simbasedir : str
        This is the directory to where the fake light curves and their
        information will be copied to.

    magrmsfrom : str or dict
        This is used to generate magnitudes and RMSes for the objects in the
        output collection of fake light curves. This arg is either a string
        pointing to an existing pickle file that must contain a dict or a dict
        variable that MUST have the following key-vals at a minimum::

            {'<magcol1_name>': {
                  'binned_sdssr_median': array of median mags for each magbin
                  'binned_lcmad_median': array of LC MAD values per magbin
             },
             '<magcol2_name>': {
                  'binned_sdssr_median': array of median mags for each magbin
                  'binned_lcmad_median': array of LC MAD values per magbin
             },
             .
             .
             ...}

        where `magcol1_name`, etc. are the same as the `magcols` listed in the
        magcols kwarg (or the default magcols for the specified
        lcformat). Examples of the magrmsfrom dict (or pickle) required can be
        generated by the
        :py:func:`astrobase.lcproc.varthreshold.variability_threshold` function.

    magrms_interpolate,magrms_fillvalue : str
        These are arguments that will be passed directly to the
        scipy.interpolate.interp1d function to generate interpolating functions
        for the mag-RMS relation. See:

        https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.interp1d.html

        for details.

    maxlcs : int
        This is the total number of light curves to choose from `lclist` and
        generate as fake LCs.

    maxvars : int
        This is the total number of fake light curves that will be marked as
        variable.

    vartypes : list of str
        This is a list of variable types to put into the collection. The
        vartypes for each fake variable star will be chosen uniformly from this
        list.

    lcformat : str
        This is the `formatkey` associated with your input real light curves'
        format, which you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `lclist`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    timecols : list of str or None
        The timecol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    magcols : list of str or None
        The magcol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    errcols : list of str or None
        The errcol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    Returns
    -------

    str
        Returns the string file name of a pickle containing all of the
        information for the fake LC collection that has been generated.

    '''

    try:
        formatinfo = get_lcformat(lcformat,
                                  use_lcformat_dir=lcformatdir)
        if formatinfo:
            (fileglob, readerfunc,
             dtimecols, dmagcols, derrcols,
             magsarefluxes, normfunc) = formatinfo
        else:
            LOGERROR("can't figure out the light curve format")
            return None
    except Exception as e:
        LOGEXCEPTION("can't figure out the light curve format")
        return None

    # override the default timecols, magcols, and errcols
    # using the ones provided to the function
    if timecols is None:
        timecols = dtimecols
    if magcols is None:
        magcols = dmagcols
    if errcols is None:
        errcols = derrcols

    if not isinstance(lclist, np.ndarray):
        lclist = np.array(lclist)

    chosenlcs = npr.choice(lclist, maxlcs, replace=False)

    fakelcdir = os.path.join(simbasedir, 'lightcurves')
    if not os.path.exists(fakelcdir):
        os.makedirs(fakelcdir)


    # get the magrms relation needed from the pickle or input dict
    if isinstance(magrmsfrom, str) and os.path.exists(magrmsfrom):
        with open(magrmsfrom,'rb') as infd:
            xmagrms = pickle.load(infd)
    elif isinstance(magrmsfrom, dict):
        xmagrms = magrmsfrom

    magrms = {}

    # get the required items from the magrms dict. interpolate the mag-rms
    # relation for the magcol so the make_fake_lc function can use it directly.
    for magcol in magcols:

        if (magcol in xmagrms and
            'binned_sdssr_median' in xmagrms[magcol] and
            'binned_lcmad_median' in xmagrms[magcol]):

            magrms[magcol] = {
                'binned_sdssr_median':np.array(
                    xmagrms[magcol]['binned_sdssr_median']
                ),
                'binned_lcmad_median':np.array(
                    xmagrms[magcol]['binned_lcmad_median']
                ),
            }

            # interpolate the mag-MAD relation
            interpolated_magmad = spi.interp1d(
                xmagrms[magcol]['binned_sdssr_median'],
                xmagrms[magcol]['binned_lcmad_median'],
                kind=magrms_interpolate,
                fill_value=magrms_fillvalue,
            )

            # save the magrms
            magrms[magcol]['interpolated_magmad'] = interpolated_magmad

            # generate the probability distribution in magbins. this is needed
            # to correctly sample the objects in this population
            magbins = np.array(xmagrms[magcol]['binned_sdssr_median'])
            bincounts = np.array(xmagrms[magcol]['binned_count'])
            binprobs = bincounts/np.sum(bincounts)

            # save the bin probabilities as well
            magrms[magcol]['magbin_probabilities'] = binprobs

        else:

            LOGWARNING('input magrms dict does not have '
                       'required info for magcol: %s' % magcol)

            magrms[magcol] = {
                'binned_sdssr_median':None,
                'binned_lcmad_median':None,
                'interpolated_magmad':None,
                'magbin_probabilities':None,
            }

    tasks = [(x, fakelcdir, {'lcformat':lcformat,
                             'timecols':timecols,
                             'magcols':magcols,
                             'errcols':errcols,
                             'magrms':magrms,
                             'randomizemags':randomizemags,
                             'randomizecoords':randomizecoords})
             for x in chosenlcs]

    # we can't parallelize because it messes up the random number generation,
    # causing all the IDs to clash. FIXME: figure out a way around this
    # (probably initial a seed in each worker process?)
    fakeresults = [collection_worker(task) for task in tasks]

    fakedb = {'simbasedir':simbasedir,
              'lcformat':lcformat,
              'timecols':timecols,
              'magcols':magcols,
              'errcols':errcols,
              'magsarefluxes':magsarefluxes}

    fobjects, fpaths = [], []
    fras, fdecls, fndets = [], [], []

    fmags, fmagmads = [], []
    ferrmeds, ferrmads = [], []

    totalvars = 0

    # these are the indices for the variable objects chosen randomly
    isvariableind = npr.randint(0,high=len(fakeresults), size=maxvars)
    isvariable = np.full(len(fakeresults), False, dtype=np.bool)
    isvariable[isvariableind] = True
    fakedb['isvariable'] = isvariable

    LOGINFO('added %s variable stars' % maxvars)

    # these are the variable types for each variable object
    vartypeind = npr.randint(0,high=len(vartypes), size=maxvars)
    vartypearr = np.array([vartypes[x] for x in vartypeind])
    fakedb['vartype'] = vartypearr

    for vt in sorted(vartypes):
        LOGINFO('%s: %s stars' % (vt, vartypearr[vartypearr == vt].size))

    # now go through the collection and get the mag/rms and err/rms for each
    # star. these will be used later to add noise to light curves
    LOGINFO('collecting info...')

    for fr in fakeresults:

        if fr is not None:

            fpath, fcols, finfo, fmoments = fr

            fobjects.append(finfo['objectid'])
            fpaths.append(fpath)

            fras.append(finfo['ra'])
            fdecls.append(finfo['decl'])
            fndets.append(finfo['ndet'])

            fmags.append(finfo['sdssr'])
            # this is per magcol
            fmagmads.append([fmoments[x]['mad'] for x in magcols])

            # these are per errcol
            ferrmeds.append([fmoments[x]['median'] for x in errcols])
            ferrmads.append([fmoments[x]['mad'] for x in errcols])


    # convert to nparrays
    fobjects = np.array(fobjects)
    fpaths = np.array(fpaths)

    fras = np.array(fras)
    fdecls = np.array(fdecls)
    fndets = np.array(fndets)

    fmags = np.array(fmags)
    fmagmads = np.array(fmagmads)
    ferrmeds = np.array(ferrmeds)
    ferrmads = np.array(ferrmads)

    # put these in the fakedb
    fakedb['objectid'] = fobjects
    fakedb['lcfpath'] = fpaths

    fakedb['ra'] = fras
    fakedb['decl'] = fdecls
    fakedb['ndet'] = fndets

    fakedb['sdssr'] = fmags
    fakedb['mad'] = fmagmads
    fakedb['errmedian'] = ferrmeds
    fakedb['errmad'] = ferrmads

    # get the mag-RMS curve for this light curve collection for each magcol
    fakedb['magrms'] = magrms

    # finally, write the collection DB to a pickle in simbasedir
    dboutfname = os.path.join(simbasedir,'fakelcs-info.pkl')
    with open(dboutfname, 'wb') as outfd:
        pickle.dump(fakedb, outfd)

    LOGINFO('wrote %s fake LCs to: %s' % (len(fakeresults), simbasedir))
    LOGINFO('fake LC info written to: %s' % dboutfname)

    return dboutfname
def make_fakelc_collection(lclist,
                           simbasedir,
                           magrmsfrom,
                           magrms_interpolate='quadratic',
                           magrms_fillvalue='extrapolate',
                           maxlcs=25000,
                           maxvars=2000,
                           randomizemags=True,
                           randomizecoords=False,
                           vartypes=('EB','RRab','RRc','cepheid',
                                     'rotator','flare','HADS',
                                     'planet','LPV'),
                           lcformat='hat-sql',
                           lcformatdir=None,
                           timecols=None,
                           magcols=None,
                           errcols=None):

    '''This prepares light curves for the recovery sim.

    Collects light curves from `lclist` using a uniform sampling among
    them. Copies them to the `simbasedir`, zeroes out their mags and errs but
    keeps their time bases, also keeps their RMS and median mags for later
    use. Calculates the mag-rms relation for the entire collection and writes
    that to the `simbasedir` as well.

    The purpose of this function is to copy over the time base and mag-rms
    relation of an existing light curve collection to use it as the basis for a
    variability recovery simulation.

    This returns a pickle written to the `simbasedir` that contains all the
    information for the chosen ensemble of fake light curves and writes all
    generated light curves to the `simbasedir/lightcurves` directory. Run the
    `add_variability_to_fakelc_collection` function after this function to add
    variability of the specified type to these generated light curves.

    Parameters
    ----------

    lclist : list of str
        This is a list of existing project light curves. This can be generated
        from :py:func:`astrobase.lcproc.catalogs.make_lclist` or similar.

    simbasedir : str
        This is the directory to where the fake light curves and their
        information will be copied to.

    magrmsfrom : str or dict
        This is used to generate magnitudes and RMSes for the objects in the
        output collection of fake light curves. This arg is either a string
        pointing to an existing pickle file that must contain a dict or a dict
        variable that MUST have the following key-vals at a minimum::

            {'<magcol1_name>': {
                  'binned_sdssr_median': array of median mags for each magbin
                  'binned_lcmad_median': array of LC MAD values per magbin
             },
             '<magcol2_name>': {
                  'binned_sdssr_median': array of median mags for each magbin
                  'binned_lcmad_median': array of LC MAD values per magbin
             },
             .
             .
             ...}

        where `magcol1_name`, etc. are the same as the `magcols` listed in the
        magcols kwarg (or the default magcols for the specified
        lcformat). Examples of the magrmsfrom dict (or pickle) required can be
        generated by the
        :py:func:`astrobase.lcproc.varthreshold.variability_threshold` function.

    magrms_interpolate,magrms_fillvalue : str
        These are arguments that will be passed directly to the
        scipy.interpolate.interp1d function to generate interpolating functions
        for the mag-RMS relation. See:

        https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.interp1d.html

        for details.

    maxlcs : int
        This is the total number of light curves to choose from `lclist` and
        generate as fake LCs.

    maxvars : int
        This is the total number of fake light curves that will be marked as
        variable.

    vartypes : list of str
        This is a list of variable types to put into the collection. The
        vartypes for each fake variable star will be chosen uniformly from this
        list.

    lcformat : str
        This is the `formatkey` associated with your input real light curves'
        format, which you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `lclist`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    timecols : list of str or None
        The timecol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    magcols : list of str or None
        The magcol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    errcols : list of str or None
        The errcol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    Returns
    -------

    str
        Returns the string file name of a pickle containing all of the
        information for the fake LC collection that has been generated.

    '''

    try:
        formatinfo = get_lcformat(lcformat,
                                  use_lcformat_dir=lcformatdir)
        if formatinfo:
            (fileglob, readerfunc,
             dtimecols, dmagcols, derrcols,
             magsarefluxes, normfunc) = formatinfo
        else:
            LOGERROR("can't figure out the light curve format")
            return None
    except Exception as e:
        LOGEXCEPTION("can't figure out the light curve format")
        return None

    # override the default timecols, magcols, and errcols
    # using the ones provided to the function
    if timecols is None:
        timecols = dtimecols
    if magcols is None:
        magcols = dmagcols
    if errcols is None:
        errcols = derrcols

    if not isinstance(lclist, np.ndarray):
        lclist = np.array(lclist)

    chosenlcs = npr.choice(lclist, maxlcs, replace=False)

    fakelcdir = os.path.join(simbasedir, 'lightcurves')
    if not os.path.exists(fakelcdir):
        os.makedirs(fakelcdir)


    # get the magrms relation needed from the pickle or input dict
    if isinstance(magrmsfrom, str) and os.path.exists(magrmsfrom):
        with open(magrmsfrom,'rb') as infd:
            xmagrms = pickle.load(infd)
    elif isinstance(magrmsfrom, dict):
        xmagrms = magrmsfrom

    magrms = {}

    # get the required items from the magrms dict. interpolate the mag-rms
    # relation for the magcol so the make_fake_lc function can use it directly.
    for magcol in magcols:

        if (magcol in xmagrms and
            'binned_sdssr_median' in xmagrms[magcol] and
            'binned_lcmad_median' in xmagrms[magcol]):

            magrms[magcol] = {
                'binned_sdssr_median':np.array(
                    xmagrms[magcol]['binned_sdssr_median']
                ),
                'binned_lcmad_median':np.array(
                    xmagrms[magcol]['binned_lcmad_median']
                ),
            }

            # interpolate the mag-MAD relation
            interpolated_magmad = spi.interp1d(
                xmagrms[magcol]['binned_sdssr_median'],
                xmagrms[magcol]['binned_lcmad_median'],
                kind=magrms_interpolate,
                fill_value=magrms_fillvalue,
            )

            # save the magrms
            magrms[magcol]['interpolated_magmad'] = interpolated_magmad

            # generate the probability distribution in magbins. this is needed
            # to correctly sample the objects in this population
            magbins = np.array(xmagrms[magcol]['binned_sdssr_median'])
            bincounts = np.array(xmagrms[magcol]['binned_count'])
            binprobs = bincounts/np.sum(bincounts)

            # save the bin probabilities as well
            magrms[magcol]['magbin_probabilities'] = binprobs

        else:

            LOGWARNING('input magrms dict does not have '
                       'required info for magcol: %s' % magcol)

            magrms[magcol] = {
                'binned_sdssr_median':None,
                'binned_lcmad_median':None,
                'interpolated_magmad':None,
                'magbin_probabilities':None,
            }

    tasks = [(x, fakelcdir, {'lcformat':lcformat,
                             'timecols':timecols,
                             'magcols':magcols,
                             'errcols':errcols,
                             'magrms':magrms,
                             'randomizemags':randomizemags,
                             'randomizecoords':randomizecoords})
             for x in chosenlcs]

    # we can't parallelize because it messes up the random number generation,
    # causing all the IDs to clash. FIXME: figure out a way around this
    # (probably initial a seed in each worker process?)
    fakeresults = [collection_worker(task) for task in tasks]

    fakedb = {'simbasedir':simbasedir,
              'lcformat':lcformat,
              'timecols':timecols,
              'magcols':magcols,
              'errcols':errcols,
              'magsarefluxes':magsarefluxes}

    fobjects, fpaths = [], []
    fras, fdecls, fndets = [], [], []

    fmags, fmagmads = [], []
    ferrmeds, ferrmads = [], []

    totalvars = 0

    # these are the indices for the variable objects chosen randomly
    isvariableind = npr.randint(0,high=len(fakeresults), size=maxvars)
    isvariable = np.full(len(fakeresults), False, dtype=np.bool)
    isvariable[isvariableind] = True
    fakedb['isvariable'] = isvariable

    LOGINFO('added %s variable stars' % maxvars)

    # these are the variable types for each variable object
    vartypeind = npr.randint(0,high=len(vartypes), size=maxvars)
    vartypearr = np.array([vartypes[x] for x in vartypeind])
    fakedb['vartype'] = vartypearr

    for vt in sorted(vartypes):
        LOGINFO('%s: %s stars' % (vt, vartypearr[vartypearr == vt].size))

    # now go through the collection and get the mag/rms and err/rms for each
    # star. these will be used later to add noise to light curves
    LOGINFO('collecting info...')

    for fr in fakeresults:

        if fr is not None:

            fpath, fcols, finfo, fmoments = fr

            fobjects.append(finfo['objectid'])
            fpaths.append(fpath)

            fras.append(finfo['ra'])
            fdecls.append(finfo['decl'])
            fndets.append(finfo['ndet'])

            fmags.append(finfo['sdssr'])
            # this is per magcol
            fmagmads.append([fmoments[x]['mad'] for x in magcols])

            # these are per errcol
            ferrmeds.append([fmoments[x]['median'] for x in errcols])
            ferrmads.append([fmoments[x]['mad'] for x in errcols])


    # convert to nparrays
    fobjects = np.array(fobjects)
    fpaths = np.array(fpaths)

    fras = np.array(fras)
    fdecls = np.array(fdecls)
    fndets = np.array(fndets)

    fmags = np.array(fmags)
    fmagmads = np.array(fmagmads)
    ferrmeds = np.array(ferrmeds)
    ferrmads = np.array(ferrmads)

    # put these in the fakedb
    fakedb['objectid'] = fobjects
    fakedb['lcfpath'] = fpaths

    fakedb['ra'] = fras
    fakedb['decl'] = fdecls
    fakedb['ndet'] = fndets

    fakedb['sdssr'] = fmags
    fakedb['mad'] = fmagmads
    fakedb['errmedian'] = ferrmeds
    fakedb['errmad'] = ferrmads

    # get the mag-RMS curve for this light curve collection for each magcol
    fakedb['magrms'] = magrms

    # finally, write the collection DB to a pickle in simbasedir
    dboutfname = os.path.join(simbasedir,'fakelcs-info.pkl')
    with open(dboutfname, 'wb') as outfd:
        pickle.dump(fakedb, outfd)

    LOGINFO('wrote %s fake LCs to: %s' % (len(fakeresults), simbasedir))
    LOGINFO('fake LC info written to: %s' % dboutfname)

    return dboutfname
def make_fakelc_collection ( lclist , simbasedir , magrmsfrom , magrms_interpolate = 'quadratic' , magrms_fillvalue = 'extrapolate' , maxlcs = 25000 , maxvars = 2000 , randomizemags = True , randomizecoords = False , vartypes = ( 'EB' , 'RRab' , 'RRc' , 'cepheid' , 'rotator' , 'flare' , 'HADS' , 'planet' , 'LPV' ) , lcformat = 'hat-sql' , lcformatdir = None , timecols = None , magcols = None , errcols = None ) : try : formatinfo = get_lcformat ( lcformat , use_lcformat_dir = lcformatdir ) if formatinfo : ( fileglob , readerfunc , dtimecols , dmagcols , derrcols , magsarefluxes , normfunc ) = formatinfo else : LOGERROR ( "can't figure out the light curve format" ) return None except Exception as e : LOGEXCEPTION ( "can't figure out the light curve format" ) return None # override the default timecols, magcols, and errcols # using the ones provided to the function if timecols is None : timecols = dtimecols if magcols is None : magcols = dmagcols if errcols is None : errcols = derrcols if not isinstance ( lclist , np . ndarray ) : lclist = np . array ( lclist ) chosenlcs = npr . choice ( lclist , maxlcs , replace = False ) fakelcdir = os . path . join ( simbasedir , 'lightcurves' ) if not os . path . exists ( fakelcdir ) : os . makedirs ( fakelcdir ) # get the magrms relation needed from the pickle or input dict if isinstance ( magrmsfrom , str ) and os . path . exists ( magrmsfrom ) : with open ( magrmsfrom , 'rb' ) as infd : xmagrms = pickle . load ( infd ) elif isinstance ( magrmsfrom , dict ) : xmagrms = magrmsfrom magrms = { } # get the required items from the magrms dict. interpolate the mag-rms # relation for the magcol so the make_fake_lc function can use it directly. for magcol in magcols : if ( magcol in xmagrms and 'binned_sdssr_median' in xmagrms [ magcol ] and 'binned_lcmad_median' in xmagrms [ magcol ] ) : magrms [ magcol ] = { 'binned_sdssr_median' : np . array ( xmagrms [ magcol ] [ 'binned_sdssr_median' ] ) , 'binned_lcmad_median' : np . array ( xmagrms [ magcol ] [ 'binned_lcmad_median' ] ) , } # interpolate the mag-MAD relation interpolated_magmad = spi . interp1d ( xmagrms [ magcol ] [ 'binned_sdssr_median' ] , xmagrms [ magcol ] [ 'binned_lcmad_median' ] , kind = magrms_interpolate , fill_value = magrms_fillvalue , ) # save the magrms magrms [ magcol ] [ 'interpolated_magmad' ] = interpolated_magmad # generate the probability distribution in magbins. this is needed # to correctly sample the objects in this population magbins = np . array ( xmagrms [ magcol ] [ 'binned_sdssr_median' ] ) bincounts = np . array ( xmagrms [ magcol ] [ 'binned_count' ] ) binprobs = bincounts / np . sum ( bincounts ) # save the bin probabilities as well magrms [ magcol ] [ 'magbin_probabilities' ] = binprobs else : LOGWARNING ( 'input magrms dict does not have ' 'required info for magcol: %s' % magcol ) magrms [ magcol ] = { 'binned_sdssr_median' : None , 'binned_lcmad_median' : None , 'interpolated_magmad' : None , 'magbin_probabilities' : None , } tasks = [ ( x , fakelcdir , { 'lcformat' : lcformat , 'timecols' : timecols , 'magcols' : magcols , 'errcols' : errcols , 'magrms' : magrms , 'randomizemags' : randomizemags , 'randomizecoords' : randomizecoords } ) for x in chosenlcs ] # we can't parallelize because it messes up the random number generation, # causing all the IDs to clash. FIXME: figure out a way around this # (probably initial a seed in each worker process?) fakeresults = [ collection_worker ( task ) for task in tasks ] fakedb = { 'simbasedir' : simbasedir , 'lcformat' : lcformat , 'timecols' : timecols , 'magcols' : magcols , 'errcols' : errcols , 'magsarefluxes' : magsarefluxes } fobjects , fpaths = [ ] , [ ] fras , fdecls , fndets = [ ] , [ ] , [ ] fmags , fmagmads = [ ] , [ ] ferrmeds , ferrmads = [ ] , [ ] totalvars = 0 # these are the indices for the variable objects chosen randomly isvariableind = npr . randint ( 0 , high = len ( fakeresults ) , size = maxvars ) isvariable = np . full ( len ( fakeresults ) , False , dtype = np . bool ) isvariable [ isvariableind ] = True fakedb [ 'isvariable' ] = isvariable LOGINFO ( 'added %s variable stars' % maxvars ) # these are the variable types for each variable object vartypeind = npr . randint ( 0 , high = len ( vartypes ) , size = maxvars ) vartypearr = np . array ( [ vartypes [ x ] for x in vartypeind ] ) fakedb [ 'vartype' ] = vartypearr for vt in sorted ( vartypes ) : LOGINFO ( '%s: %s stars' % ( vt , vartypearr [ vartypearr == vt ] . size ) ) # now go through the collection and get the mag/rms and err/rms for each # star. these will be used later to add noise to light curves LOGINFO ( 'collecting info...' ) for fr in fakeresults : if fr is not None : fpath , fcols , finfo , fmoments = fr fobjects . append ( finfo [ 'objectid' ] ) fpaths . append ( fpath ) fras . append ( finfo [ 'ra' ] ) fdecls . append ( finfo [ 'decl' ] ) fndets . append ( finfo [ 'ndet' ] ) fmags . append ( finfo [ 'sdssr' ] ) # this is per magcol fmagmads . append ( [ fmoments [ x ] [ 'mad' ] for x in magcols ] ) # these are per errcol ferrmeds . append ( [ fmoments [ x ] [ 'median' ] for x in errcols ] ) ferrmads . append ( [ fmoments [ x ] [ 'mad' ] for x in errcols ] ) # convert to nparrays fobjects = np . array ( fobjects ) fpaths = np . array ( fpaths ) fras = np . array ( fras ) fdecls = np . array ( fdecls ) fndets = np . array ( fndets ) fmags = np . array ( fmags ) fmagmads = np . array ( fmagmads ) ferrmeds = np . array ( ferrmeds ) ferrmads = np . array ( ferrmads ) # put these in the fakedb fakedb [ 'objectid' ] = fobjects fakedb [ 'lcfpath' ] = fpaths fakedb [ 'ra' ] = fras fakedb [ 'decl' ] = fdecls fakedb [ 'ndet' ] = fndets fakedb [ 'sdssr' ] = fmags fakedb [ 'mad' ] = fmagmads fakedb [ 'errmedian' ] = ferrmeds fakedb [ 'errmad' ] = ferrmads # get the mag-RMS curve for this light curve collection for each magcol fakedb [ 'magrms' ] = magrms # finally, write the collection DB to a pickle in simbasedir dboutfname = os . path . join ( simbasedir , 'fakelcs-info.pkl' ) with open ( dboutfname , 'wb' ) as outfd : pickle . dump ( fakedb , outfd ) LOGINFO ( 'wrote %s fake LCs to: %s' % ( len ( fakeresults ) , simbasedir ) ) LOGINFO ( 'fake LC info written to: %s' % dboutfname ) return dboutfname
This prepares light curves for the recovery sim.

    Collects light curves from `lclist` using a uniform sampling among
    them. Copies them to the `simbasedir`, zeroes out their mags and errs but
    keeps their time bases, also keeps their RMS and median mags for later
    use. Calculates the mag-rms relation for the entire collection and writes
    that to the `simbasedir` as well.

    The purpose of this function is to copy over the time base and mag-rms
    relation of an existing light curve collection to use it as the basis for a
    variability recovery simulation.

    This returns a pickle written to the `simbasedir` that contains all the
    information for the chosen ensemble of fake light curves and writes all
    generated light curves to the `simbasedir/lightcurves` directory. Run the
    `add_variability_to_fakelc_collection` function after this function to add
    variability of the specified type to these generated light curves.

    Parameters
    ----------

    lclist : list of str
        This is a list of existing project light curves. This can be generated
        from :py:func:`astrobase.lcproc.catalogs.make_lclist` or similar.

    simbasedir : str
        This is the directory to where the fake light curves and their
        information will be copied to.

    magrmsfrom : str or dict
        This is used to generate magnitudes and RMSes for the objects in the
        output collection of fake light curves. This arg is either a string
        pointing to an existing pickle file that must contain a dict or a dict
        variable that MUST have the following key-vals at a minimum::

            {'<magcol1_name>': {
                  'binned_sdssr_median': array of median mags for each magbin
                  'binned_lcmad_median': array of LC MAD values per magbin
             },
             '<magcol2_name>': {
                  'binned_sdssr_median': array of median mags for each magbin
                  'binned_lcmad_median': array of LC MAD values per magbin
             },
             .
             .
             ...}

        where `magcol1_name`, etc. are the same as the `magcols` listed in the
        magcols kwarg (or the default magcols for the specified
        lcformat). Examples of the magrmsfrom dict (or pickle) required can be
        generated by the
        :py:func:`astrobase.lcproc.varthreshold.variability_threshold` function.

    magrms_interpolate,magrms_fillvalue : str
        These are arguments that will be passed directly to the
        scipy.interpolate.interp1d function to generate interpolating functions
        for the mag-RMS relation. See:

        https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.interp1d.html

        for details.

    maxlcs : int
        This is the total number of light curves to choose from `lclist` and
        generate as fake LCs.

    maxvars : int
        This is the total number of fake light curves that will be marked as
        variable.

    vartypes : list of str
        This is a list of variable types to put into the collection. The
        vartypes for each fake variable star will be chosen uniformly from this
        list.

    lcformat : str
        This is the `formatkey` associated with your input real light curves'
        format, which you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `lclist`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    timecols : list of str or None
        The timecol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    magcols : list of str or None
        The magcol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    errcols : list of str or None
        The errcol keys to use from the input lcdict in generating the fake
        light curve. Fake LCs will be generated for each each
        timecol/magcol/errcol combination in the input light curves.

    Returns
    -------

    str
        Returns the string file name of a pickle containing all of the
        information for the fake LC collection that has been generated.
This prepares light curves for the recovery sim .