How to increase index row maximum size in Postgresql?
That would require significant changes, and I doubt it can be done easily.

See these excerpts from src/include/access/itup.h:

/*
 * Index tuple header structure
 *
 * All index tuples start with IndexTupleData.  If the HasNulls bit is set,
 * this is followed by an IndexAttributeBitMapData.  The index attribute
 * values follow, beginning at a MAXALIGN boundary.
 *
 * Note that the space allocated for the bitmap does not vary with the number
 * of attributes; that is because we don't have room to store the number of
 * attributes in the header.  Given the MAXALIGN constraint there's no space
 * savings to be had anyway, for usual values of INDEX_MAX_KEYS.
 */

typedef struct IndexTupleData
{
    ItemPointerData t_tid;      /* reference TID to heap tuple */

    /* ---------------
     * t_info is laid out in the following fashion:
     *
     * 15th (high) bit: has nulls
     * 14th bit: has var-width attributes
     * 13th bit: AM-defined meaning
     * 12-0 bit: size of tuple
     * ---------------
     */

    unsigned short t_info;      /* various info about tuple */

} IndexTupleData;               /* MORE DATA FOLLOWS AT END OF STRUCT */
[...]

/*
 * t_info manipulation macros
 */
#define INDEX_SIZE_MASK 0x1FFF
#define INDEX_AM_RESERVED_BIT 0x2000    /* reserved for index-AM specific
                                         * usage */
#define INDEX_VAR_MASK  0x4000
#define INDEX_NULL_MASK 0x8000


The limit you are hitting is INDEX_SIZE_MASK, and to increase it, you'd have to change the tuple header so that t_info has more than two bytes.

Perhaps it is as simple as that, but it might have repercussions in other parts of the code.