Can I have both a case-insensitive system volume and a case-sensitive volume within an encrypted CoreStorage volume?
Can I have both a case-insensitive system volume and a case-sensitive volume within an encrypted CoreStorage volume?


Core Storage: 


can not create a volume within a volume
can create a logical volume (LV) within a logical volume group (LVG). 



  … to keep the main system volume case-insensitive …


That's reasonable. 


  … to have my home dir be case-sensitive …


Maybe desirable, but please beware: some software for OS X may not work as expected in that environment. I thought first of Adobe, found examples from 2012 and 2013: 


Dear Adobe - Alastair’s Place
Adobe Acrobat XI Pro Installation Failure (Case Sensitive User Directory) | Adobe Community


An example – OS X 10.9.5, Mavericks

I used Disk Utility to create two partitions from a USB flash drive (at /dev/disk12 at the time), then erased each of the unencrypted file systems – and opted to encrypt. 

For both encryptions I chose the same passphrase that is used to encrypt the FileVault 2 OS X startup volume.  

The relevant part of output from diskutil coreStorage list:

+-- Logical Volume Group 391198AA-F179-466A-BD30-1B9BF02D9CAF
|   =========================================================
|   Name:         journaled, encrypted
|   Status:       Online
|   Size:         2004877312 B (2.0 GB)
|   Free Space:   0 B (0 B)
|   |
|   +-&lt; Physical Volume 0A2101EC-811E-4E12-9CFD-A3CAEE4299D9
|   |   ----------------------------------------------------
|   |   Index:    0
|   |   Disk:     disk12s2
|   |   Status:   Online
|   |   Size:     2004877312 B (2.0 GB)
|   |
|   +-&gt; Logical Volume Family 83E9BA42-4F8D-4AF4-BBEA-F831A4C5E1F5
|       ----------------------------------------------------------
|       Encryption Status:       Unlocked
|       Encryption Type:         AES-XTS
|       Conversion Status:       Complete
|       Conversion Direction:    -none-
|       Has Encrypted Extents:   Yes
|       Fully Secure:            Yes
|       Passphrase Required:     Yes
|       |
|       +-&gt; Logical Volume C098FD1F-49E4-48D7-A3ED-1F8AC38FCA14
|           ---------------------------------------------------
|           Disk:                  disk13
|           Status:                Online
|           Size (Total):          1669324800 B (1.7 GB)
|           Conversion Progress:   -none-
|           Revertible:            No
|           LV Name:               journaled, encrypted
|           Volume Name:           journaled, encrypted
|           Content Hint:          Apple_HFS
|
+-- Logical Volume Group CA81191D-B39E-4B19-B4B6-FEC1FF3D2478
    =========================================================
    Name:         case-sensitive, journaled, encrypted
    Status:       Online
    Size:         1526685696 B (1.5 GB)
    Free Space:   0 B (0 B)
    |
    +-&lt; Physical Volume 4DDC2838-F234-430B-89E5-0C0AF59ECC28
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk12s3
    |   Status:   Online
    |   Size:     1526685696 B (1.5 GB)
    |
    +-&gt; Logical Volume Family FA8E36E2-724C-44B6-BC74-C7310BAE26D6
        ----------------------------------------------------------
        Encryption Status:       Unlocked
        Encryption Type:         AES-XTS
        Conversion Status:       Complete
        Conversion Direction:    -none-
        Has Encrypted Extents:   Yes
        Fully Secure:            Yes
        Passphrase Required:     Yes
        |
        +-&gt; Logical Volume A4C96B6E-06F4-4E49-B57F-AA5DDC05EF66
            ---------------------------------------------------
            Disk:                  disk14
            Status:                Online
            Size (Total):          1191133184 B (1.2 GB)
            Conversion Progress:   -none-
            Revertible:            No
            LV Name:               case-sensitive, journaled, encrypted
            Volume Name:           case-sensitive, journaled, encrypted
            Content Hint:          Apple_HFSX


Limitations

I rebooted the Mac to check whether either logical volume family (LVF) was automatically unlocked before loginwindow is launched by launchd. As expected, no automation of that sort. 

Issues

With Mavericks I can find no way to create more than one LV within an LVF. 

The following articles state that it's possible to do so: 


Undocumented CoreStorage Commands (2011-08-05)
Carter Allen - Practical Usage of CoreStorage (2011-08-14)
OS X CoreStorage Volume - shrink and create new - Fabian Franke (2013-02-25)


In the Franke article it's remarkable that the same UUID is used to both (a) shrink an existing LV; and (b) create another LV in the same LVF. 

Unless I'm missing something, this approach is impossible with Mavericks. For example: 

sh-3.2$ diskutil coreStorage resizeVolume DAED4621-90CF-43D1-BC40-F7ACABD7174F 600G
The Core Storage Logical Volume UUID is DAED4621-90CF-43D1-BC40-F7ACABD7174F
Started CoreStorage operation
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume Yosemite appears to be OK
Shrinking file system
Shrinking Logical Volume
Resizing Core Storage Logical Volume structures
Resized Core Storage Logical Volume to 600000000000 bytes
Finished CoreStorage operation
sh-3.2$ diskutil coreStorage createVolume DAED4621-90CF-43D1-BC40-F7ACABD7174F jhfsx case-sensitive 65G
DAED4621-90CF-43D1-BC40-F7ACABD7174F does not appear to be a valid Core Storage Logical Volume Group UUID or name
sh-3.2$ diskutil cs list | grep DAED4621-90CF-43D1-BC40-F7ACABD7174F
        +-&gt; Logical Volume DAED4621-90CF-43D1-BC40-F7ACABD7174F


Assuming a bug in Core Storage in Mavericks

We should review this answer – maybe add another – after Yosemite is released. 

(I should not expect a fix in Mavericks.)