Storing embeddings in Pinecone. Here's the process:

1. Loaded API Keys: 

- Used dotenv to load Pinecone credentials and environment securely.

2. Initialized Pinecone: 

- Connected to Pinecone to manage indexes and upload data.

3. Checked/Created Index:

- Index name: cre (Commercial Real Estate).

- Embedding dimension: 1536 (matches text-embedding-ada-002).

- Ensured the index exists or created one using cosine similarity.

4. Processed Embedding Files:

- Read embedding JSON files.

- Processed in batches (100 vectors per batch) for efficiency.

5. Prepared and Uploaded Data:

- Added unique IDs, vectors, and metadata for retrieval.

- Used index.upsert() to upload batches.

6. Logged Progress: 

- Tracked and confirmed all embeddings uploaded successfully.

Pinecone setup is complete. Next, we will query data from Pinecone!