feat: add APIEmbedder for remote embedding with multi-endpoint support

- Introduced APIEmbedder class to handle embeddings via a remote HTTP API.
- Implemented token packing to optimize batch sizes based on token limits.
- Added support for multiple API endpoints with round-robin dispatching.
- Included retry logic for API calls with exponential backoff on failures.
- Enhanced indexing pipeline with file exclusion checks and smart chunking strategies.
- Updated tests to cover new APIEmbedder functionality and ensure robustness.

codex-lens-v2/pyproject.toml

@@ -44,6 +44,9 @@ faiss-cpu = [
 faiss-gpu = [
     "faiss-gpu>=1.7.4",
 ]
+embed-api = [
+    "httpx>=0.25",
+]
 reranker-api = [
     "httpx>=0.25",
 ]