tfrere commited on
Commit
d99ffd8
·
1 Parent(s): fd99e72
Files changed (2) hide show
  1. Dockerfile +4 -1
  2. backend/README.md +0 -352
Dockerfile CHANGED
@@ -17,6 +17,9 @@ RUN useradd -m -u 1000 user
17
  # Install uv instead of poetry
18
  RUN pip install uv
19
 
 
 
 
20
  # Create and configure cache directory
21
  RUN mkdir -p /app/.cache && \
22
  chown -R user:user /app
@@ -59,4 +62,4 @@ USER user
59
  EXPOSE 7860
60
 
61
  # Start both servers with wait-for
62
- CMD ["sh", "-c", "uvicorn app.asgi:app --host 0.0.0.0 --port 7861 & while ! nc -z localhost 7861; do sleep 1; done && cd frontend && npm run serve"]
 
17
  # Install uv instead of poetry
18
  RUN pip install uv
19
 
20
+ # Install uvicorn explicitly
21
+ RUN pip install uvicorn
22
+
23
  # Create and configure cache directory
24
  RUN mkdir -p /app/.cache && \
25
  chown -R user:user /app
 
62
  EXPOSE 7860
63
 
64
  # Start both servers with wait-for
65
+ CMD ["sh", "-c", "uvicorn main:app --host 0.0.0.0 --port 7861 & while ! nc -z localhost 7861; do sleep 1; done && cd frontend && npm run serve"]
backend/README.md CHANGED
@@ -1,352 +0,0 @@
1
- # Backend - Open LLM Leaderboard 🏆
2
-
3
- FastAPI backend for the Open LLM Leaderboard. This service is part of a larger architecture that includes a React frontend. For complete project installation, see the [main README](../README.md).
4
-
5
- ## ✨ Features
6
-
7
- - 📊 REST API for LLM models leaderboard management
8
- - 🗳️ Voting and ranking system
9
- - 🔄 HuggingFace Hub integration
10
- - 🚀 Caching and performance optimizations
11
-
12
- ## 🏗 Architecture
13
-
14
- ```mermaid
15
- flowchart TD
16
- Client(["**Frontend**<br><br>React Application"]) --> API["**API Server**<br><br>FastAPI REST Endpoints"]
17
-
18
- subgraph Backend
19
- API --> Core["**Core Layer**<br><br>• Middleware<br>• Cache<br>• Rate Limiting"]
20
- Core --> Services["**Services Layer**<br><br>• Business Logic<br>• Data Processing"]
21
-
22
- subgraph Services Layer
23
- Services --> Models["**Model Service**<br><br>• Model Submission<br>• Evaluation Pipeline"]
24
- Services --> Votes["**Vote Service**<br><br>• Vote Management<br>• Data Synchronization"]
25
- Services --> Board["**Leaderboard Service**<br><br>• Rankings<br>• Performance Metrics"]
26
- end
27
-
28
- Models --> Cache["**Cache Layer**<br><br>• In-Memory Store<br>• Auto Invalidation"]
29
- Votes --> Cache
30
- Board --> Cache
31
-
32
- Models --> HF["**HuggingFace Hub**<br><br>• Models Repository<br>• Datasets Access"]
33
- Votes --> HF
34
- Board --> HF
35
- end
36
-
37
- style Client fill:#f9f,stroke:#333,stroke-width:2px
38
- style Models fill:#bbf,stroke:#333,stroke-width:2px
39
- style Votes fill:#bbf,stroke:#333,stroke-width:2px
40
- style Board fill:#bbf,stroke:#333,stroke-width:2px
41
- style HF fill:#bfb,stroke:#333,stroke-width:2px
42
- ```
43
-
44
- ## 🛠️ HuggingFace Datasets
45
-
46
- The application uses several datasets on the HuggingFace Hub:
47
-
48
- ### 1. Requests Dataset (`{HF_ORGANIZATION}/requests`)
49
-
50
- - **Operations**:
51
- - 📤 `POST /api/models/submit`: Adds a JSON file for each new model submission
52
- - 📥 `GET /api/models/status`: Reads files to get models status
53
- - **Format**: One JSON file per model with submission details
54
- - **Updates**: On each new model submission
55
-
56
- ### 2. Votes Dataset (`{HF_ORGANIZATION}/votes`)
57
-
58
- - **Operations**:
59
- - 📤 `POST /api/votes/{model_id}`: Adds a new vote
60
- - 📥 `GET /api/votes/model/{provider}/{model}`: Reads model votes
61
- - 📥 `GET /api/votes/user/{user_id}`: Reads user votes
62
- - **Format**: JSONL with one vote per line
63
- - **Sync**: Bidirectional between local cache and Hub
64
-
65
- ### 3. Contents Dataset (`{HF_ORGANIZATION}/contents`)
66
-
67
- - **Operations**:
68
- - 📥 `GET /api/leaderboard`: Reads raw data
69
- - 📥 `GET /api/leaderboard/formatted`: Reads and formats data
70
- - **Format**: Main dataset containing all scores and metrics
71
- - **Updates**: Automatic after model evaluations
72
-
73
- ### 4. Official Providers Dataset (`{HF_ORGANIZATION}/official-providers`)
74
-
75
- - **Operations**:
76
- - 📥 Read-only access for highlighted models
77
- - **Format**: List of models selected by maintainers
78
- - **Updates**: Manual by maintainers
79
-
80
- ## 🛠 Local Development
81
-
82
- ### Prerequisites
83
-
84
- - Python 3.9+
85
- - [Poetry](https://python-poetry.org/docs/#installation)
86
-
87
- ### Standalone Installation (without Docker)
88
-
89
- ```bash
90
- # Install dependencies
91
- poetry install
92
-
93
- # Setup configuration
94
- cp .env.example .env
95
-
96
- # Start development server
97
- poetry run uvicorn app.asgi:app --host 0.0.0.0 --port 7860 --reload
98
- ```
99
-
100
- Server will be available at http://localhost:7860
101
-
102
- ## ⚙️ Configuration
103
-
104
- | Variable | Description | Default |
105
- | ------------ | ------------------------------------ | ----------- |
106
- | ENVIRONMENT | Environment (development/production) | development |
107
- | HF_TOKEN | HuggingFace authentication token | - |
108
- | PORT | Server port | 7860 |
109
- | LOG_LEVEL | Logging level (INFO/DEBUG/WARNING) | INFO |
110
- | CORS_ORIGINS | Allowed CORS origins | ["*"] |
111
- | CACHE_TTL | Cache Time To Live in seconds | 300 |
112
-
113
- ## 🔧 Middleware
114
-
115
- The backend uses several middleware layers for optimal performance and security:
116
-
117
- - **CORS Middleware**: Handles Cross-Origin Resource Sharing
118
- - **GZIP Middleware**: Compresses responses > 500 bytes
119
- - **Rate Limiting**: Prevents API abuse
120
- - **Caching**: In-memory caching with automatic invalidation
121
-
122
- ## 📝 Logging
123
-
124
- The application uses a structured logging system with:
125
-
126
- - Formatted console output
127
- - Different log levels per component
128
- - Request/Response logging
129
- - Performance metrics
130
- - Error tracking
131
-
132
- ## 📁 File Structure
133
-
134
- ```
135
- backend/
136
- ├── app/ # Source code
137
- │ ├── api/ # Routes and endpoints
138
- │ │ └── endpoints/ # Endpoint handlers
139
- │ ├── core/ # Configurations
140
- │ ├── services/ # Business logic
141
- │ └── utils/ # Utilities
142
- └── tests/ # Tests
143
- ```
144
-
145
- ## 📚 API
146
-
147
- Swagger documentation available at http://localhost:7860/docs
148
-
149
- ### Main Endpoints & Data Structures
150
-
151
- #### Leaderboard
152
-
153
- - `GET /api/leaderboard/formatted` - Formatted data with computed fields and metadata
154
-
155
- ```typescript
156
- Response {
157
- models: [{
158
- id: string, // eval_name
159
- model: {
160
- name: string, // fullname
161
- sha: string, // Model sha
162
- precision: string, // e.g. "fp16", "int8"
163
- type: string, // e.g. "fined-tuned-on-domain-specific-dataset"
164
- weight_type: string,
165
- architecture: string,
166
- average_score: number,
167
- has_chat_template: boolean
168
- },
169
- evaluations: {
170
- ifeval: {
171
- name: "IFEval",
172
- value: number, // Raw score
173
- normalized_score: number
174
- },
175
- bbh: {
176
- name: "BBH",
177
- value: number,
178
- normalized_score: number
179
- },
180
- math: {
181
- name: "MATH Level 5",
182
- value: number,
183
- normalized_score: number
184
- },
185
- gpqa: {
186
- name: "GPQA",
187
- value: number,
188
- normalized_score: number
189
- },
190
- musr: {
191
- name: "MUSR",
192
- value: number,
193
- normalized_score: number
194
- },
195
- mmlu_pro: {
196
- name: "MMLU-PRO",
197
- value: number,
198
- normalized_score: number
199
- }
200
- },
201
- features: {
202
- is_not_available_on_hub: boolean,
203
- is_merged: boolean,
204
- is_moe: boolean,
205
- is_flagged: boolean,
206
- is_official_provider: boolean
207
- },
208
- metadata: {
209
- upload_date: string,
210
- submission_date: string,
211
- generation: string,
212
- base_model: string,
213
- hub_license: string,
214
- hub_hearts: number,
215
- params_billions: number,
216
- co2_cost: number // CO₂ cost in kg
217
- }
218
- }]
219
- }
220
- ```
221
-
222
- - `GET /api/leaderboard` - Raw data from the HuggingFace dataset
223
- ```typescript
224
- Response {
225
- models: [{
226
- eval_name: string,
227
- Precision: string,
228
- Type: string,
229
- "Weight type": string,
230
- Architecture: string,
231
- Model: string,
232
- fullname: string,
233
- "Model sha": string,
234
- "Average ⬆️": number,
235
- "Hub License": string,
236
- "Hub ❤️": number,
237
- "#Params (B)": number,
238
- "Available on the hub": boolean,
239
- Merged: boolean,
240
- MoE: boolean,
241
- Flagged: boolean,
242
- "Chat Template": boolean,
243
- "CO₂ cost (kg)": number,
244
- "IFEval Raw": number,
245
- IFEval: number,
246
- "BBH Raw": number,
247
- BBH: number,
248
- "MATH Lvl 5 Raw": number,
249
- "MATH Lvl 5": number,
250
- "GPQA Raw": number,
251
- GPQA: number,
252
- "MUSR Raw": number,
253
- MUSR: number,
254
- "MMLU-PRO Raw": number,
255
- "MMLU-PRO": number,
256
- "Maintainer's Highlight": boolean,
257
- "Upload To Hub Date": string,
258
- "Submission Date": string,
259
- Generation: string,
260
- "Base Model": string
261
- }]
262
- }
263
- ```
264
-
265
- #### Models
266
-
267
- - `GET /api/models/status` - Get all models grouped by status
268
- ```typescript
269
- Response {
270
- pending: [{
271
- name: string,
272
- submitter: string,
273
- revision: string,
274
- wait_time: string,
275
- submission_time: string,
276
- status: "PENDING" | "EVALUATING" | "FINISHED",
277
- precision: string
278
- }],
279
- evaluating: Array<Model>,
280
- finished: Array<Model>
281
- }
282
- ```
283
- - `GET /api/models/pending` - Get pending models only
284
- - `POST /api/models/submit` - Submit model
285
-
286
- ```typescript
287
- Request {
288
- user_id: string,
289
- model_id: string,
290
- base_model?: string,
291
- precision?: string,
292
- model_type: string
293
- }
294
-
295
- Response {
296
- status: string,
297
- message: string
298
- }
299
- ```
300
-
301
- - `GET /api/models/{model_id}/status` - Get model status
302
-
303
- #### Votes
304
-
305
- - `POST /api/votes/{model_id}` - Vote
306
-
307
- ```typescript
308
- Request {
309
- vote_type: "up" | "down",
310
- user_id: string // HuggingFace username
311
- }
312
-
313
- Response {
314
- success: boolean,
315
- message: string
316
- }
317
- ```
318
-
319
- - `GET /api/votes/model/{provider}/{model}` - Get model votes
320
- ```typescript
321
- Response {
322
- total_votes: number,
323
- up_votes: number,
324
- down_votes: number
325
- }
326
- ```
327
- - `GET /api/votes/user/{user_id}` - Get user votes
328
- ```typescript
329
- Response Array<{
330
- model_id: string,
331
- vote_type: string,
332
- timestamp: string
333
- }>
334
- ```
335
-
336
- ## 🔒 Authentication
337
-
338
- The backend uses HuggingFace token-based authentication for secure API access. Make sure to:
339
-
340
- 1. Set your HF_TOKEN in the .env file
341
- 2. Include the token in API requests via Bearer authentication
342
- 3. Keep your token secure and never commit it to version control
343
-
344
- ## 🚀 Performance
345
-
346
- The backend implements several optimizations:
347
-
348
- - In-memory caching with configurable TTL (Time To Live)
349
- - Batch processing for model evaluations
350
- - Rate limiting for API endpoints
351
- - Efficient database queries with proper indexing
352
- - Automatic cache invalidation for votes