caldervf commited on
Commit
f98b3e9
·
1 Parent(s): 74c716c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +11 -282
README.md CHANGED
@@ -1,282 +1,11 @@
1
- ![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/Maven-Building-LLMS-into-Production/Cicero-LLM-Synthesizer/code-linting.yml)
2
-
3
- # Cicero LLM Synthesizer
4
-
5
- ## Contents
6
-
7
- - [Setup](#setup)
8
- - [Setup for local code development](#setup-for-local-code-development)
9
- - [Makefile](#makefile)
10
- - [Starting up the Docker container and initializing the repository](#starting-up-the-docker-container-and-initializing-the-repository)
11
- - [Starting the API service](#starting-the-api-service)
12
- - [Starting up all the services](#starting-up-all-the-services)
13
- - [Tests](#tests)
14
- - [Helpful Commands](#helpful-commands)
15
- - [VS Code Extensions](#vs-code-extensions)
16
- - [GPT3.5 summaries](#gpt35-summaries)
17
- - [Resources](#resources)
18
-
19
- ## Setup
20
-
21
- Ensure you have python and pip installed.
22
-
23
- ```shell
24
- python --version
25
- pip --version
26
- ```
27
-
28
- From the root directory run the following command to install the
29
- dependencies: `pip install -r requirements.txt`
30
-
31
- You can run the app using this command: `python -m uvicorn src.api.index:app --reload`
32
-
33
- Once running you can navigate to `http://127.0.0.1:8000/docs` to view the
34
- interactive API documentation.
35
-
36
- ## Setup for local code development
37
-
38
- There are some steps that need to be done prior to being able to
39
- properly run and develop the code in this repository.
40
-
41
- The following is a list of steps that have to happen prior to starting to
42
- work / test the pipelines of this repository:
43
-
44
- ### Makefile
45
-
46
- The project comes with a `Makefile` (**not supported in Windows!**)
47
- that can be used for executing commands that will make the interaction
48
- with this project much smoother. Keep in mind that folders with spaces in their names may cause issues.
49
-
50
- One can see all of the available options by:
51
-
52
- ```bash
53
- $: make
54
-
55
- Available rules:
56
-
57
- add-licenses Add licenses to Python files
58
- all-start Starts both the API service and the local development service
59
- all-stop Stops both the API service and the local development service
60
- all-web Open up all web endpoints
61
- api-build Build API Docker image
62
- api-start Start API Docker image container
63
- api-stop Stop API Docker image container
64
- api-web Open API in web browser
65
- app-app-build Build App Docker image
66
- app-app-start Start App Docker image container
67
- app-app-stop Stop App Docker image container
68
- app-app-web Open App in web browser
69
- clean Removes artifacts from the build stage, and other common Python artifacts.
70
- clean-build Remove build artifacts
71
- clean-images Clean left-over images
72
- clean-model-files Remove files related to pre-trained models
73
- clean-pyc Removes Python file artifacts
74
- clean-secrets Removes secret artifacts - Serverless
75
- clean-test Remove test and coverage artifacts
76
- create-environment Creates the Python environment
77
- create-envrc Set up the envrc file for the project.
78
- delete-environment Deletes the Python environment
79
- delete-envrc Delete the local envrc file of the project
80
- destroy Remove ALL of the artifacts + Python environments
81
- docker-local-dev-build Build local development Docker image
82
- docker-local-dev-login Start a shell session into the docker container
83
- docker-local-dev-start Start service for local development
84
- docker-local-dev-stop Stop service for local development
85
- docker-prune Clean Docker images
86
- git-flow-install Install git-flow
87
- init Initialize the repository for code development
88
- lint Run the 'pre-commit' linting step manually
89
- pip-upgrade Upgrade the version of the 'pip' package
90
- pre-commit-install Installing the pre-commit Git hook
91
- pre-commit-uninstall Uninstall the pre-commit Git hook
92
- prepare_data Run the data preparation on the input dataset
93
- requirements Install Python dependencies into the Python environment
94
- run_faiss_and_embeddings Run the script for creating a FAISS index and text embeddings of the dataset
95
- show-params Show the set of input parameters
96
- sort-requirements Sort the project packages requirements file
97
- ```
98
-
99
- > **NOTE**: If you're using `Windows`, you may have to copy and modify to some
100
- > extents the commands that are part of the `Makefile` for some tasks.
101
-
102
- ### Starting up the Docker container and initializing the repository
103
-
104
- In order to work on current / new features, one can use *Docker* to
105
- start a new container and start the local development process.
106
-
107
- To build the Docker image, one must follow the following steps:
108
-
109
- 1. Start the Docker daemon. If you're using Mac, one can use the
110
- Docker Desktop App.
111
- 2. Go the project's directory and run the following command using the `Makefile`:
112
- ```bash
113
- # Go the project's directory
114
- cd /path/to/directory
115
-
116
- # Build the Docker iamge and start a container
117
- make docker-local-dev-start
118
- ```
119
- 3. Log into the container
120
- ```bash
121
- # Log into the container
122
- make docker-local-dev-login
123
- ```
124
-
125
- 4. Once you're inside the container, you'll see the following prompt:
126
-
127
- ```bash
128
- # Log into the container
129
- ???$: make docker-local-dev-login
130
- direnv: error /opt/program/.envrc is blocked. Run `direnv allow` to approve its content
131
- ```
132
- > One will see the `direnv` error because `direnv` is installed and one must
133
- > *allow* the changes to take effect.
134
-
135
- 5. Allow for the `direnv` changes
136
- ```bash
137
- # Accept the changes
138
- $: direnv allow
139
- direnv: loading /opt/program/.envrc
140
- ```
141
-
142
- 6. The last thing is to initialize the repository. This can easily be done
143
- with the `init` command:
144
-
145
- ```bash
146
- $: make init
147
- ```
148
- This will do the following tasks:
149
- - Clean Python files
150
- - Initialize the `.envrc` file used by `direnv`.
151
- - Delete an existing python environment for the project, if it exists.
152
- - Creates a new environment, if applicable
153
- - Apply `direnv allow` to allow for `direnv` modifications.
154
- - Install package requirements via `pip`
155
- - Install `pre-commit` for code-linting and code-checking.
156
- - Install `git-flow`, whenever possible.
157
-
158
- These steps allow for the user to be able to develop new feature within
159
- Docker, which makes it easier for developers to have the exact same set of
160
- tools available.
161
-
162
- ## Starting the API service
163
-
164
- The project comes with an out-of-the-box solution for starting and stopping
165
- the API endpoint via Docker.
166
-
167
- To start the container with the API endpoint, one must run the following
168
- command:
169
-
170
- ```bash
171
- # Start API service
172
- make api-start
173
- ```
174
-
175
- This service will start a Docker container that exposes the internal port
176
- `7860` to the local host's port `7860`. Once the image has been built and
177
- a container has started, one can go to the service's main page by using
178
- the following command:
179
-
180
- ```bash
181
- # Go the URL of the API endpoint
182
- make api-web
183
- ```
184
-
185
- > This will direct the user to the following URL:
186
- > [http://localhost:7860/docs](http://localhost:7860/docs)
187
-
188
- In order to *stop* the API service, one can run the following command:
189
-
190
- ```bash
191
- # Stop the API service
192
- make api-stop
193
- ```
194
-
195
- As one customizes the FastAPI with new features and more, these changes
196
- will be automatically displayed in the URL from above.
197
-
198
- ### Starting up all the services
199
-
200
- Similar to the sections from above, one can spin up or spin down all the
201
- services at once with the help of 2 commands, i.e. `all-start` and `all-stop`.
202
-
203
- In order to spin up both the *api* service and that for *local development*,
204
- one can run:
205
-
206
- ```bash
207
- make all-start
208
- ```
209
-
210
- This command will execute both services and one will be able to log into the
211
- container for local development, as well to connect to the API via the
212
- browser.
213
-
214
- Similarly, in order to spin down all of the services, one can simply run:
215
-
216
- ```bash
217
- make all-stop
218
- ```
219
-
220
- This will stop both services and delete any unused Docker containers.
221
-
222
- ## Tests
223
-
224
- Unit tests can be found under the `src` folder alongside source code.
225
- Test files end with `_test`. The following command will run all of the tests.
226
-
227
- ```shell
228
- python -m pytest -v -s
229
- ```
230
-
231
- The `-v` argument is for verbose output. The `-s` argument is for turning
232
- off the capture mode so that print statements are printed to the console.
233
-
234
- A Makefile command also exists to run these. See `make test`.
235
-
236
- ## Helpful Commands
237
-
238
- Here is a list of commands that may be helpful when interacting with this project.
239
-
240
- ### Docker
241
-
242
- List all Docker containers:
243
-
244
- ```shell
245
- docker ps -a
246
- ```
247
-
248
- ## VS Code Extensions
249
-
250
- To help facilitate local development you can install
251
- the [Visual Studio Code Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
252
- extension for VS Code. This will allow you to connect to the local development Docker container and more easily develop features.
253
-
254
- ## GPT3.5 summaries
255
-
256
- To generate the GPT3.5 summaries for all articles, use the following commands:
257
-
258
- ```
259
- cd src
260
- python3 -m utils.gpt35_summaries.cleanup_and_summarize
261
- ```
262
-
263
- The output CSV file is placed in `src/utils/gpt35_summaries/df_embed_out.csv`
264
- The pre-generated summaries for all articles are in `df_embed_out2.csv` in the same directory.
265
-
266
- For an example of a focussed summary, please see `src/focused_summary_example.py`.
267
-
268
- ## Resources
269
-
270
- - [direnv](https://github.com/direnv/direnv)
271
- - [Docker](https://docs.docker.com/reference/)
272
- - [Docker Compose](https://docs.docker.com/compose/)
273
- - [FastAPI](https://fastapi.tiangolo.com/)
274
- - [flake8](https://flake8.pycqa.org/en/latest/)
275
- - [git](https://git-scm.com/)
276
- - [GitHub Actions](https://docs.github.com/en/actions)
277
- - [isort](https://pycqa.github.io/isort/index.html)
278
- - [Makefile](https://www.gnu.org/software/make/manual/make.html)
279
- - [Markdown](https://www.markdownguide.org/)
280
- - [pre-commit](https://pre-commit.com)
281
- - [Python](https://www.python.org/)
282
- - [tmux](https://github.com/tmux/tmux/wiki/Getting-Started)
 
1
+ ---
2
+ title: Cicero Synthesizer Space
3
+ emoji: 🏆
4
+ colorFrom: purple
5
+ colorTo: green
6
+ sdk: docker
7
+ pinned: false
8
+ license: mit
9
+ ---
10
+
11
+ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference