Spaces:
Sleeping
Sleeping
Update README.md
Browse files
README.md
CHANGED
@@ -1,282 +1,11 @@
|
|
1 |
-
|
2 |
-
|
3 |
-
|
4 |
-
|
5 |
-
|
6 |
-
|
7 |
-
|
8 |
-
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
-
- [Starting up all the services](#starting-up-all-the-services)
|
13 |
-
- [Tests](#tests)
|
14 |
-
- [Helpful Commands](#helpful-commands)
|
15 |
-
- [VS Code Extensions](#vs-code-extensions)
|
16 |
-
- [GPT3.5 summaries](#gpt35-summaries)
|
17 |
-
- [Resources](#resources)
|
18 |
-
|
19 |
-
## Setup
|
20 |
-
|
21 |
-
Ensure you have python and pip installed.
|
22 |
-
|
23 |
-
```shell
|
24 |
-
python --version
|
25 |
-
pip --version
|
26 |
-
```
|
27 |
-
|
28 |
-
From the root directory run the following command to install the
|
29 |
-
dependencies: `pip install -r requirements.txt`
|
30 |
-
|
31 |
-
You can run the app using this command: `python -m uvicorn src.api.index:app --reload`
|
32 |
-
|
33 |
-
Once running you can navigate to `http://127.0.0.1:8000/docs` to view the
|
34 |
-
interactive API documentation.
|
35 |
-
|
36 |
-
## Setup for local code development
|
37 |
-
|
38 |
-
There are some steps that need to be done prior to being able to
|
39 |
-
properly run and develop the code in this repository.
|
40 |
-
|
41 |
-
The following is a list of steps that have to happen prior to starting to
|
42 |
-
work / test the pipelines of this repository:
|
43 |
-
|
44 |
-
### Makefile
|
45 |
-
|
46 |
-
The project comes with a `Makefile` (**not supported in Windows!**)
|
47 |
-
that can be used for executing commands that will make the interaction
|
48 |
-
with this project much smoother. Keep in mind that folders with spaces in their names may cause issues.
|
49 |
-
|
50 |
-
One can see all of the available options by:
|
51 |
-
|
52 |
-
```bash
|
53 |
-
$: make
|
54 |
-
|
55 |
-
Available rules:
|
56 |
-
|
57 |
-
add-licenses Add licenses to Python files
|
58 |
-
all-start Starts both the API service and the local development service
|
59 |
-
all-stop Stops both the API service and the local development service
|
60 |
-
all-web Open up all web endpoints
|
61 |
-
api-build Build API Docker image
|
62 |
-
api-start Start API Docker image container
|
63 |
-
api-stop Stop API Docker image container
|
64 |
-
api-web Open API in web browser
|
65 |
-
app-app-build Build App Docker image
|
66 |
-
app-app-start Start App Docker image container
|
67 |
-
app-app-stop Stop App Docker image container
|
68 |
-
app-app-web Open App in web browser
|
69 |
-
clean Removes artifacts from the build stage, and other common Python artifacts.
|
70 |
-
clean-build Remove build artifacts
|
71 |
-
clean-images Clean left-over images
|
72 |
-
clean-model-files Remove files related to pre-trained models
|
73 |
-
clean-pyc Removes Python file artifacts
|
74 |
-
clean-secrets Removes secret artifacts - Serverless
|
75 |
-
clean-test Remove test and coverage artifacts
|
76 |
-
create-environment Creates the Python environment
|
77 |
-
create-envrc Set up the envrc file for the project.
|
78 |
-
delete-environment Deletes the Python environment
|
79 |
-
delete-envrc Delete the local envrc file of the project
|
80 |
-
destroy Remove ALL of the artifacts + Python environments
|
81 |
-
docker-local-dev-build Build local development Docker image
|
82 |
-
docker-local-dev-login Start a shell session into the docker container
|
83 |
-
docker-local-dev-start Start service for local development
|
84 |
-
docker-local-dev-stop Stop service for local development
|
85 |
-
docker-prune Clean Docker images
|
86 |
-
git-flow-install Install git-flow
|
87 |
-
init Initialize the repository for code development
|
88 |
-
lint Run the 'pre-commit' linting step manually
|
89 |
-
pip-upgrade Upgrade the version of the 'pip' package
|
90 |
-
pre-commit-install Installing the pre-commit Git hook
|
91 |
-
pre-commit-uninstall Uninstall the pre-commit Git hook
|
92 |
-
prepare_data Run the data preparation on the input dataset
|
93 |
-
requirements Install Python dependencies into the Python environment
|
94 |
-
run_faiss_and_embeddings Run the script for creating a FAISS index and text embeddings of the dataset
|
95 |
-
show-params Show the set of input parameters
|
96 |
-
sort-requirements Sort the project packages requirements file
|
97 |
-
```
|
98 |
-
|
99 |
-
> **NOTE**: If you're using `Windows`, you may have to copy and modify to some
|
100 |
-
> extents the commands that are part of the `Makefile` for some tasks.
|
101 |
-
|
102 |
-
### Starting up the Docker container and initializing the repository
|
103 |
-
|
104 |
-
In order to work on current / new features, one can use *Docker* to
|
105 |
-
start a new container and start the local development process.
|
106 |
-
|
107 |
-
To build the Docker image, one must follow the following steps:
|
108 |
-
|
109 |
-
1. Start the Docker daemon. If you're using Mac, one can use the
|
110 |
-
Docker Desktop App.
|
111 |
-
2. Go the project's directory and run the following command using the `Makefile`:
|
112 |
-
```bash
|
113 |
-
# Go the project's directory
|
114 |
-
cd /path/to/directory
|
115 |
-
|
116 |
-
# Build the Docker iamge and start a container
|
117 |
-
make docker-local-dev-start
|
118 |
-
```
|
119 |
-
3. Log into the container
|
120 |
-
```bash
|
121 |
-
# Log into the container
|
122 |
-
make docker-local-dev-login
|
123 |
-
```
|
124 |
-
|
125 |
-
4. Once you're inside the container, you'll see the following prompt:
|
126 |
-
|
127 |
-
```bash
|
128 |
-
# Log into the container
|
129 |
-
???$: make docker-local-dev-login
|
130 |
-
direnv: error /opt/program/.envrc is blocked. Run `direnv allow` to approve its content
|
131 |
-
```
|
132 |
-
> One will see the `direnv` error because `direnv` is installed and one must
|
133 |
-
> *allow* the changes to take effect.
|
134 |
-
|
135 |
-
5. Allow for the `direnv` changes
|
136 |
-
```bash
|
137 |
-
# Accept the changes
|
138 |
-
$: direnv allow
|
139 |
-
direnv: loading /opt/program/.envrc
|
140 |
-
```
|
141 |
-
|
142 |
-
6. The last thing is to initialize the repository. This can easily be done
|
143 |
-
with the `init` command:
|
144 |
-
|
145 |
-
```bash
|
146 |
-
$: make init
|
147 |
-
```
|
148 |
-
This will do the following tasks:
|
149 |
-
- Clean Python files
|
150 |
-
- Initialize the `.envrc` file used by `direnv`.
|
151 |
-
- Delete an existing python environment for the project, if it exists.
|
152 |
-
- Creates a new environment, if applicable
|
153 |
-
- Apply `direnv allow` to allow for `direnv` modifications.
|
154 |
-
- Install package requirements via `pip`
|
155 |
-
- Install `pre-commit` for code-linting and code-checking.
|
156 |
-
- Install `git-flow`, whenever possible.
|
157 |
-
|
158 |
-
These steps allow for the user to be able to develop new feature within
|
159 |
-
Docker, which makes it easier for developers to have the exact same set of
|
160 |
-
tools available.
|
161 |
-
|
162 |
-
## Starting the API service
|
163 |
-
|
164 |
-
The project comes with an out-of-the-box solution for starting and stopping
|
165 |
-
the API endpoint via Docker.
|
166 |
-
|
167 |
-
To start the container with the API endpoint, one must run the following
|
168 |
-
command:
|
169 |
-
|
170 |
-
```bash
|
171 |
-
# Start API service
|
172 |
-
make api-start
|
173 |
-
```
|
174 |
-
|
175 |
-
This service will start a Docker container that exposes the internal port
|
176 |
-
`7860` to the local host's port `7860`. Once the image has been built and
|
177 |
-
a container has started, one can go to the service's main page by using
|
178 |
-
the following command:
|
179 |
-
|
180 |
-
```bash
|
181 |
-
# Go the URL of the API endpoint
|
182 |
-
make api-web
|
183 |
-
```
|
184 |
-
|
185 |
-
> This will direct the user to the following URL:
|
186 |
-
> [http://localhost:7860/docs](http://localhost:7860/docs)
|
187 |
-
|
188 |
-
In order to *stop* the API service, one can run the following command:
|
189 |
-
|
190 |
-
```bash
|
191 |
-
# Stop the API service
|
192 |
-
make api-stop
|
193 |
-
```
|
194 |
-
|
195 |
-
As one customizes the FastAPI with new features and more, these changes
|
196 |
-
will be automatically displayed in the URL from above.
|
197 |
-
|
198 |
-
### Starting up all the services
|
199 |
-
|
200 |
-
Similar to the sections from above, one can spin up or spin down all the
|
201 |
-
services at once with the help of 2 commands, i.e. `all-start` and `all-stop`.
|
202 |
-
|
203 |
-
In order to spin up both the *api* service and that for *local development*,
|
204 |
-
one can run:
|
205 |
-
|
206 |
-
```bash
|
207 |
-
make all-start
|
208 |
-
```
|
209 |
-
|
210 |
-
This command will execute both services and one will be able to log into the
|
211 |
-
container for local development, as well to connect to the API via the
|
212 |
-
browser.
|
213 |
-
|
214 |
-
Similarly, in order to spin down all of the services, one can simply run:
|
215 |
-
|
216 |
-
```bash
|
217 |
-
make all-stop
|
218 |
-
```
|
219 |
-
|
220 |
-
This will stop both services and delete any unused Docker containers.
|
221 |
-
|
222 |
-
## Tests
|
223 |
-
|
224 |
-
Unit tests can be found under the `src` folder alongside source code.
|
225 |
-
Test files end with `_test`. The following command will run all of the tests.
|
226 |
-
|
227 |
-
```shell
|
228 |
-
python -m pytest -v -s
|
229 |
-
```
|
230 |
-
|
231 |
-
The `-v` argument is for verbose output. The `-s` argument is for turning
|
232 |
-
off the capture mode so that print statements are printed to the console.
|
233 |
-
|
234 |
-
A Makefile command also exists to run these. See `make test`.
|
235 |
-
|
236 |
-
## Helpful Commands
|
237 |
-
|
238 |
-
Here is a list of commands that may be helpful when interacting with this project.
|
239 |
-
|
240 |
-
### Docker
|
241 |
-
|
242 |
-
List all Docker containers:
|
243 |
-
|
244 |
-
```shell
|
245 |
-
docker ps -a
|
246 |
-
```
|
247 |
-
|
248 |
-
## VS Code Extensions
|
249 |
-
|
250 |
-
To help facilitate local development you can install
|
251 |
-
the [Visual Studio Code Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
|
252 |
-
extension for VS Code. This will allow you to connect to the local development Docker container and more easily develop features.
|
253 |
-
|
254 |
-
## GPT3.5 summaries
|
255 |
-
|
256 |
-
To generate the GPT3.5 summaries for all articles, use the following commands:
|
257 |
-
|
258 |
-
```
|
259 |
-
cd src
|
260 |
-
python3 -m utils.gpt35_summaries.cleanup_and_summarize
|
261 |
-
```
|
262 |
-
|
263 |
-
The output CSV file is placed in `src/utils/gpt35_summaries/df_embed_out.csv`
|
264 |
-
The pre-generated summaries for all articles are in `df_embed_out2.csv` in the same directory.
|
265 |
-
|
266 |
-
For an example of a focussed summary, please see `src/focused_summary_example.py`.
|
267 |
-
|
268 |
-
## Resources
|
269 |
-
|
270 |
-
- [direnv](https://github.com/direnv/direnv)
|
271 |
-
- [Docker](https://docs.docker.com/reference/)
|
272 |
-
- [Docker Compose](https://docs.docker.com/compose/)
|
273 |
-
- [FastAPI](https://fastapi.tiangolo.com/)
|
274 |
-
- [flake8](https://flake8.pycqa.org/en/latest/)
|
275 |
-
- [git](https://git-scm.com/)
|
276 |
-
- [GitHub Actions](https://docs.github.com/en/actions)
|
277 |
-
- [isort](https://pycqa.github.io/isort/index.html)
|
278 |
-
- [Makefile](https://www.gnu.org/software/make/manual/make.html)
|
279 |
-
- [Markdown](https://www.markdownguide.org/)
|
280 |
-
- [pre-commit](https://pre-commit.com)
|
281 |
-
- [Python](https://www.python.org/)
|
282 |
-
- [tmux](https://github.com/tmux/tmux/wiki/Getting-Started)
|
|
|
1 |
+
---
|
2 |
+
title: Cicero Synthesizer Space
|
3 |
+
emoji: 🏆
|
4 |
+
colorFrom: purple
|
5 |
+
colorTo: green
|
6 |
+
sdk: docker
|
7 |
+
pinned: false
|
8 |
+
license: mit
|
9 |
+
---
|
10 |
+
|
11 |
+
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|