Spaces:
Sleeping
Sleeping
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,282 +1,11 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
- [Starting up all the services](#starting-up-all-the-services)
|
| 13 |
-
- [Tests](#tests)
|
| 14 |
-
- [Helpful Commands](#helpful-commands)
|
| 15 |
-
- [VS Code Extensions](#vs-code-extensions)
|
| 16 |
-
- [GPT3.5 summaries](#gpt35-summaries)
|
| 17 |
-
- [Resources](#resources)
|
| 18 |
-
|
| 19 |
-
## Setup
|
| 20 |
-
|
| 21 |
-
Ensure you have python and pip installed.
|
| 22 |
-
|
| 23 |
-
```shell
|
| 24 |
-
python --version
|
| 25 |
-
pip --version
|
| 26 |
-
```
|
| 27 |
-
|
| 28 |
-
From the root directory run the following command to install the
|
| 29 |
-
dependencies: `pip install -r requirements.txt`
|
| 30 |
-
|
| 31 |
-
You can run the app using this command: `python -m uvicorn src.api.index:app --reload`
|
| 32 |
-
|
| 33 |
-
Once running you can navigate to `http://127.0.0.1:8000/docs` to view the
|
| 34 |
-
interactive API documentation.
|
| 35 |
-
|
| 36 |
-
## Setup for local code development
|
| 37 |
-
|
| 38 |
-
There are some steps that need to be done prior to being able to
|
| 39 |
-
properly run and develop the code in this repository.
|
| 40 |
-
|
| 41 |
-
The following is a list of steps that have to happen prior to starting to
|
| 42 |
-
work / test the pipelines of this repository:
|
| 43 |
-
|
| 44 |
-
### Makefile
|
| 45 |
-
|
| 46 |
-
The project comes with a `Makefile` (**not supported in Windows!**)
|
| 47 |
-
that can be used for executing commands that will make the interaction
|
| 48 |
-
with this project much smoother. Keep in mind that folders with spaces in their names may cause issues.
|
| 49 |
-
|
| 50 |
-
One can see all of the available options by:
|
| 51 |
-
|
| 52 |
-
```bash
|
| 53 |
-
$: make
|
| 54 |
-
|
| 55 |
-
Available rules:
|
| 56 |
-
|
| 57 |
-
add-licenses Add licenses to Python files
|
| 58 |
-
all-start Starts both the API service and the local development service
|
| 59 |
-
all-stop Stops both the API service and the local development service
|
| 60 |
-
all-web Open up all web endpoints
|
| 61 |
-
api-build Build API Docker image
|
| 62 |
-
api-start Start API Docker image container
|
| 63 |
-
api-stop Stop API Docker image container
|
| 64 |
-
api-web Open API in web browser
|
| 65 |
-
app-app-build Build App Docker image
|
| 66 |
-
app-app-start Start App Docker image container
|
| 67 |
-
app-app-stop Stop App Docker image container
|
| 68 |
-
app-app-web Open App in web browser
|
| 69 |
-
clean Removes artifacts from the build stage, and other common Python artifacts.
|
| 70 |
-
clean-build Remove build artifacts
|
| 71 |
-
clean-images Clean left-over images
|
| 72 |
-
clean-model-files Remove files related to pre-trained models
|
| 73 |
-
clean-pyc Removes Python file artifacts
|
| 74 |
-
clean-secrets Removes secret artifacts - Serverless
|
| 75 |
-
clean-test Remove test and coverage artifacts
|
| 76 |
-
create-environment Creates the Python environment
|
| 77 |
-
create-envrc Set up the envrc file for the project.
|
| 78 |
-
delete-environment Deletes the Python environment
|
| 79 |
-
delete-envrc Delete the local envrc file of the project
|
| 80 |
-
destroy Remove ALL of the artifacts + Python environments
|
| 81 |
-
docker-local-dev-build Build local development Docker image
|
| 82 |
-
docker-local-dev-login Start a shell session into the docker container
|
| 83 |
-
docker-local-dev-start Start service for local development
|
| 84 |
-
docker-local-dev-stop Stop service for local development
|
| 85 |
-
docker-prune Clean Docker images
|
| 86 |
-
git-flow-install Install git-flow
|
| 87 |
-
init Initialize the repository for code development
|
| 88 |
-
lint Run the 'pre-commit' linting step manually
|
| 89 |
-
pip-upgrade Upgrade the version of the 'pip' package
|
| 90 |
-
pre-commit-install Installing the pre-commit Git hook
|
| 91 |
-
pre-commit-uninstall Uninstall the pre-commit Git hook
|
| 92 |
-
prepare_data Run the data preparation on the input dataset
|
| 93 |
-
requirements Install Python dependencies into the Python environment
|
| 94 |
-
run_faiss_and_embeddings Run the script for creating a FAISS index and text embeddings of the dataset
|
| 95 |
-
show-params Show the set of input parameters
|
| 96 |
-
sort-requirements Sort the project packages requirements file
|
| 97 |
-
```
|
| 98 |
-
|
| 99 |
-
> **NOTE**: If you're using `Windows`, you may have to copy and modify to some
|
| 100 |
-
> extents the commands that are part of the `Makefile` for some tasks.
|
| 101 |
-
|
| 102 |
-
### Starting up the Docker container and initializing the repository
|
| 103 |
-
|
| 104 |
-
In order to work on current / new features, one can use *Docker* to
|
| 105 |
-
start a new container and start the local development process.
|
| 106 |
-
|
| 107 |
-
To build the Docker image, one must follow the following steps:
|
| 108 |
-
|
| 109 |
-
1. Start the Docker daemon. If you're using Mac, one can use the
|
| 110 |
-
Docker Desktop App.
|
| 111 |
-
2. Go the project's directory and run the following command using the `Makefile`:
|
| 112 |
-
```bash
|
| 113 |
-
# Go the project's directory
|
| 114 |
-
cd /path/to/directory
|
| 115 |
-
|
| 116 |
-
# Build the Docker iamge and start a container
|
| 117 |
-
make docker-local-dev-start
|
| 118 |
-
```
|
| 119 |
-
3. Log into the container
|
| 120 |
-
```bash
|
| 121 |
-
# Log into the container
|
| 122 |
-
make docker-local-dev-login
|
| 123 |
-
```
|
| 124 |
-
|
| 125 |
-
4. Once you're inside the container, you'll see the following prompt:
|
| 126 |
-
|
| 127 |
-
```bash
|
| 128 |
-
# Log into the container
|
| 129 |
-
???$: make docker-local-dev-login
|
| 130 |
-
direnv: error /opt/program/.envrc is blocked. Run `direnv allow` to approve its content
|
| 131 |
-
```
|
| 132 |
-
> One will see the `direnv` error because `direnv` is installed and one must
|
| 133 |
-
> *allow* the changes to take effect.
|
| 134 |
-
|
| 135 |
-
5. Allow for the `direnv` changes
|
| 136 |
-
```bash
|
| 137 |
-
# Accept the changes
|
| 138 |
-
$: direnv allow
|
| 139 |
-
direnv: loading /opt/program/.envrc
|
| 140 |
-
```
|
| 141 |
-
|
| 142 |
-
6. The last thing is to initialize the repository. This can easily be done
|
| 143 |
-
with the `init` command:
|
| 144 |
-
|
| 145 |
-
```bash
|
| 146 |
-
$: make init
|
| 147 |
-
```
|
| 148 |
-
This will do the following tasks:
|
| 149 |
-
- Clean Python files
|
| 150 |
-
- Initialize the `.envrc` file used by `direnv`.
|
| 151 |
-
- Delete an existing python environment for the project, if it exists.
|
| 152 |
-
- Creates a new environment, if applicable
|
| 153 |
-
- Apply `direnv allow` to allow for `direnv` modifications.
|
| 154 |
-
- Install package requirements via `pip`
|
| 155 |
-
- Install `pre-commit` for code-linting and code-checking.
|
| 156 |
-
- Install `git-flow`, whenever possible.
|
| 157 |
-
|
| 158 |
-
These steps allow for the user to be able to develop new feature within
|
| 159 |
-
Docker, which makes it easier for developers to have the exact same set of
|
| 160 |
-
tools available.
|
| 161 |
-
|
| 162 |
-
## Starting the API service
|
| 163 |
-
|
| 164 |
-
The project comes with an out-of-the-box solution for starting and stopping
|
| 165 |
-
the API endpoint via Docker.
|
| 166 |
-
|
| 167 |
-
To start the container with the API endpoint, one must run the following
|
| 168 |
-
command:
|
| 169 |
-
|
| 170 |
-
```bash
|
| 171 |
-
# Start API service
|
| 172 |
-
make api-start
|
| 173 |
-
```
|
| 174 |
-
|
| 175 |
-
This service will start a Docker container that exposes the internal port
|
| 176 |
-
`7860` to the local host's port `7860`. Once the image has been built and
|
| 177 |
-
a container has started, one can go to the service's main page by using
|
| 178 |
-
the following command:
|
| 179 |
-
|
| 180 |
-
```bash
|
| 181 |
-
# Go the URL of the API endpoint
|
| 182 |
-
make api-web
|
| 183 |
-
```
|
| 184 |
-
|
| 185 |
-
> This will direct the user to the following URL:
|
| 186 |
-
> [http://localhost:7860/docs](http://localhost:7860/docs)
|
| 187 |
-
|
| 188 |
-
In order to *stop* the API service, one can run the following command:
|
| 189 |
-
|
| 190 |
-
```bash
|
| 191 |
-
# Stop the API service
|
| 192 |
-
make api-stop
|
| 193 |
-
```
|
| 194 |
-
|
| 195 |
-
As one customizes the FastAPI with new features and more, these changes
|
| 196 |
-
will be automatically displayed in the URL from above.
|
| 197 |
-
|
| 198 |
-
### Starting up all the services
|
| 199 |
-
|
| 200 |
-
Similar to the sections from above, one can spin up or spin down all the
|
| 201 |
-
services at once with the help of 2 commands, i.e. `all-start` and `all-stop`.
|
| 202 |
-
|
| 203 |
-
In order to spin up both the *api* service and that for *local development*,
|
| 204 |
-
one can run:
|
| 205 |
-
|
| 206 |
-
```bash
|
| 207 |
-
make all-start
|
| 208 |
-
```
|
| 209 |
-
|
| 210 |
-
This command will execute both services and one will be able to log into the
|
| 211 |
-
container for local development, as well to connect to the API via the
|
| 212 |
-
browser.
|
| 213 |
-
|
| 214 |
-
Similarly, in order to spin down all of the services, one can simply run:
|
| 215 |
-
|
| 216 |
-
```bash
|
| 217 |
-
make all-stop
|
| 218 |
-
```
|
| 219 |
-
|
| 220 |
-
This will stop both services and delete any unused Docker containers.
|
| 221 |
-
|
| 222 |
-
## Tests
|
| 223 |
-
|
| 224 |
-
Unit tests can be found under the `src` folder alongside source code.
|
| 225 |
-
Test files end with `_test`. The following command will run all of the tests.
|
| 226 |
-
|
| 227 |
-
```shell
|
| 228 |
-
python -m pytest -v -s
|
| 229 |
-
```
|
| 230 |
-
|
| 231 |
-
The `-v` argument is for verbose output. The `-s` argument is for turning
|
| 232 |
-
off the capture mode so that print statements are printed to the console.
|
| 233 |
-
|
| 234 |
-
A Makefile command also exists to run these. See `make test`.
|
| 235 |
-
|
| 236 |
-
## Helpful Commands
|
| 237 |
-
|
| 238 |
-
Here is a list of commands that may be helpful when interacting with this project.
|
| 239 |
-
|
| 240 |
-
### Docker
|
| 241 |
-
|
| 242 |
-
List all Docker containers:
|
| 243 |
-
|
| 244 |
-
```shell
|
| 245 |
-
docker ps -a
|
| 246 |
-
```
|
| 247 |
-
|
| 248 |
-
## VS Code Extensions
|
| 249 |
-
|
| 250 |
-
To help facilitate local development you can install
|
| 251 |
-
the [Visual Studio Code Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
|
| 252 |
-
extension for VS Code. This will allow you to connect to the local development Docker container and more easily develop features.
|
| 253 |
-
|
| 254 |
-
## GPT3.5 summaries
|
| 255 |
-
|
| 256 |
-
To generate the GPT3.5 summaries for all articles, use the following commands:
|
| 257 |
-
|
| 258 |
-
```
|
| 259 |
-
cd src
|
| 260 |
-
python3 -m utils.gpt35_summaries.cleanup_and_summarize
|
| 261 |
-
```
|
| 262 |
-
|
| 263 |
-
The output CSV file is placed in `src/utils/gpt35_summaries/df_embed_out.csv`
|
| 264 |
-
The pre-generated summaries for all articles are in `df_embed_out2.csv` in the same directory.
|
| 265 |
-
|
| 266 |
-
For an example of a focussed summary, please see `src/focused_summary_example.py`.
|
| 267 |
-
|
| 268 |
-
## Resources
|
| 269 |
-
|
| 270 |
-
- [direnv](https://github.com/direnv/direnv)
|
| 271 |
-
- [Docker](https://docs.docker.com/reference/)
|
| 272 |
-
- [Docker Compose](https://docs.docker.com/compose/)
|
| 273 |
-
- [FastAPI](https://fastapi.tiangolo.com/)
|
| 274 |
-
- [flake8](https://flake8.pycqa.org/en/latest/)
|
| 275 |
-
- [git](https://git-scm.com/)
|
| 276 |
-
- [GitHub Actions](https://docs.github.com/en/actions)
|
| 277 |
-
- [isort](https://pycqa.github.io/isort/index.html)
|
| 278 |
-
- [Makefile](https://www.gnu.org/software/make/manual/make.html)
|
| 279 |
-
- [Markdown](https://www.markdownguide.org/)
|
| 280 |
-
- [pre-commit](https://pre-commit.com)
|
| 281 |
-
- [Python](https://www.python.org/)
|
| 282 |
-
- [tmux](https://github.com/tmux/tmux/wiki/Getting-Started)
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: Cicero Synthesizer Space
|
| 3 |
+
emoji: 🏆
|
| 4 |
+
colorFrom: purple
|
| 5 |
+
colorTo: green
|
| 6 |
+
sdk: docker
|
| 7 |
+
pinned: false
|
| 8 |
+
license: mit
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|