text:
<file_sep>---
title: "9/6/2018 notes"
author: "<NAME>"
date: "9/6/2018"
output: html_document
runtime: shiny
---
```{r libraries}
library(dplyr)
library(ggplot2)
```
# Using python in RMD
The library reticulate enables to use python with r, you also have to direct to where it is using the *use_python* method.
```{r python_enabler}
library(reticulate)
use_python("/Users/brennandonnell/anaconda/bin/python")
```
Printing the first ten natural numbers numbers using r
```{r r_code}
x <- 1:10
print(x)
```
Printing the first ten natural numbers using python
```{python python_code}
import numpy as np
y = np.arange(1,11)
print(y)
```
# Shiny:
## What is it?
This is what you use if you can't afford tableau and/or you are on a budget.

* R package for *interactive web apps*

## 3 major road blocks
1. **User INterface (UI) aka look and feel**
    + Defines the components of the user interface
        + Page titles
        + Input options
        + Outputs
    + Defines what the user will see and interact with
2. **Server, aka brain**
    + Contains the information to build the app
    + Defines what happens in R
3. **Global/shinyapp aka enabler**

```{r shiny_example_1}
library(shiny)
# User Interface
ui_demo <- fluidPage(
 # "UNH Analytics & Data Science"
  #
  tags$em(h2("UNH Analytics & Data Science"))
  )
# Server
server_demo <- function(input, output){
}

# Enabler, creates the app.
shinyApp(ui=ui_demo, server=server_demo)
```
You can also use tags for this, see the comments.
```{r html_tags}
names(tags)
```
## Making user input

```{r shiny_example_2}
ui2 <- fluidPage(
  sidebarLayout(
    sidebarPanel(textInput(inputId = "n1", label = "input name")),
    mainPanel(textOutput(outputId = "displaytext"))
    )
  )

server2 <- function(input, output) {
  output$displaytext <- renderText(
    print(paste("Hello", input$n1))
  )
}
shinyApp(ui=ui2, server = server2)
```
## another example
make it do it for two strings together
```{r shiny_example_3}
ui3 <- fluidPage(
  sidebarLayout(
    sidebarPanel(textInput(inputId = "n1", label = "input first"),
                 textInput(inputId = "n2", label = "input second")),
    mainPanel(textOutput(outputId = "displaytext"))
    )
  )

server3 <- function(input, output) {
  output$displaytext <- renderText(
    print(paste(input$n1, input$n2))
  )
}
shinyApp(ui=ui3, server = server3)
```
## make a calculator
doesn't work yet
```{r shiny_example_4}
ui4 <- fluidPage(
  sidebarLayout(
    sidebarPanel(
      numericInput(inputId = "n1", label = "First number", 0),
      numericInput(inputId = "n2", label = "Second number", 0)),
    mainPanel(textOutput(outputId = "calculate"))
    )
  )

server4 <- function(input, output) {
  output$calculate <- renderText(
    sum(input$n1, input$n2)
  )
}
shinyApp(ui=ui4, server = server4)
```
# Moving on to the csv

```{r reading_csv, warning=FALSE, error=FALSE, message=FALSE}
library(readr)
fastfood_data <- read_csv("/Users/brennandonnell/Desktop/grad_school/R/fast_food (1).csv")
```
Doing stuff in shiny!

```{r radiobuttons}
ui_food <- fluidPage(
  # Sidebar layout with input and output definitions ----
  sidebarLayout(
    # Sidebar panel for inputs ----
    sidebarPanel(
      radioButtons("chain_ui", "Select Chain", choices = unique(fastfood_data$Chain),
                         selected = unique(fastfood_data$Chain)[1],inline = FALSE)

      ),
    # Main panel for displaying outputs ----
    mainPanel(
      # Output: Text ----
      h2("Scatter Plot"),
      plotOutput(outputId = "scatter")
      )
  )
)
server_food <- function(input,output){
  filtered_data <- reactive(filter(fastfood_data, Chain == input$chain_ui))
  output$scatter <- renderPlot(
    {
      ggplot(filtered_data(), aes(x=Calories,y=Saturated_Fat))+ geom_point()+
                    theme_classic()+scale_x_continuous(limits = c(0, 1500)) +
                    scale_y_continuous(limits = c(0,35))
    }
  )
}

shinyApp(ui = ui_food, server = server_food)
```

#What if we want to make a server of it?
Copyt he following into a new folder.
```{r za_warudo}
library(shiny)
library(dplyr)
library(ggplot2)
library(readr)
fastfood_data <- read_csv("fast_food.csv")
```
in the previous cell, make separate .r scripts for the server, ui, and the above will be the global

---
#shinyapps.io stuff
Go to [shinyapps.io](https://www.shinyapps.io) and make an account. It's pretty nifty and will let you share your apps. You can also link it up to your GitHub account, so you should also do that.

Go to Account, and then token. Don't share your token.
Now, in RStudio, under *Tools*, select *Global Options*, and then pick *Publishing*. You'll be able to *Connect* your shinyapps.io account to this, by putting the super secret code into there.

## This is where the fun begins.
Now, open one of your global files. a little button that says *Run app* will appear From there it will do the thing. **But wait, there's more!** If you click on the *publish* icon, you can publish it to your shinyapps.io account, and then run it on that website, and other people can also run it.

**Note:** whenever you publish stuff, you don't need the enabler.

---
# For next week...

* Create multiple tabs: one with a barplot, another with a scatterplot
* get your internet to work
* make a shinyapps.io account
    + Hint: use tabsetPanel() and tabPanel() as necessary within the mainPanel.

ON to tableauprep<file_sep># r-notebooks
Class notes from my graduate classes taken in R Markdown.

**Note**: .csv files are unfortunately not included as they are not my property.

Since these are my class notes, there may be mistakes in them.
Markdown is a really wonderful thing, and I would highly recommend people give it a try.
<file_sep>---
title: "8/30/18 notes"
author: "<NAME>"
output: pdf_document
---

# NOTES
## Intro

Replication is important when writing code. Make sure it's legible and easily understandable, and that your variables make sense.
you also want to show what worked **and** what didn't.

Side note, go back and redo the code in python/r and write a report about it. You could get it published too. Go to a conference with it as opposed to a journal.

## R Markdown
What is it?

it's markdown mixed with R code. Its output is similar to HTML, but way easier to write.
Good thing about it is that you can run your R code **AND** your python code.

# THIS IS HEADING 1
## this is a second heading.
From the pop-up window, if you select **Open in browser** it opens in your browser.

## lists and asterisks

* Putting *one* asterisk around text gives you *italicized* text.
* Putting **two** asterisks around text gives you **bolded** text.
    + To give you a list, give a space between the start of the list

[link](www.rstudio.com) is how you make a link.

***
# Example: mexample.txt
## create the stuff in r markdown.
Applicants must hold a baccalaureate degree (no specific field of study is required) from an accredited college or university with a **3.0 GPA** or higher. Applicants should have demonstrated quantitative aptitude in undergraduate coursework or similar work experience in analytic field. Prior coursework may include: ***statistics, chemistry, physics, mathematics*** or other quantitative courses.
Submit an application to the [UNH Graduate School](http://gradschool.unh.edu/apply.php) with the following:

1. Prior college transcripts
2. Two letters of recommendation
3. Resume

The document was last update on: `r Sys.time()`

***
# How about actually using R code in this?
## simple.
Here's how it is done. This is called an **R Chunk**
The following function (*paste*) concatenates two strings: "AB" and "CD".
```{r}
fn <- "AB"
ln <- "CD"
print(paste(fn, ln))

x <- "Univ of"
y <- "New Hampshire"
paste(x, y)
```
Now let's say that this code didn't work, but we still want to keep it. How do we hide it? in the R chunk, put {r echo=FALSE}.
```{r echo=FALSE}
fn <- "AB"
ln <- "CD"
print(paste(fn, ln))

x <- "Univ of"
y <- "New Hampshire"
paste(x, y)
```
But now, we're still getting the output. WE don't even want that. Now try {r echo=FALSE, eval=FALSE}. With the new eval=FALSE, the code in the chunk doesn't even get executed.
```{r echo=FALSE, eval=FALSE}
fn <- "AB"
ln <- "CD"
print(paste(fn, ln))

x <- "Univ of"
y <- "New Hampshire"
paste(x, y)
```
To turn off warnings, such as from loading libraries, if you include warning=FALSE in the curly braces for the chunk, the warnings won't appear.

Lastly, to showcase the code but *not* the results, use {r results='hide'} in the chunk.

***
# question
## in R markdown...
use data(mtcars) and do some stuff
SCATTERPLOT of **MPG vs HP**
```{r warning=FALSE, results='hide', mtcars, fig.height=4}
library(ggplot2)
#plot(mtcars$mpg, mtcars$hp)
ggplot(mtcars, aes(hp, mpg)) + geom_point(aes())
```
Bar Pot of **gear**
```{r}
ggplot(mtcars, aes(gear)) + geom_bar()
```
***

# BENCHMARKING

## WHY DO WE NEED IT?

Is your code **too slow**? Does your system **Crash** whenever you run R?
Well, *try keeping your libraries up-to-date*.
system.time(command) times it.
```{r warning=FALSE}
library(readr)
```
which runs faster, read.csv() or read_csv()?
```{r warning=FALSE}
setwd("/Users/brennandonnell/Desktop/grad_school/R")
system.time(
  read.csv("NH_data_072418.csv")
  )

system.time(
  read_csv("NH_data_072418.csv")
)
```

check out the library(microbenchmark). this'll do it a lot better.

```{r warning=FALSE, eval=FALSE}
library(microbenchmark)
microbenchmark(read.csv("NH_data_072418.csv"),
               read_csv("NH_data_072418.csv"),
               times = 3)
```

an important thing to know for parallel processing is *how many cores does your computer have*?
Luckily, thereis an easy way to figure it out, thanks to the benchmarkme library
```{r}
library(benchmarkme)
get_ram()
get_cpu()
```
# Code

* Which part of the code is taking the longest time to execute?
* Can you change the way you are executing the code?
    + Which is taking up the most time?

this is where the library profvis comes in.

* It allows you to benchmark *each line* of our code for...
    + d
    + d

how do you use it?
profvis({
your code here
})

```{r warning=FALSE, eval=FALSE}
library(profvis)
profvis({
  library(readr)
  x <- read_csv("Durham_Weather.csv")
  x$below_zero_vector <- ifelse(x$TMIN < 0, 1, 0)
  for (i in 1:length(x$TMIN)){
    if (x$TMIN[1] < 0){
      x$below_zero_loop[i] <- 1
    }else{
      x$below_zero_loop[i] <- 0
    }
    }
})
```

***
write code to make a scatterplot of tmin vs tmax using ggplot and plot. According to profvis, which is better?

```{r warning=FALSE, eval=FALSE}
profvis({
   x <- read_csv("Durham_Weather.csv")
   durham_weather <- filter(x, !is.na(TMAX) | !is.na(TMIN))
   plot(durham_weather$TMIN, durham_weather$TMAX)
   ggplot(durham_weather, aes(TMIN, TMAX)) + geom_point()
})
```
All odd occurances aside, it
what does *version* do? `r version`
it prints out the version & other stuff.

***

# Question:

* Does every year have 365 days? (no, leap year)
* Does every day have 24 hours? (no, dst)
* Does every minute have 60 seconds? (no, leap second. )

SO... what is the right day or time? what date and time is universal? what is the global standard time? [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) is the golden standard.

* yyyy-mm-dd
    + Example: 24 July 2018 in ISO 8601 is 2018-07-24, no need of separators (or - if needed).
* HH:MM:SS
    + Example: 9.15 PM in ISO 8601 standard is 21:15:00
```{r}
x <- Sys.Date()
```

***

# fun

Use the appropriate lubridate function to parse each of the following dates:

* d1 <- "August 29, 2018"
* d2 <- "2018-Aug-29"
* d3 <- "29-Aug-2018"
* d4 <- c("August 29 (2018)","August 30 (2018)")
* d5 <- "08/29/18"

(if you don't remember lubridate, google the following: DMY, MDY, YMD)

```{r}
library(lubridate)
d1 <- mdy("August 29, 2018")
d2 <- ymd("2018-Aug-29")
d3 <- dmy("29-Aug-2018")
d4 <- mdy(c("August 29 (2018)", "August 30 (2018)"))
d5 <- mdy("08/29/18")
```

Now try reading in durham weather again and telling the class of the variable

```{r}
y <- read.csv("Durham_Weather.csv")
typeof(y$Temp_Date)

x <- read_csv("Durham_Weather.csv")
typeof(x$Temp_Date)
typeof(x$Raw_Date)
```
SO, read.csv gives us an integer, and read_csv gives us a double. better.
now, extract and create new columns for

* day
* month (label)
* year
* weekday (label)
* yearday

(hints: In lubridate, day, month, year, wday, yday)
```{r warning=FALSE}
x1 <- read_csv("Durham_Weather.csv")
x1$Day     <- day(x$Temp_Date)
x1$Month   <- month(x$Temp_Date, label = TRUE)
x1$Year    <- year(x$Temp_Date)
x1$Weekday <- wday(x$Temp_Date, label = TRUE)
x1$Yearday <- yday(x$Temp_Date)
```

Now try doing that with the rawdate.
```{r warning=FALSE}
x2 <- read_csv("Durham_Weather.csv")
x2$Day     <- day(x$Raw_Date)
x2$Month   <- month(x$Raw_Date, label = TRUE)
x2$Year    <- year(x$Raw_Date)
x2$Weekday <- wday(x$Raw_Date, label = TRUE)
x2$Yearday <- yday(x$Raw_Date)
```
INterestingly enough, the date in Raw_Date switches midway through, resulting in a lot of missing data. How do we fix this?

d2$raw_date_fixed <- parse_date_time... just check the notes.

***

# Work on the following before coming back:

## checking for holidays?

* for the Durham_Weather dataset, create a column "isholiday" which equals 1 if NYSE is closed, or 0 if NYSE is open for the years 2014 & 2015
* SOme of the functions of interest to solve this problem include...
    + holidayNYSE from library(timeDate)
    + %in%

holidayNYSE takes in a year and gives back the timedates of the holidays
```{r warning=FALSE, message=FALSE}
library(timeDate)
hw           <- read_csv("Durham_Weather.csv")
hw$Day       <- day(hw$Temp_Date)
hw$Month_lab <- month(hw$Temp_Date, label = TRUE)
hw$Month     <- month(hw$Temp_Date)
hw$Year      <- year(hw$Temp_Date)
hw$Weekday   <- wday(hw$Temp_Date, label = TRUE)
hw$Yearday   <- yday(hw$Temp_Date)
hw$YMD       <- ymd(paste(hw$Year, hw$Month, hw$Day, sep = "-"))
hw$isweekend <- ifelse(hw$Weekday %in% c("Sat", "Sun"), 1, 0)
hw$isholiday <- ifelse(hw$YMD %in% as.Date(holidayNYSE(2014:2015)), 1,
                       ifelse(hw$isweekend == 1, 1, 0))
```

branch_name: refs/heads/master

languages: RMarkdown Markdown

repo_language: RMarkdown


text:
<repo_name>MIKNOTAURO/djangocon-2016-demo<file_sep>/fabfile/test.py
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from fabric.api import task

from .utils import docker_exec, build_venv

__all__ = ('run_all', 'es', 'failed', 'force_db', 'no_es', 'tdd')

def run_tests(name="live", flags='', args=''):
    with build_venv(name):
        docker_exec("py.test %s src/olympia %s" % (flags, args))

@task
def tdd(name="live", args=''):
    """to run the entire test suite, but stop on the first error"""
    run_tests(name, "-x --pdb", args)

@task
def run_all(name="live", args=''):
    """to run the entire test suite"""
    run_tests(name, args=args)

@task
def es(name="live", args=''):
    """to run the ES tests"""
    run_tests(name, "-m es_tests", args)

@task
def failed(name="live", args=''):
    """to rerun the failed tests from the previous run"""
    run_tests(name, "--lf", args)

@task
def force_db(name="live", args=''):
    """to run the entire test suite with a new database"""
    run_tests(name, "--create-db", args)

@task
def no_es(name="live", args=''):
    """to run all but the ES tests"""
    run_tests(name, "-m 'no es_tests'", args)
<file_sep>/wsgi/stub.ini
[uwsgi]
plugins = logfile,http,python
master = true
uid = olympia
gid = olympia
http = :1999
processes = 1
wsgi-file = /code/wsgi/stub.py
procname = uwsgi stub
logto = /code/deploy/logs/uwsgi-stub.log
<file_sep>/wsgi/uwsgi_status.ini
[uwsgi]
plugins = logfile,http,python
master = true
uid = olympia
gid = olympia
http = :9999
processes = 1
wsgi-file = /code/wsgi/uwsgi_status.py
procname = uwsgi status
logto = /code/deploy/logs/uwsgi-status.log
<file_sep>/fabfile/__init__.py
# -*- coding: utf-8 -*-
from __future__ import unicode_literals, print_function

from distutils.spawn import find_executable
from glob import glob
import socket
import time
import os
import re

from fabric.api import abort, cd, env, hide, lcd, local, task, settings, quiet

from .build_port_convert import build_to_port, base26_encode
from .utils import docker_exec, build_venv, dealias_build
# from . import test  # noqa

ROOT_DIR = os.path.dirname(os.path.dirname(__file__))

NUM_ADDONS = NUM_THEMES = 10

env.docker_shell_env = {
    'NPM_CONFIG_PREFIX': '/code/deploy/deps/',
    'PIP_CACHE_DIR': '/code/deploy/deps/cache/',
    'LANG': 'en_US.UTF-8',
    'LC_ALL': 'en_US.UTF-8',
}

@task
def init():
    """to initialize a docker image"""
    build_wheels()
    docker_exec("mkdir -p /code/deploy/assets/{static,media}")
    create_build("a")
    create_build("b")
    docker_exec(
        "ln -sfvn a /code/deploy/builds/_live",
        "ln -sfvn b /code/deploy/builds/_stage")
    initialize_db()
    populate_data()
    activate("a")
    activate("b")
    nginx_reload()

@task
def build_wheels():
    """Build the wheels for all dependencies"""
    tmp_dir = docker_exec("mktemp -d", capture=True)
    with cd(tmp_dir):
        docker_exec(
            "cp /code/addons-server/requirements/*.txt .",
            "cp /code/src/requirements/hadjango.txt .",
            "mkdir -p /code/deploy/deps/wheelhouse",
            r"perl -pi -e 's/^ .+$//g;s/ \\//g;s/setuptools==23\.0\.0/setuptools==25\.1\.1/g;' *.txt",
            "pip wheel -f /code/deploy/deps/wheelhouse --wheel-dir=/code/deploy/deps/wheelhouse --no-deps -r dev.txt",
            "pip wheel -f /code/deploy/deps/wheelhouse --wheel-dir=/code/deploy/deps/wheelhouse --no-deps -r docs.txt",
            "pip wheel -f /code/deploy/deps/wheelhouse --wheel-dir=/code/deploy/deps/wheelhouse --no-deps -r hadjango.txt",
            ("pip wheel -f /code/deploy/deps/wheelhouse --wheel-dir=/code/deploy/deps/wheelhouse --no-deps -r"
             " <(perl -pe 's/^\-e //g;' prod_without_hash.txt)"))

@task
def create_build(name):
    """Creates a new build of the Mozilla addons-server"""
    symlink_static_dirs(name)
    copy_src(name)
    git_init(name)
    with build_venv(name):
        npm_install(name)
        pip_install(name)
        build_assets(name)
        port = build_to_port(name)
        docker_exec(
            'echo "upstream web { server web:%d; }" > live.conf' % port,
            'echo "upstream webstage { server web:%d; }" > stage.conf' % port,
            'date +%FT%T > .DEPLOY_TAG')
        docker_exec("ln -svfn /code/conf/uwsgi/vassal.skel vassal.ini")

def symlink_static_dirs(name):
    """
    Creates symlinks for the static and media directories of build ``name`` so
    that nginx static serving works correctly.

    The directory './deploy/assets/static/%(name)s' will be created, along with
    the following symlink

        - ./deploy/builds/%(name)s/assets/static -> ./deploy/assets/static/%(name)s
    """
    build_dir = "/code/deploy/builds/%s" % name
    static_dir = "/code/deploy/assets/static/%s" % name
    link_dir = "%s/assets/static" % build_dir
    docker_exec(
        "mkdir -p %s" % static_dir,
        "mkdir -p %s/assets" % build_dir,
        "ln -sfvn /code/deploy/assets/media %s/assets/media" % build_dir,
        "[ -L %(link_dir)s ] || ln -s %(static_dir)s %(link_dir)s" % {
            'link_dir': link_dir,
            'static_dir': static_dir,
        })

    for d in glob("%s/deploy/assets/static/*" % ROOT_DIR):
        link_dir = "/code/deploy/assets/static/%s/%s" % (os.path.basename(d), name)
        docker_exec("[ -L %(link_dir)s ] || ln -s %(static_dir)s %(link_dir)s" % {
            'link_dir': link_dir,
            'static_dir': static_dir,
        })

def copy_src(name):
    build_dir = "/code/deploy/builds/%s" % name
    docker_exec(
        "git ls-files"
        " | perl -pne 's/\"//g; s/e\\\\314\\\\201/é/g;'"  # Fix jétpack.xpi path
        " | rsync -a --info=progress2 --files-from=- . %s" % build_dir)
    docker_exec("cp /code/src/local_settings.py %s" % build_dir)
    with build_venv(name):
        docker_exec("cp -R /code/src/hadjango .")
        docker_exec("cp /code/src/requirements/*.txt requirements")
        docker_exec("cp /code/fabfile/build_port_convert.py hadjango")
        docker_exec(
            r"perl -pi -e "
            r"'s/^ .+$//g;s/ \\//g;"
            r"s/setuptools==23\.0\.0/setuptools==25\.1\.1/g;' "
            "requirements/*.txt")

def git_init(name):
    """Create fake git repo so that jingo minify can cache bust images"""
    if not os.path.exists("%s/deploy/builds/%s/.git" % (ROOT_DIR, name)):
        with build_venv(name):
            docker_exec(
                'git init',
                'cat /dev/urandom | head -c256 > .random',
                'git add .random',
                'git commit -m "Initial commit"')

def pip_install(name):
    build_wheels()
    args = "-f /code/deploy/deps/wheelhouse --no-index --no-deps"
    with build_venv(name):
        olympia_egg_link = (
            "/code/deploy/builds/%s/lib/python2.7/site-packages/olympia.egg-link" % name)
        docker_exec((
            '[ -f %(egg_link)s ]'
            ' || pip install %(args)s -e .') % {'egg_link': olympia_egg_link, 'args': args})
        docker_exec(
            "pip install %s -r requirements/dev.txt" % args,
            "pip install %s -r requirements/docs.txt" % args,
            "pip install %s -r requirements/prod_without_hash.txt" % args,
            "pip install %s -r requirements/hadjango.txt" % args)

def npm_install(name="_live"):
    if os.path.exists(os.path.join(ROOT_DIR, 'deploy', 'builds', name, 'node_modules')):
        return
    with build_venv(name):
        # npm install has a bug when run inside docker on a mounted volume
        # (see <https://github.com/npm/npm/issues/9863>). So we run npm
        # install in a temp directory, then move it back.
        tmp_dir = docker_exec("mktemp -d", capture=True)
        docker_exec("cp package.json %s" % tmp_dir)
        with cd(tmp_dir):
            docker_exec("npm install")
        docker_exec("mv %s/node_modules ." % tmp_dir)

def initialize_db():
    """to create a new database"""
    with build_venv("_live"):
        docker_exec(
            "python manage.py reset_db",
            "python manage.py syncdb --noinput",
            "python manage.py loaddata initial.json",
            "python manage.py import_prod_versions",
            "schematic --fake src/olympia/migrations/",
            "python manage.py createsuperuser",
            "python manage.py loaddata zadmin/users")

def populate_data():
    """to populate a new database"""
    with build_venv("_live"):
        docker_exec(
            # reindex --wipe will force the ES mapping to be re-installed. Useful to
            # make sure the mapping is correct before adding a bunch of add-ons.
            "python manage.py reindex --wipe --force --noinput",
            "python manage.py generate_addons --app firefox %s" % NUM_ADDONS,
            "python manage.py generate_addons --app thunderbird %s" % NUM_ADDONS,
            "python manage.py generate_addons --app android %s" % NUM_ADDONS,
            "python manage.py generate_addons --app seamonkey %s" % NUM_ADDONS,
            "python manage.py generate_themes %s" % NUM_THEMES,
            # Now that addons have been generated, reindex.
            "python manage.py reindex --force --noinput",
            # Also update category counts (denormalized field)
            "python manage.py cron category_totals")

def build_assets(name):
    with build_venv(name):
        docker_exec(
            "python manage.py compress_assets",
            "python manage.py collectstatic --noinput")

@task
def swap_live(name=None):
    """Swap the specified build (default: _stage) with _live"""
    if name is None:
        name = dealias_build("_stage")
    live = dealias_build("_live")
    with cd("/code/deploy/builds"):
        docker_exec(
            "ln -snvf %s _stage" % live,
            "ln -snvf %s _live" % name)
    nginx_reload()

@task
def stage(name):
    """Stage the designated build"""
    live = dealias_build("_live")
    stage = dealias_build("_stage")
    if name == live:
        abort("Cannot stage the live build; use 'fab swap_live' instead")
    if name == stage:
        abort("Build %s is already staged" % name)
    if not os.path.islink("%s/deploy/builds/%s/zerg.ini" % (ROOT_DIR, name)):
        activate(name)
        time.sleep(10)
        warmup(name)
    with cd("/code/deploy/builds"):
        docker_exec("ln -snvf %s _stage" % name)

@task
def activate(name):
    """Start uWSGI zerg instances for a build"""
    with build_venv(name):
        if not os.path.islink("%s/deploy/builds/%s/vassal.ini" % (ROOT_DIR, name)):
            docker_exec("ln -svfn /code/conf/uwsgi/vassal.skel vassal.ini")
            time.sleep(3)
        docker_exec("ln -svfn /code/conf/uwsgi/zerg.skel zerg.ini")
        # Wait for the emperor to poll again
        time.sleep(3)

@task
def deactivate(name):
    """Stop uWSGI zerg instance for a build"""
    if not os.path.islink("%s/deploy/builds/%s/zerg.ini" % (ROOT_DIR, name)):
        return
    with build_venv(name):
        docker_exec("rm -f zerg.ini")
        time.sleep(3)

@task
def stop(name):
    """Stop all uwsgi vassals and zergs for a build"""
    deactivate(name)
    if not os.path.islink("%s/deploy/builds/%s/vassal.ini" % (ROOT_DIR, name)):
        return
    with build_venv(name):
        docker_exec("rm -f vassal.ini")
        time.sleep(3)

@task
def warmup(name="_stage"):
    """Warm up a build, to be used before swapping to live"""
    with build_venv(name):
        docker_exec("python manage.py warmup")

@task
def nginx_reload():
    """Reload nginx config"""
    docker_exec("nginx -s reload", server="nginx")

@task
def ip():
    """The ip where the addons site can be accessed"""
    docker_host = os.environ.get("DOCKER_HOST") or ""
    if not docker_host:
        with quiet():
            docker_env = local("docker-machine env addons", capture=True)
        if docker_env:
            match = re.search(r'DOCKER_HOST="(tcp://[^"]+?)"', docker_env)
            if match:
                docker_host = match.group(1)

    match = re.search(r'tcp://([^:]+):', docker_host)
    if match:
        print(match.group(1))
    else:
        try:
            # host used by dlite
            _, _, ips = socket.gethostbyname_ex("local.docker")
        except:
            abort("Could not determine docker-machine host; perhaps localhost?")
        else:
            print(ips[0])

@task
def find_free_build():
    """Find what the next free build directory name is."""
    current_builds = set([os.path.basename(p) for p in glob("%s/deploy/builds/[a-z]*" % ROOT_DIR)])
    i = 0
    while True:
        build = base26_encode(i)
        if build not in current_builds:
            print(build)
            return
        i += 1

@task
def rm(name):
    """Delete a build"""
    live_name = dealias_build("_live")
    stage_name = dealias_build("_stage")
    if name in (live_name, stage_name):
        abort("Cannot delete the current live or stage build")
    stop(name)
    docker_exec("rm -rf /code/deploy/builds/%s" % name)

@task
def djshell(name="_live"):
    """Connect to a running addons-server django shell"""
    with build_venv(name):
        docker_exec("python manage.py shell")

@task
def shell(server='web'):
    """Connect to a running addons-server docker shell"""
    cwd = '/code' if server == 'web' else '/'
    with cd(cwd):
        docker_exec("bash", root=True, server=server)

@task
def make_incremental_build(name, basis="live"):
    """
    Archive a build, hard-linking unchanged files from the "basis" build (default live)

    This can significantly reduce the disk space used by multiple builds.

    On mac, requires ``brew install coreutils``
    """
    cp_bin = find_executable('gcp')
    if find_executable('gcp'):
        cp_bin = "gcp"
    else:
        cp_bin = "cp"

    live_name = dealias_build("_live")
    stage_name = dealias_build("_stage")
    if name in (live_name, stage_name):
        abort("Cannot turn the live or stage build into an incremental build")

    basis = dealias_build(basis)

    stop(name)

    with lcd("%s/deploy/builds" % ROOT_DIR):
        local("mv %(name)s %(name)s~" % {'name': name})
        with settings(hide("stderr"), warn_only=True):
            cp_ret = local("%(cp_bin)s -al %(basis)s %(name)s" % {
                'basis': basis,
                'name': name,
                'cp_bin': cp_bin,
            })
        if not cp_ret.succeeded:
            local("mv %(name)s~ %(name)s" % {'name': name})
            abort("Local cp bin does not support -l flag (on mac: brew install coreutils)")
        local("rsync -acH --delete %(name)s~/ %(name)s" % {'name': name})
        local("rm -rf %(name)s~" % {'name': name})
<file_sep>/src/hadjango/uwsgi/bootstrap.py
"""
A script which preloads most modules in Django.

This file is executed via execfile() from other files in this directory.

It is assumed that the original file which is executing this file has set
os.environ['DJANGO_SETTINGS_MODULE'] prior to calling execfile().
"""
import os
from importlib import import_module

os.environ["CELERY_LOADER"] = "django"

def run_mgmt_validate():
    import django.core.management
    utility = django.core.management.ManagementUtility()
    command = utility.fetch_command('runserver')
    command.validate(display_num_errors=True)

def load_templatetags():
    import_module('django.template.base').get_templatetags_modules()

def load_admin():
    import_module('django.contrib.admin').autodiscover()

def load_i18n(lang_code):
    import_module('django.utils.translation').activate(lang_code)

def load_urls():
    import_module('django.core.urlresolvers').resolve('/')

def setup():
    import django

    django.setup()

    from django.conf import settings

    load_templatetags()
    if 'django.contrib.admin' in settings.INSTALLED_APPS:
        load_admin()
    load_i18n(settings.LANGUAGE_CODE)

    load_urls()

    run_mgmt_validate()

setup()
<file_sep>/docker/README.md
These are files that are either copied into docker containers, or are
executed as the entry points in `docker-compose.yml`.
<file_sep>/fabfile/README.md
Defines all of the commands available to fabric (run `fab --list` from the root
of the repository to see a full list).

All tasks are defined in `fabfile/__init__.py`.
<file_sep>/wsgi/README.md
Contains the uwsgi configuration files (`*.ini`) and wsgi files `*.py` for
simple wsgi applications that start on load.

stub.ini / stub.py

        A “hello world” wsgi application that is the initial upstream for
        nginx before `fab init` has been run.

uwsgi_status.ini / uwsgi_status.py

        A very simple app, bound on port `:9999`, that merges the output of
        the various uwsgi stat sockets at `/var/run/uwsgi/*.stats` and combines
        that with information pulled from the process table (see
        `src/hadjango/uwsgi/wsgi.py`), outputing the result as json.
        This script returns the data that powers the uwsgi dashboard
        (at http://live.addons/uwsgi/).

<file_sep>/wsgi/stub.py
#!/usr/bin/env python
"""
A very simple wsgi application, used for the initial upstream before the
environments have been initialized.
"""

html = """<!DOCTYPE html>

<title>High Availability Django Demo</title>

Welcome to the High Availability Django Demo!

To continue, run:

fab init

from the root of the checked out repository.

"""

def application(environ, start_response):
    status = '200 OK'
    headers = [
        ('Content-Type', 'text/html'),
        ('Content-Length', str(len(html))),
    ]
    start_response(status, headers)
    return [html]
<file_sep>/fabfile/utils.py
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from contextlib import contextmanager
import os
import re

import six

from fabric.api import env, local
from fabric.context_managers import cd, path, quiet
from fabric.state import win32

ROOT_DIR = os.path.dirname(os.path.dirname(__file__))

IN_DOCKER = os.path.exists('/addons-server-centos7-container')

if IN_DOCKER:
    DOCKER_NAME = ""
else:
    COMPOSE_PROJECT_NAME = os.path.basename(ROOT_DIR.replace('-', '').replace('_', ''))
    DOCKER_NAME = "%s_%%(server)s_1" % COMPOSE_PROJECT_NAME

def docker_exec(*cmd, **kwargs):
    server = kwargs.pop('server', 'web')
    root = kwargs.pop('root', False if server == 'web' else True)
    default_cwd = "addons-server" if server == "web" else "/"
    cwd = env['cwd'] if env['cwd'] else default_cwd
    if not cmd[0].startswith('cd '):
        cmd = ("cd %s" % cwd, ) + cmd
    cmd = " && ".join(cmd)
    if server == "web":
        cmd = _prefix_env_vars(cmd)
    cmd = cmd.replace("'", "'\"'\"'")
    if root:
        full_cmd = "bash -c '%s'" % cmd
    else:
        full_cmd = """su olympia -c 'bash -c '"'"'%s'"'"''""" % cmd.replace("'", "'\"'\"'")
    if not IN_DOCKER:
        container_name = DOCKER_NAME % {'server': server}
        full_cmd = "docker exec -t -i %s %s" % (container_name, full_cmd)
    if (six.PY2 and isinstance(full_cmd, unicode)):
        full_cmd = full_cmd.encode('utf-8')
    return local(full_cmd, **kwargs)

@contextmanager
def build_venv(name):
    old_build_name = env.get('build_name')
    env['build_name'] = name
    build_dir = "/code/deploy/builds/%s" % name
    with cd("%s" % build_dir):
        docker_exec("[ -f bin/pip ] || virtualenv . --never-download")
        with path(os.path.join(build_dir, 'bin'), behavior='prepend'):
            yield
    env['build_name'] = old_build_name

_find_unsafe = re.compile(r'[a-zA-Z0-9_^@%+=:,./-]').search

def quote(s):
    """Return a shell-escaped version of the string *s*."""
    if not s:
        return "''"

    if _find_unsafe(s) is None:
        return s

    # use single quotes, and put single quotes into double quotes
    # the string $'b is then quoted as '$'"'"'b'

    return "'" + s.replace("'", "'\"'\"'") + "'"

def _shell_escape(string):
    """
    Escape double quotes, backticks and dollar signs in given ``string``.

    For example::

        >>> _shell_escape('abc$')
        'abc\\\\$'
        >>> _shell_escape('"')
        '\\\\"'
    """
    for char in ('"', '$', '`'):
        string = string.replace(char, '\%s' % char)
    return string

def _prefix_env_vars(command, local=False):
    """
    Prefixes ``command`` with any shell environment vars, e.g. ``PATH=foo ``.

    Currently, this only applies the PATH updating implemented in
    `~fabric.context_managers.path` and environment variables from
    `~fabric.context_managers.shell_env`.

    Will switch to using Windows style 'SET' commands when invoked by
    ``local()`` and on a Windows localhost.
    """
    env_vars = {}

    # path(): local shell env var update, appending/prepending/replacing $PATH
    path = env.path
    if path:
        if env.path_behavior == 'append':
            path = '$PATH:\"%s\"' % path
        elif env.path_behavior == 'prepend':
            path = '\"%s\":$PATH' % path
        elif env.path_behavior == 'replace':
            path = '\"%s\"' % path

        env_vars['PATH'] = path

    # shell_env()
    env_vars.update(env.get('docker_shell_env'))

    if env_vars:
        set_cmd, exp_cmd = '', ''
        if win32 and local:
            set_cmd = 'SET '
        else:
            exp_cmd = 'export '

        exports = ' '.join(
            '%s%s="%s"' % (set_cmd, k, v if k == 'PATH' else _shell_escape(v))
            for k, v in env_vars.iteritems()
        )
        shell_env_str = '%s%s && ' % (exp_cmd, exports)
    else:
        shell_env_str = ''

    return "%s%s" % (shell_env_str, command)

def dealias_build(name):
    if name in ("live", "stage"):
        name = "_%s" % name
    path = "%s/deploy/builds/%s" % (ROOT_DIR, name)
    with quiet():
        return os.path.basename(
            docker_exec("cd %s && pwd -P" % path, capture=True))
<file_sep>/src/hadjango/management/commands/warmup.py
import random

from django.conf import settings
from django.core.management.base import BaseCommand

import requests
from requests_futures.sessions import FuturesSession

from hadjango.build_port_convert import build_to_port

class Command(BaseCommand):
    """
    Command that primes processes/cache for a given build
    """

    help = 'Warms up processes/cache for a given host/port'

    def add_arguments(self, parser):
        parser.add_argument('--build', '-b', dest='build',
            default=settings.BUILD_NAME, help='Build to warm up: (a|b|c|...)')
        parser.add_argument('--concurrents', '-c', dest='concurrents',
            default='6', help='Number of concurrent requests to make')

    def handle(self, **options):
        server = "web"
        build = options['build']
        port = build_to_port(build)
        max_workers = int(options['concurrents'])

        session = FuturesSession(max_workers=max_workers)

        futures = []
        random_version = random.randint(0, 65535)
        for i in range(0, max_workers):
            version = '' if i == 0 else '?v=%d%d' % (i, random_version)
            futures.append(session.get('http://%(server)s:%(port)d/en-US/firefox/%(version)s' % {
                'server': server,
                'port': port,
                'version': version,
            }))

        for i, future in enumerate(futures):
            # wait for the response to complete
            try:
                response = future.result()
            except requests.ConnectionError:
                status = 'XXX'
                time = 0
            else:
                status = response.status_code
                time = int(round(response.elapsed.total_seconds() * 1000.0))
            self.stdout.write("%(server)s: (%(status)s) %(time)6d ms: worker %(i)s\n" % {
                'server': server,
                'status': status,
                'time': time,
                'i': i + 1,
            })

        self.stdout.write("Successfully warmed up.")
<file_sep>/README.md
# Django UWSGI deploy demo

This repository demonstrates the principles behind the talk “High-Availability
Django” given by <NAME> of _The Atlantic_ at the 2016 Djangocon.

In order to simulate the deployment of a large-ish Django project, it uses the
source for [addons.mozilla.org](https://addons.mozilla.org/), [found
here](https://github.com/mozilla/addons-server).

## Running

In order to run this demo VM, Docker, Fabric, and VirtualBox must be
installed. Once installed, the VM can be provisioned by running (from the
root of the checked out repository):

```shell
git submodule init
git submodule update
# Skip the next two steps if you don't want docker-machine
docker-machine create --driver virtualbox addons
eval $(docker-machine env addons)
docker-compose up -d
fab init
```

During the `fab init` step you will be prompted once when the database is initialized,
and then again to create the superuser.

Using `docker-machine` is not required unless you are using the older Docker Toolbox
on Mac (or you have installed it with `brew install docker`). Keep in mind however
that by default, running `docker-compose up` without using a docker-machine VM will
bind to port 80 on your localhost, which means it will need to be unused, so
you may wish to use `docker-machine` even if you don't need to.
Alternatively, if you are using Docker for Mac, you can use
[https://github.com/nlf/dlite](https://github.com/nlf/dlite) to have docker use a
separate bridged network (and thus separate ips for its containers).

Grab the ip address using the command `fab ip` (unless you are not using `docker-machine`
or `dlite`, in which case it will be `127.0.0.1`), and then add to your `/etc/hosts` files:

```
192.168.64.7  live.addons stage.addons
```

Replacing `192.168.64.7` with the ip address for docker.

After running `fab init`, you should be able to visit http://live.addons/ and http://stage.addons/
to see the two initial builds (note that there may be a delay while the code initializes).
You can view a live-updating uwsgi status dashboard at http://live.addons/uwsgi/

## Creating new builds

Builds are stored in folders, as part of the sequence {a, b, c, ..., z, aa, ab, ...}

In many scenarios, builds will cycle between a, b, and c, and the unused build
(the one linked neither to live or stage) can be removed or archived after each build.

To find the next unused build folder, run `fab find_free_build`.

Then, to create the build, run, e.g. `fab create_build:z` where “z” should be
whatever build name was returned from `find_free_build`.

## Staging and deploying builds

To set a build as active (which will create a wsgi configuration that
spawns vassal in uWSGI emperor mode), run `fab activate:z`.

To then stage this build, run `fab stage:z` (again, the string after the “:”
being the build name), which will unlink the current stage build and link the
specified build to stage.

To “warm up” a build by hitting it with concurrent requests, use the command
`fab warmup:z` (the build name is optional and defaults to the stage build).

To swap the stage and live builds, run `fab swap_live`. By default, this will
swap live with the stage build, but it is possible to specify a different
build by, e.g. `fab swap_live:z`.

To spin down an old build (it cannot be the current live or stage builds), run
`fab deactivate:z`.

## `fab init` and `fab create_build`

(Note: This repository's directory will be mounted as `/code` within the docker container,
so when you see paths like `/code/deploy/...` below, keep in mind that this refers also
to `./deploy/...` within this repository.)

Running `fab init` executes a number of steps to get a stage and live build up and running:

- Builds all of the requirements into wheels, output to /code/deploy/deps/wheelhouse, which
  then allows for faster `pip install` when creating new addons-server builds
- Creates two builds, “a” and “b”, in directories `/code/deploy/builds/{a,b}`
  (`fab create_build:a` and `fab create_build:b`). The `create_build` command
  does the following:

  - Creates symlinks for the static and media directories of the build so
    that nginx static serving works correctly.

      *Note*: All static assets (static files and user-uploaded media) live
      outside of the `builds` directory, in `/code/deploy/assets`. This is
      to allow for the common use case where static files and user media use
      a distinct mount or partition from the rest of the code. In order to
      keep all static file urls distinct (beyond whatever staticfile cache busting
      is used), the static files for all builds are kept in separate directories,
      e.g. `/code/deploy/assets/static/a`.

  - Uses `rsync` to copy the source in the `addons-server` submodule to the
    build directory, and copies the contents of `/code/src` on top of that.
  - From the build directory, runs `git init` and commits a randomly generated
    file. Mozilla’s addons-server uses the git commit hash to cache-bust minified
    assets, so this ensures that this step won’t fail and that the hash will be
    unique for each build.
  - Runs `npm install` as well as `pip install` with the requirements files, using
    the `--no-index` flag and our wheeldir of `code/deploy/deps/wheelhouse` to
    keep this speedy.
  - Runs `manage.py collectstatic` and `manage.py compress_assets` (the latter being a
    jingo-minify management command specific to the mozilla addons-server project)
    from the build’s virtualenv.
  - Determines what port number corresponds to the current build name, and creates
    `/code/deploy/builds/a/live.conf` and `/code/deploy/builds/a/stage.conf`, which
    are nginx configuration files that specify the upstream. This will only come
    into play if the build gets designated as the stage or live build (since nginx
    includes `/code/deploy/builds/_live/live.conf` and `/code/deploy/builds/_stage/stage.conf`).
  - Creates symlinks to the uwsgi skeleton file `/code/conf/uwsgi/vassal.skel` at
    `/code/deploy/builds/{a,b}/vassal.ini`. This step causes the uWSGI emperor to spawn
    a vassal that will manage the zerg instances. At this point the application on the
    ``web`` container will be listening on ports `2000` and `2010` (for the a and b builds,
    respectively), but because there are no zerg instances attached to the vassal
    http requests will not yet route.

- After the builds “a” and “b” are created and linked, respectively, to
  `/code/deploy/builds/_live` and `/code/deploy/builds/_stage`,
  `fab init` initiates the database and populates it with sample test data,
  executed using the manage.py in the `_live` (read “a”) build’s virtualenv.
- It then symlinks to the uwsgi skeleton file `/code/conf/uwsgi/zerg.skel` from
  `/code/deploy/builds/{a,b}/zerg.ini`. This “activates” (`fab activate:a`) the
  build, spawning zerg instances that attach to the vassal.
- Lastly, `nginx -s reload` is run on the nginx container, updating the upstreams.

## Troubleshooting

* Problem: You encounter the error “Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.22)”
* Solution: `export DOCKER_API_VERSION=1.22`
<file_sep>/wsgi/uwsgi_status.py
#!/usr/bin/env python

import sys
import re
import errno
import glob
import socket
import json
import logging

import psutil

logging.basicConfig()
logger = logging.getLogger()

re_procname = re.compile(r"(?i)"
    r"uwsgi (?P<type>(?:worker|master))"
    r"(?: (?P<worker_num>\d+))? "
    r"<(?P<build>[^\.]+)>"
    r"(?: \[(?P<deploy_tag>[^\]]+)\])?")

class UwsgiProcess(object):

    def __init__(self, pinfo):
        self.pid = pinfo['pid']
        self.status = pinfo['status']
        self.name = pinfo['cmdline'][0]
        self.create_time = pinfo['create_time']
        m = re_procname.match(self.name)
        if not m:
            raise Exception("Bad uwsgi process name")
        matches = m.groupdict()
        self.is_master = matches['type'] == 'master'
        if self.is_master:
            self.workers = []
        else:
            self.worker_num = int(matches['worker_num'])
            self.ppid = pinfo['ppid']
            self.deploy_tag = matches['deploy_tag']
        self.build = matches['build']

def get_uwsgi_procs():
    procs = {}
    for proc in psutil.process_iter():
        try:
            pinfo = proc.as_dict(
                attrs=['pid', 'ppid', 'name', 'cmdline', 'create_time', 'status'])
        except psutil.NoSuchProcess:
            pass
        else:
            if not len(pinfo['cmdline']):
                continue
            if pinfo['cmdline'][0].startswith('u') and pinfo['ppid'] != 1:
                if re_procname.match(pinfo['cmdline'][0]):
                    procs[pinfo['pid']] = UwsgiProcess(pinfo)

    stat_sock_files = glob.glob('/var/run/uwsgi/*.stats')
    for stat_sock_file in stat_sock_files:
        js = ''
        try:
            s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
            s.connect(stat_sock_file)

            while True:
                d = s.recv(4096)
                if len(d) < 1:
                    break
                js += d.decode('utf-8')
        except IOError as e:
            if e.errno == errno.ECONNREFUSED:
                logger.error("Connection refused to socket %s" % stat_sock_file)
            elif e.errno != errno.EINTR:
                raise
            continue
        except:
            raise Exception("Unable to get uwsgi statistics for socket %s"
                % stat_sock_file)

        raw_stats = json.loads(js)
        if raw_stats['pid'] not in procs:
            continue

        workers = raw_stats.pop('workers')

        master_proc = procs[raw_stats['pid']]
        master_proc.status = 'active'
        master_proc.__dict__.update(raw_stats)

        for worker in workers:
            pid = worker['pid']
            if pid in procs:
                uwsgi_worker = procs.pop(pid)
                uwsgi_worker.__dict__.update(worker)
                master_proc.workers.append(uwsgi_worker.__dict__)
                master_proc.deploy_tag = uwsgi_worker.deploy_tag

    for pid in reversed(sorted(procs.keys())):
        proc = procs[pid]
        if not proc.is_master and proc.ppid in procs:
            procs[proc.ppid].workers.append(procs.pop(pid).__dict__)

    build_data = {}
    for pid in reversed(sorted(procs.keys())):
        proc = procs[pid]
        build = proc.build
        if build not in build_data:
            build_data[build] = []

        build_data[build].append(proc.__dict__)

    data = []
    for build in sorted(build_data.keys()):
        data.append({
            "name": build,
            "processes": build_data[build],
        })

    return {
        "configs": data
    }

def application(environ, start_response):
    status = '200 OK'
    output = json.dumps(get_uwsgi_procs())
    headers = [
        ('Content-Type', 'application/json'),
        ('Content-Length', str(len(output)))
    ]
    start_response(status, headers)
    return [output]

if __name__ == '__main__':
    sys.stdout.write(json.dumps(get_uwsgi_procs()))
<file_sep>/fabfile/build_port_convert.py
#!/usr/bin/env python
"""Convert a build name (e.g. 'a', 'b', etc.) to a port number, or vice-versa"""
from __future__ import print_function

import os
import sys

START_PORT = 2000
PORT_INCREMENT = 10
MAX_PORT = 65535
MAX_PORT_NUMS = (MAX_PORT - START_PORT) // PORT_INCREMENT

NUM_TO_CHAR = dict([((i), chr(i + ord('a'))) for i in range(0, 26)])
CHAR_TO_NUM = dict(map(reversed, NUM_TO_CHAR.items()))

def base26_decode(string):
    num = 0
    i = 0
    while i < len(string):
        char = string[i]
        offset = 0
        if i == 0:
            offset = 1
        power = len(string) - i - 1
        num += (26 ** power) * (CHAR_TO_NUM[char] + offset)
        if offset:
            while power > 1:
                power -= 1
                num += 26 ** power
        i += 1
    return num - 1

def base26_encode(num):
    string = ''
    num += 1
    while num:
        num, val = divmod((num - 1), 26)
        string = "%s%s" % (NUM_TO_CHAR[val], string)
    return string

def port_to_build(port):
    if port < START_PORT:
        usage("Port number must be greater than or equal to %d" % START_PORT)
    if port > MAX_PORT:
        usage("Port number must be less than %d" % MAX_PORT)
    num, index = divmod((port - START_PORT), PORT_INCREMENT)
    return (base26_encode(num), index)

def build_to_port(name):
    if name[-1].isdigit():
        index = int(name[-1])
        name = name[:-1]
    else:
        index = 0

    num = base26_decode(name)
    port = START_PORT + (num * PORT_INCREMENT) + index
    return port

USAGE = """%(error)s
Usage: %(script_name)s [port|build_name [index]]

build_name  A string composed of lowercase letters, as part of the sequence
            a, b, c, ..., z, aa, ab, ac, ..., zz, aaa, ..., %(max_build_name)s
index       Optional (defaults to 0), an integer between 0 and %(max_index)s
            which is added to the port number for the build_name
port        An integer, greater than %(start)s, which is converted into a
            build_name and index.
""".strip()

def usage(error=""):
    if error:
        error = "ERROR: %s\n" % error
    script_name = os.path.basename(sys.argv[0])
    print(USAGE % {
        "script_name": script_name,
        "max_build_name": base26_encode(MAX_PORT_NUMS),
        "max_index": PORT_INCREMENT,
        "start": START_PORT,
        "error": error,
    })
    sys.exit(1)

if __name__ == '__main__':
    if len(sys.argv) != 2:
        usage()
    arg = sys.argv[-1]
    if arg.isdigit():
        print("%s %d" % port_to_build(int(arg)))
    elif len(arg) > 3:
        usage()
    else:
        print(build_to_port(arg))
<file_sep>/src/hadjango/uwsgi/wsgi.py
import re
import os
import sys
import site
import uwsgi

project_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))

sys.path.insert(0, project_root)
site.addsitedir(os.path.join(
        project_root,
        "lib",
        "python" + sys.version[0:3],
        "site-packages"))

def get_site_name():
    settings_module = os.environ.get('DJANGO_SETTINGS_MODULE') or ''
    matches = re.match(r'settings\.(\w+)\.live', settings_module)
    if matches:
        site_name = matches.group(1)
    else:
        site_name = 'unknown'
    return site_name

site_name = get_site_name()

def set_uwsgi_proc_name():
    build_dir = os.path.basename(project_root)
    try:
        deploy_tag = open(os.path.join(project_root, '.DEPLOY_TAG')).read().strip()
    except IOError:
        deploy_tag = '?'

    os.environ['DEPLOY_TAG'] = deploy_tag

    uwsgi.setprocname("uwsgi worker %(worker_id)d <%(build)s> [%(tag)s]" % {
        'worker_id': uwsgi.worker_id(),
        'build': build_dir,
        'tag': deploy_tag,
    })

set_uwsgi_proc_name()

execfile(os.path.join(os.path.dirname(__file__), 'bootstrap.py'))

_application = None

def application(environ, start_response):
    global _application

    uwsgi.set_logvar('worker_id', str(uwsgi.worker_id()))

    if not os.environ.get('DJANGO_SETTINGS_MODULE'):
        os.environ['DJANGO_SETTINGS_MODULE'] = environ.get('DJANGO_SETTINGS_MODULE', 'settings')

    if _application is None:
        try:
            from django.core.wsgi import get_wsgi_application
        except ImportError:
            import django.core.handlers.wsgi
            _application = django.core.handlers.wsgi.WSGIHandler()
        else:
            _application = get_wsgi_application()

    return _application(environ, start_response)
<file_sep>/docker-compose.yml
version: '2'

services:
  web:
    # build: .
    image: hadjango/djangocon-2016-demo
    entrypoint: ./docker/start-docker.sh
    command: supervisord -n -c /code/docker/supervisor.conf
    privileged: true
    expose:
      - "1999"
      - "2000"
      - "2010"
      - "2020"
      - "2030"
      - "2040"
      - "2050"
      - "2060"
      - "2070"
      - "2080"
      - "2090"
      - "2100"
      - "2110"
      - "2120"
      - "2130"
      - "2140"
      - "2150"
      - "2160"
      - "2170"
      - "2180"
      - "2190"
      - "2200"
      - "2210"
      - "2220"
      - "2230"
      - "2240"
      - "2250"
      - "9999"
    volumes:
      - .:/code
    links:
      - memcached
      - mysqld
      - elasticsearch
      - redis
    environment:
      - PYTHONUNBUFFERED=1
      - RECURSION_LIMIT=10000
      - TERM=xterm-256color

  nginx:
    # build: ./nginx
    image: hadjango/addons-nginx
    entrypoint: /code/docker/start-nginx.sh
    command: nginx -g 'daemon off;'
    ports:
      - "80"
    ports:
      - "80:80"
    volumes:
      - .:/code
    links:
      - web:web

  memcached:
    image: memcached:1.4

  mysqld:
    image: mysql:5.6
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes
      - MYSQL_DATABASE=olympia

  elasticsearch:
    image: elasticsearch:1.6

  redis:
    image: redis:2.8
<file_sep>/docker/start-nginx.sh
#!/bin/bash
# If running within docker-machine, the current directory will be a mounted
# volume, owned by the user running Docker on the host machine.
#
# In that case, we don't want to trample all over the contents of this
# directory with files owned by root. So we create a new user with the same
# UID, and drop privileges before running any commands.
#
# If we are not running in docker-machine, we create a user with uid 1000
# and execute commands as that user.

# Get the numeric user ID of the current directory.
uid=$(ls -nd /code | awk '{ print $3 }')
gid=$(ls -nd /code | awk '{ print $4 }')

# If the current working directory is owned by root, that means we're running
# with plain-old docker, not docker-machine.
if [ "$uid" == "0" ]; then
    uid=1000
    gid=1000
fi

group_name=$(getent group $gid)

groupadd -g 1000 olympia 2>&1 2>/dev/null

# Create an `olympia` user with that ID, and the current directory
# as its home directory.
if [ -z "$group_name" ]; then
    useradd -Md $(pwd) -u $uid -g $gid -G olympia olympia 2>&1 2>/dev/null
else
    useradd -Md $(pwd) -u $uid -g $gid olympia 2>&1 2>/dev/null
fi

if [ ! -L /code/deploy/builds/_live ]; then
    mkdir -p /code/deploy/tmp/live
    mkdir -p /code/deploy/tmp/stage
    echo "upstream web { server web:1999; }" > /code/deploy/tmp/live/live.conf;
    echo "upstream webstage { server web:1999; }" > /code/deploy/tmp/stage/stage.conf;

    ln -sfvn /code/deploy/tmp/live /code/deploy/builds/_live
    ln -sfvn /code/deploy/tmp/stage /code/deploy/builds/_stage
fi

chown -R $uid:$gid /code/deploy/tmp
chown $uid:$gid /code/deploy/builds/*

exec su root -c 'exec "$@"' sh -- "$@"
<file_sep>/src/local_settings.py
import os

from olympia.lib.settings_base import INSTALLED_APPS

INSTALLED_APPS = ('hadjango', ) + INSTALLED_APPS + ('olympia.landfill', )

BROKER_URL = 'amqp://olympia:olympia@rabbitmq/olympia'
CELERY_RESULT_BACKEND = 'redis://redis:6379/1'
REDIS_LOCATION = 'redis://redis:6379/0?socket_timeout=0.5'
ES_HOSTS = ['elasticsearch:9200']
ES_URLS = ['http://%s' % h for h in ES_HOSTS]
SITE_DIR = 'http://olympia.dev'

CACHES = {
    'default': {
        'BACKEND': 'caching.backends.memcached.MemcachedCache',
        'LOCATION': 'memcached:11211',
    }
}

ROOT = os.path.realpath(os.path.dirname(__file__))

BUILD_NAME = os.path.basename(ROOT)
STATIC_ROOT = os.path.join(ROOT, "assets", "static")
MEDIA_ROOT = os.path.join(ROOT, "assets", "media")
STATIC_URL = "/static/%s/" % BUILD_NAME

CELERY_ALWAYS_EAGER = True

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'olympia',
        'USER': 'root',
        'PASSWORD': '',
        'HOST': 'mysqld',
        'PORT': '',
        'OPTIONS': {'sql_mode': 'STRICT_ALL_TABLES'},
        'TEST_CHARSET': 'utf8',
        'TEST_COLLATION': 'utf8_general_ci',
        # Run all views in a transaction unless they are decorated not to.
        'ATOMIC_REQUESTS': True,
        # Pool our database connections up for 300 seconds
        'CONN_MAX_AGE': 300,
    },
}

# A database to be used by the services scripts, which does not use Django.
# The settings can be copied from DATABASES, but since its not a full Django
# database connection, only some values are supported.
SERVICES_DATABASE = {
    'NAME': DATABASES['default']['NAME'],
    'USER': DATABASES['default']['USER'],
    'PASSWORD': DATABASES['default']['PASSWORD'],
    'HOST': DATABASES['default']['HOST'],
    'PORT': DATABASES['default']['PORT'],
}
<file_sep>/.gitconfig
[user]
	email = <EMAIL>
	name = Nobody
<file_sep>/docker/start-docker.sh
#!/bin/sh
# If running within docker-machine, the current directory will be a mounted
# volume, owned by the user running Docker on the host machine.
#
# In that case, we don't want to trample all over the contents of this
# directory with files owned by root. So we create a new user with the same
# UID, and drop privileges before running any commands.
#
# If we are not running in docker-machine, we create a user with uid 1000
# and execute commands as that user.

# Get the numeric user ID of the current directory.
uid=$(ls -nd . | awk '{ print $3 }')
gid=$(ls -nd . | awk '{ print $4 }')

# If the current working directory is owned by root, that means we're running
# with plain-old docker, not docker-machine.
if [ "$uid" == "0" ]; then
    uid=1000
    gid=1000
fi

group_name=$(getent group $gid)

groupadd -g 1000 olympia

# Create an `olympia` user with that ID, and the current directory
# as its home directory.
if [ -z "$group_name" ]; then
    useradd -Md $(pwd) -u $uid -g $gid -G olympia olympia
else
    useradd -Md $(pwd) -u $uid -g $gid olympia
fi

if [ ! -d /deps ]; then
    mkdir /deps
    chown $uid:$gid /deps
fi

mkdir -p /var/run/supervisor
chown $uid:$gid /var/run/supervisor
chown $uid:$gid /var/run/uwsgi

# Switch to that user and execute our actual command.
exec su root -c 'exec "$@"' sh -- "$@"

languages:
Shell
INI
Markdown
Git Config
YAML
Python

repo_language: Shell


text:
<file_sep>'use strict';

const UsuarioDelAPI = require('../../models/UsuarioDelAPI');
const i18n = require('../../lib/i18nSetup')();
const jwt = require('jsonwebtoken');

class LoginController {
	// GET /api/authenticate
	index(req, res, next) {
		res.locals.user = ''; // para que la vista tenga el nombre de usuario
		res.locals.error = '';
		res.render('APILogin');
	}

	// POST /api/authenticate
	async post(req, res, next) {
		// async jwtCreation(req, res, next) {
		const userName = req.body.user; // Esto se hace en Ajax

		console.log('SERVIDOR en loginController.js ---> El usuario es:', userName);

		// const user = await UsuarioDelAPI.findOne({ user: userName });

		// Creamos el JWT token
		//jwt.sign({ _id: user._id }, process.env.JWT_SECRET, {
		jwt.sign(
			{ _id: userName._id }, // user._id },
			'miclave',    // Clave del servidor
			{
				expiresIn: '2d'
			},
			(err, token_id) => {
				if (err) {
					return next(err);
				}

				//window.localStorage.setItem('myJWT', token_id);
				console.log('token_id PASADO al navegador:', token_id);
				// respondemos con un JWT
				//req.session.jwtToken = token_id;
				res.setHeader('x-access-token', token_id);
				res.json({ ok: true, token_id: token_id });
				// res.send(token_id);

			}
		);
	}

	logout(req, res, next) {
		delete req.session.authUser;
		req.session.regenerate(function(err) {
			if (err) {
				return next(err);
			}
			res.redirect('/');
		});
	}
} // End of the class

module.exports = new LoginController();
<file_sep>'use strict';

/* let register = function(Handlebars) {
    let helpers = {
        i18nHelper: function() { return 'AABB';}
    }
};
for (let prop in register.helpers) {
    Handlebars.registerHelper(prop, helpers[prop]);
}

module.exports.register = register;
module.exports.helpers = register(null);
*/

exports.yell = function(msg) {
    return msg.toUpperCase();
}<file_sep>'use strict';

const mongoose = require('mongoose');
// var hash = require('hash.js')

const usuarioSchema = mongoose.Schema({
  user: String,
});

/*usuarioSchema.statics.hashPassword = function(plain) {
  return hash.sha256().update(plain).digest('hex');
}   */

var UsuarioDelAPI = mongoose.model('UsuarioDelAPI', usuarioSchema);

module.exports = UsuarioDelAPI;
<file_sep>
  <xmp theme="united" style="display:none;">
    {{readme}}
  </xmp>

<file_sep>const i18n = require('./i18nSetup')();

module.exports = {
    //Setup our default layout
    defaultLayout: 'index',

    //More handlebars configuration
    // ..

    //Register handlebars helpers
  helpers: {
    //Register your helpers
    // ..
    //Helper for multiple languages
    i18n: function(str){
        //return i18n.__.apply(this, arguments);
        return i18n.__(str);   // Both lines do the same thing
    },
    getLocales: function() {
        return getLocales();
    }
  }
}

// module.exports = hbsconf;
<file_sep>'use strict';

const express = require('express');
const path = require('path');
const logger = require('morgan');
const cookieParser = require('cookie-parser');
const bodyParser = require('body-parser');    // We can get info from POST and/or URL parameters
const session = require('express-session');
const sessionAuth = require('./lib/sessionAuth');
const jwtAuth = require('./lib/jwtAuth');
const mongoose = require('mongoose');
const MongoStore = require('connect-mongo')(session);

// Conexión a la base de datos
const db = require('./lib/connectMongoose'); // ??

require('dotenv').config(); // inicializamos variables de entrono desde el fichero .env

// Cargamos las definiciones de todos nuestros modelos
require('./models/Anuncio');
require('./models/Usuario');
require('./models/UsuarioDelAPI');

// view engine setup
// Source: https://webapplog.com/jade-handlebars-express/
let helpers = require('./public/hs/helpers');

const app = express();
//let dirName = __dirname;  //.includes('api') ? path.dirname(module.parent.__dirname) : __dirname;

// Load handlebars engine and load our configuration
// from the file hbsconf.js under a libs folder at the same level of the application
const handlebars = require('express-handlebars').create(require('./lib/hbsconf.js'));

//Set our application to use our configured handlebars module as the view engine
app.engine('handlebars', handlebars.engine);
app.set('view engine', 'handlebars');

// app.use(logger('dev'));
if (process.env.LOG_FORMAT !== 'nolog') {
	app.use(logger(process.env.LOG_FORMAT || 'dev'));
}
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser());

app.use(express.static(path.join(__dirname, 'public')));

const i18n = require('./lib/i18nSetup')();
app.use(i18n.init);

// Global Template variables
app.locals.title = 'NodePop';

// Login
const loginController = require('./routes/loginController');
const APILoginController = require('./routes/apiv1/loginController');

app.get('/api/authenticate', APILoginController.index);
app.post('/api/authenticate', APILoginController.post);
app.get('/api/logout', APILoginController.logout);

//app.post('/api/authenticate', loginController.APIPost);   //jwtCreation);
//app.use('/api/anuncios', jwtAuth(), require('./routes/apiv1/anuncios'));

// catch API 404 and forward to error handler
// app.use('/api', function (req, res, next) {
app.use('/api', function(req, res, next) {
	const err = new Error(__('not_found'));
	err.status = 404;
	next(err);
});

// middleware de control de sesiones
app.use(
	session({
		name: 'nodepop',
		secret: '<KEY>',
		resave: false,
		saveUninitialized: false,
		cookie: { maxAge: 2 * 24 * 3600 * 1000 }, // dos dias de inactividad
		store: new MongoStore({
			// url: cadena de conexión
			mongooseConnection: mongoose.connection,
			autoReconnect: true,
			clear_interval: 3600
		})
	})
);

// usamos las rutas de un controlador
app.get('/login', loginController.index);   // Si comenta esta línea, el navegador da error: "TOO_MANY_REDIRECTS" !!
app.post('/login', loginController.post);
app.get('/logout', loginController.logout);

app.use('/', sessionAuth(), require('./routes/index')); // Ponía sessionAuth, sin paréntesis, y no cargaba ninguna página !!
app.use('/anuncios', require('./routes/index'));
// app.use('/hola', require('./routes/hola').router);

// Web
// app.use('/', require('./routes/anuncios'));
// app.use('/anuncios', require('./routes/anuncios'));

// catch 404 and forward to error handler
app.use(function(req, res, next) {
	const err = new Error(__('not_found'));
	err.status = 404;
	next(err);
});

// error handler
app.use(function(err, req, res, next) {
	if (err.array) {
		// validation error
		err.status = 422;
		const errInfo = err.array({ onlyFirstError: true })[0];
		err.message = isAPI(req)
			? { message: __('not_valid'), errors: err.mapped() }
			: `${__('not_valid')} - ${errInfo.param} ${errInfo.msg}`;
	}

	// establezco el status a la respuesta
	err.status = err.status || 500;
	res.status(err.status);

	// si es un 500 lo pinto en el log
	if (err.status && err.status >= 500) console.error(err);

	// si es una petición al API respondo JSON...
	if (isAPI(req)) {
		res.json({ success: false, error: 'Petición al API. ' + err.message });
		return;
	}

	// ...y si no respondo con HTML...

	// set locals, only providing error in development
	res.locals.message = err.message;
	res.locals.error = req.app.get('env') === 'development' ? err : {};

	// render the error page
	res.render('error');
});

function isAPI(req) {
	return req.originalUrl.indexOf('/api') === 0;
}

module.exports = app;
<file_sep>
{{> header}}

{{i18n "Ads List"}}

{{i18n "Listing"}}: {{total}}  {{i18n "Of"}}  {{total}}

          {{i18n "Name"}}
          {{i18n "For Sale"}}
          {{i18n "Price"}}
          {{i18n "Photo"}}
          {{i18n "Tags"}}
          #

        {{#each anuncios}}

            {{i18n this.nombre}}
            {{this.venta}}
            {{this.precio}} / 100 €

            {{#each this.tags}}
              {{i18n this}}
            {{/each}}

            {{this.anuncio._id}}

        {{/each}}

{{i18n "Examples"}}:

{{i18n "All the ads"}}

{{i18n "paginated"}}

{{i18n "Sorted by increasing price"}}

{{i18n "Sorted by decreasing price"}}

{{i18n "With the mobile tag"}}

  {{i18n "Go back home"}}

<file_sep>

Nodeapi

Lista de anuncios

Listando: anuncios.length  de  total /p>

          Nombre
          Venta
          Precio
          Foto
          Tags
          #

{{#each anuncios}}

            {{this.nombre}}
            anuncio.venta ? 'Si' : 'No'
            <{{this.precio}} / 100 €

            {{this.tags}}
            {{this.anuncio._id}}

{{/each}}

Ejemplos:

Todos (max. 1000)

Paginado

Ordenado por precio ascendente

Ordenado por precio descendente

Con tag mobile

  Volver a la home

<file_sep>{{> header}}

{{i18n 'Please login to access application'}}

      <form method="POST">

          {{i18n 'Email address'}}
          <input type="email" class="form-control" id="email" aria-describedby="emailHelp" placeholder="Enter email"
                name="email"
                value={{email}} >

          {{i18n 'Password'}}
          <input type="<PASSWORD>" class="form-control" id="password" placeholder="{{i18n '<PASSWORD>'}}"
                name="password">

        <button type="submit" id="login" class="btn btn-primary">{{i18n 'Submit'}}</button>

{{error}}

      </form>

 <file_sep>

{{title}}

      {{i18n 'Logout'}}

      {{#each getLocales}}
      {{this}}
      {{/each}}

<file_sep>'use strict';

const connectionPromise = require('./connectAMQP');

const q = 'q3';

// Publicador
// IIFE Inmediatelly Invoked Function Expression
(async () => {
	// nos aseguramos de que está conectado
	const conn = await connectionPromise;

	// conectarnos a un canal
	const ch2 = await conn.createChannel();

	// conecto a una cola
	await ch2.assertQueue(q, {
        // auto_delete: true,
		durable: false // NO sobrevive a reinicios
	});

	// maximo teórico del tamaño de un mensaje:
	// 9,223,372,036,854,775,807 B

	//setInterval(() => {
	//const mensaje = {
	// const directorio = 'home/jose/1-KeepCoding/Proyecto-articulos/src/img/';
	const rutaFoto = '/home/jose/1-KeepCoding/Proyecto-articulos/src/img/libro1.jpg';
	//};
	// mandar mensaje
	// const res = ch.sendToQueue(q, new Buffer(JSON.stringify(mensaje)), {
	const res = ch2.sendToQueue(q, new Buffer(rutaFoto), {
		// persistent: true // para sobrevivir a reinicios
	});
	console.log(`enviado: ${rutaFoto} ${res}`);
})().catch((err) => console.error(err));
<file_sep>'use strict';

function crearJWTToken() {
	$.ajax({
		method: 'post',
		url: '/api/authenticate',
		data: {
			// Valores que se pasan al cuerpo de la petición del servidor (res.body)
			user: $('#user').val()
		},
		success: function(data) {
			// "data" designa al objeto JSON enviado por el servidor
			console.log('Hola, soy el cliente');
			console.log('Token recibido:', data.token_id);
			localStorage.setItem('token2', data.token_id);
		},
		error: function(err) {
			console.log('Error:', err);
		}
	});
}

function getRequestToTheAPI() {
	let token = localStorage.getItem('token');
	$.ajax({
		method: 'get',
		url: '/api/anuncios',
		data: {
			// Valores que se pasan al cuerpo de la petición del servidor (req.body)
		},
		headers: { 'x-access-token': token },
		// Valores que se pasan a la cabecera de la petición del servidor (req.headers)
		success: function(data) {
			// "data" designa al objeto JSON mandado por el servidor
			console.log('DATOS recibidos:', data._id);
		},
		error: function(err) {
			console.log('Error:', err);
		}
	});
}

$(document).ready(function() {
	// Esta línea es imprescindible !!!
	// document.getElementById('prueba').addEventListener('click', function() {
	$('#API-login').on('click', function() {
		crearJWTToken();
	});

	let token = localStorage.getItem('token');
	// JR Duda: qué evento está asociado a esta función ajax
	/*$.ajax({
		method: 'get',
		url: '/api/anuncios',
		data: {
			// Valores que se pasan al cuerpo de la petición del servidor (req.body)
		},
		headers: { 'x-access-token': localStorage.getItem('token') },  // token },
		// Valores que se pasan a la cabecera de la petición del servidor (req.headers)
		success: function(data) {
			// "data" designa al objeto JSON mandado por el servidor
			console.log('DATOS recibidos:', data._id);
		},
		error: function(err) {
			console.log('Error:', err);
		}
	}); */

	/*$(document).on('readystatechange', function() {
		console.log('Dirección cambiada !!');
	});*/
});
<file_sep>#for f in *.ejs; do echo "$f"; done

#for f in *.ejs; do new=$(echo $f | sed -e 's/\.ejs$/\.hbrs/'); echo ${new}; done

convert() {
## Command to rename all the template files to "handlebars"
for f in $(ls $PWD | grep $1); do
    new=$(echo $f | sed -e "s/$1/$2/");
##    new=$(echo $f | sed -e 's/$1/EXT/');
 ## Esto no funciona, pq con comillas simples no hay interpolación de variables !!
    mv -v $f $new;
  done;
}

convert '..handlebars' '.handlebars'
<file_sep>'use strict';

const router = require('express').Router();
const fs = require('fs');
const Anuncio = require('mongoose').model('Anuncio');
const i18n = require('../lib/i18nSetup')();
//i18n('en');
/* GET anuncios page. */
router.get('/', async function(req, res, next) {
	try {
		const start = parseInt(req.query.start) || 0;
		const limit = parseInt(req.query.limit) || 1000; // nuestro api devuelve max 1000 registros
		const sort = req.query.sort || '_id';
		const includeTotal = true;

		const filters = {};
		if (req.query.tag) {
		}
		if (req.query.venta) {
			filters.venta = req.query.venta;
		}

		const { total, rows } = await Anuncio.list(filters, start, limit, sort, includeTotal);
		res.render('anuncios', { total, anuncios: rows }); // Original
	} catch (err) {
		return res.next(err);
	}
});

router.get('/lang/:locale', (req, res, next) => {
	//console.log('LA VISTA TIENE ESTO:', res.locals);
	const locale = req.params.locale;
	console.log('LCambio a ', locale);
	const referer = req.query.redir || req.get('referer');
	res.cookie('nodepop-lang', locale, { maxAge: 900000, httpOnly: true });
	res.redirect(referer);
});

module.exports = router;
<file_sep>'use strict';

const jwt = require('jsonwebtoken');

module.exports = function(token) {
	// devuelve un middleware que si no hay usuario responde con error
	return function(req, res, next) {
		// let existingToken = '<KEY>';
		// req.header('x-access-token');
		let existingToken = req.session.jwtToken;
		// res.setHeader('Acces-Control-Allow-Headers', 'x-access-token, mytoken');
		let token = existingToken || req.body.token || req.query.token || req.get('x-access-token');
		res.setHeader('x-access-token', token);
		console.log('Token q usaremos:', token);

		if (!token) {
			const err = new Error('no token provided');
			next(err);
			return;
		}

		// tengo token
		// jwt.verify(token, process.env.JWT_SECRET, (err, decoded) => {
		jwt.verify(token, 'mi<PASSWORD>', (err, decoded) => {
			if (err) {
				return next(err);
			}
			// guardo el id del usuario en request para que
			// los siguientes middlewares puedan usarlo
			// res.json(decoded);

			console.log('Token correcto-1111111', req.session.jwtToken);
			req.userId = decoded._id;
			console.log('Token correcto', decoded._id);
			// el usuario está autenticado
			next();
			// if everything is good, save to request for use in other routes
			// req.decoded = decoded;
		});
	};
};
<file_sep>'use strict';

const connectionPromise = require('./connectAMQP');
const Jimp = require('jimp');

const q = 'q3';

// Consumidor
// IIFE Inmediatelly Invoked Function Expression
(async () => {
	// nos aseguramos de que está conectado
	const conn = await connectionPromise;

	// conectarnos a un canal
	const ch2 = await conn.createChannel();

	// conecto a una cola
	await ch2.assertQueue(q, { durable: false });

	// le decimos a rabbitMQ cuántos mensaje puede darme en paralelo
	ch2.prefetch(1);

	await ch2.consume(q, function(msg) {
		let photo = msg.content.toString();
		console.log('Consumidor:', photo);
		// procesamos el mensaje
		Jimp.read(photo)
			.then(function(photo) {
				photo
					.resize(100, 100) // resize
					.quality(60) // set JPEG quality
					.write('thumbnail.jpg'); // save
				// confirmamos a rabbit que está procesado
				ch2.ack(msg); // ch2.nack(msh)
			})
			.catch(function(err) {
				console.error(err);
			});
	});
})();
<file_sep>{{> header}}

{{i18n 'Please login to access the API'}}

      <form method="POST">

          {{i18n 'User'}}
          <input type="text" class="form-control" id="user" aria-describedby="emailHelp" placeholder={{i18n 'Enter user'}}
                name="user"
                value={{user}} >

        <button type="submit" id="API-login" class="btn btn-primary">{{i18n 'Submit'}}</button>

{{error}}

      </form>

 <file_sep>'use strict';

//const mongoose = require('mongoose');
const readLine = require('readline');
//const async = require('async');
//require('./lib/i18nSetup');

const conn = require('./lib/connectMongoose');
const Usuario = require('./models/Usuario');   // Cargamos las definiciones de todos nuestros modelos

conn.once('open', async function() {
  // uso try/catch para cazar los errores de async/await
  try {

    await initUsuarios();
    // otros inits ...
    conn.close();

  } catch(err) {
    console.log('Hubo un error:', err);
    process.exit(1);
  }
});

async function initUsuarios() {
  const deleted = await Usuario.deleteMany();
  console.log(`Eliminados ${deleted.result.n} usuarios.`);

  const inserted = await Usuario.insertMany([
    { name: 'admin',
      email: '<EMAIL>',
      password: <PASSWORD>('<PASSWORD>') }
  ]);
  console.log(`Insertados ${inserted.length} usuarios.`);
}

/*conn.once('open', async function () {

  const rl = readLine.createInterface({
    input: process.stdin,
    output: process.stdout
  });

  rl.question('Are you sure you want to empty conn? (no) ', function (answer) {
    rl.close();
    if (answer.toLowerCase() === 'yes') {
      runInstallScript();
    } else {
      console.log('conn install aborted!');
      return process.exit(0);
    }
  });

});

function runInstallScript() {

  async.series([
      initAnuncios
    ], (err) => {
      if (err) {
        console.error( __('generic', { err }) );
        return process.exit(1);
      }

      return process.exit(0);
    }
  );

}

function initAnuncios(cb) {
  const Anuncio = mongoose.model('Anuncio');

  Anuncio.remove({}, ()=> {

    console.log('Anuncios borrados.');

    // Cargar anuncios.json
    const fichero = './anuncios.json';
    console.log('Cargando ' + fichero + '...');

    Anuncio.cargaJson(fichero, (err, numLoaded)=> {
      if (err) return cb(err);

      console.log(`Se han cargado ${numLoaded} anuncios.`);
      return cb(null, numLoaded);
    });

  }); */

languages: Handlebars Shell JavaScript

repo_language: Handlebars


text:
<file_sep>import random

def quick_sort(lists):
    if len(lists) <= 1:
        return lists
    else:
        left = [i for i in lists[1:] if i< lists[0]]
        right = [i for i in lists[1:] if i>= lists[0]]
        return quick_sort(left)+lists[0:1]+quick_sort(right)

def insert_sort(lists):
    for i in range(1,len(lists)):
        key = lists[i]
        j = i-1
        while j>=0:
            if key < lists[j]:
                lists[j+1] = lists[j]
                lists[j] = key
            else:
                break
            j = j -1
    return lists

def merge(left, right):
    i, j = 0, 0
    result = []
    while i < len(left) and j < len(right):
        if left[i] <= right[j]:
            result.append(left[i])
            i += 1
        else:
            result.append(right[j])
            j += 1
    result += left[i:]
    result += right[j:]
    return result

def merge_sort(lists):
    # 归并排序
    if len(lists) <= 1:
        return lists
    num = len(lists) // 2
    left = merge_sort(lists[:num])
    right = merge_sort(lists[num:])
    return merge(left, right)

if __name__ == '__main__':

    print('hello')
    n = 10
    arr = [0]*n
    for i in range(10):
        # arr = [5,6,9,2,1,8,3]
        arr[i] = random.randint(1,100)
    print(arr)

    # print(quick_sort(arr))
    # print(insert_sort(arr))
    print(merge_sort(arr))

repo_language: Python

repo_name: szdmgy/sort


text:
<repo_name>mtensor/prototypical-networks<file_sep>/protonets/data/__init__.py
from . import omniglot
<file_sep>/protonets/models/__init__.py
from prototypical_networks.protonets.models.factory import get_model, register_model

import prototypical_networks.protonets.models.few_shot
<file_sep>/setup.py
from setuptools import setup

setup(name='protonets',
      version='0.0.1',
      author='<NAME>',
      author_email='<EMAIL>',
      license='MIT',
      packages=['protonets'],
      install_requires=[
          'torch',
          'tqdm'
      ])
<file_sep>/README.md
# Prototypical Networks for Few-shot Learning

Code for the NIPS 2017 paper [Prototypical Networks for Few-shot Learning](http://papers.nips.cc/paper/6996-prototypical-networks-for-few-shot-learning.pdf).

If you use this code, please cite our paper:

```
@inproceedings{snell2017prototypical,
  title={Prototypical Networks for Few-shot Learning},
  author={<NAME>},
  booktitle={Advances in Neural Information Processing Systems},
  year={2017}
 }
 ```

## Training a prototypical network

### Install dependencies

* This code has been tested on Ubuntu 16.04 with Python 3.6 and PyTorch 0.4.
* Install [PyTorch and torchvision](http://pytorch.org/).
* Install [torchnet](https://github.com/pytorch/tnt) by running `pip install git+https://github.com/pytorch/tnt.git@master`.
* Install the protonets package by running `python setup.py install` or `python setup.py develop`.

### Set up the Omniglot dataset

* Run `sh download_omniglot.sh`.

### Train the model

* Run `python scripts/train/few_shot/run_train.py`. This will run training and place the results into `results`.
  * You can specify a different output directory by passing in the option `--log.exp_dir EXP_DIR`, where `EXP_DIR` is your desired output directory.
  * If you are running on a GPU you can pass in the option `--data.cuda`.
* Re-run in trainval mode `python scripts/train/few_shot/run_trainval.py`. This will save your model into `results/trainval` by default.

### Evaluate

* Run evaluation as: `python scripts/predict/few_shot/run_eval.py --model.model_path results/trainval/best_model.pt`.

languages: Markdown Python

repo_language: Markdown

repo_name: mtensor/prototypical-networks