问:
How to handle Ajax and normal pagination in Scrapy?
I am scraping this website https://www.saksfifthavenue.com/on/demandware.store/Sites-SaksFifthAvenue-Site/en_US/Search-UpdateGrid?cgid=2534374306418048&start=0&sz=24&hideLess=true it is api url I guess which uses ajax pagination and normal pagination as well. Website using ajax pagination to load the items partially once all the items loaded then page shows the conventional numbered list of page numbers. I have write the scraper and handling the pagination by simply checking if show more div is present in the page or not. Here is the html source for that div:

```<code><div class="show-more invisible">
  <div class="text-center">
    <button class="btn btn-outline-primary col-12 col-sm-4" data-url="https://www.saksfifthavenue.com/on/demandware.store/Sites-SaksFifthAvenue-Site/en_US/Search-UpdateGrid?cgid=2534374306418048&start=21&sz=24&hideLess=true">
More Results
</button>
</div>
</div>
</code>```

If div is presented in the page I am fetching the data-url and parse items from that url. Once the show more div is no more spider will look for next page div which is look like this:

```<code><p class="page-item d-flex  next">
<a class="d-inline-block" aria-label="Next" href="https://www.saksfifthavenue.com/c/women-s-apparel?start=37724&sz=24"></a>
</p>
</code>```

This is little bit tricky to work with it since I am using api url and next page url is the normal website url so I am only extracting start value from it and update the params. Here is the spider:

```<code>import scrapy
from urllib.parse import urlencode
import re
import json
import html

class SaksFifthAvenueSpider(scrapy.Spider):
    name = "safa-feeds"

    # custom settings
    custom_settings = {
        "LOG_FILE": "safa-feeds-spider.log",
        "ITEM_PIPELINES": {
            "sdt.pipelines.SdtWasabiS3Pipeline": 300,
        },
    }

    headers = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
        "Connection": "keep-alive",
        "Upgrade-Insecure-Requests": "1",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "cross-site",
        "Sec-GPC": "1",
    }

    params = {
        "cgid": "",
        "start": "0",
        "sz": "24",
        "hideLess": "true",
    }

    base_url = "https://www.saksfifthavenue.com/on/demandware.store/Sites-SaksFifthAvenue-Site/en_US/Search-UpdateGrid?"

    def start_requests(self):
        cgid_list = [
            "2534374306418048",
            # "2534374306624247",
            # "2534374306622828",
            # "1608313652004",
            # "2534374306418050",
            # "2534374306418162",
            # "2534374306418054",
            # "1654108780232",
            # "2534374306418205",
            # "2534374306418206",
            # "2534374306418217",
            # "2534374306418192",
            # "1608313652004",
            # "2534374306418053",
        ]
        for i, cgid in enumerate(cgid_list):
            self.params["cgid"] = cgid
            category_url = self.base_url + urlencode(self.params)
            yield scrapy.Request(
                url=category_url, headers=self.headers, callback=self.parse_page_items
            )

    def parse_page_items(self, response):
        item_links = set(
            [
                "https://www.saksfifthavenue.com" + u.split("?")[0]
                for u in response.css("a.thumb-link.mw-100::attr(href)").extract()
            ]
        )
        inner_load = response.css("div.show-more ::attr(data-url)").get()
        if inner_load:
            print(inner_load)
            yield scrapy.Request(
                url=inner_load,
                headers=self.headers,
                callback=self.parse_page_items,
            )
        next_page_no = response.css('a[aria-label="Next"]::attr(href)').get()
        if next_page_no:
            self.params["start"] = next_page_no.split("&")[0].split("=")[-1]
            next_url = self.base_url + urlencode(self.params)
            print(next_url)
            yield scrapy.Request(
                url=next_url, headers=self.headers, callback=self.parse_page_items
            )

        for link in item_links:
            yield scrapy.Request(
                url=link, headers=self.headers, callback=self.parse_product_details
            )

    def parse_product_details(self, response):
        item = {}
        json_text = response.css('script[type="application/ld+json"]').get()
        if json_text is not None:
            clean_text = json_text.replace(
                '<script type="application/ld+json">', ""
            ).replace("</script>", "")
            json_blob = json.loads(clean_text)
            prod_id = response.css("div.container.product-detail::attr(data-pid)").get()
            colors = response.css(
                "button::attr(data-adobelaunchproductcolor)"
            ).extract()
            sizes = response.css("li::attr(data-attr-value)").extract()
            item["product_id"] = prod_id
            item["product_brand"] = response.css("a.product-brand::text").get()
            item["product_name"] = response.css("h1.product-name::text").get()
            json_breadcrumbs_text = (
                response.css('script[type="application/ld+json"]')
                .extract()[-1]
                .replace('<script type="application/ld+json">', "")
                .replace("</script>", "")
            )
            bc_json_blob = json.loads(json_breadcrumbs_text)
            if "itemListElement" in bc_json_blob:
                item["categories"] = [
                    {f"category_{idx}": cat["name"]}
                    for idx, cat in enumerate(bc_json_blob["itemListElement"], 1)
                ]
            else:
                item["categories"] = []
            item["slug"] = json_blob["offers"]["url"].split(".com")[-1]
            desc = json_blob["description"]
            item["description"] = re.sub("<[^<]+?>", " ", html.unescape(desc))
            item["product_variants"] = []
            item["color"] = response.css(
                "span.text2.color-value.attribute-displayValue::text"
            ).get()
            item["sizes"] = []
            item["gender"] = ""
            if "itemListElement" in bc_json_blob:
                bc_li = [b["name"] for b in bc_json_blob["itemListElement"]]

                if "Women's Clothing" or "Beauty" in bc_li:
                    item["gender"] = "Female"
                elif any("Men" in s for s in bc_li) or any("Men's" in s for s in bc_li):
                    item["gender"] = "Male"
                else:
                    item["gender"] = "Female"
                if (
                    "Kids" in bc_li
                    and any("Boys" in s for s in bc_li)
                    or any("Baby Boy" in s for s in bc_li)
                ):
                    item["gender"] = "Boy"
                elif (
                    "Kids" in bc_li
                    and any("Girls" in s for s in bc_li)
                    or any("Baby Girl" in s for s in bc_li)
                ):
                    item["gender"] = "Girl"

                elif (
                    any("Kids" in s for s in bc_li)
                    and not any("Baby Girl" in s for s in bc_li)
                    and not any("Baby Boy" in s for s in bc_li)
                    and not any("Boys" in s for s in bc_li)
                    and not any("Girls" in s for s in bc_li)
                ):
                    item["gender"] = "Kids"

                elif any("Accessories" in s for s in bc_li):
                    item["gender"] = ""

            price_json_text = (
                response.css('script[type="text/javascript"]')
                .extract()[2]
                .replace('<script type="text/javascript">\npageDataObj = ', "")
                .replace(";\n</script>", "")
            )
            price_json_blob = json.loads(price_json_text)
            item["tag"] = price_json_blob["products"][0]["tags"]["feature_type"]
            item["price"] = [
                {
                    "original_price": p["original_price"],
                    "price": p["price"],
                    "currency": json_blob["offers"]["priceCurrency"],
                }
                for p in price_json_blob["products"]
            ]
            variant_urls = []
            for color in colors:
                for i_size in sizes:
                    variant_url = (
                        response.url
                        + "?dwvar_"
                        + prod_id
                        + "_color="
                        + color.upper()
                        + "&dwvar_"
                        + prod_id
                        + f"_size={i_size}&pid="
                        + prod_id
                    )
                    variant_urls.append(variant_url)
            item["images"] = json_blob["image"]
            size_selector = response.css(
                "ul.radio-group-list.size-attribute.swatch-display-three.show-size-dropdown"
            )
            if size_selector:
                for s in size_selector.css("li"):
                    all_size_var = s.css("::text").getall()
                    if not s.css("[disabled]"):
                        available = all_size_var
                        clean = list(
                            filter(None, [c.replace("\n", "") for c in available])
                        )
                        for out_si in clean:
                            item["sizes"].append(
                                {"size": out_si, "status": "AVAILABLE"}
                            )
                    else:
                        out_of_stock = all_size_var
                        clean = list(
                            filter(None, [c.replace("\n", "") for c in out_of_stock])
                        )
                        for in_si in clean:
                            item["sizes"].append(
                                {"size": in_si, "status": "NOT_AVAILABLE"}
                            )
            clean_sizes_urls = []
            if response.css("div.form-group.show-size-dropdown-holder"):
                size_dropdown = response.css(
                    "ul.radio-group-list.size-attribute.swatch-display-three ::text"
                ).extract()
                clean_sizes = list(
                    filter(None, [s.replace("\n", "") for s in size_dropdown])
                )
                for dd_si in clean_sizes:
                    if item["color"]:
                        variant_url = (
                            response.url
                            + "?dwvar_"
                            + prod_id
                            + "_color="
                            + item["color"].upper()
                            + "&dwvar_"
                            + prod_id
                            + f"_size={dd_si}&pid="
                            + prod_id
                        )
                        clean_sizes_urls.append(variant_url)
            if variant_urls:
                yield scrapy.Request(
                    variant_urls[0],
                    headers=self.headers,
                    cb_kwargs={
                        "item": item,
                        "variant_urls": variant_urls[1:],
                        "clean_sizes_urls": clean_sizes_urls,
                    },
                    callback=self.parse_product_variants,
                )
            elif clean_sizes_urls:
                yield scrapy.Request(
                    clean_sizes_urls[0],
                    headers=self.headers,
                    cb_kwargs={"item": item, "clean_sizes_urls": clean_sizes_urls[1:]},
                    callback=self.parse_clean_sizes,
                )

            yield item

    def parse_product_variants(self, response, item, variant_urls, clean_sizes_urls):
        size = "".join(
            list(
                filter(
                    None,
                    [
                        s.replace("\n", "")
                        for s in response.css("li").css("[selected] ::text").extract()
                    ],
                )
            )
        )
        disabled = (
            response.css("li").css("[disabled]").css("[selected] ::text").getall()
        )
        final_price = ""
        final_price = response.css(
            "span.formatted_sale_price.formatted_price.js-final-sale-price.bfx-price.bfx-list-price::text"
        ).get()
        if final_price is None:
            final_price = response.css(
                "span.formatted_sale_price.formatted_price.js-final-sale-price.bfx-price.bfx-sale-price::text"
            ).get()
        try:
            old_price = response.css(
                "span.formatted_price.bfx-price.bfx-list-price ::text"
            ).get()
        except:
            old_price = ""

        if not disabled:
            item["product_variants"].append(
                {
                    "color": response.css(
                        "span.text2.color-value.attribute-displayValue::text"
                    ).get(),
                    "size": size,
                    "status": "AVAILABLE",
                    "final_price": final_price,
                    "old_price": old_price,
                }
            )
        else:
            item["product_variants"].append(
                {
                    "color": response.css(
                        "span.text2.color-value.attribute-displayValue::text"
                    ).get(),
                    "size": size,
                    "status": "NOT_AVAILABLE",
                    "final_price": final_price,
                    "old_price": old_price,
                }
            )
        if len(variant_urls) > 0:
            yield scrapy.Request(
                variant_urls[0],
                headers=self.headers,
                cb_kwargs={
                    "item": item,
                    "variant_urls": variant_urls[1:],
                    "clean_sizes_urls": clean_sizes_urls,
                },
                callback=self.parse_product_variants,
            )
        elif len(clean_sizes_urls) > 0:
            yield scrapy.Request(
                clean_sizes_urls[0],
                headers=self.headers,
                cb_kwargs={"item": item, "clean_sizes_urls": clean_sizes_urls[1:]},
                callback=self.parse_clean_sizes,
            )
        yield item

    def parse_clean_sizes(self, response, item, clean_sizes_urls):
        size = "".join(
            list(
                filter(
                    None,
                    [
                        s.replace("\n", "")
                        for s in response.css("li").css("[selected] ::text").extract()
                    ],
                )
            )
        )
        disabled = (
            response.css("li").css("[disabled]").css("[selected] ::text").getall()
        )
        final_price = ""
        final_price = response.css(
            "span.formatted_sale_price.formatted_price.js-final-sale-price.bfx-price.bfx-list-price::text"
        ).get()
        if final_price is None:
            final_price = response.css(
                "span.formatted_sale_price.formatted_price.js-final-sale-price.bfx-price.bfx-sale-price::text"
            ).get()
        try:
            old_price = response.css(
                "span.formatted_price.bfx-price.bfx-list-price ::text"
            ).get()
        except:
            old_price = ""

        if not disabled:
            item["product_variants"].append(
                {
                    "color": item["color"],
                    "size": size,
                    "status": "AVAILABLE",
                    "final_price": final_price,
                    "old_price": old_price,
                }
            )
        else:
            item["product_variants"].append(
                {
                    "color": item["color"],
                    "size": size,
                    "status": "NOT_AVAILABLE",
                    "final_price": final_price,
                    "old_price": old_price,
                }
            )
        if len(clean_sizes_urls) > 0:
            yield scrapy.Request(
                clean_sizes_urls[0],
                headers=self.headers,
                cb_kwargs={"item": item, "clean_sizes_urls": clean_sizes_urls[1:]},
                callback=self.parse_clean_sizes,
            )

        yield item
</code>```

Here is the page with list of numbers pagination: https://www.saksfifthavenue.com/on/demandware.store/Sites-SaksFifthAvenue-Site/en_US/Search-UpdateGrid?cgid=2534374306418048&start=37698&sz=24&hideLess=true

My problem is, there is around 38,723 items as of now but I am getting only 2k or 3k items I tried to debug by looking into the selectors checking if spider reaching to the next_page_no and inner_load and processing it is working fine but still not fetching all the pages spider quits early way before scraping all the pages. I couldn't find the issue. Can anyone please help me out here? Thank you!

Body:
I am scraping this website https://www.saksfifthavenue.com/on/demandware.store/Sites-SaksFifthAvenue-Site/en_US/Search-UpdateGrid?cgid=2534374306418048&start=0&sz=24&hideLess=true it is api url I guess which uses ajax pagination and normal pagination as well. Website using ajax pagination to load the items partially once all the items loaded then page shows the conventional numbered list of page numbers. I have write the scraper and handling the pagination by simply checking if show more div is present in the page or not. Here is the html source for that div:

    <button class="btn btn-outline-primary col-12 col-sm-4" data-url="https://www.saksfifthavenue.com/on/demandware.store/Sites-SaksFifthAvenue-Site/en_US/Search-UpdateGrid?cgid=2534374306418048&start=21&sz=24&hideLess=true">
More Results
</button>

If div is presented in the page I am fetching the data-url and parse items from that url. Once the show more div is no more spider will look for next page div which is look like this:

This is little bit tricky to work with it since I am using api url and next page url is the normal website url so I am only extracting start value from it and update the params. Here is the spider:

import scrapy
from urllib.parse import urlencode
import re
import json
import html

class SaksFifthAvenueSpider(scrapy.Spider):
    name = "safa-feeds"

    # custom settings
    custom_settings = {
        "LOG_FILE": "safa-feeds-spider.log",
        "ITEM_PIPELINES": {
            "sdt.pipelines.SdtWasabiS3Pipeline": 300,
        },
    }

    headers = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
        "Connection": "keep-alive",
        "Upgrade-Insecure-Requests": "1",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "cross-site",
        "Sec-GPC": "1",
    }

    params = {
        "cgid": "",
        "start": "0",
        "sz": "24",
        "hideLess": "true",
    }

    base_url = "https://www.saksfifthavenue.com/on/demandware.store/Sites-SaksFifthAvenue-Site/en_US/Search-UpdateGrid?"

    def start_requests(self):
        cgid_list = [
            "2534374306418048",
            # "2534374306624247",
            # "2534374306622828",
            # "1608313652004",
            # "2534374306418050",
            # "2534374306418162",
            # "2534374306418054",
            # "1654108780232",
            # "2534374306418205",
            # "2534374306418206",
            # "2534374306418217",
            # "2534374306418192",
            # "1608313652004",
            # "2534374306418053",
        ]
        for i, cgid in enumerate(cgid_list):
            self.params["cgid"] = cgid
            category_url = self.base_url + urlencode(self.params)
            yield scrapy.Request(
                url=category_url, headers=self.headers, callback=self.parse_page_items
            )

    def parse_page_items(self, response):
        item_links = set(
            [
                "https://www.saksfifthavenue.com" + u.split("?")[0]
                for u in response.css("a.thumb-link.mw-100::attr(href)").extract()
            ]
        )
        inner_load = response.css("div.show-more ::attr(data-url)").get()
        if inner_load:
            print(inner_load)
            yield scrapy.Request(
                url=inner_load,
                headers=self.headers,
                callback=self.parse_page_items,
            )
        next_page_no = response.css('a[aria-label="Next"]::attr(href)').get()
        if next_page_no:
            self.params["start"] = next_page_no.split("&")[0].split("=")[-1]
            next_url = self.base_url + urlencode(self.params)
            print(next_url)
            yield scrapy.Request(
                url=next_url, headers=self.headers, callback=self.parse_page_items
            )

        for link in item_links:
            yield scrapy.Request(
                url=link, headers=self.headers, callback=self.parse_product_details
            )

    def parse_product_details(self, response):
        item = {}
        json_text = response.css('script[type="application/ld+json"]').get()
        if json_text is not None:
            clean_text = json_text.replace(
                '", "")
            json_blob = json.loads(clean_text)
            prod_id = response.css("div.container.product-detail::attr(data-pid)").get()
            colors = response.css(
                "button::attr(data-adobelaunchproductcolor)"
            ).extract()
            sizes = response.css("li::attr(data-attr-value)").extract()
            item["product_id"] = prod_id
            item["product_brand"] = response.css("a.product-brand::text").get()
            item["product_name"] = response.css("h1.product-name::text").get()
            json_breadcrumbs_text = (
                response.css('script[type="application/ld+json"]')
                .extract()[-1]
                .replace('", "")
            )
            bc_json_blob = json.loads(json_breadcrumbs_text)
            if "itemListElement" in bc_json_blob:
                item["categories"] = [
                    {f"category_{idx}": cat["name"]}
                    for idx, cat in enumerate(bc_json_blob["itemListElement"], 1)
                ]
            else:
                item["categories"] = []
            item["slug"] = json_blob["offers"]["url"].split(".com")[-1]
            desc = json_blob["description"]
            item["description"] = re.sub("<[^<]+?>", " ", html.unescape(desc))
            item["product_variants"] = []
            item["color"] = response.css(
                "span.text2.color-value.attribute-displayValue::text"
            ).get()
            item["sizes"] = []
            item["gender"] = ""
            if "itemListElement" in bc_json_blob:
                bc_li = [b["name"] for b in bc_json_blob["itemListElement"]]

                if "Women's Clothing" or "Beauty" in bc_li:
                    item["gender"] = "Female"
                elif any("Men" in s for s in bc_li) or any("Men's" in s for s in bc_li):
                    item["gender"] = "Male"
                else:
                    item["gender"] = "Female"
                if (
                    "Kids" in bc_li
                    and any("Boys" in s for s in bc_li)
                    or any("Baby Boy" in s for s in bc_li)
                ):
                    item["gender"] = "Boy"
                elif (
                    "Kids" in bc_li
                    and any("Girls" in s for s in bc_li)
                    or any("Baby Girl" in s for s in bc_li)
                ):
                    item["gender"] = "Girl"

                elif (
                    any("Kids" in s for s in bc_li)
                    and not any("Baby Girl" in s for s in bc_li)
                    and not any("Baby Boy" in s for s in bc_li)
                    and not any("Boys" in s for s in bc_li)
                    and not any("Girls" in s for s in bc_li)
                ):
                    item["gender"] = "Kids"

                elif any("Accessories" in s for s in bc_li):
                    item["gender"] = ""

            price_json_text = (
                response.css('script[type="text/javascript"]')
                .extract()[2]
                .replace('", "")
            )
            price_json_blob = json.loads(price_json_text)
            item["tag"] = price_json_blob["products"][0]["tags"]["feature_type"]
            item["price"] = [
                {
                    "original_price": p["original_price"],
                    "price": p["price"],
                    "currency": json_blob["offers"]["priceCurrency"],
                }
                for p in price_json_blob["products"]
            ]
            variant_urls = []
            for color in colors:
                for i_size in sizes:
                    variant_url = (
                        response.url
                        + "?dwvar_"
                        + prod_id
                        + "_color="
                        + color.upper()
                        + "&dwvar_"
                        + prod_id
                        + f"_size={i_size}&pid="
                        + prod_id
                    )
                    variant_urls.append(variant_url)
            item["images"] = json_blob["image"]
            size_selector = response.css(
                "ul.radio-group-list.size-attribute.swatch-display-three.show-size-dropdown"
            )
            if size_selector:
                for s in size_selector.css("li"):
                    all_size_var = s.css("::text").getall()
                    if not s.css("[disabled]"):
                        available = all_size_var
                        clean = list(
                            filter(None, [c.replace("\n", "") for c in available])
                        )
                        for out_si in clean:
                            item["sizes"].append(
                                {"size": out_si, "status": "AVAILABLE"}
                            )
                    else:
                        out_of_stock = all_size_var
                        clean = list(
                            filter(None, [c.replace("\n", "") for c in out_of_stock])
                        )
                        for in_si in clean:
                            item["sizes"].append(
                                {"size": in_si, "status": "NOT_AVAILABLE"}
                            )
            clean_sizes_urls = []
            if response.css("div.form-group.show-size-dropdown-holder"):
                size_dropdown = response.css(
                    "ul.radio-group-list.size-attribute.swatch-display-three ::text"
                ).extract()
                clean_sizes = list(
                    filter(None, [s.replace("\n", "") for s in size_dropdown])
                )
                for dd_si in clean_sizes:
                    if item["color"]:
                        variant_url = (
                            response.url
                            + "?dwvar_"
                            + prod_id
                            + "_color="
                            + item["color"].upper()
                            + "&dwvar_"
                            + prod_id
                            + f"_size={dd_si}&pid="
                            + prod_id
                        )
                        clean_sizes_urls.append(variant_url)
            if variant_urls:
                yield scrapy.Request(
                    variant_urls[0],
                    headers=self.headers,
                    cb_kwargs={
                        "item": item,
                        "variant_urls": variant_urls[1:],
                        "clean_sizes_urls": clean_sizes_urls,
                    },
                    callback=self.parse_product_variants,
                )
            elif clean_sizes_urls:
                yield scrapy.Request(
                    clean_sizes_urls[0],
                    headers=self.headers,
                    cb_kwargs={"item": item, "clean_sizes_urls": clean_sizes_urls[1:]},
                    callback=self.parse_clean_sizes,
                )

            yield item

    def parse_product_variants(self, response, item, variant_urls, clean_sizes_urls):
        size = "".join(
            list(
                filter(
                    None,
                    [
                        s.replace("\n", "")
                        for s in response.css("li").css("[selected] ::text").extract()
                    ],
                )
            )
        )
        disabled = (
            response.css("li").css("[disabled]").css("[selected] ::text").getall()
        )
        final_price = ""
        final_price = response.css(
            "span.formatted_sale_price.formatted_price.js-final-sale-price.bfx-price.bfx-list-price::text"
        ).get()
        if final_price is None:
            final_price = response.css(
                "span.formatted_sale_price.formatted_price.js-final-sale-price.bfx-price.bfx-sale-price::text"
            ).get()
        try:
            old_price = response.css(
                "span.formatted_price.bfx-price.bfx-list-price ::text"
            ).get()
        except:
            old_price = ""

        if not disabled:
            item["product_variants"].append(
                {
                    "color": response.css(
                        "span.text2.color-value.attribute-displayValue::text"
                    ).get(),
                    "size": size,
                    "status": "AVAILABLE",
                    "final_price": final_price,
                    "old_price": old_price,
                }
            )
        else:
            item["product_variants"].append(
                {
                    "color": response.css(
                        "span.text2.color-value.attribute-displayValue::text"
                    ).get(),
                    "size": size,
                    "status": "NOT_AVAILABLE",
                    "final_price": final_price,
                    "old_price": old_price,
                }
            )
        if len(variant_urls) > 0:
            yield scrapy.Request(
                variant_urls[0],
                headers=self.headers,
                cb_kwargs={
                    "item": item,
                    "variant_urls": variant_urls[1:],
                    "clean_sizes_urls": clean_sizes_urls,
                },
                callback=self.parse_product_variants,
            )
        elif len(clean_sizes_urls) > 0:
            yield scrapy.Request(
                clean_sizes_urls[0],
                headers=self.headers,
                cb_kwargs={"item": item, "clean_sizes_urls": clean_sizes_urls[1:]},
                callback=self.parse_clean_sizes,
            )
        yield item

    def parse_clean_sizes(self, response, item, clean_sizes_urls):
        size = "".join(
            list(
                filter(
                    None,
                    [
                        s.replace("\n", "")
                        for s in response.css("li").css("[selected] ::text").extract()
                    ],
                )
            )
        )
        disabled = (
            response.css("li").css("[disabled]").css("[selected] ::text").getall()
        )
        final_price = ""
        final_price = response.css(
            "span.formatted_sale_price.formatted_price.js-final-sale-price.bfx-price.bfx-list-price::text"
        ).get()
        if final_price is None:
            final_price = response.css(
                "span.formatted_sale_price.formatted_price.js-final-sale-price.bfx-price.bfx-sale-price::text"
            ).get()
        try:
            old_price = response.css(
                "span.formatted_price.bfx-price.bfx-list-price ::text"
            ).get()
        except:
            old_price = ""

        if not disabled:
            item["product_variants"].append(
                {
                    "color": item["color"],
                    "size": size,
                    "status": "AVAILABLE",
                    "final_price": final_price,
                    "old_price": old_price,
                }
            )
        else:
            item["product_variants"].append(
                {
                    "color": item["color"],
                    "size": size,
                    "status": "NOT_AVAILABLE",
                    "final_price": final_price,
                    "old_price": old_price,
                }
            )
        if len(clean_sizes_urls) > 0:
            yield scrapy.Request(
                clean_sizes_urls[0],
                headers=self.headers,
                cb_kwargs={"item": item, "clean_sizes_urls": clean_sizes_urls[1:]},
                callback=self.parse_clean_sizes,
            )

        yield item

Here is the page with list of numbers pagination: https://www.saksfifthavenue.com/on/demandware.store/Sites-SaksFifthAvenue-Site/en_US/Search-UpdateGrid?cgid=2534374306418048&start=37698&sz=24&hideLess=true

My problem is, there is around 38,723 items as of now but I am getting only 2k or 3k items I tried to debug by looking into the selectors checking if spider reaching to the next_page_no and inner_load and processing it is working fine but still not fetching all the pages spider quits early way before scraping all the pages. I couldn't find the issue. Can anyone please help me out here? Thank you!


来源: stackexchange

问:
Can I create a data set using WITH before doing an INSERT in MYSQL?
I'm trying to insert data from a spreadsheet into a MySQL table.  I wan't to verify the data combination doesn't already exists so I create the "stuff" dataset which contains the data from Excel.

MySQL doesn't seem to like that I'm using a WITH table (sorry don't know what these are called) before doing the insert.  Is it possible?

```<code>with stuff as(
select  '1005-BASE-BLACK' as external_sku, 'PMD WEBSITE' as retailer, '1005-Base-Black' as sku, 'ACCESSORIES' as cat, 'Charging Stand' as sub_cat
)

insert into retail_to_internal_sku(retailer, retail_sku, internal_sku, date_added)

select s.retailer, s.external_sku, s.sku, curdate()
from stuff s
where not exists (select * from retail_to_internal_sku r
                    where r.internal_sku = s.sku
                    and r.retail_sku = s.external_sku
                    and r.retailer= s.retailer);
</code>```

ContentLicense: CC BY-SA 4.0

Body:
I'm trying to insert data from a spreadsheet into a MySQL table.  I wan't to verify the data combination doesn't already exists so I create the "stuff" dataset which contains the data from Excel.

MySQL doesn't seem to like that I'm using a WITH table (sorry don't know what these are called) before doing the insert.  Is it possible?

with stuff as(
select  '1005-BASE-BLACK' as external_sku, 'PMD WEBSITE' as retailer, '1005-Base-Black' as sku, 'ACCESSORIES' as cat, 'Charging Stand' as sub_cat
)

insert into retail_to_internal_sku(retailer, retail_sku, internal_sku, date_added)

select s.retailer, s.external_sku, s.sku, curdate()
from stuff s
where not exists (select * from retail_to_internal_sku r
                    where r.internal_sku = s.sku
                    and r.retail_sku = s.external_sku
                    and r.retailer= s.retailer);


问:
Perfect subset sum problem with limit on subset size
Currently, I implemented the perfect subset sum problem using the implementation provided on geeksfor geeks (1, java implementation). The number of subsets with sum=k increases exponentially with the set size from which the subsets are taken. As I consider the subsets in increasing size, I would like to be able to first generate all subsets with size n, then all subsets with size n+1, etc. This way, if some criteria is not satisfied for size n, I would no longer have to generate all subsets with size n+1.

The question remains how to efficiently generate all subsets with sum k and size n only. I have check several threads here on stackoverflow, but none gave me a useful and insightful approach.

Any suggestions would be more than welcome!


问:
How to cancel thunks on rote change
I use reduxjs/toolkit for handling redux/actions in my application. Now I've come to the point when I need to cancel pending actions on route change (some or all). It shouldn't come with request cancel with AbortController - simple action failure is just fine. So I started to implement it using middlewares, but there's no such thing as cancelling task without promise from a dispatch. I've got a lot of actions to cancel and it seems odd to cancel each action one by one on unmount. Is there a way to cancel them all (or some) at once?
Here's a piece of code I used to try to achieve the result</p>
```<code>import { createListenerMiddleware, isPending } from '@reduxjs/toolkit';
import { routerLocationSelector } from '../selectors';
import { Store } from '../../types';

const listenerMiddleware = createListenerMiddleware<Store>();
const cancelMiddleware = createListenerMiddleware<Store>();

const actions = {};

listenerMiddleware.startListening({
  effect: (action) => {
    if (isPending(action)) {
      actions[action.meta.requestId] = action;
    } else {
      delete actions[action.meta.requestId];
    }
  },
  predicate: () => true,
});

cancelMiddleware.startListening({
  effect: () => {
    console.log('this is the moment i want all my actions to cancel');
  },
  predicate: (action, currentState, prevState) =>
    routerLocationSelector(currentState) !== routerLocationSelector(prevState),
});

export { listenerMiddleware, cancelMiddleware };
</code>```

Title: How to cancel thunks on rote change

Body:
I use reduxjs/toolkit for handling redux/actions in my application. Now I've come to the point when I need to cancel pending actions on route change (some or all). It shouldn't come with request cancel with AbortController - simple action failure is just fine. So I started to implement it using middlewares, but there's no such thing as cancelling task without promise from a dispatch. I've got a lot of actions to cancel and it seems odd to cancel each action one by one on unmount. Is there a way to cancel them all (or some) at once?
Here's a piece of code I used to try to achieve the result

import { createListenerMiddleware, isPending } from '@reduxjs/toolkit';
import { routerLocationSelector } from '../selectors';
import { Store } from '../../types';

const listenerMiddleware = createListenerMiddleware<Store>();
const cancelMiddleware = createListenerMiddleware<Store>();

const actions = {};

listenerMiddleware.startListening({
  effect: (action) => {
    if (isPending(action)) {
      actions[action.meta.requestId] = action;
    } else {
      delete actions[action.meta.requestId];
    }
  },
  predicate: () => true,
});

cancelMiddleware.startListening({
  effect: () => {
    console.log('this is the moment i want all my actions to cancel');
  },
  predicate: (action, currentState, prevState) =>
    routerLocationSelector(currentState) !== routerLocationSelector(prevState),
});

export { listenerMiddleware, cancelMiddleware };


问:
Unwanted blank table row in Docmosis
I am using Docmosis to generate data into a table. A blank row appears underneath the table heading and between each row containing data.

I suspect my issue is occurring because of the statements using: <<$isLender>>
I have considered using <<else_ to handle behaviour when <<$isLender>> is true or false however can't seem to find a working solution to remove the blank row. Any advice would be appreciated.

Name
Address

<<rr_submittingConveyancer.representees>>

<<$isLender=false>><<rs_party.roles>><<cs_{type=’lender’}>><<$isLender=true>><<es_{type=’lender’}>><<es_party.roles>>

<<cr_$isLender>>

Full code for the table

Name
Address

Joe Bloggs
A1 address of the property

Big Company
B1 Current proprietor's address

Title: Unwanted blank table row in Docmosis


问:
making a image adjust to width and height changes
ok so i have this image and i need it to be she height of it's available space but, at the same time,  i need it to overflow in the y axis without losing its aspect ratio. I managed to make it shrink it horizontally without losing aspect ratio (using overflow) but when i strech it too much on the x axis it gest warped. Can you guys help me with that:

```<code> <main>
        <div class="imageContainer"><img src="../files/img215.jpg" alt=""></div>

 </main>
</code>```
```<code>main > div {

    height: 85vh;
    overflow: hidden;
}

main > div > img {

    height: 100%;
    min-width: 100%;
    object-fit: cover;

}
</code>```

i tried object-fit: cover;

but i did'nt tget the response i was expecting cause i just need it to crop downwards not evenly

Body:
ok so i have this image and i need it to be she height of it's available space but, at the same time,  i need it to overflow in the y axis without losing its aspect ratio. I managed to make it shrink it horizontally without losing aspect ratio (using overflow) but when i strech it too much on the x axis it gest warped. Can you guys help me with that:

main > div {

    height: 85vh;
    overflow: hidden;
}

main > div > img {

    height: 100%;
    min-width: 100%;
    object-fit: cover;

}

i tried object-fit: cover;

but i did'nt tget the response i was expecting cause i just need it to crop downwards not evenly


问:
How to select values inside list?
How to select values inside MultiSelect? I just want to select some items inside a dropdownlist from URL Variables

Issue: following line is not selecting items Colors.add(TempArray[i]);

debug: If I remove this line than it works ok and selects values. if (!string.IsNullOrEmpty(result)) ... foreach (var item2 in TempArray) but I need this line to check for null values. Code will get error otherwise

Also If I remove Post Method and it goes to Get directly than it works fine (items get selected). how can I make it work by using post method

Form

```<code><form asp-page="./Index" method="post">
<select asp-for="Colors" asp-items="@Model.Colors_SELECT " class="MultiSelect" multiple>
     ....
</select>
...
</code>```

back-end variables

```<code>[BindProperty(SupportsGet = true)]
public List<string>? Colors { get; set; }
public SelectList? Colors_SELECT { get; set; }
</code>```

OnPost() Method - here I am change the URL format. for ex: localhost/index?Colors=red&Colors=Black to localhost/index?Colors=red,Black

```<code>    public async Task<IActionResult> OnPostAsync()
    {
        var CurrentFilters = new Dictionary<string, string>();
        var ColorsTemp = string.Join(",", Colors);
        CurrentFilters.Add("Colors", ColorsTemp);

        string query = "";
        foreach (var p in CurrentFilters)
        {
            query += $"{p.Key}={p.Value}&";
        }
        query = query.TrimEnd(query[query.Length - 1]); //remove last '&'
        var url = $"{HttpContext.Request.Path}?{query}"; // create URL
        return Redirect(url); // Send new url - call get Method()
    }
</code>```

on OnGet() method

```<code>public async Task OnGetAsync()
{
    // here i want to get URL values and select those items
    string result = Request.Query["Colors"];
    if (!string.IsNullOrEmpty(result))
    {
        string[] TempArray = result.Split(",");
         foreach (var item2 in TempArray)
         {
             Colors.add(TempArray[i]);
         }
    }
}
</code>```

Title: How to select values inside list?

Body:
How to select values inside MultiSelect? I just want to select some items inside a dropdownlist from URL Variables

Issue: following line is not selecting items Colors.add(TempArray[i]);

debug: If I remove this line than it works ok and selects values. if (!string.IsNullOrEmpty(result)) ... foreach (var item2 in TempArray) but I need this line to check for null values. Code will get error otherwise

Also If I remove Post Method and it goes to Get directly than it works fine (items get selected). how can I make it work by using post method

Form

<form asp-page="./Index" method="post">
<select asp-for="Colors" asp-items="@Model.Colors_SELECT " class="MultiSelect" multiple>
     ....
</select>
...

back-end variables

[BindProperty(SupportsGet = true)]
public List<string>? Colors { get; set; }
public SelectList? Colors_SELECT { get; set; }

OnPost() Method - here I am change the URL format. for ex: localhost/index?Colors=red&Colors=Black to localhost/index?Colors=red,Black

    public async Task<IActionResult> OnPostAsync()
    {
        var CurrentFilters = new Dictionary<string, string>();
        var ColorsTemp = string.Join(",", Colors);
        CurrentFilters.Add("Colors", ColorsTemp);

        string query = "";
        foreach (var p in CurrentFilters)
        {
            query += $"{p.Key}={p.Value}&";
        }
        query = query.TrimEnd(query[query.Length - 1]); //remove last '&'
        var url = $"{HttpContext.Request.Path}?{query}"; // create URL
        return Redirect(url); // Send new url - call get Method()
    }

on OnGet() method

public async Task OnGetAsync()
{
    // here i want to get URL values and select those items
    string result = Request.Query["Colors"];
    if (!string.IsNullOrEmpty(result))
    {
        string[] TempArray = result.Split(",");
         foreach (var item2 in TempArray)
         {
             Colors.add(TempArray[i]);
         }
    }
}


问:
How can I use Unity to train an autonomous driving AI (for a small robot) that i can then export onto a Raspberry Pi? I have 4 ToF Sensors on servos
So I have a Robot that is supposed to follow a BLE Tag, and it gets the vector from my antenna in what direction the tag is. My problem is that if the tag goes around a corner, the robot would just crash into the wall. So my plan is to use 4 ToF sensors on servos (180°), one at each corner, plus 4 fixed HC-04 sensors to look if there is a wall and so on.

I know that BMW uses Unity to train their autonomous driving models, but i found no info on how I can then export the AI to use with Python onto a Raspi 4 once it's done training to let it run with the real data from my real sensors.

I've used a bit of Unity before, so creating worlds and simulating the Robot with it's sensors and simulating the data each sensor returns shouldn't be a problem.

Would be great if someone could help :)

I haven't tried anything yet, I'm just trying to figure out how to export that and I wasn't able to find anything on the Internet


问:
Google Play paid subscription works in closed Alpha but not in Production
I've implemented a paid Google Play subscription option into my app, I tested it in debug, release and closed alpha with test card and it always worked. But as I push my app from Alpha into Production the popup payment dialog stops showing up.

This menu of Google Play payment use see in the screenshot just doesn't show up, without any errors. I'm not sure what causes the problem because it seems it happens only in Production version and I don't know how to debug it.
The menu that doesn't popup in production

I've tested the subscription in Android Studio and also uploaded in into Closed Alpha track. Each time it worked fine and asked me to pay with the test card.


问:
Get relative object after creating it in Async SQL Alchemy
I have model:
`class User(Base):
tablename = 'user'

```<code>id = Column(Integer, Sequence('id'), primary_key=True)
name = Column(String(50), nullable=False)
coffee_id = Column(Integer, ForeignKey('coffee.id'))
coffee = relationship('Coffee', backref='users')`
</code>```

in SQL Alchemy I create object:

        new_user = User(name="user", coffee_id=1) async with session.begin(): session.add(new_user)

If i am trying get new_user.coffee i get exception, that object is not downloaded.

Of course i can do:

user_with_coffee = select(User) \ .filter_by(id=new_user.id) \ .options(selectinload(User.coffee)) new_user = await session.execute(user_with_coffee)

But maybe i can add some parameter in add method of session or something else to avoid second query  in db ?

Body:
I have model:
`class User(Base):
tablename = 'user'

id = Column(Integer, Sequence('id'), primary_key=True)
name = Column(String(50), nullable=False)
coffee_id = Column(Integer, ForeignKey('coffee.id'))
coffee = relationship('Coffee', backref='users')`

in SQL Alchemy I create object:

        new_user = User(name="user", coffee_id=1) async with session.begin(): session.add(new_user)

If i am trying get new_user.coffee i get exception, that object is not downloaded.

Of course i can do:

user_with_coffee = select(User) \ .filter_by(id=new_user.id) \ .options(selectinload(User.coffee)) new_user = await session.execute(user_with_coffee)

But maybe i can add some parameter in add method of session or something else to avoid second query  in db ?


问:
How can I find the most frequent county using data from another table?
I have three tables:

Customer (cnp, customer_no, city, county)

Accounts (account_number, customer_no, balance)

Cards (card_id, account_number, card_number, exp_date)

Select the COUNTY where are the most CARDS that expired in the current month.

Tried this method:

SELECT TOP 1 Customer.county, COUNT(Customer.county) AS county_count
FROM Customer
JOIN Accounts ON Customer.customer_no = Accounts.customer_no
JOIN Cards ON Accounts.account_number = Cards.account_number
WHERE Cards.exp_date BETWEEN DATEADD(month, -1, GETDATE()) AND GETDATE()
GROUP BY Customer.county
ORDER BY county_count DESC;
```


问:
How can I give these two functions access to shared data?
I have three functions, handleSubmit handleCancel, and solve. HandleSubmit triggers a compute-heavy process, solve. handleCancel should stop that process early.

My simplified code looks like this:

```<code>function handleSubmit() {
  computeHeavyProcess();
}
</code>```
```<code>function handleCancel() {
  // not sure what to put here, or how to design this
}
</code>```
```<code>function solve() {
  while (computationNotDone) {
    // do computations
    setSolutionState(resultOfCurrentComputations); // update React state every iteration
    await sleep(0); // unblock UI by chunking compute
  }
}
</code>```

I would like for the following interaction:

When a user clicks triggers handleSubmit, the computation starts.

The computation runs, updating state every so often, chunking the compute and allowing the UI to be responsive

If the user clicks triggers handleCancel, the computation process is killed.

One way to do this is to have a global variable, allowedToRun, and invoking handleCancel will set this to false. Then, inside solve, I can check for while (computationNotDone && allowedToRun).

However, I would like to not use global variables, as I believe it suggests my code is designed poorly.

I would appreciate any advice on how to design this system without global variables.

Also, I would like to import solve, so I cannot use a module-level variable without wrapping it in an object, which I want to avoid.

Body:
I have three functions, handleSubmit handleCancel, and solve. HandleSubmit triggers a compute-heavy process, solve. handleCancel should stop that process early.

My simplified code looks like this:

function handleSubmit() {
  computeHeavyProcess();
}

function handleCancel() {
  // not sure what to put here, or how to design this
}

function solve() {
  while (computationNotDone) {
    // do computations
    setSolutionState(resultOfCurrentComputations); // update React state every iteration
    await sleep(0); // unblock UI by chunking compute
  }
}

I would like for the following interaction:

When a user clicks triggers handleSubmit, the computation starts.

The computation runs, updating state every so often, chunking the compute and allowing the UI to be responsive

If the user clicks triggers handleCancel, the computation process is killed.

One way to do this is to have a global variable, allowedToRun, and invoking handleCancel will set this to false. Then, inside solve, I can check for while (computationNotDone && allowedToRun).

However, I would like to not use global variables, as I believe it suggests my code is designed poorly.

I would appreciate any advice on how to design this system without global variables.

Also, I would like to import solve, so I cannot use a module-level variable without wrapping it in an object, which I want to avoid.


问:
special ordered set of type 2 yalmip
I am trying to implement special ordered set of type 2 SOS2 in my optimization problem to linearyze it.
I have: w = x1*x2 = y1^2 - y2^2
Therefore: y1 = 1/2(x1 + x2) and y1 = 1/2(x1 - x2)
I want to apply this method to y1^2 and y2^2 and I am considering x of my problem y1 and y2. I am using cplex as solver and is saying is infeasible.

My code is a model of a distributed system

```<code>%% Reset cmd
% Funciona
clear all
close all
clc

%% Define base values
SBase=5e3; % kVA
VBase=4160; % V
ZBase=VBase^2/(SBase)/1e3; % Ohm
YBase = 1/ZBase; % Siemens
IBase=SBase/(sqrt(3)*VBase);

%% Data
T= 24; %Time considered
N=3;

% kW

DA = -[0.0770 0.0256 0.0770 0.0320 0.0320 0.0770 0.0256 0.0320 0.0770 0.0320]/10;
DR = -[0.0440 0.0172 0.0440 0.0220 0.0220 0.0440 0.0172 0.0220 0.0440 0.0220]/10;

DA = [DA(1:6), DA(1:6), DA(1:9), DA(1:6), DA(1:2), DA, DR(1:6), DR(1:9), DA, DR(1:6), DR(1:9), DA  ];
DR = [DR(1:6), DR(1:9), DR(1:9), DR(1:6), DR(1:9), DR(1:6), DR(1:9), DR, DR(1:6), DR(1:9)  ];

% PV
P_res = [0  0   0   0   0.01      0.04  0.06    0.07    0.1     0.12    0.17  0.14  0.17    0.15    0.1 0.09    0.08    0.05    0.05     0.06   0 0 0   0]/20;
Q_res = P_res *tan(acos(0.95));

% 12 Nodes

Y = [  3.9638 -     11.424i      -3.9638 +     11.424i            0 +          0i
      -3.9638 +     11.424i       42.193 -     61.341i      -15.004 +     13.801i
            0 +          0i      -15.004 +     13.801i       40.009 -     36.803i     ]*YBase;

%% Variables

Ir=sdpvar(N,T);
Ii=sdpvar(N,T);
Vr=sdpvar(N,T);
Vi=sdpvar(N,T);

V=sdpvar(N,T);
I=sdpvar(N,T);

P=sdpvar(N,T);
Q=sdpvar(N,T);

%% Constraints
distancia = 1000;

xi = linspace(-10,10,distancia)';
fi = xi.^2;

lambda1 = sdpvar(distancia, N*T);
lambda2 = sdpvar(distancia, N*T);
lambda3 = sdpvar(distancia, N*T);
lambda4 = sdpvar(distancia, N*T);
lambda5 = sdpvar(distancia, N*T);
lambda6 = sdpvar(distancia, N*T);
lambda7 = sdpvar(distancia, N*T);
lambda8 = sdpvar(distancia, N*T);

z1 = sdpvar(N,T);
z2 = sdpvar(N,T);
z3 = sdpvar(N,T);
z4 = sdpvar(N,T);
z5 = sdpvar(N,T);
z6 = sdpvar(N,T);
z7 = sdpvar(N,T);
z8 = sdpvar(N,T);

x1 = sdpvar(N,T);
x2 = sdpvar(N,T);
x3 = sdpvar(N,T);
x4 = sdpvar(N,T);
x5 = sdpvar(N,T);
x6 = sdpvar(N,T);
x7 = sdpvar(N,T);
x8 = sdpvar(N,T);

F = [
    x1 == 1/2*(Vr + Ir),...
    x2 == 1/2*(Vr - Ir),...
    x3 == 1/2*(Vi + Ii),...
    x4 == 1/2*(Vi - Ii),...
    x5 == 1/2*(Vr + Ii),...
    x6 == 1/2*(Vr - Ii),...
    x7 == 1/2*(Vi + Ir),...
    x8 == 1/2*(Vi - Ir)];

for n = 1:N
    for t = 1:T

        F = [F,...
        sos2(lambda1(:,n*t)),...
        sos2(lambda2(:,n*t)),...
        sos2(lambda3(:,n*t)),...
        sos2(lambda4(:,n*t)),...
        sos2(lambda5(:,n*t)),...
        sos2(lambda6(:,n*t)),...
        sos2(lambda7(:,n*t)),...
        sos2(lambda8(:,n*t)),...

        ];
    end
end

for n = 1:N
    for t = 1:T
        F = [F,...
        sum(lambda1(:,n*t)) == 1,...
        lambda1(:,n*t) >= 0,...
        sum(lambda2(:,n*t)) == 1,...
        lambda2(:,n*t) >= 0,...
        sum(lambda3(:,n*t)) == 1,...
        lambda3(:,n*t) >= 0,...
        sum(lambda4(:,n*t)) == 1,...
        lambda4(:,n*t) >= 0,...
        sum(lambda5(:,n*t)) == 1,...
        lambda5(:,n*t) >= 0,...
        sum(lambda6(:,n*t)) == 1,...
        lambda6(:,n*t) >= 0,...
        sum(lambda7(:,n*t)) == 1,...
        lambda7(:,n*t) >= 0,...
        sum(lambda8(:,n*t)) == 1,...
        lambda8(:,n*t) >= 0,...

        z1(n,t) == lambda1(:,n*t)'*fi,...
        z2(n,t) == lambda2(:,n*t)'*fi,...
        z3(n,t) == lambda3(:,n*t)'*fi,...
        z4(n,t) == lambda4(:,n*t)'*fi,...
        z5(n,t) == lambda5(:,n*t)'*fi,...
        z6(n,t) == lambda6(:,n*t)'*fi,...
        z7(n,t) == lambda7(:,n*t)'*fi,...
        z8(n,t) == lambda8(:,n*t)'*fi,...

        x1(n,t) == lambda1(:,n*t)'*xi,...
        x2(n,t) == lambda2(:,n*t)'*xi,...
        x3(n,t) == lambda3(:,n*t)'*xi,...
        x4(n,t) == lambda4(:,n*t)'*xi,...
        x5(n,t) == lambda5(:,n*t)'*xi,...
        x6(n,t) == lambda6(:,n*t)'*xi,...
        x7(n,t) == lambda7(:,n*t)'*xi,...
        x8(n,t) == lambda8(:,n*t)'*xi,...

        ];
    end
end

Constraint = [];
for n = 1:N
    for t = 1:T
        Constraint = [Constraint,...

        P(n, t) == z1(n, t) - z2(n, t) + z3(n, t) - z4(n, t) ,...
        Q(n, t) == -z5(n, t) + z6(n, t) + z7(n, t) - z8(n, t),...

    ];
    end
end

Constraint = [Constraint,...
V==Vr+1i*Vi,...
I==Ir+1i*Ii,...
I == (Y*V),...%.*Ysum.*Loadsum;

Vr(1, :) == 1,... % 1<=
Vi(1, :) == 0,... % 0<=

%with EV
P(2, :) == DA(10:T+9),...
P(3, :) == DA(10:T+9)/2,...

Q(2, :) == DR(10:T+9),...
Q(3, :) == DR(10:T+9)/2,...

0.9<=Vr(:,:)<=1.1,...
-1.1<=Vi(:,:)<=1.1,...

];

%% Cost equation

L = sum(P(1, :));

%% Solution
ops = sdpsettings('solver','cplex');
tic
prob = optimize([Constraint, F], L, ops);
toc
</code>```

Anyone knows something related to this?
Thanks

Title: special ordered set of type 2 yalmip

Body:
I am trying to implement special ordered set of type 2 SOS2 in my optimization problem to linearyze it.
I have: w = x1*x2 = y1^2 - y2^2
Therefore: y1 = 1/2(x1 + x2) and y1 = 1/2(x1 - x2)
I want to apply this method to y1^2 and y2^2 and I am considering x of my problem y1 and y2. I am using cplex as solver and is saying is infeasible.

My code is a model of a distributed system

%% Reset cmd
% Funciona
clear all
close all
clc

%% Define base values
SBase=5e3; % kVA
VBase=4160; % V
ZBase=VBase^2/(SBase)/1e3; % Ohm
YBase = 1/ZBase; % Siemens
IBase=SBase/(sqrt(3)*VBase);

%% Data
T= 24; %Time considered
N=3;

% kW

DA = -[0.0770 0.0256 0.0770 0.0320 0.0320 0.0770 0.0256 0.0320 0.0770 0.0320]/10;
DR = -[0.0440 0.0172 0.0440 0.0220 0.0220 0.0440 0.0172 0.0220 0.0440 0.0220]/10;

DA = [DA(1:6), DA(1:6), DA(1:9), DA(1:6), DA(1:2), DA, DR(1:6), DR(1:9), DA, DR(1:6), DR(1:9), DA  ];
DR = [DR(1:6), DR(1:9), DR(1:9), DR(1:6), DR(1:9), DR(1:6), DR(1:9), DR, DR(1:6), DR(1:9)  ];

% PV
P_res = [0  0   0   0   0.01      0.04  0.06    0.07    0.1     0.12    0.17  0.14  0.17    0.15    0.1 0.09    0.08    0.05    0.05     0.06   0 0 0   0]/20;
Q_res = P_res *tan(acos(0.95));

% 12 Nodes

Y = [  3.9638 -     11.424i      -3.9638 +     11.424i            0 +          0i
      -3.9638 +     11.424i       42.193 -     61.341i      -15.004 +     13.801i
            0 +          0i      -15.004 +     13.801i       40.009 -     36.803i     ]*YBase;

%% Variables

Ir=sdpvar(N,T);
Ii=sdpvar(N,T);
Vr=sdpvar(N,T);
Vi=sdpvar(N,T);

V=sdpvar(N,T);
I=sdpvar(N,T);

P=sdpvar(N,T);
Q=sdpvar(N,T);

%% Constraints
distancia = 1000;

xi = linspace(-10,10,distancia)';
fi = xi.^2;

lambda1 = sdpvar(distancia, N*T);
lambda2 = sdpvar(distancia, N*T);
lambda3 = sdpvar(distancia, N*T);
lambda4 = sdpvar(distancia, N*T);
lambda5 = sdpvar(distancia, N*T);
lambda6 = sdpvar(distancia, N*T);
lambda7 = sdpvar(distancia, N*T);
lambda8 = sdpvar(distancia, N*T);

z1 = sdpvar(N,T);
z2 = sdpvar(N,T);
z3 = sdpvar(N,T);
z4 = sdpvar(N,T);
z5 = sdpvar(N,T);
z6 = sdpvar(N,T);
z7 = sdpvar(N,T);
z8 = sdpvar(N,T);

x1 = sdpvar(N,T);
x2 = sdpvar(N,T);
x3 = sdpvar(N,T);
x4 = sdpvar(N,T);
x5 = sdpvar(N,T);
x6 = sdpvar(N,T);
x7 = sdpvar(N,T);
x8 = sdpvar(N,T);

F = [
    x1 == 1/2*(Vr + Ir),...
    x2 == 1/2*(Vr - Ir),...
    x3 == 1/2*(Vi + Ii),...
    x4 == 1/2*(Vi - Ii),...
    x5 == 1/2*(Vr + Ii),...
    x6 == 1/2*(Vr - Ii),...
    x7 == 1/2*(Vi + Ir),...
    x8 == 1/2*(Vi - Ir)];

for n = 1:N
    for t = 1:T

        F = [F,...
        sos2(lambda1(:,n*t)),...
        sos2(lambda2(:,n*t)),...
        sos2(lambda3(:,n*t)),...
        sos2(lambda4(:,n*t)),...
        sos2(lambda5(:,n*t)),...
        sos2(lambda6(:,n*t)),...
        sos2(lambda7(:,n*t)),...
        sos2(lambda8(:,n*t)),...

        ];
    end
end

for n = 1:N
    for t = 1:T
        F = [F,...
        sum(lambda1(:,n*t)) == 1,...
        lambda1(:,n*t) >= 0,...
        sum(lambda2(:,n*t)) == 1,...
        lambda2(:,n*t) >= 0,...
        sum(lambda3(:,n*t)) == 1,...
        lambda3(:,n*t) >= 0,...
        sum(lambda4(:,n*t)) == 1,...
        lambda4(:,n*t) >= 0,...
        sum(lambda5(:,n*t)) == 1,...
        lambda5(:,n*t) >= 0,...
        sum(lambda6(:,n*t)) == 1,...
        lambda6(:,n*t) >= 0,...
        sum(lambda7(:,n*t)) == 1,...
        lambda7(:,n*t) >= 0,...
        sum(lambda8(:,n*t)) == 1,...
        lambda8(:,n*t) >= 0,...

        z1(n,t) == lambda1(:,n*t)'*fi,...
        z2(n,t) == lambda2(:,n*t)'*fi,...
        z3(n,t) == lambda3(:,n*t)'*fi,...
        z4(n,t) == lambda4(:,n*t)'*fi,...
        z5(n,t) == lambda5(:,n*t)'*fi,...
        z6(n,t) == lambda6(:,n*t)'*fi,...
        z7(n,t) == lambda7(:,n*t)'*fi,...
        z8(n,t) == lambda8(:,n*t)'*fi,...

        x1(n,t) == lambda1(:,n*t)'*xi,...
        x2(n,t) == lambda2(:,n*t)'*xi,...
        x3(n,t) == lambda3(:,n*t)'*xi,...
        x4(n,t) == lambda4(:,n*t)'*xi,...
        x5(n,t) == lambda5(:,n*t)'*xi,...
        x6(n,t) == lambda6(:,n*t)'*xi,...
        x7(n,t) == lambda7(:,n*t)'*xi,...
        x8(n,t) == lambda8(:,n*t)'*xi,...

        ];
    end
end

Constraint = [];
for n = 1:N
    for t = 1:T
        Constraint = [Constraint,...

        P(n, t) == z1(n, t) - z2(n, t) + z3(n, t) - z4(n, t) ,...
        Q(n, t) == -z5(n, t) + z6(n, t) + z7(n, t) - z8(n, t),...

    ];
    end
end

Constraint = [Constraint,...
V==Vr+1i*Vi,...
I==Ir+1i*Ii,...
I == (Y*V),...%.*Ysum.*Loadsum;

Vr(1, :) == 1,... % 1<=
Vi(1, :) == 0,... % 0<=

%with EV
P(2, :) == DA(10:T+9),...
P(3, :) == DA(10:T+9)/2,...

Q(2, :) == DR(10:T+9),...
Q(3, :) == DR(10:T+9)/2,...

0.9<=Vr(:,:)<=1.1,...
-1.1<=Vi(:,:)<=1.1,...

];

%% Cost equation

L = sum(P(1, :));

%% Solution
ops = sdpsettings('solver','cplex');
tic
prob = optimize([Constraint, F], L, ops);
toc

Anyone knows something related to this?
Thanks


问:
Apollo js refetchQueries does not fire requests
I am using Apollo Client without React, just plain ol JS. I have a classic table and modal that adds things to the backend. Once I add the thing using client.mutate() I'd like to update the table. If the table is not on the last page, it doesn't really matter, the user will eventually click there to get to the new row. But if the user is on the last page or has visited the last page at any point, going back to that page will not display the new row.

In React I was able to do something like

```<code>useMyMutation({
  variables: {},
  refetchQueries: [getOperationName(SomeDocument)]
})
</code>```

And what this did was refetch the provided query name with its last used variables. I'd like to do something similar and I've been reading the docs and trying this and that but can't even get a query to trigger on the network.

```<code>client.mutate(
  mutation: MyMutation,
  variables: {a:1},
  refetchQueries: (e: any) => "all", // nothing I put here ever does anything
)
</code>```

Body:
I am using Apollo Client without React, just plain ol JS. I have a classic table and modal that adds things to the backend. Once I add the thing using client.mutate() I'd like to update the table. If the table is not on the last page, it doesn't really matter, the user will eventually click there to get to the new row. But if the user is on the last page or has visited the last page at any point, going back to that page will not display the new row.

In React I was able to do something like

useMyMutation({
  variables: {},
  refetchQueries: [getOperationName(SomeDocument)]
})

And what this did was refetch the provided query name with its last used variables. I'd like to do something similar and I've been reading the docs and trying this and that but can't even get a query to trigger on the network.

client.mutate(
  mutation: MyMutation,
  variables: {a:1},
  refetchQueries: (e: any) => "all", // nothing I put here ever does anything
)


问:
android.os.BinderProxy cannot be cast to android.app.servertransaction.ClientTransaction
In Poco X3 phone found a black out screen after changing language from settings.

steps to produce

From Dashboard to goes to detail screen

Go to Device settings , update language

back to app

back press navigate to dashboard

black out screen for few seconds. How to fix this issue !!!!

android studio logs

```<code>java.lang.ClassCastException: android.os.BinderProxy cannot be cast to android.app.servertransaction.ClientTransaction
        at android.app.ActivityInjector.checkAccessControl(ActivityInjector.java:24)
        at android.app.Activity.onResume(Activity.java:1995)
        at androidx.fragment.app.n.onResume(FragmentActivity.java:2)
        at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1457)
        at android.app.Activity.performResume(Activity.java:8264)
        at android.app.ActivityThread.performResumeActivity(ActivityThread.java:4509)
        at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:4551)
</code>```

Body:
In Poco X3 phone found a black out screen after changing language from settings.

steps to produce

From Dashboard to goes to detail screen

Go to Device settings , update language

back to app

back press navigate to dashboard

black out screen for few seconds. How to fix this issue !!!!

android studio logs

java.lang.ClassCastException: android.os.BinderProxy cannot be cast to android.app.servertransaction.ClientTransaction
        at android.app.ActivityInjector.checkAccessControl(ActivityInjector.java:24)
        at android.app.Activity.onResume(Activity.java:1995)
        at androidx.fragment.app.n.onResume(FragmentActivity.java:2)
        at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1457)
        at android.app.Activity.performResume(Activity.java:8264)
        at android.app.ActivityThread.performResumeActivity(ActivityThread.java:4509)
        at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:4551)


问:
Is it possible to run an Apollo model as a function? - gives errors about global environment
Let's say I have this Apollo model (from http://www.apollochoicemodelling.com/files/examples/4%20Ranking%20and%20rating/OL.r)

```<code># ################################################################# #
#### LOAD LIBRARY AND DEFINE CORE SETTINGS                       ####
# ################################################################# #

### Clear memory
rm(list = ls())

### Load Apollo library
library(apollo)

### Initialise code
apollo_initialise()

### Set core controls
apollo_control = list(
  modelName       = "OL",
  modelDescr      = "Ordered logit model fitted to attitudinal question in drug choice data",
  indivID         = "ID",
  outputDirectory = "output"
)

# ################################################################# #
#### LOAD DATA AND APPLY ANY TRANSFORMATIONS                     ####
# ################################################################# #

### Loading data from package
### if data is to be loaded from a file (e.g. called data.csv),
### the code would be: database = read.csv("data.csv",header=TRUE)
database = ?apollo_drugChoiceData
### for data dictionary, use ?apollo_drugChoiceData

# ################################################################# #
#### DEFINE MODEL PARAMETERS                                     ####
# ################################################################# #

### Vector of parameters, including any that are kept fixed in estimation
apollo_beta = c(beta_reg_user   = 0,
                beta_university = 0,
                beta_age_50     = 0,
                tau_quality_1   =-2,
                tau_quality_2   =-1,
                tau_quality_3   = 1,
                tau_quality_4   = 2)

### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
apollo_fixed = c()

# ################################################################# #
#### GROUP AND VALIDATE INPUTS                                   ####
# ################################################################# #

apollo_inputs = apollo_validateInputs()

# ################################################################# #
#### DEFINE MODEL AND LIKELIHOOD FUNCTION                        ####
# ################################################################# #

apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

  ### Attach inputs and detach after function exit
  apollo_attach(apollo_beta, apollo_inputs)
  on.exit(apollo_detach(apollo_beta, apollo_inputs))

  ### Create list of probabilities P
  P = list()

  ### Calculate probabilities using Ordered Logit model
  ol_settings = list(outcomeOrdered = attitude_quality,
                     utility        = beta_reg_user*regular_user + beta_university*university_educated + beta_age_50*over_50,
                     tau            = list(tau_quality_1, tau_quality_2, tau_quality_3, tau_quality_4),
                     rows           = (task==1))

  P[["model"]] = apollo_ol(ol_settings, functionality)

  ### Take product across observation for same individual
  P = apollo_panelProd(P, apollo_inputs, functionality)

  ### Prepare and return outputs of function
  P = apollo_prepareProb(P, apollo_inputs, functionality)
  return(P)
}

# ################################################################# #
#### MODEL ESTIMATION                                            ####
# ################################################################# #

model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)

# ################################################################# #
#### MODEL OUTPUTS                                               ####
# ################################################################# #

# ----------------------------------------------------------------- #
#---- FORMATTED OUTPUT (TO SCREEN)                               ----
# ----------------------------------------------------------------- #

apollo_modelOutput(model)

# ----------------------------------------------------------------- #
#---- FORMATTED OUTPUT (TO FILE, using model name)               ----
# ----------------------------------------------------------------- #

apollo_saveOutput(model)
</code>```

However, since I need to make many version of the same model with a very small change, I would love to make it into a function. Say like this:

```<code>Apollofunction <- function(beta,variable){
      # ################################################################# #
    #### LOAD LIBRARY AND DEFINE CORE SETTINGS                       ####
    # ################################################################# #

    ### Clear memory
    rm(list = ls())

    ### Load Apollo library
    library(apollo)

    ### Initialise code
    apollo_initialise()

    ### Set core controls
    apollo_control = list(
      modelName       = "OL",
      modelDescr      = "Ordered logit model fitted to attitudinal question in drug choice data",
      indivID         = "ID",
      outputDirectory = "output"
    )

    # ################################################################# #
    #### LOAD DATA AND APPLY ANY TRANSFORMATIONS                     ####
    # ################################################################# #

    ### Loading data from package
    ### if data is to be loaded from a file (e.g. called data.csv),
    ### the code would be: database = read.csv("data.csv",header=TRUE)
    database = ?apollo_drugChoiceData
    ### for data dictionary, use ?apollo_drugChoiceData

    # ################################################################# #
    #### DEFINE MODEL PARAMETERS                                     ####
    # ################################################################# #

    ### Vector of parameters, including any that are kept fixed in estimation
    apollo_beta = c(beta            = 0,
                    beta_university = 0,
                    beta_age_50     = 0,
                    tau_quality_1   =-2,
                    tau_quality_2   =-1,
                    tau_quality_3   = 1,
                    tau_quality_4   = 2)

    ### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
    apollo_fixed = c()

    # ################################################################# #
    #### GROUP AND VALIDATE INPUTS                                   ####
    # ################################################################# #

    apollo_inputs = apollo_validateInputs()

    # ################################################################# #
    #### DEFINE MODEL AND LIKELIHOOD FUNCTION                        ####
    # ################################################################# #

    apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

      ### Attach inputs and detach after function exit
      apollo_attach(apollo_beta, apollo_inputs)
      on.exit(apollo_detach(apollo_beta, apollo_inputs))

      ### Create list of probabilities P
      P = list()

      ### Calculate probabilities using Ordered Logit model
      ol_settings = list(outcomeOrdered = attitude_quality,
                         utility        = beta*variable + beta_university*university_educated + beta_age_50*over_50,
                         tau            = list(tau_quality_1, tau_quality_2, tau_quality_3, tau_quality_4),
                         rows           = (task==1))

      P[["model"]] = apollo_ol(ol_settings, functionality)

      ### Take product across observation for same individual
      P = apollo_panelProd(P, apollo_inputs, functionality)

      ### Prepare and return outputs of function
      P = apollo_prepareProb(P, apollo_inputs, functionality)
      return(P)
    }

    # ################################################################# #
    #### MODEL ESTIMATION                                            ####
    # ################################################################# #

    model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)

    # ################################################################# #
    #### MODEL OUTPUTS                                               ####
    # ################################################################# #

    # ----------------------------------------------------------------- #
    #---- FORMATTED OUTPUT (TO SCREEN)                               ----
    # ----------------------------------------------------------------- #

    apollo_modelOutput(model)

    # ----------------------------------------------------------------- #
    #---- FORMATTED OUTPUT (TO FILE, using model name)               ----
    # ----------------------------------------------------------------- #

    apollo_saveOutput(model)
  }
</code>```

Then I could be able to call this apollo function with multiple variations:

```<code>Apollofunction(beta_reg_user, regular_user)
Apollofunction(etc, etc)
</code>```

However, when I then try to run the function, it gives me the error Error in apollo_validateInputs() :  No variable called database found in user workspace (i.e. global environment).

I then tried to upload the database in the global environment and delete the rm(list = ls()), but then I just get this error: Error in apollo_validateInputs() :  No variable called apollo_beta found in user workspace (i.e. global environment). - so there seems to be a problem with how the function interacts with the global environment. Any way to change that? Maybe it has something to do with the use of "="?

The data is available at the link for OL - the csv file. http://www.apollochoicemodelling.com/examples.html

Body:
Let's say I have this Apollo model (from http://www.apollochoicemodelling.com/files/examples/4%20Ranking%20and%20rating/OL.r)

# ################################################################# #
#### LOAD LIBRARY AND DEFINE CORE SETTINGS                       ####
# ################################################################# #

### Clear memory
rm(list = ls())

### Load Apollo library
library(apollo)

### Initialise code
apollo_initialise()

### Set core controls
apollo_control = list(
  modelName       = "OL",
  modelDescr      = "Ordered logit model fitted to attitudinal question in drug choice data",
  indivID         = "ID",
  outputDirectory = "output"
)

# ################################################################# #
#### LOAD DATA AND APPLY ANY TRANSFORMATIONS                     ####
# ################################################################# #

### Loading data from package
### if data is to be loaded from a file (e.g. called data.csv),
### the code would be: database = read.csv("data.csv",header=TRUE)
database = ?apollo_drugChoiceData
### for data dictionary, use ?apollo_drugChoiceData

# ################################################################# #
#### DEFINE MODEL PARAMETERS                                     ####
# ################################################################# #

### Vector of parameters, including any that are kept fixed in estimation
apollo_beta = c(beta_reg_user   = 0,
                beta_university = 0,
                beta_age_50     = 0,
                tau_quality_1   =-2,
                tau_quality_2   =-1,
                tau_quality_3   = 1,
                tau_quality_4   = 2)

### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
apollo_fixed = c()

# ################################################################# #
#### GROUP AND VALIDATE INPUTS                                   ####
# ################################################################# #

apollo_inputs = apollo_validateInputs()

# ################################################################# #
#### DEFINE MODEL AND LIKELIHOOD FUNCTION                        ####
# ################################################################# #

apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

  ### Attach inputs and detach after function exit
  apollo_attach(apollo_beta, apollo_inputs)
  on.exit(apollo_detach(apollo_beta, apollo_inputs))

  ### Create list of probabilities P
  P = list()

  ### Calculate probabilities using Ordered Logit model
  ol_settings = list(outcomeOrdered = attitude_quality,
                     utility        = beta_reg_user*regular_user + beta_university*university_educated + beta_age_50*over_50,
                     tau            = list(tau_quality_1, tau_quality_2, tau_quality_3, tau_quality_4),
                     rows           = (task==1))

  P[["model"]] = apollo_ol(ol_settings, functionality)

  ### Take product across observation for same individual
  P = apollo_panelProd(P, apollo_inputs, functionality)

  ### Prepare and return outputs of function
  P = apollo_prepareProb(P, apollo_inputs, functionality)
  return(P)
}

# ################################################################# #
#### MODEL ESTIMATION                                            ####
# ################################################################# #

model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)

# ################################################################# #
#### MODEL OUTPUTS                                               ####
# ################################################################# #

# ----------------------------------------------------------------- #
#---- FORMATTED OUTPUT (TO SCREEN)                               ----
# ----------------------------------------------------------------- #

apollo_modelOutput(model)

# ----------------------------------------------------------------- #
#---- FORMATTED OUTPUT (TO FILE, using model name)               ----
# ----------------------------------------------------------------- #

apollo_saveOutput(model)

However, since I need to make many version of the same model with a very small change, I would love to make it into a function. Say like this:

Apollofunction <- function(beta,variable){
      # ################################################################# #
    #### LOAD LIBRARY AND DEFINE CORE SETTINGS                       ####
    # ################################################################# #

    ### Clear memory
    rm(list = ls())

    ### Load Apollo library
    library(apollo)

    ### Initialise code
    apollo_initialise()

    ### Set core controls
    apollo_control = list(
      modelName       = "OL",
      modelDescr      = "Ordered logit model fitted to attitudinal question in drug choice data",
      indivID         = "ID",
      outputDirectory = "output"
    )

    # ################################################################# #
    #### LOAD DATA AND APPLY ANY TRANSFORMATIONS                     ####
    # ################################################################# #

    ### Loading data from package
    ### if data is to be loaded from a file (e.g. called data.csv),
    ### the code would be: database = read.csv("data.csv",header=TRUE)
    database = ?apollo_drugChoiceData
    ### for data dictionary, use ?apollo_drugChoiceData

    # ################################################################# #
    #### DEFINE MODEL PARAMETERS                                     ####
    # ################################################################# #

    ### Vector of parameters, including any that are kept fixed in estimation
    apollo_beta = c(beta            = 0,
                    beta_university = 0,
                    beta_age_50     = 0,
                    tau_quality_1   =-2,
                    tau_quality_2   =-1,
                    tau_quality_3   = 1,
                    tau_quality_4   = 2)

    ### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
    apollo_fixed = c()

    # ################################################################# #
    #### GROUP AND VALIDATE INPUTS                                   ####
    # ################################################################# #

    apollo_inputs = apollo_validateInputs()

    # ################################################################# #
    #### DEFINE MODEL AND LIKELIHOOD FUNCTION                        ####
    # ################################################################# #

    apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

      ### Attach inputs and detach after function exit
      apollo_attach(apollo_beta, apollo_inputs)
      on.exit(apollo_detach(apollo_beta, apollo_inputs))

      ### Create list of probabilities P
      P = list()

      ### Calculate probabilities using Ordered Logit model
      ol_settings = list(outcomeOrdered = attitude_quality,
                         utility        = beta*variable + beta_university*university_educated + beta_age_50*over_50,
                         tau            = list(tau_quality_1, tau_quality_2, tau_quality_3, tau_quality_4),
                         rows           = (task==1))

      P[["model"]] = apollo_ol(ol_settings, functionality)

      ### Take product across observation for same individual
      P = apollo_panelProd(P, apollo_inputs, functionality)

      ### Prepare and return outputs of function
      P = apollo_prepareProb(P, apollo_inputs, functionality)
      return(P)
    }

    # ################################################################# #
    #### MODEL ESTIMATION                                            ####
    # ################################################################# #

    model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)

    # ################################################################# #
    #### MODEL OUTPUTS                                               ####
    # ################################################################# #

    # ----------------------------------------------------------------- #
    #---- FORMATTED OUTPUT (TO SCREEN)                               ----
    # ----------------------------------------------------------------- #

    apollo_modelOutput(model)

    # ----------------------------------------------------------------- #
    #---- FORMATTED OUTPUT (TO FILE, using model name)               ----
    # ----------------------------------------------------------------- #

    apollo_saveOutput(model)
  }

Then I could be able to call this apollo function with multiple variations:

Apollofunction(beta_reg_user, regular_user)
Apollofunction(etc, etc)

However, when I then try to run the function, it gives me the error Error in apollo_validateInputs() :  No variable called database found in user workspace (i.e. global environment).

I then tried to upload the database in the global environment and delete the rm(list = ls()), but then I just get this error: Error in apollo_validateInputs() :  No variable called apollo_beta found in user workspace (i.e. global environment). - so there seems to be a problem with how the function interacts with the global environment. Any way to change that? Maybe it has something to do with the use of "="?

The data is available at the link for OL - the csv file. http://www.apollochoicemodelling.com/examples.html


问:
How to bind the "Go to Line" feature in Visual Studio Code to a single keystroke?
Is there a way to shorten the process of moving the cursor to a specific line in a file using the "Go to Line" feature in Visual Studio Code? Currently, the process involves pressing Ctrl + G, typing the line number in the input box, and pressing Enter. Is there a quicker way to do this, perhaps using a different keybinding or a command palette command?

Any help would be appreciated. Thanks!


问:
AWS Glue Job - create parquet from Data Catalog Table to S3
Im struggling with what supposed to be a simple task (maybe i do not understand whats under the hood in the glue data catalog).

I have an S3 bucket with parquet files

I ran a glue crawler through it to create a catalogue table so i could query it in Athena, etc...

Im trying to create a glue job that retreives the data from that table (catalogue table), partitiones is, and saves in another S3 bucket. I receive this exception every time i use the catalogue as source:

```<code>Py4JJavaError: An error occurred while calling o497.pyWriteDynamicFrame.
: org.apache.spark.SparkException: Job aborted.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

...

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 29 in stage 39.0 failed 4 times, most recent failure: Lost task 29.3 in stage 39.0 (TID 437, 172.34.18.186, executor 1): java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary
    at org.apache.parquet.column.Dictionary.decodeToDouble(Dictionary.java:57)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetDictionary.decodeToDouble(ParquetDictionary.java:46)
    at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.getDouble(OnHeapColumnVector.java:460)

...blabla

</code>```

However, if i use the initial S3 bucket as source, it works. I pulled the data from both sources and compared schemas (they are the same, at least on the surface).

Here is my test glue job code:

```<code>import sys
import pandas as pd
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
import pyspark.sql.functions as f
from awsglue.dynamicframe import DynamicFrame
from pyspark.sql.types import StructField, StructType, StringType,LongType

sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)

## Read the data
from_catalogue = glueContext.create_dynamic_frame.from_catalog(database = "ss-demo-database", table_name = "tlc_green_data", transformation_ctx = "datasource0")

from_s3 = glueContext.create_dynamic_frame.from_options(
    format_options={},
    connection_type="s3",
    format="parquet",
    connection_options={
        "paths": ["s3://test/tlc_green_data/green_tripdata_2014-02.parquet"],
        "recurse": False,
    },
    transformation_ctx="from_s3",
)

from_s3.printSchema()

from_catalogue.printSchema()

S3_location = "s3://test/demo/"

# Result are store back to the S3 bucket (this works if from s3, doesnt work if from catalogue)
datasink = glueContext.write_dynamic_frame_from_options(
               frame=from_catalogue,
               connection_type="s3",
               connection_options={
                  "path": S3_location
               },
               format="parquet",
               format_options = {"compression": "snappy"},
               transformation_ctx ="datasink")

job.commit()

</code>```

I tried converting to spark df and with overwrite option - same thing
I tried only selecting 10 rows - same thing

Your help is greatly appreciated!

Body:
Im struggling with what supposed to be a simple task (maybe i do not understand whats under the hood in the glue data catalog).

I have an S3 bucket with parquet files

I ran a glue crawler through it to create a catalogue table so i could query it in Athena, etc...

Im trying to create a glue job that retreives the data from that table (catalogue table), partitiones is, and saves in another S3 bucket. I receive this exception every time i use the catalogue as source:

Py4JJavaError: An error occurred while calling o497.pyWriteDynamicFrame.
: org.apache.spark.SparkException: Job aborted.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

...

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 29 in stage 39.0 failed 4 times, most recent failure: Lost task 29.3 in stage 39.0 (TID 437, 172.34.18.186, executor 1): java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary
    at org.apache.parquet.column.Dictionary.decodeToDouble(Dictionary.java:57)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetDictionary.decodeToDouble(ParquetDictionary.java:46)
    at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.getDouble(OnHeapColumnVector.java:460)

...blabla

However, if i use the initial S3 bucket as source, it works. I pulled the data from both sources and compared schemas (they are the same, at least on the surface).

Here is my test glue job code:

import sys
import pandas as pd
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
import pyspark.sql.functions as f
from awsglue.dynamicframe import DynamicFrame
from pyspark.sql.types import StructField, StructType, StringType,LongType

sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)

## Read the data
from_catalogue = glueContext.create_dynamic_frame.from_catalog(database = "ss-demo-database", table_name = "tlc_green_data", transformation_ctx = "datasource0")

from_s3 = glueContext.create_dynamic_frame.from_options(
    format_options={},
    connection_type="s3",
    format="parquet",
    connection_options={
        "paths": ["s3://test/tlc_green_data/green_tripdata_2014-02.parquet"],
        "recurse": False,
    },
    transformation_ctx="from_s3",
)

from_s3.printSchema()

from_catalogue.printSchema()

S3_location = "s3://test/demo/"

# Result are store back to the S3 bucket (this works if from s3, doesnt work if from catalogue)
datasink = glueContext.write_dynamic_frame_from_options(
               frame=from_catalogue,
               connection_type="s3",
               connection_options={
                  "path": S3_location
               },
               format="parquet",
               format_options = {"compression": "snappy"},
               transformation_ctx ="datasink")

job.commit()

I tried converting to spark df and with overwrite option - same thing
I tried only selecting 10 rows - same thing

Your help is greatly appreciated!


问:
Localhosting in Visual Studio with IIS isn't using the .dll files from my .NET Website (I'm new to this)
Some Background
I recently inherited a large ASP .NET Visual Basic website (WSP). Unfortunately, I was unable to talk with the person previously working on it and now have a tight deadline. I know general VB code but am new to hosting.

The site itself is running on a server I can access, and updating files there is no problem. Before I update live files though, I'd like to test them locally.

TLDR: I'm brand new to testing ASP .NET projects locally

My Main Issue
The ASP .Net website project I inherited has a Bin folder with all the necessary dll files that seem to be from/are NuGet packages (Still learning here, sorry). When I run ISS Express in Visual Studio to localhost the project with Visual Studio (2022), I'm getting errors as if the packages are missing or not found, despite them being in the Bin folder already. How exactly do I get ISS Express/Visual Studio localhost to use the packages (in the form of dll files) in the Bin folder of the proj?

What I Tried and More Context

Before I fully understood that the dll files were connected to the packages I need, I did install some with the NuGet package manager, and that seemed to solve some problems but not all of them as I didn't have the full package list and was just going off the error messages I was getting and the assemblies in the web.config file for the project. While this does work, if the packages (and the right versions) are already in the Bin folder as dll files, I'd far rather use those!

I've also tried researching to find how I can point ISS/Visual Studio local hosting to the bin folder for packages, but ended up more confused than helped.

TLDR: The packages/dll files in the Bin folder are what I want, but I need help making sense out of how to get Visual Studio to localhost using those dll files


问:
Score SARIMAX model in python
I have coefficients for a SARIMAX model in python with AR1, MA1, an intercept term and two exogenous variables (no seasonal components). The model is built on the timeframe 2011 Q1 to 2020 Q4. Now, I just want to make predictions from 2021 Q1 onwards until 2022 Q4 using the coefficients in hand (without re estimating the model again with the SARIMAX function).

I checked the SARIMAX function in python but I couldn't find a way to simply make forecasts with the available coefficients.

Any help would be much appreciated.

Title: Score SARIMAX model in python


问:
How to use zoomer jquery plugin in product gallery to zoom the large image every time i click on it?
Product gallery view code

```<code><div class="row">

        <div class="col-6">

            <div class="row">
                <div class="col-12">

                    @{
                        string[] AllImages = Model.ProductImage.Split(',');

                }
                    <img src="@AllImages[0]" class="largeimage" style="min-height:300px; max-height:500px;" alt="" />

                </div>
            </div>

            <div class="row">
                <div class="col-12">

                    @for (int i = 0; i<AllImages.Length; i++)
                    {
                        <img src="@AllImages[i]" class="img-thumbnail smallimages" style="max-height:100px;" alt="" />

                    }

                </div>
            </div>

        </div>
</code>```

Jquery section

```<code>@section Scripts{

    <script>

        $(function () {

            $(".smallimages").click(function () {
                var clickedImage = $(this);

                $(".largeimage").attr("src", clickedImage.attr("src"));

             $(".largeimage").imageZoom({ zoom: 300 });
            });

         });</script>
               }
</code>```

I want to zoom the large image every time i click on the existing images but by using this code i only zoom the first selected image next image is not selectable.

Body:
Product gallery view code

                    @{
                        string[] AllImages = Model.ProductImage.Split(',');

                }

                    @for (int i = 0; i<AllImages.Length; i++)
                    {

                    }

Jquery section

@section Scripts{

               }

I want to zoom the large image every time i click on the existing images but by using this code i only zoom the first selected image next image is not selectable.