问:
Spark-nlp: can't load pretrained recognize entity model from disk in pyspark
I have a spark cluster set up and would like to integrate spark-nlp to run named entity recognition. I need to access the model from disk rather than download it from the internet at runtime. I have downloaded the recognize_entities_dl model from the model download page and placed the unzipped files where spark should be able to access it. When I run the following code:

```<code>ner = NerDLModel.pretrained('/path/to/unzipped/files')
</code>```

I see the Can not find the model to download please check the name! message, indicating it can't find the files followed by a stacktrace further down in the code. I've also tried the PretrainedPipeline class with similar results.

A few important details for what they're worth:

spark version: 2.4.4

sparknlp version: 2.3.3

Spark is running in a docker container within a kubernetes pod. I can exec into this container and run commands manually to reproduce the problem. It looks like _internal._GetResourceSize is returning a -1, causing the loader to exit. I also get some warnings about http, but all I'm trying to do is access a local file so not sure what that would have to do with things:

```<code>>>> _internal._GetResourceSize('/path/in/container/recognize_entities_dl_en_2.1.0_2.4_1562946909722', 'en', remote_loc=None).apply()
19/12/02 20:29:03 WARN ApacheUtils: NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
19/12/02 20:29:03 WARN ApacheUtils: NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
'-1'
>>>
</code>```

答:
You are trying to load a pre-trained pipeline inside an annotator. There are two types of pre-trained resources: model and pipeline. The pre-trained model can be load inside the annotator which later will be used inside of a pipeline, however, the pre-trained pipeline can be just loaded easily and used after.

Example of the pre-trained pipeline (online-requires internet):

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States")
)).toDF("id", "text")

// Pay attention, for loading a pre-trained pipeline we use PretrainedPipeline
val pipeline = PretrainedPipeline("recognize_entities_dl", lang="en")

val annotation = pipeline.transform(testData)

annotation.show()

/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.4.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(entity_recognizer_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 6 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|            sentence|               token|          embeddings|                 ner|       entities|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[document, 0, 10...|[[token, 0, 5, Go...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|Donald John Trump...|[[document, 0, 92...|[[document, 0, 92...|[[token, 0, 5, Do...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 16, D...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select("entities.result").show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/

```
<ul>
<li>Example of the pre-trained pipeline (offline-loading a saved pipeline):</li>
</ul>
<pre class="lang-scala prettyprint-override"><code>import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States")
)).toDF("id", "text")

// Here we are loading a pre-trained pipeline we already downloaded manually for offline use

val pipeline = PretrainedPipeline.load("/path/in/container/recognize_entities_dl_en_2.1.0_2.4_1562946909722")

val annotation = pipeline.transform(testData)

annotation.show()

/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.4.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(entity_recognizer_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 6 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|            sentence|               token|          embeddings|                 ner|       ner_converter|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[document, 0, 10...|[[token, 0, 5, Go...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|Donald John Trump...|[[document, 0, 92...|[[document, 0, 92...|[[token, 0, 5, Do...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 16, D...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select("entities.result").show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/
</code>```

Example of loading pre-trained model for NerDLModel

// Online
val ner = NerDLModel.pretrained(name="ner_dl", lang="en")
// Offline - manualy downloaded
val ner = NerDLModel.load("/path/ner_dl_en_2.4.0_2.4_1580251789753")
```

Let me know if you have any questions or problems with your input data and I'll update my answer.

References:

https://github.com/JohnSnowLabs/spark-nlp

https://nlp.johnsnowlabs.com/models

https://github.com/JohnSnowLabs/spark-nlp-workshop

Body:
I have a spark cluster set up and would like to integrate spark-nlp to run named entity recognition. I need to access the model from disk rather than download it from the internet at runtime. I have downloaded the recognize_entities_dl model from the model download page and placed the unzipped files where spark should be able to access it. When I run the following code:

ner = NerDLModel.pretrained('/path/to/unzipped/files')

I see the Can not find the model to download please check the name! message, indicating it can't find the files followed by a stacktrace further down in the code. I've also tried the PretrainedPipeline class with similar results.

A few important details for what they're worth:

spark version: 2.4.4

sparknlp version: 2.3.3

Spark is running in a docker container within a kubernetes pod. I can exec into this container and run commands manually to reproduce the problem. It looks like _internal._GetResourceSize is returning a -1, causing the loader to exit. I also get some warnings about http, but all I'm trying to do is access a local file so not sure what that would have to do with things:

>>> _internal._GetResourceSize('/path/in/container/recognize_entities_dl_en_2.1.0_2.4_1562946909722', 'en', remote_loc=None).apply()
19/12/02 20:29:03 WARN ApacheUtils: NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
19/12/02 20:29:03 WARN ApacheUtils: NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
'-1'
>>>

Body:
You are trying to load a pre-trained pipeline inside an annotator. There are two types of pre-trained resources: model and pipeline. The pre-trained model can be load inside the annotator which later will be used inside of a pipeline, however, the pre-trained pipeline can be just loaded easily and used after.

Example of the pre-trained pipeline (online-requires internet):

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States")
)).toDF("id", "text")

// Pay attention, for loading a pre-trained pipeline we use PretrainedPipeline
val pipeline = PretrainedPipeline("recognize_entities_dl", lang="en")

val annotation = pipeline.transform(testData)

annotation.show()

/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.4.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(entity_recognizer_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 6 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|            sentence|               token|          embeddings|                 ner|       entities|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[document, 0, 10...|[[token, 0, 5, Go...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|Donald John Trump...|[[document, 0, 92...|[[document, 0, 92...|[[token, 0, 5, Do...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 16, D...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select("entities.result").show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/

Example of the pre-trained pipeline (offline-loading a saved pipeline):

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States")
)).toDF("id", "text")

// Here we are loading a pre-trained pipeline we already downloaded manually for offline use

val pipeline = PretrainedPipeline.load("/path/in/container/recognize_entities_dl_en_2.1.0_2.4_1562946909722")

val annotation = pipeline.transform(testData)

annotation.show()

/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.4.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(entity_recognizer_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 6 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|            sentence|               token|          embeddings|                 ner|       ner_converter|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[document, 0, 10...|[[token, 0, 5, Go...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|Donald John Trump...|[[document, 0, 92...|[[document, 0, 92...|[[token, 0, 5, Do...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 16, D...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select("entities.result").show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/

Example of loading pre-trained model for NerDLModel

// Online
val ner = NerDLModel.pretrained(name="ner_dl", lang="en")
// Offline - manualy downloaded
val ner = NerDLModel.load("/path/ner_dl_en_2.4.0_2.4_1580251789753")

Let me know if you have any questions or problems with your input data and I'll update my answer.

References:

https://github.com/JohnSnowLabs/spark-nlp

https://nlp.johnsnowlabs.com/models

https://github.com/JohnSnowLabs/spark-nlp-workshop


来源: stackexchange

问:
How can I "Replace" Exception with my own code?
I would like to use my own catch-exception when exceptions occours.

My test code is like this...

```<code>Public Class Form1
    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
        Try
            Dim sb As System.Text.StringBuilder
            sb.Append("test")

        Catch ex As Exception
            ex.Alert("Yo")

        End Try
    End Sub
End Class

<Serializable>
Public Class Exception
    Inherits System.Exception

    Public Sub Alert(text As String)
        MsgBox(text)
    End Sub

    Public Sub New(message As String)
        MyBase.New(message)
    End Sub

    Public Sub New(message As String, innerException As Exception)
        MyBase.New(message, innerException)
    End Sub

    Public Sub New()
    End Sub

    Protected Sub New(serializationInfo As Runtime.Serialization.SerializationInfo, streamingContext As Runtime.Serialization.StreamingContext)
    End Sub

End Class
</code>```

But it doesn't catch the error.

How can I replace the "Exception" class with my own class?

答:
You've successfully created a custom exception type (though just calling it Exception is likely to lead to confusion, you should give it a more specific name).  But nothing is throwing your exception type.  None of the built-in .NET code was built with your custom exception in mind, so nothing is going to throw that type.

Your code can throw your custom exception type.  For example, throwing an exception based on some condition:

```<code>Try
    If someCondition Then
        Throw New Exception()
    End If
Catch ex As Exception
    ex.Alert("Yo")
End Try
</code>```

Generally something like this would happen deeper in the call stack, such as in a method you've called and not on the same level as the Try/Catch here.  (Consider the general advice that one should not use exceptions for control flow, which is kind of what's happening in this contrived example.)

But the point is that your code would throw your custom exception, built-in .NET code certainly wouldn't do that.

As an aside, coupling your exception type with MsgBox effectively couples it with a specific UI technology.  That may be fine for your needs, but it's mixing concerns that generally shouldn't be mixed.  An exception itself should really just carry information about the problem, the exception handling logic should handle displaying that information, or logging it, or in any way responding to the problem.

Currently this custom exception type doesn't add anything to the information about the problem, it only couples itself to the UI.  So when you consider its usage... All it does is replace MsgBox but does so only in places where you are also able to use MsgBox, so it effectively gains you nothing.  In the long run you may be starting off in the wrong direction in terms of effective exception handling.

ContentLicense: CC BY-SA 4.0

Body:
I would like to use my own catch-exception when exceptions occours.

My test code is like this...

Public Class Form1
    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
        Try
            Dim sb As System.Text.StringBuilder
            sb.Append("test")

        Catch ex As Exception
            ex.Alert("Yo")

        End Try
    End Sub
End Class

<Serializable>
Public Class Exception
    Inherits System.Exception

    Public Sub Alert(text As String)
        MsgBox(text)
    End Sub

    Public Sub New(message As String)
        MyBase.New(message)
    End Sub

    Public Sub New(message As String, innerException As Exception)
        MyBase.New(message, innerException)
    End Sub

    Public Sub New()
    End Sub

    Protected Sub New(serializationInfo As Runtime.Serialization.SerializationInfo, streamingContext As Runtime.Serialization.StreamingContext)
    End Sub

End Class

But it doesn't catch the error.

How can I replace the "Exception" class with my own class?

Body:
You've successfully created a custom exception type (though just calling it Exception is likely to lead to confusion, you should give it a more specific name).  But nothing is throwing your exception type.  None of the built-in .NET code was built with your custom exception in mind, so nothing is going to throw that type.

Your code can throw your custom exception type.  For example, throwing an exception based on some condition:

Try
    If someCondition Then
        Throw New Exception()
    End If
Catch ex As Exception
    ex.Alert("Yo")
End Try

Generally something like this would happen deeper in the call stack, such as in a method you've called and not on the same level as the Try/Catch here.  (Consider the general advice that one should not use exceptions for control flow, which is kind of what's happening in this contrived example.)

But the point is that your code would throw your custom exception, built-in .NET code certainly wouldn't do that.

As an aside, coupling your exception type with MsgBox effectively couples it with a specific UI technology.  That may be fine for your needs, but it's mixing concerns that generally shouldn't be mixed.  An exception itself should really just carry information about the problem, the exception handling logic should handle displaying that information, or logging it, or in any way responding to the problem.

Currently this custom exception type doesn't add anything to the information about the problem, it only couples itself to the UI.  So when you consider its usage... All it does is replace MsgBox but does so only in places where you are also able to use MsgBox, so it effectively gains you nothing.  In the long run you may be starting off in the wrong direction in terms of effective exception handling.


问:
Useful data structure for the following case
Which can be the beste data structures for the following case.

1.Should have operations like search, insert and delete. Mostly searching activities will be there.Around 90% of the operations will be search and rest are delete and insert.

2 Insertion,deletion and searching will  be based on the  key of the objects. Each key will point to a object. The keys will be sorted.

Any suggestion for optimal data structure will be highly appreciated.

答:
Not sure by what you mean with "data structures"

I would suggest MySQL.
Read more here: WikiPedia

ContentLicense: CC BY-SA 3.0


问:
RecyleView showing all items duplicates
I am trying to display posts from a server in listView. So I used recycle-view to achieve that. Everything is working fine except that ll items are displaying twice.

I counted the total fetched items from server, and the count is 5, but adapter.getItemCount is showing 10.

After searching hours on the internet, I tried following :

```<code>@Override
public long getItemId(int position) {
    return position;
}
@Override
public int getItemViewType(int position) {
    return position;
}
</code>```

and

homeFragmentAdapter.setHasStableIds(true);

Below is my fragment...

```<code>package com.example.projectName;

import static android.content.Context.MODE_PRIVATE;
import static android.webkit.ConsoleMessage.MessageLevel.LOG;
import static com.bumptech.glide.load.resource.drawable.DrawableTransitionOptions.withCrossFade;

public class HomeFollowersFragment extends Fragment implements InfiniteScrollListener.OnLoadMoreListener, RecyclerViewItemListener {

    private static final String TAG = "HomeFollowersFragment";
    private static final String URL = "https://api.androidhive.info/json/movies_2017.json";
    private RecyclerView recyclerView;
    private ProgressBar postLoader;
    FFmpeg ffmpeg;
//    private List<Movie> movieList;
//    private HomeAdapter mAdapter;

    private List<PostList> postListGlobal = new ArrayList<>();
    List<VerticalDataModal> verticalDataModals;
    List<HorizontalDataModal> horizontalDataModals;
    private SwipeRefreshLayout swipeMore;
    private InfiniteScrollListener infiniteScrollListener;
    private HomeFragmentAdapter homeFragmentAdapter;

    SharedPreferences sharedPreferences;
    private Boolean isLoggedIn = false;
    private String email = "";
    private String token = "";
    private String userId = "";

    private Dialog customLoader;
    SkeletonScreen skeletonScreen;

    private int pastVisiblesItems, visibleItemCount, totalItemCount;
    private boolean loading = false;
    private EndlessScrollListener scrollListener;

    SharedPreferences sp;
    SharedPreferences.Editor Ed;

    public HomeFollowersFragment() {
        //super();
    }
    /**
     * @return A new instance of fragment HomeFollowersFragment.
     */
    public static HomeFollowersFragment newInstance() {
        return new HomeFollowersFragment();
    }

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
    }

    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
                             Bundle savedInstanceState) {
        // Inflate the layout for this fragment
        View view = inflater.inflate(R.layout.fragment_home, container, false);
//        ((AppCompatActivity) getActivity()).getSupportActionBar().show();

        try{
            sharedPreferences = getActivity().getSharedPreferences("Login", MODE_PRIVATE);
            email = sharedPreferences.getString("email", null);
            token = sharedPreferences.getString("token", null);
            isLoggedIn = sharedPreferences.getBoolean("isLoggedIn", false);
            userId = sharedPreferences.getString("id", null);
        }catch (Exception e){
            e.printStackTrace();
            Log.d("StackError", "StackError: "+e);
        }
        sp = getActivity().getSharedPreferences("Posts", MODE_PRIVATE);
        if(!isLoggedIn || token == null || userId == null){
            Intent intent = new Intent(getActivity(), RegisterActivity.class);
            intent.putExtra("loginFrom", "profile");
            startActivity(intent);
        }

        recyclerView = view.findViewById(R.id.recycler_view);
        postLoader = view.findViewById(R.id.post_loader);
        swipeMore = view.findViewById(R.id.swipe_layout);

        homeFragmentAdapter = new HomeFragmentAdapter(postListGlobal, this, "home");

        if(sp.contains("postListGlobal"))
            skeletonScreen = Skeleton.bind(recyclerView)
                    .adapter(homeFragmentAdapter)
                    .shimmer(true)
                    .angle(20)
                    .frozen(false)
                    .duration(1200)
                    .count(10)
                    .load(R.layout.item_skelton_home_page)
                    .show(); //default count is 10
        RecyclerView.LayoutManager mLayoutManager = new GridLayoutManager(getActivity(), 2);
        StaggeredGridLayoutManager sLayoutManager = new StaggeredGridLayoutManager(2, StaggeredGridLayoutManager.VERTICAL);

        recyclerView.setLayoutManager(sLayoutManager);
        homeFragmentAdapter.setHasStableIds(true);
        recyclerView.setAdapter(homeFragmentAdapter);
        recyclerView.setNestedScrollingEnabled(false);

        customLoader = new Dialog(getActivity(), R.style.crystal_range_seek_bar);
        customLoader.setCancelable(false);
        View loaderView = getLayoutInflater().inflate(R.layout.custom_loading_layout, null);
        customLoader.getWindow().getAttributes().windowAnimations = R.style.crystal_range_seek_bar;
        customLoader.getWindow().setBackgroundDrawableResource(R.color.translucent_black);

        ImageView imageLoader = loaderView.findViewById(R.id.logo_loader);
        Glide.with(this).load(R.drawable.logo_loader).into(imageLoader);
        customLoader.setContentView(loaderView);

        if(homeFragmentAdapter.getItemCount() == 0 && !loading){
            // server fetchdata
            Log.d(TAG, "no item available..");
            postLoader.setVisibility(View.VISIBLE);
            loading = true;
            fetchStoreItems();
        }else{
            postLoader.setVisibility(View.GONE);
        }
        swipeMore.setOnRefreshListener(new SwipeRefreshLayout.OnRefreshListener() {
            @Override
            public void onRefresh() {
                Log.d(TAG, "on refresh...");
                fetchStoreItems();
            }
        });
        return view;
    }
    @Override
    public void onItemClicked(int position) {
        Log.d(TAG, "click position: "+position);
        Toast.makeText(getActivity(),postListGlobal.get(position).getTitle(),Toast.LENGTH_SHORT).show();
//        Toast.makeText(getActivity(),""+position, Toast.LENGTH_SHORT).show();

    }
    public int getLastVisibleItem(int[] lastVisibleItemPositions) {
        int maxSize = 0;
        for (int i = 0; i < lastVisibleItemPositions.length; i++) {
            if (i == 0) {
                maxSize = lastVisibleItemPositions[i];
            }
            else if (lastVisibleItemPositions[i] > maxSize) {
                maxSize = lastVisibleItemPositions[i];
            }
        }
        return maxSize;
    }
    @Override
    public void onLoadMore() {
        homeFragmentAdapter.addNullData();
        new Handler().postDelayed(new Runnable() {
            @Override
            public void run() {
                homeFragmentAdapter.removeNull();
                Toast.makeText(getContext(), "load more here...", Toast.LENGTH_LONG).show();
//                fetchStoreItems();
                swipeMore.setRefreshing(false);
            }
        }, 2000);
    }
    private void fetchStoreItems() {
        RequestQueue queue = Volley.newRequestQueue(getActivity());
        Log.d(TAG, "Post Data Followers: "+Constant.FETCH_POSTS_API);
        CacheRequest cacheRequest = new CacheRequest(0, Constant.FETCH_POSTS_API, new Response.Listener<NetworkResponse>() {
            @Override
            public void onResponse(NetworkResponse response) {
                try {
                    final String jsonString = new String(response.data,
                            HttpHeaderParser.parseCharset(response.headers));
                    if (response == null) {
                        Toast.makeText(getActivity(), "Couldn't fetch the store items! Pleas try again.", Toast.LENGTH_LONG).show();
                        loading = false;
                        return;
                    }
                    JSONObject postObj = new JSONObject(jsonString);
                    System.out.println("post full data... : " + postObj);
                    if (postObj.getBoolean("Status")) {
                        try {
                            postLoader.setVisibility(View.GONE);
                            JSONArray arrayResponse = postObj.optJSONArray("Data");
                            int dataArrLength = arrayResponse.length();
                            if(dataArrLength == 0){
                                Toast.makeText(getActivity(), "No posts available at this time, you can create yout own post by clicking on mic button", Toast.LENGTH_SHORT).show();
                            }
                            postListGlobal.clear();
                            Log.d(TAG, "Total Posts count: "+dataArrLength);
                            for(int i=0; i<dataArrLength; i++) {
                                try {
                                    JSONObject dataListObj = arrayResponse.optJSONObject(i);
                                    System.out.println("post full data... : " + dataListObj);

                                    JSONObject postDetailObj = dataListObj.optJSONObject("post_detail");
                                    JSONObject followDtatusObj = dataListObj.optJSONObject("follow_status");
                                    JSONArray postFilesArr = dataListObj.optJSONArray("post_files");
                                    JSONObject userDatasObj = postDetailObj.optJSONObject("user");

                                    String userId = userDatasObj.optString("id");
                                    String userName = userDatasObj.optString("email");
                                    String userImage = userDatasObj.optString("email");

                                    boolean followStatus = followDtatusObj.optBoolean("follow");

                                    String postId = postDetailObj.optString("id");
                                    String postTitle = postDetailObj.optString("post_title");
                                    String postDescription = postDetailObj.optString("post_description");
                                    String postCoverUrl = postDetailObj.optString("post_coverurl", "1");
                                    String postViewType = postDetailObj.optString("view_type", "1");
                                    String postAllowComment = postDetailObj.optString("allow_comments", "1");
                                    String postAllowDownload = postDetailObj.optString("allow_download", "1");
                                    String postTotalPost = postDetailObj.optString("total_post", "1");
                                    String postPostSection = postDetailObj.optString("post_section", "image");
                                    String postActiveStatus = postDetailObj.optString("is_active", "1");

                                    String postTotalViews = postDetailObj.optString("total_watched","0");
                                    String postTotalShare = postDetailObj.optString("total_share","0");
                                    String postTotalDownload = postDetailObj.optString("total_download","0");
                                    String postTotalReaction = postDetailObj.optString("total_reaction","0");

                                    String postTotalLike = postDetailObj.optString("total_like","0");
                                    String postTotalSmile = postDetailObj.optString("smile_reaction","0");
                                    String postTotalLaugh = postDetailObj.optString("laugh_reaction","0");
                                    String postTotalSad = postDetailObj.optString("sad_reaction","0");
                                    String postTotalLove = postDetailObj.optString("love_reaction","0");
                                    String postTotalShock = postDetailObj.optString("shock_reaction","0");

                                    int totalPostFiles = Integer.parseInt(postTotalPost);
                                    int postArrLength = postFilesArr.length();
                                    String postImageUrl = null;
                                    String postMusicUrl = null;

                                    String commonUrl = "http://serverName.com/";
                                    if(postArrLength >= 1){
                                        JSONObject dataFilesListObj = postFilesArr.optJSONObject(0);
//                                        System.out.println("post files full data... : " + dataFilesListObj);
                                        String postFileId = dataFilesListObj.optString("id");
                                        postImageUrl = dataFilesListObj.optString("image_file_path");
                                        postMusicUrl = dataFilesListObj.optString("music_file_path");
                                        System.out.println("post files full data... : " + dataFilesListObj);
                                    }
                                    System.out.println("post files full data... : " + commonUrl+postMusicUrl);
                                    System.out.println("post files full data... : " + commonUrl+postImageUrl);
                                    PostList postList = new PostList();
                                    postList.setId(postId);
                                    postList.setTitle(postTitle);
                                    postList.setTotalPost(""+dataArrLength);
                                    postList.setTotalView(postTotalViews);
                                    postList.setTotalReaction(postTotalReaction);
                                    postList.setMusicPath(commonUrl+postMusicUrl);

                                    postList.setImagePath(commonUrl+postImageUrl);
                                    if(postImageUrl == null){
                                        postList.setImagePath("https://amazonBucket.s3.location.amazonaws.com/images/pic1.jpg");
                                    }
                                    postList.setUserId(userId);
                                    postList.setUserName(userName);
                                    postList.setPostDataObject(arrayResponse);
                                    postListGlobal.add(postList);

                                    Log.d(TAG, "Total Posts: "+dataListObj);

                                } catch (Exception e) {
                                    e.printStackTrace();
                                    Log.d(TAG, "Post Data Error1: "+e);
                                    Toast.makeText(getActivity(), "File now found", Toast.LENGTH_LONG).show();
                                    loading = false;
                                }

                            }

                        } catch (Exception e){
                            e.printStackTrace();
                            Log.d(TAG, "Post Data Error2: "+e);
                            Toast.makeText(getActivity(), R.string.server_error, Toast.LENGTH_LONG).show();
                            loading = false;
                        }
                    }else{
                        try {
                            Toast.makeText(getActivity(), new JSONObject(jsonString).getString("Message"), Toast.LENGTH_LONG).show();
                        } catch (JSONException ex) {
                            ex.printStackTrace();
                            Log.d(TAG, "Post Data Error3: "+ex);
                            Toast.makeText(getActivity(), R.string.server_error, Toast.LENGTH_SHORT).show();
                        }
                        loading = false;
                    }

                    // refreshing recycler view
                    homeFragmentAdapter.removeNull();
                    homeFragmentAdapter.addData(postListGlobal);
                    homeFragmentAdapter.notifyDataSetChanged();
//                    save in local memory
//                    saveArrayList(postListGlobal, "postListGlobal");

                } catch (Exception e) {
                    e.printStackTrace();
                    Log.d(TAG, "Post Data Error4: "+e);
                }
                loading = true;
                homeFragmentAdapter.notifyDataSetChanged();
                swipeMore.setRefreshing(false);
            }
        }, new Response.ErrorListener() {
            @Override
            public void onErrorResponse(VolleyError error) {
                Toast.makeText(getActivity(), "onErrorResponse: "+ error, Toast.LENGTH_SHORT).show();
                swipeMore.setRefreshing(false);
                loading = true;
                homeFragmentAdapter.notifyDataSetChanged();
                postLoader.setVisibility(View.VISIBLE);
                loading = false;
                Log.d(TAG, "Post Data Error5: "+error);
            }
        }){
            @Override
            public Map<String, String> getHeaders() throws AuthFailureError {
                Map<String, String>  params = new HashMap<String, String>();
                String finalToken = "Bearer  "+token;
                params.put("Authorization", finalToken);
                params.put("Content-Type", "application/json");

                return params;
            }
        };
        // Add the request to the RequestQueue.
        queue.add(cacheRequest);

    }
    private class CacheRequest extends Request<NetworkResponse> {
        private final Response.Listener<NetworkResponse> mListener;
        private final Response.ErrorListener mErrorListener;

        public CacheRequest(int method, String url, Response.Listener<NetworkResponse> listener, Response.ErrorListener errorListener) {
            super(method, url, errorListener);
            this.mListener = listener;
            this.mErrorListener = errorListener;
        }
        @Override
        protected Response<NetworkResponse> parseNetworkResponse(NetworkResponse response) {
            Cache.Entry cacheEntry = HttpHeaderParser.parseCacheHeaders(response);
            if (cacheEntry == null) {
                cacheEntry = new Cache.Entry();
            }
            final long cacheHitButRefreshed = 3 * 60 * 1000; // in 3 minutes cache will be hit, but also refreshed on background
            final long cacheExpired = 24 * 60 * 60 * 1000; // in 24 hours this cache entry expires completely
            long now = System.currentTimeMillis();
            final long softExpire = now + cacheHitButRefreshed;
            final long ttl = now + cacheExpired;
            cacheEntry.data = response.data;
            cacheEntry.softTtl = softExpire;
            cacheEntry.ttl = ttl;
            String headerValue;
            headerValue = response.headers.get("Date");
            if (headerValue != null) {
                cacheEntry.serverDate = HttpHeaderParser.parseDateAsEpoch(headerValue);
            }
            headerValue = response.headers.get("Last-Modified");
            if (headerValue != null) {
                cacheEntry.lastModified = HttpHeaderParser.parseDateAsEpoch(headerValue);
            }
            cacheEntry.responseHeaders = response.headers;
            return Response.success(response, cacheEntry);
        }
        @Override
        protected void deliverResponse(NetworkResponse response) {
            mListener.onResponse(response);
        }
        @Override
        protected VolleyError parseNetworkError(VolleyError volleyError) {
            Log.d(TAG, "Post Data volleyError: "+volleyError);
            return super.parseNetworkError(volleyError);
        }
        @Override
        public void deliverError(VolleyError error) {
            mErrorListener.onErrorResponse(error);
        }
    }
}
</code>```

and Adapter Class

```<code>package com.example.ProjectName;

public class HomeFragmentAdapter extends RecyclerView.Adapter <HomeFragmentAdapter.HomeViewHolder>{

    //    private ArrayList<Integer> dataList;
    private List<PostList> postListGlobal;
    int VIEW_TYPE_LOADING;
    int VIEW_TYPE_ITEM;
    Context context;
    private RecyclerViewItemListener callback;
    FFmpeg ffmpeg;
    String callingPage;

    public HomeFragmentAdapter(List<PostList> postListGlobal, RecyclerViewItemListener callback, String callingPage) {
        this.postListGlobal = postListGlobal;
        this.callback = callback;
        this.callingPage = callingPage;
//        setHasStableIds(true);
    }

    @NonNull
    @Override
    public HomeFragmentAdapter.HomeViewHolder onCreateViewHolder(@NonNull ViewGroup parent, int viewType) {
        View root = null;
        context = parent.getContext();
        root = LayoutInflater.from(parent.getContext()).inflate(R.layout.adapter_home_tile_list, parent, false);
        return new DataViewHolder(root);
    }

    @Override
    public void onBindViewHolder(@NonNull HomeFragmentAdapter.HomeViewHolder holder, int position) {
        if (holder instanceof DataViewHolder) {
            final PostList postList = postListGlobal.get(position);
            holder.postTitle.setText(postList.getTitle());
            holder.postWatch.setText(postList.getTotalView());
            holder.postReaction.setText(postList.getTotalReaction());

            String imageUrl = postList.getImagePath();
//            String imageUrl = Constant.SERVER_URL+"/"+postList.getImagePath();
            String musicUrl = postList.getMusicPath();
//            String musicUrl = Constant.SERVER_URL+"/"+postList.getMusicPath();
            Log.d(TAG, "Post url: "+imageUrl+" -- "+musicUrl);
//            int totalMusicTime = getDurationVal(musicUrl, "second");
            holder.postTime.setText(postList.getTotalPost());

            holder.thumbnail.setOnClickListener(new View.OnClickListener() {
                @Override
                public void onClick(View view) {
                    callback.onItemClicked(position);
                    Log.d("homeView", "screenName : "+callingPage);
                    if(callingPage.equals("home")){
                        Log.d("homeView", "screenName : "+position);
                        Intent intent = new Intent(context, MainViewActivity.class);
                        intent.putExtra("loginFrom", "homeView");
                        intent.putExtra("postDataObj", postList.getPostDataObject().toString());
                        intent.putExtra("postPosition", ""+position);
                        intent.putExtra("tabId", "1");
                        context.startActivity(intent);
                    }
                }
            });

            Drawable mDefaultBackground = context.getResources().getDrawable(R.drawable.influencers);
            CircularProgressDrawable circularProgressDrawable = new CircularProgressDrawable(context);
            circularProgressDrawable.setStrokeWidth(5f);
            Glide.with(context)
                    .load(imageUrl)
                    .listener(new RequestListener<Drawable>() {
                        @Override
                        public boolean onLoadFailed(@Nullable GlideException e, Object model, Target<Drawable> target, boolean isFirstResource) {
//                            progressBar.setVisibility(View.GONE);
                            return false;
                        }

                        @Override
                        public boolean onResourceReady(Drawable resource, Object model, Target<Drawable> target, DataSource dataSource, boolean isFirstResource) {
//                            progressBar.setVisibility(View.GONE);
                            return false;
                        }
                    })
                    .error(mDefaultBackground)
                    .into(holder.thumbnail);
        }else{
            //Do whatever you want. Or nothing !!
        }
    }

    @Override
    public int getItemCount() {
        return postListGlobal.size();
    }

    class DataViewHolder extends HomeViewHolder {
        public DataViewHolder(View itemView) {
            super(itemView);
        }

    }

    class ProgressViewHolder extends HomeViewHolder {
        public ProgressViewHolder(View itemView) {
            super(itemView);
        }
    }

    class HomeViewHolder extends RecyclerView.ViewHolder {
        public TextView postTitle, postTime, postWatch, postReaction;
        public ImageView thumbnail;
        public HomeViewHolder(View itemView) {
            super(itemView);
            postTitle = itemView.findViewById(R.id.post_title);
            postTime = itemView.findViewById(R.id.total_time);
            postWatch = itemView.findViewById(R.id.total_watch);
            postReaction = itemView.findViewById(R.id.total_reaction);
            thumbnail = itemView.findViewById(R.id.thumbnail);
    }
    @Override
    public long getItemId(int position) {
        return position;
    }
    @Override
    public int getItemViewType(int position) {
        return position;
    }
    public void addNullData() {
    }
    public void removeNull() {
        notifyItemRemoved(postListGlobal.size());
    }
    public void addData(List<PostList> postLists) {
        postListGlobal.addAll(postLists);
        notifyDataSetChanged();
    }
}
</code>```

After trying everything, I was still not able to resolve the issue. Any help/suggestions are welcome. Let me know If I left out any needed code--if so I can update it here.

答:
```<code>`postListGlobal.add(postList);` below this line add ` homeFragmentAdapter.notifyDataSetChanged();` and remove `  homeFragmentAdapter.removeNull(); homeFragmentAdapter.addData(postListGlobal);homeFragmentAdapter.notifyDataSetChanged();` this code.Because in this case  list added twice without notifying datasetchange check with your code by removing this.
</code>```

Title: RecyleView showing all items duplicates

Body:
I am trying to display posts from a server in listView. So I used recycle-view to achieve that. Everything is working fine except that ll items are displaying twice.

I counted the total fetched items from server, and the count is 5, but adapter.getItemCount is showing 10.

After searching hours on the internet, I tried following :

@Override
public long getItemId(int position) {
    return position;
}
@Override
public int getItemViewType(int position) {
    return position;
}

and

homeFragmentAdapter.setHasStableIds(true);

Below is my fragment...

package com.example.projectName;

import static android.content.Context.MODE_PRIVATE;
import static android.webkit.ConsoleMessage.MessageLevel.LOG;
import static com.bumptech.glide.load.resource.drawable.DrawableTransitionOptions.withCrossFade;

public class HomeFollowersFragment extends Fragment implements InfiniteScrollListener.OnLoadMoreListener, RecyclerViewItemListener {

    private static final String TAG = "HomeFollowersFragment";
    private static final String URL = "https://api.androidhive.info/json/movies_2017.json";
    private RecyclerView recyclerView;
    private ProgressBar postLoader;
    FFmpeg ffmpeg;
//    private List<Movie> movieList;
//    private HomeAdapter mAdapter;

    private List<PostList> postListGlobal = new ArrayList<>();
    List<VerticalDataModal> verticalDataModals;
    List<HorizontalDataModal> horizontalDataModals;
    private SwipeRefreshLayout swipeMore;
    private InfiniteScrollListener infiniteScrollListener;
    private HomeFragmentAdapter homeFragmentAdapter;

    SharedPreferences sharedPreferences;
    private Boolean isLoggedIn = false;
    private String email = "";
    private String token = "";
    private String userId = "";

    private Dialog customLoader;
    SkeletonScreen skeletonScreen;

    private int pastVisiblesItems, visibleItemCount, totalItemCount;
    private boolean loading = false;
    private EndlessScrollListener scrollListener;

    SharedPreferences sp;
    SharedPreferences.Editor Ed;

    public HomeFollowersFragment() {
        //super();
    }
    /**
     * @return A new instance of fragment HomeFollowersFragment.
     */
    public static HomeFollowersFragment newInstance() {
        return new HomeFollowersFragment();
    }

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
    }

    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
                             Bundle savedInstanceState) {
        // Inflate the layout for this fragment
        View view = inflater.inflate(R.layout.fragment_home, container, false);
//        ((AppCompatActivity) getActivity()).getSupportActionBar().show();

        try{
            sharedPreferences = getActivity().getSharedPreferences("Login", MODE_PRIVATE);
            email = sharedPreferences.getString("email", null);
            token = sharedPreferences.getString("token", null);
            isLoggedIn = sharedPreferences.getBoolean("isLoggedIn", false);
            userId = sharedPreferences.getString("id", null);
        }catch (Exception e){
            e.printStackTrace();
            Log.d("StackError", "StackError: "+e);
        }
        sp = getActivity().getSharedPreferences("Posts", MODE_PRIVATE);
        if(!isLoggedIn || token == null || userId == null){
            Intent intent = new Intent(getActivity(), RegisterActivity.class);
            intent.putExtra("loginFrom", "profile");
            startActivity(intent);
        }

        recyclerView = view.findViewById(R.id.recycler_view);
        postLoader = view.findViewById(R.id.post_loader);
        swipeMore = view.findViewById(R.id.swipe_layout);

        homeFragmentAdapter = new HomeFragmentAdapter(postListGlobal, this, "home");

        if(sp.contains("postListGlobal"))
            skeletonScreen = Skeleton.bind(recyclerView)
                    .adapter(homeFragmentAdapter)
                    .shimmer(true)
                    .angle(20)
                    .frozen(false)
                    .duration(1200)
                    .count(10)
                    .load(R.layout.item_skelton_home_page)
                    .show(); //default count is 10
        RecyclerView.LayoutManager mLayoutManager = new GridLayoutManager(getActivity(), 2);
        StaggeredGridLayoutManager sLayoutManager = new StaggeredGridLayoutManager(2, StaggeredGridLayoutManager.VERTICAL);

        recyclerView.setLayoutManager(sLayoutManager);
        homeFragmentAdapter.setHasStableIds(true);
        recyclerView.setAdapter(homeFragmentAdapter);
        recyclerView.setNestedScrollingEnabled(false);

        customLoader = new Dialog(getActivity(), R.style.crystal_range_seek_bar);
        customLoader.setCancelable(false);
        View loaderView = getLayoutInflater().inflate(R.layout.custom_loading_layout, null);
        customLoader.getWindow().getAttributes().windowAnimations = R.style.crystal_range_seek_bar;
        customLoader.getWindow().setBackgroundDrawableResource(R.color.translucent_black);

        ImageView imageLoader = loaderView.findViewById(R.id.logo_loader);
        Glide.with(this).load(R.drawable.logo_loader).into(imageLoader);
        customLoader.setContentView(loaderView);

        if(homeFragmentAdapter.getItemCount() == 0 && !loading){
            // server fetchdata
            Log.d(TAG, "no item available..");
            postLoader.setVisibility(View.VISIBLE);
            loading = true;
            fetchStoreItems();
        }else{
            postLoader.setVisibility(View.GONE);
        }
        swipeMore.setOnRefreshListener(new SwipeRefreshLayout.OnRefreshListener() {
            @Override
            public void onRefresh() {
                Log.d(TAG, "on refresh...");
                fetchStoreItems();
            }
        });
        return view;
    }
    @Override
    public void onItemClicked(int position) {
        Log.d(TAG, "click position: "+position);
        Toast.makeText(getActivity(),postListGlobal.get(position).getTitle(),Toast.LENGTH_SHORT).show();
//        Toast.makeText(getActivity(),""+position, Toast.LENGTH_SHORT).show();

    }
    public int getLastVisibleItem(int[] lastVisibleItemPositions) {
        int maxSize = 0;
        for (int i = 0; i < lastVisibleItemPositions.length; i++) {
            if (i == 0) {
                maxSize = lastVisibleItemPositions[i];
            }
            else if (lastVisibleItemPositions[i] > maxSize) {
                maxSize = lastVisibleItemPositions[i];
            }
        }
        return maxSize;
    }
    @Override
    public void onLoadMore() {
        homeFragmentAdapter.addNullData();
        new Handler().postDelayed(new Runnable() {
            @Override
            public void run() {
                homeFragmentAdapter.removeNull();
                Toast.makeText(getContext(), "load more here...", Toast.LENGTH_LONG).show();
//                fetchStoreItems();
                swipeMore.setRefreshing(false);
            }
        }, 2000);
    }
    private void fetchStoreItems() {
        RequestQueue queue = Volley.newRequestQueue(getActivity());
        Log.d(TAG, "Post Data Followers: "+Constant.FETCH_POSTS_API);
        CacheRequest cacheRequest = new CacheRequest(0, Constant.FETCH_POSTS_API, new Response.Listener<NetworkResponse>() {
            @Override
            public void onResponse(NetworkResponse response) {
                try {
                    final String jsonString = new String(response.data,
                            HttpHeaderParser.parseCharset(response.headers));
                    if (response == null) {
                        Toast.makeText(getActivity(), "Couldn't fetch the store items! Pleas try again.", Toast.LENGTH_LONG).show();
                        loading = false;
                        return;
                    }
                    JSONObject postObj = new JSONObject(jsonString);
                    System.out.println("post full data... : " + postObj);
                    if (postObj.getBoolean("Status")) {
                        try {
                            postLoader.setVisibility(View.GONE);
                            JSONArray arrayResponse = postObj.optJSONArray("Data");
                            int dataArrLength = arrayResponse.length();
                            if(dataArrLength == 0){
                                Toast.makeText(getActivity(), "No posts available at this time, you can create yout own post by clicking on mic button", Toast.LENGTH_SHORT).show();
                            }
                            postListGlobal.clear();
                            Log.d(TAG, "Total Posts count: "+dataArrLength);
                            for(int i=0; i<dataArrLength; i++) {
                                try {
                                    JSONObject dataListObj = arrayResponse.optJSONObject(i);
                                    System.out.println("post full data... : " + dataListObj);

                                    JSONObject postDetailObj = dataListObj.optJSONObject("post_detail");
                                    JSONObject followDtatusObj = dataListObj.optJSONObject("follow_status");
                                    JSONArray postFilesArr = dataListObj.optJSONArray("post_files");
                                    JSONObject userDatasObj = postDetailObj.optJSONObject("user");

                                    String userId = userDatasObj.optString("id");
                                    String userName = userDatasObj.optString("email");
                                    String userImage = userDatasObj.optString("email");

                                    boolean followStatus = followDtatusObj.optBoolean("follow");

                                    String postId = postDetailObj.optString("id");
                                    String postTitle = postDetailObj.optString("post_title");
                                    String postDescription = postDetailObj.optString("post_description");
                                    String postCoverUrl = postDetailObj.optString("post_coverurl", "1");
                                    String postViewType = postDetailObj.optString("view_type", "1");
                                    String postAllowComment = postDetailObj.optString("allow_comments", "1");
                                    String postAllowDownload = postDetailObj.optString("allow_download", "1");
                                    String postTotalPost = postDetailObj.optString("total_post", "1");
                                    String postPostSection = postDetailObj.optString("post_section", "image");
                                    String postActiveStatus = postDetailObj.optString("is_active", "1");

                                    String postTotalViews = postDetailObj.optString("total_watched","0");
                                    String postTotalShare = postDetailObj.optString("total_share","0");
                                    String postTotalDownload = postDetailObj.optString("total_download","0");
                                    String postTotalReaction = postDetailObj.optString("total_reaction","0");

                                    String postTotalLike = postDetailObj.optString("total_like","0");
                                    String postTotalSmile = postDetailObj.optString("smile_reaction","0");
                                    String postTotalLaugh = postDetailObj.optString("laugh_reaction","0");
                                    String postTotalSad = postDetailObj.optString("sad_reaction","0");
                                    String postTotalLove = postDetailObj.optString("love_reaction","0");
                                    String postTotalShock = postDetailObj.optString("shock_reaction","0");

                                    int totalPostFiles = Integer.parseInt(postTotalPost);
                                    int postArrLength = postFilesArr.length();
                                    String postImageUrl = null;
                                    String postMusicUrl = null;

                                    String commonUrl = "http://serverName.com/";
                                    if(postArrLength >= 1){
                                        JSONObject dataFilesListObj = postFilesArr.optJSONObject(0);
//                                        System.out.println("post files full data... : " + dataFilesListObj);
                                        String postFileId = dataFilesListObj.optString("id");
                                        postImageUrl = dataFilesListObj.optString("image_file_path");
                                        postMusicUrl = dataFilesListObj.optString("music_file_path");
                                        System.out.println("post files full data... : " + dataFilesListObj);
                                    }
                                    System.out.println("post files full data... : " + commonUrl+postMusicUrl);
                                    System.out.println("post files full data... : " + commonUrl+postImageUrl);
                                    PostList postList = new PostList();
                                    postList.setId(postId);
                                    postList.setTitle(postTitle);
                                    postList.setTotalPost(""+dataArrLength);
                                    postList.setTotalView(postTotalViews);
                                    postList.setTotalReaction(postTotalReaction);
                                    postList.setMusicPath(commonUrl+postMusicUrl);

                                    postList.setImagePath(commonUrl+postImageUrl);
                                    if(postImageUrl == null){
                                        postList.setImagePath("https://amazonBucket.s3.location.amazonaws.com/images/pic1.jpg");
                                    }
                                    postList.setUserId(userId);
                                    postList.setUserName(userName);
                                    postList.setPostDataObject(arrayResponse);
                                    postListGlobal.add(postList);

                                    Log.d(TAG, "Total Posts: "+dataListObj);

                                } catch (Exception e) {
                                    e.printStackTrace();
                                    Log.d(TAG, "Post Data Error1: "+e);
                                    Toast.makeText(getActivity(), "File now found", Toast.LENGTH_LONG).show();
                                    loading = false;
                                }

                            }

                        } catch (Exception e){
                            e.printStackTrace();
                            Log.d(TAG, "Post Data Error2: "+e);
                            Toast.makeText(getActivity(), R.string.server_error, Toast.LENGTH_LONG).show();
                            loading = false;
                        }
                    }else{
                        try {
                            Toast.makeText(getActivity(), new JSONObject(jsonString).getString("Message"), Toast.LENGTH_LONG).show();
                        } catch (JSONException ex) {
                            ex.printStackTrace();
                            Log.d(TAG, "Post Data Error3: "+ex);
                            Toast.makeText(getActivity(), R.string.server_error, Toast.LENGTH_SHORT).show();
                        }
                        loading = false;
                    }

                    // refreshing recycler view
                    homeFragmentAdapter.removeNull();
                    homeFragmentAdapter.addData(postListGlobal);
                    homeFragmentAdapter.notifyDataSetChanged();
//                    save in local memory
//                    saveArrayList(postListGlobal, "postListGlobal");

                } catch (Exception e) {
                    e.printStackTrace();
                    Log.d(TAG, "Post Data Error4: "+e);
                }
                loading = true;
                homeFragmentAdapter.notifyDataSetChanged();
                swipeMore.setRefreshing(false);
            }
        }, new Response.ErrorListener() {
            @Override
            public void onErrorResponse(VolleyError error) {
                Toast.makeText(getActivity(), "onErrorResponse: "+ error, Toast.LENGTH_SHORT).show();
                swipeMore.setRefreshing(false);
                loading = true;
                homeFragmentAdapter.notifyDataSetChanged();
                postLoader.setVisibility(View.VISIBLE);
                loading = false;
                Log.d(TAG, "Post Data Error5: "+error);
            }
        }){
            @Override
            public Map<String, String> getHeaders() throws AuthFailureError {
                Map<String, String>  params = new HashMap<String, String>();
                String finalToken = "Bearer  "+token;
                params.put("Authorization", finalToken);
                params.put("Content-Type", "application/json");

                return params;
            }
        };
        // Add the request to the RequestQueue.
        queue.add(cacheRequest);

    }
    private class CacheRequest extends Request<NetworkResponse> {
        private final Response.Listener<NetworkResponse> mListener;
        private final Response.ErrorListener mErrorListener;

        public CacheRequest(int method, String url, Response.Listener<NetworkResponse> listener, Response.ErrorListener errorListener) {
            super(method, url, errorListener);
            this.mListener = listener;
            this.mErrorListener = errorListener;
        }
        @Override
        protected Response<NetworkResponse> parseNetworkResponse(NetworkResponse response) {
            Cache.Entry cacheEntry = HttpHeaderParser.parseCacheHeaders(response);
            if (cacheEntry == null) {
                cacheEntry = new Cache.Entry();
            }
            final long cacheHitButRefreshed = 3 * 60 * 1000; // in 3 minutes cache will be hit, but also refreshed on background
            final long cacheExpired = 24 * 60 * 60 * 1000; // in 24 hours this cache entry expires completely
            long now = System.currentTimeMillis();
            final long softExpire = now + cacheHitButRefreshed;
            final long ttl = now + cacheExpired;
            cacheEntry.data = response.data;
            cacheEntry.softTtl = softExpire;
            cacheEntry.ttl = ttl;
            String headerValue;
            headerValue = response.headers.get("Date");
            if (headerValue != null) {
                cacheEntry.serverDate = HttpHeaderParser.parseDateAsEpoch(headerValue);
            }
            headerValue = response.headers.get("Last-Modified");
            if (headerValue != null) {
                cacheEntry.lastModified = HttpHeaderParser.parseDateAsEpoch(headerValue);
            }
            cacheEntry.responseHeaders = response.headers;
            return Response.success(response, cacheEntry);
        }
        @Override
        protected void deliverResponse(NetworkResponse response) {
            mListener.onResponse(response);
        }
        @Override
        protected VolleyError parseNetworkError(VolleyError volleyError) {
            Log.d(TAG, "Post Data volleyError: "+volleyError);
            return super.parseNetworkError(volleyError);
        }
        @Override
        public void deliverError(VolleyError error) {
            mErrorListener.onErrorResponse(error);
        }
    }
}

and Adapter Class

package com.example.ProjectName;

public class HomeFragmentAdapter extends RecyclerView.Adapter <HomeFragmentAdapter.HomeViewHolder>{

    //    private ArrayList<Integer> dataList;
    private List<PostList> postListGlobal;
    int VIEW_TYPE_LOADING;
    int VIEW_TYPE_ITEM;
    Context context;
    private RecyclerViewItemListener callback;
    FFmpeg ffmpeg;
    String callingPage;

    public HomeFragmentAdapter(List<PostList> postListGlobal, RecyclerViewItemListener callback, String callingPage) {
        this.postListGlobal = postListGlobal;
        this.callback = callback;
        this.callingPage = callingPage;
//        setHasStableIds(true);
    }

    @NonNull
    @Override
    public HomeFragmentAdapter.HomeViewHolder onCreateViewHolder(@NonNull ViewGroup parent, int viewType) {
        View root = null;
        context = parent.getContext();
        root = LayoutInflater.from(parent.getContext()).inflate(R.layout.adapter_home_tile_list, parent, false);
        return new DataViewHolder(root);
    }

    @Override
    public void onBindViewHolder(@NonNull HomeFragmentAdapter.HomeViewHolder holder, int position) {
        if (holder instanceof DataViewHolder) {
            final PostList postList = postListGlobal.get(position);
            holder.postTitle.setText(postList.getTitle());
            holder.postWatch.setText(postList.getTotalView());
            holder.postReaction.setText(postList.getTotalReaction());

            String imageUrl = postList.getImagePath();
//            String imageUrl = Constant.SERVER_URL+"/"+postList.getImagePath();
            String musicUrl = postList.getMusicPath();
//            String musicUrl = Constant.SERVER_URL+"/"+postList.getMusicPath();
            Log.d(TAG, "Post url: "+imageUrl+" -- "+musicUrl);
//            int totalMusicTime = getDurationVal(musicUrl, "second");
            holder.postTime.setText(postList.getTotalPost());

            holder.thumbnail.setOnClickListener(new View.OnClickListener() {
                @Override
                public void onClick(View view) {
                    callback.onItemClicked(position);
                    Log.d("homeView", "screenName : "+callingPage);
                    if(callingPage.equals("home")){
                        Log.d("homeView", "screenName : "+position);
                        Intent intent = new Intent(context, MainViewActivity.class);
                        intent.putExtra("loginFrom", "homeView");
                        intent.putExtra("postDataObj", postList.getPostDataObject().toString());
                        intent.putExtra("postPosition", ""+position);
                        intent.putExtra("tabId", "1");
                        context.startActivity(intent);
                    }
                }
            });

            Drawable mDefaultBackground = context.getResources().getDrawable(R.drawable.influencers);
            CircularProgressDrawable circularProgressDrawable = new CircularProgressDrawable(context);
            circularProgressDrawable.setStrokeWidth(5f);
            Glide.with(context)
                    .load(imageUrl)
                    .listener(new RequestListener<Drawable>() {
                        @Override
                        public boolean onLoadFailed(@Nullable GlideException e, Object model, Target<Drawable> target, boolean isFirstResource) {
//                            progressBar.setVisibility(View.GONE);
                            return false;
                        }

                        @Override
                        public boolean onResourceReady(Drawable resource, Object model, Target<Drawable> target, DataSource dataSource, boolean isFirstResource) {
//                            progressBar.setVisibility(View.GONE);
                            return false;
                        }
                    })
                    .error(mDefaultBackground)
                    .into(holder.thumbnail);
        }else{
            //Do whatever you want. Or nothing !!
        }
    }

    @Override
    public int getItemCount() {
        return postListGlobal.size();
    }

    class DataViewHolder extends HomeViewHolder {
        public DataViewHolder(View itemView) {
            super(itemView);
        }

    }

    class ProgressViewHolder extends HomeViewHolder {
        public ProgressViewHolder(View itemView) {
            super(itemView);
        }
    }

    class HomeViewHolder extends RecyclerView.ViewHolder {
        public TextView postTitle, postTime, postWatch, postReaction;
        public ImageView thumbnail;
        public HomeViewHolder(View itemView) {
            super(itemView);
            postTitle = itemView.findViewById(R.id.post_title);
            postTime = itemView.findViewById(R.id.total_time);
            postWatch = itemView.findViewById(R.id.total_watch);
            postReaction = itemView.findViewById(R.id.total_reaction);
            thumbnail = itemView.findViewById(R.id.thumbnail);
    }
    @Override
    public long getItemId(int position) {
        return position;
    }
    @Override
    public int getItemViewType(int position) {
        return position;
    }
    public void addNullData() {
    }
    public void removeNull() {
        notifyItemRemoved(postListGlobal.size());
    }
    public void addData(List<PostList> postLists) {
        postListGlobal.addAll(postLists);
        notifyDataSetChanged();
    }
}

After trying everything, I was still not able to resolve the issue. Any help/suggestions are welcome. Let me know If I left out any needed code--if so I can update it here.


问:
How to iterate a big list to make it smaller for a REST call using Java streams?
I have logic something like this to process in chunks.

```<code>List<String> bigList = getList(); // Returns a big list of 1000 recs
int startIndex=0;
boolean entireListNotProcessed=false;
while(entireListNotProcessed) {
   int endIndex = startIndex + 100;
   if(endIndex > myList.size()-1) {
      endIndex=myList.size()-1;
   }
   List<String> subList= bigList.subList(startIndex, endIndex);
   //call Rest API with subList and update entireListNotProcessed flag...
}
</code>```

Is there a better way of doing this iteration using java streams?

答:
You can use Guava's Iterables.partition or Lists.partition</p>

```<code>Lists.partition(bigList, 100).forEach(subList -> callRestApi(subList));
</code>```

Body:
I have logic something like this to process in chunks.

List<String> bigList = getList(); // Returns a big list of 1000 recs
int startIndex=0;
boolean entireListNotProcessed=false;
while(entireListNotProcessed) {
   int endIndex = startIndex + 100;
   if(endIndex > myList.size()-1) {
      endIndex=myList.size()-1;
   }
   List<String> subList= bigList.subList(startIndex, endIndex);
   //call Rest API with subList and update entireListNotProcessed flag...
}

Is there a better way of doing this iteration using java streams?

Body:
You can use Guava's Iterables.partition or Lists.partition

Lists.partition(bigList, 100).forEach(subList -> callRestApi(subList));


问:
Powershell -Select String (SLS) spliting one line into multiples
I've been looking for a way to perform basic grep with Powershell.

I have an input text file but want to extract only some of them
For instance,

```<code>2020-02-04 05:42:12,132 INFO UselessLog   8=FIX4.4|35=D|49=sender1|56=recipient|....
2020-02-04 05:42:12,134 INFO UselessLog2  8=FIX5.0|35=AB|49=sender2|56=recipient2|....
2020-02-04 05:42:12,136 INFO UselessLog2  8=FIX5.0|35=AB|49=sender3|56=recipient2|....
</code>```

I'm trying to parse the file with

```<code>sls "8=FIX" fixlogs -ca | select -exp line |sls "=sender1" > parseFix
</code>```

However, when I open the parseFix file, sls just transformed one line into multiple ones; e.g., the following two lines represent a single match:

```<code>2020-02-04 05:42:12,132 INFO UselessLog
8=FIX4.4|35=D|49=sender1|56=recipient|...
</code>```

I'm fairly certain there's a config in Powershell to advise not to split but I've investigated and failed to find the one.

I have no access to Unix/Linux server to do a grep magic also I have checked within StackOverFlow but didn't find anything matching my Question.

答:
You need another select -exp line call after your 2nd sls (Select-String) call to make sure that the matching lines are written as-is to the output file:

```<code>sls "8=FIX" fixlogs -ca | select -exp line |sls "=sender1" | select -exp line > parseFix
</code>```

Note: In PowerShell [Core] 7.0+ you can use Select-String -Raw to directly output the matching lines, without the wrapper object that provide metadata about the match that you get by default. That is, there is then no need to extract the lines via separate select -exp Line (Select-Object -ExpandProperty Line) calls.

Why your output lines were broken apart:

Unlike grep or findstr, Select-String outputs match-information objects (of type Microsoft.PowerShell.Commands.MatchInfo) that wrap the matching lines, not just the matching lines themselves.

These objects, because they aren't single values such as a string or a number, loosely speaking, are subject to PowerShell's default output formatting  when you send them to a file with the > operator, which is effectively an alias for Out-File, and this formatting (the same representation you'd see in the console) involves automatic word wrapping, i.e. breaking long lines into multiple ones to fit the display width (even though a file has no inherent width; Out-File -Width would allow you to control this width).

Body:
I've been looking for a way to perform basic grep with Powershell.

I have an input text file but want to extract only some of them
For instance,

2020-02-04 05:42:12,132 INFO UselessLog   8=FIX4.4|35=D|49=sender1|56=recipient|....
2020-02-04 05:42:12,134 INFO UselessLog2  8=FIX5.0|35=AB|49=sender2|56=recipient2|....
2020-02-04 05:42:12,136 INFO UselessLog2  8=FIX5.0|35=AB|49=sender3|56=recipient2|....

I'm trying to parse the file with

sls "8=FIX" fixlogs -ca | select -exp line |sls "=sender1" > parseFix

However, when I open the parseFix file, sls just transformed one line into multiple ones; e.g., the following two lines represent a single match:

2020-02-04 05:42:12,132 INFO UselessLog
8=FIX4.4|35=D|49=sender1|56=recipient|...

I'm fairly certain there's a config in Powershell to advise not to split but I've investigated and failed to find the one.

I have no access to Unix/Linux server to do a grep magic also I have checked within StackOverFlow but didn't find anything matching my Question.

Body:
You need another select -exp line call after your 2nd sls (Select-String) call to make sure that the matching lines are written as-is to the output file:

sls "8=FIX" fixlogs -ca | select -exp line |sls "=sender1" | select -exp line > parseFix

Note: In PowerShell [Core] 7.0+ you can use Select-String -Raw to directly output the matching lines, without the wrapper object that provide metadata about the match that you get by default. That is, there is then no need to extract the lines via separate select -exp Line (Select-Object -ExpandProperty Line) calls.

Why your output lines were broken apart:

Unlike grep or findstr, Select-String outputs match-information objects (of type Microsoft.PowerShell.Commands.MatchInfo) that wrap the matching lines, not just the matching lines themselves.

These objects, because they aren't single values such as a string or a number, loosely speaking, are subject to PowerShell's default output formatting  when you send them to a file with the > operator, which is effectively an alias for Out-File, and this formatting (the same representation you'd see in the console) involves automatic word wrapping, i.e. breaking long lines into multiple ones to fit the display width (even though a file has no inherent width; Out-File -Width would allow you to control this width).


问:
No qualifying bean of type 'ru.spb.repository.UserRepository' available: expected at least 1 bean which qualifies as autowire candidate
Repository class:

```<code> package ru.spb.repository.user;

@Repository
public class AnketUserRepository implements UserRepository {

     @Autowired
     private CrudUserRepository crudRepository;

 }
</code>```

Interface:

```<code>package ru.spb.repository;

public interface UserRepository {
}
</code>```

Service:

```<code>package ru.spb.service;

@Service
public class UserService {

   private final UserRepository repository;

   @Autowired
   public UserService (UserRepository repository){
       this.repository = repository;
   }
}
</code>```

Configs:
in spring-app.xml:

```<code> <context:component-scan base-package="ru.spb.service"/>
</code>```

in spring-db.xml:

```<code> <context:component-scan base-package="ru.spb.repository.user"/>
</code>```

But I catch the following exception:

```<code> NoSuchBeanDefinitionException: No qualifying bean of type 'ru.spb.repository.UserRepository' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
</code>```

I tried to add @Repository on UserRepository-interface, but without success.

答:
Your component scan is configured only to the package of the implementing class, but you are autowiring the interface which is on a different (non component scanned) package.

Move the @Repository annotation to the inteface, and add ru.spb.repository to your component-scan

Body:
Repository class:

 package ru.spb.repository.user;

@Repository
public class AnketUserRepository implements UserRepository {

     @Autowired
     private CrudUserRepository crudRepository;

 }

Interface:

package ru.spb.repository;

public interface UserRepository {
}

Service:

package ru.spb.service;

@Service
public class UserService {

   private final UserRepository repository;

   @Autowired
   public UserService (UserRepository repository){
       this.repository = repository;
   }
}

Configs:
in spring-app.xml:

 <context:component-scan base-package="ru.spb.service"/>

in spring-db.xml:

 <context:component-scan base-package="ru.spb.repository.user"/>

But I catch the following exception:

 NoSuchBeanDefinitionException: No qualifying bean of type 'ru.spb.repository.UserRepository' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}

I tried to add @Repository on UserRepository-interface, but without success.


问:
{Flex: 1} not working with relative positioning
I want the second View to be relatively positioned at -50 below the first View, but the second View does not extend completely at the bottom.

```<code>import React from 'react';
import {View, Text} from 'react-native';

export const OverlayStorybook = () => {
  return (
    <View style={{flex: 1}}>
      <View style={{height: 100, backgroundColor: 'red'}}>
        <Text>OverlayStorybook</Text>
      </View>
      <View style={{flex: 1, backgroundColor: 'blue', top: 0}} />
    </View>
  );
};
</code>```

```<code>import React from 'react';
import {View, Text} from 'react-native';

export const OverlayStorybook = () => {
  return (
    <View style={{flex: 1}}>
      <View style={{height: 100, backgroundColor: 'red'}}>
        <Text>OverlayStorybook</Text>
      </View>
      <View style={{flex: 1, backgroundColor: 'blue', top: -50}} />
    </View>
  );
};
</code>```

You can see at the bottom there are 50 pixels that are white.

答:
This is one option I figured out: https://snack.expo.io/@zvona/full-flex

So removing flex and putting absolute positioning, then adding 50 (instead of subtracting it). Of course that "50" depends on the position you want to fit it in. So basically it's error prone if height of header element is changed.

```<code><View style={{position: 'absolute', backgroundColor: 'rgba(32, 32, 128, 0.8)', top: 50, left: 0, right: 0, bottom: 0}} />
</View>
</code>```

Body:
I want the second View to be relatively positioned at -50 below the first View, but the second View does not extend completely at the bottom.

import React from 'react';
import {View, Text} from 'react-native';

export const OverlayStorybook = () => {
  return (
    <View style={{flex: 1}}>
      <View style={{height: 100, backgroundColor: 'red'}}>
        <Text>OverlayStorybook</Text>
      </View>
      <View style={{flex: 1, backgroundColor: 'blue', top: 0}} />
    </View>
  );
};

import React from 'react';
import {View, Text} from 'react-native';

export const OverlayStorybook = () => {
  return (
    <View style={{flex: 1}}>
      <View style={{height: 100, backgroundColor: 'red'}}>
        <Text>OverlayStorybook</Text>
      </View>
      <View style={{flex: 1, backgroundColor: 'blue', top: -50}} />
    </View>
  );
};

You can see at the bottom there are 50 pixels that are white.

Body:
This is one option I figured out: https://snack.expo.io/@zvona/full-flex

So removing flex and putting absolute positioning, then adding 50 (instead of subtracting it). Of course that "50" depends on the position you want to fit it in. So basically it's error prone if height of header element is changed.

<View style={{position: 'absolute', backgroundColor: 'rgba(32, 32, 128, 0.8)', top: 50, left: 0, right: 0, bottom: 0}} />
</View>


问:
Where's definitive reference on string formatting
does anyone know of a good online resource that simply and definitevly explains how to use the string formatter method...?

I need to write a series of "records" into a set ascii text files. I need to "delimit" each "record" with a cr-lf sequence in a windows 2008 server environment.
Therefore I'm trying to figure out how to add a \r\n character string at the end of each "record". I tried a "record_string.append(CR) and LF" ; but it didn't work.

Thanks much

Guy

答:
Just do the:
record_string = record_string + "\n"

on widnows \n means CR-LF

Or you can use FileWriter to use writeLine(record);


问:
Kubernetes nginx ingress controller as loadbalancer gets random ports
I'm trying to expose kubernetes dashboard publicly via an ingress on a single master bare-metal cluster. The issue is that the LoadBalancer (nginx ingress controller) service I'm using is not opening the 80/443 ports which I would expect it to open/use. Instead it takes some random ports from the 30-32k range. I know I can set this range with --service-node-port-range but I'm quite certain I didn't have to do this a year ago on another server. Am I missing something here?

Currently this is my stack/setup (clean install of Ubuntu 16.04):

Nginx Ingress Controller (installed via helm)

MetalLB

Kubernetes Dashboard

Kubernetes Dashboard Ingress to deploy it publicly on <domain>

Cert-Manager (installed via helm)

k8s-dashboard-ingress.yaml

```<code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: letsencrypt-staging
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  name: kubernetes-dashboard-ingress
  namespace: kubernetes-dashboard
spec:
  rules:
  - host: <domain>
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 443
        path: /
  tls:
  - hosts:
    - <domain>
    secretName: kubernetes-dashboard-staging-cert
</code>```

This is what my kubectl get svc -A looks like:

```<code>NAMESPACE              NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
cert-manager           cert-manager                    ClusterIP      10.101.142.87    <none>          9402/TCP                     23h
cert-manager           cert-manager-webhook            ClusterIP      10.104.104.232   <none>          443/TCP                      23h
default                kubernetes                      ClusterIP      10.96.0.1        <none>          443/TCP                      6d6h
ingress-nginx          nginx-ingress-controller        LoadBalancer   10.100.64.210    10.65.106.240   80:31122/TCP,443:32697/TCP   16m
ingress-nginx          nginx-ingress-default-backend   ClusterIP      10.111.73.136    <none>          80/TCP                       16m
kube-system            kube-dns                        ClusterIP      10.96.0.10       <none>          53/UDP,53/TCP,9153/TCP       6d6h
kubernetes-dashboard   cm-acme-http-solver-kw8zn       NodePort       10.107.15.18     <none>          8089:30074/TCP               140m
kubernetes-dashboard   dashboard-metrics-scraper       ClusterIP      10.96.228.215    <none>          8000/TCP                     5d18h
kubernetes-dashboard   kubernetes-dashboard            ClusterIP      10.99.250.49     <none>          443/TCP                      4d6h
</code>```

Here are some more examples of what's happening:

curl -D- http://<public_ip>:31122 -H 'Host: <domain>'

returns 308, as the protocol is http not https. This is expected

curl -D- http://<public_ip> -H 'Host: <domain>'

curl: (7) Failed to connect to <public_ip> port 80: Connection refused

port 80 is closed

curl -D- --insecure https://10.65.106.240 -H "Host: <domain>"

reaching the dashboard through an internal IP obviously works and I get the correct k8s-dashboard html.

--insecure is due to the let's encrypt not working yet as the acme challenge on port 80 is unreachable.

So to recap, how do I get 2. working? E.g. reaching the service through 80/443?

EDIT: Nginx Ingress Controller .yaml

```<code>apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-02-12T20:20:45Z"
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.30.1
    component: controller
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-controller
  namespace: ingress-nginx
  resourceVersion: "1785264"
  selfLink: /api/v1/namespaces/ingress-nginx/services/nginx-ingress-controller
  uid: b3ce0ff2-ad3e-46f7-bb02-4dc45c1e3a62
spec:
  clusterIP: 10.100.64.210
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 31122
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 32697
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: nginx-ingress
    component: controller
    release: nginx-ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 10.65.106.240
</code>```

EDIT 2: metallb configmap yaml

```<code>kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      -  10.65.106.240-10.65.106.250
</code>```

答:
So, to solve the 2nd question, as I suggested, you can use hostNetwork: true parameter to map container port to the host it is running on. Note that this is not a recommended practice, and you should always avoid to do this, unless you have a reason.

Example:

```<code>apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  hostNetwork: true
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
      hostPort: 80           # this parameter is optional, but recommended when using host network
      name: nginx
</code>```

When I deploy this yaml, I can check where the pod is running and curl that host's port 80.

```<code>root@v1-16-master:~# kubectl get po -owide
NAME                     READY   STATUS    RESTARTS   AGE     IP                NODE             NOMINATED NODE   READINESS GATES
nginx                    1/1     Running   0          105s    10.132.0.50       v1-16-worker-2   <none>           <none>
</code>```

Note: now I know the pod is running on worker node 2. I just need its IP address.

```<code>root@v1-16-master:~# kubectl get no -owide
NAME             STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
v1-16-master     Ready    master   52d   v1.16.4   10.132.0.48   xxxx        Ubuntu 16.04.6 LTS   4.15.0-1052-gcp   docker://19.3.5
v1-16-worker-1   Ready    <none>   52d   v1.16.4   10.132.0.49   xxxx        Ubuntu 16.04.6 LTS   4.15.0-1052-gcp   docker://19.3.5
v1-16-worker-2   Ready    <none>   52d   v1.16.4   10.132.0.50   xxxx        Ubuntu 16.04.6 LTS   4.15.0-1052-gcp   docker://19.3.5
v1-16-worker-3   Ready    <none>   20d   v1.16.4   10.132.0.51   xxxx        Ubuntu 16.04.6 LTS   4.15.0-1052-gcp   docker://19.3.5
root@v1-16-master:~# curl 10.132.0.50 2>/dev/null | grep title
<title>Welcome to nginx!</title>
root@v1-16-master:~# kubectl delete po nginx
pod "nginx" deleted
root@v1-16-master:~# curl 10.132.0.50
curl: (7) Failed to connect to 10.132.0.50 port 80: Connection refused
</code>```

And of course it also works if I go to the public IP on my browser.

Body:
I'm trying to expose kubernetes dashboard publicly via an ingress on a single master bare-metal cluster. The issue is that the LoadBalancer (nginx ingress controller) service I'm using is not opening the 80/443 ports which I would expect it to open/use. Instead it takes some random ports from the 30-32k range. I know I can set this range with --service-node-port-range but I'm quite certain I didn't have to do this a year ago on another server. Am I missing something here?

Currently this is my stack/setup (clean install of Ubuntu 16.04):

Nginx Ingress Controller (installed via helm)

MetalLB

Kubernetes Dashboard

Kubernetes Dashboard Ingress to deploy it publicly on <domain>

Cert-Manager (installed via helm)

k8s-dashboard-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: letsencrypt-staging
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  name: kubernetes-dashboard-ingress
  namespace: kubernetes-dashboard
spec:
  rules:
  - host: <domain>
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 443
        path: /
  tls:
  - hosts:
    - <domain>
    secretName: kubernetes-dashboard-staging-cert

This is what my kubectl get svc -A looks like:

NAMESPACE              NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
cert-manager           cert-manager                    ClusterIP      10.101.142.87    <none>          9402/TCP                     23h
cert-manager           cert-manager-webhook            ClusterIP      10.104.104.232   <none>          443/TCP                      23h
default                kubernetes                      ClusterIP      10.96.0.1        <none>          443/TCP                      6d6h
ingress-nginx          nginx-ingress-controller        LoadBalancer   10.100.64.210    10.65.106.240   80:31122/TCP,443:32697/TCP   16m
ingress-nginx          nginx-ingress-default-backend   ClusterIP      10.111.73.136    <none>          80/TCP                       16m
kube-system            kube-dns                        ClusterIP      10.96.0.10       <none>          53/UDP,53/TCP,9153/TCP       6d6h
kubernetes-dashboard   cm-acme-http-solver-kw8zn       NodePort       10.107.15.18     <none>          8089:30074/TCP               140m
kubernetes-dashboard   dashboard-metrics-scraper       ClusterIP      10.96.228.215    <none>          8000/TCP                     5d18h
kubernetes-dashboard   kubernetes-dashboard            ClusterIP      10.99.250.49     <none>          443/TCP                      4d6h

Here are some more examples of what's happening:

curl -D- http://<public_ip>:31122 -H 'Host: <domain>'

returns 308, as the protocol is http not https. This is expected

curl -D- http://<public_ip> -H 'Host: <domain>'

curl: (7) Failed to connect to <public_ip> port 80: Connection refused

port 80 is closed

curl -D- --insecure https://10.65.106.240 -H "Host: <domain>"

reaching the dashboard through an internal IP obviously works and I get the correct k8s-dashboard html.

--insecure is due to the let's encrypt not working yet as the acme challenge on port 80 is unreachable.

So to recap, how do I get 2. working? E.g. reaching the service through 80/443?

EDIT: Nginx Ingress Controller .yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-02-12T20:20:45Z"
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.30.1
    component: controller
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-controller
  namespace: ingress-nginx
  resourceVersion: "1785264"
  selfLink: /api/v1/namespaces/ingress-nginx/services/nginx-ingress-controller
  uid: b3ce0ff2-ad3e-46f7-bb02-4dc45c1e3a62
spec:
  clusterIP: 10.100.64.210
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 31122
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 32697
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: nginx-ingress
    component: controller
    release: nginx-ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 10.65.106.240

EDIT 2: metallb configmap yaml

kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      -  10.65.106.240-10.65.106.250

Body:
So, to solve the 2nd question, as I suggested, you can use hostNetwork: true parameter to map container port to the host it is running on. Note that this is not a recommended practice, and you should always avoid to do this, unless you have a reason.

Example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  hostNetwork: true
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
      hostPort: 80           # this parameter is optional, but recommended when using host network
      name: nginx

When I deploy this yaml, I can check where the pod is running and curl that host's port 80.

root@v1-16-master:~# kubectl get po -owide
NAME                     READY   STATUS    RESTARTS   AGE     IP                NODE             NOMINATED NODE   READINESS GATES
nginx                    1/1     Running   0          105s    10.132.0.50       v1-16-worker-2   <none>           <none>

Note: now I know the pod is running on worker node 2. I just need its IP address.

root@v1-16-master:~# kubectl get no -owide
NAME             STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
v1-16-master     Ready    master   52d   v1.16.4   10.132.0.48   xxxx        Ubuntu 16.04.6 LTS   4.15.0-1052-gcp   docker://19.3.5
v1-16-worker-1   Ready    <none>   52d   v1.16.4   10.132.0.49   xxxx        Ubuntu 16.04.6 LTS   4.15.0-1052-gcp   docker://19.3.5
v1-16-worker-2   Ready    <none>   52d   v1.16.4   10.132.0.50   xxxx        Ubuntu 16.04.6 LTS   4.15.0-1052-gcp   docker://19.3.5
v1-16-worker-3   Ready    <none>   20d   v1.16.4   10.132.0.51   xxxx        Ubuntu 16.04.6 LTS   4.15.0-1052-gcp   docker://19.3.5
root@v1-16-master:~# curl 10.132.0.50 2>/dev/null | grep title
<title>Welcome to nginx!</title>
root@v1-16-master:~# kubectl delete po nginx
pod "nginx" deleted
root@v1-16-master:~# curl 10.132.0.50
curl: (7) Failed to connect to 10.132.0.50 port 80: Connection refused

And of course it also works if I go to the public IP on my browser.