@SuppressWarnings({ "unchecked", "static-access" }) @Test public void uploadDataWithIdTest() throws Exception{ List<Map<String, Object>> docs = new ArrayList<>(); Map<String,Object> doc = new HashMap<>(); doc.put("id", "id"); docs.add(doc); HttpEntity jsonEntity = new StringEntity("{}", ContentType.APPLICATION_JSON); when(response.getEntity()).thenReturn(jsonEntity); when(restClient.performRequest(anyString(), anyString(), anyMap(), any(HttpEntity.class), Matchers.<Header>anyVararg())).thenReturn(response); ReflectionTestUtils.setField(esManager, "restClient", restClient); when(sl.getStatusCode()).thenReturn(200); when(response.getStatusLine()).thenReturn(sl); esManager.uploadData("index", "type", docs,"id",true); }
public static Map<String, Object> uploadData(String index, String type, List<Map<String, Object>> docs, String loaddate) { Map<String, Object> status = new LinkedHashMap<>(); List<String> errors = new ArrayList<>(); String actionTemplate = "{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\", \"_id\" : \"%s\" } }%n"; LOGGER.info("*********UPLOADING*** {}" ,type); String keys = ConfigManager.getKeyForType(index, type); String[] _keys = keys.split(","); if (null != docs && !docs.isEmpty()) { LOGGER.info("*********# of docs *** {}" , docs.size()); StringBuilder bulkRequest = new StringBuilder(); int i = 0; for (Map<String, Object> doc : docs) { String id = Util.concatenate(doc, _keys, "_"); StringBuilder _doc = new StringBuilder(createESDoc(doc)); _doc.deleteCharAt(_doc.length() - 1); _doc.append(",\"latest\":true,\"_loaddate\":\"" + loaddate + "\" }"); bulkRequest.append(String.format(actionTemplate, index, type, id)); bulkRequest.append(_doc + "\n"); i++; if (i % 1000 == 0 || bulkRequest.toString().getBytes().length / (1024 * 1024) > 5) { bulkUpload(errors, bulkRequest); bulkRequest = new StringBuilder(); } } if (bulkRequest.length() > 0) { bulkUpload(errors, bulkRequest); } LOGGER.info("Updating status"); refresh(index); updateLatestStatus(index, type, loaddate); status.put("uploaded_docs", i); if (!errors.isEmpty()) status.put("errors", errors); } return status; }
ESManager implements Constants { public static Map<String, Object> uploadData(String index, String type, List<Map<String, Object>> docs, String loaddate) { Map<String, Object> status = new LinkedHashMap<>(); List<String> errors = new ArrayList<>(); String actionTemplate = "{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\", \"_id\" : \"%s\" } }%n"; LOGGER.info("*********UPLOADING*** {}" ,type); String keys = ConfigManager.getKeyForType(index, type); String[] _keys = keys.split(","); if (null != docs && !docs.isEmpty()) { LOGGER.info("*********# of docs *** {}" , docs.size()); StringBuilder bulkRequest = new StringBuilder(); int i = 0; for (Map<String, Object> doc : docs) { String id = Util.concatenate(doc, _keys, "_"); StringBuilder _doc = new StringBuilder(createESDoc(doc)); _doc.deleteCharAt(_doc.length() - 1); _doc.append(",\"latest\":true,\"_loaddate\":\"" + loaddate + "\" }"); bulkRequest.append(String.format(actionTemplate, index, type, id)); bulkRequest.append(_doc + "\n"); i++; if (i % 1000 == 0 || bulkRequest.toString().getBytes().length / (1024 * 1024) > 5) { bulkUpload(errors, bulkRequest); bulkRequest = new StringBuilder(); } } if (bulkRequest.length() > 0) { bulkUpload(errors, bulkRequest); } LOGGER.info("Updating status"); refresh(index); updateLatestStatus(index, type, loaddate); status.put("uploaded_docs", i); if (!errors.isEmpty()) status.put("errors", errors); } return status; } }
ESManager implements Constants { public static Map<String, Object> uploadData(String index, String type, List<Map<String, Object>> docs, String loaddate) { Map<String, Object> status = new LinkedHashMap<>(); List<String> errors = new ArrayList<>(); String actionTemplate = "{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\", \"_id\" : \"%s\" } }%n"; LOGGER.info("*********UPLOADING*** {}" ,type); String keys = ConfigManager.getKeyForType(index, type); String[] _keys = keys.split(","); if (null != docs && !docs.isEmpty()) { LOGGER.info("*********# of docs *** {}" , docs.size()); StringBuilder bulkRequest = new StringBuilder(); int i = 0; for (Map<String, Object> doc : docs) { String id = Util.concatenate(doc, _keys, "_"); StringBuilder _doc = new StringBuilder(createESDoc(doc)); _doc.deleteCharAt(_doc.length() - 1); _doc.append(",\"latest\":true,\"_loaddate\":\"" + loaddate + "\" }"); bulkRequest.append(String.format(actionTemplate, index, type, id)); bulkRequest.append(_doc + "\n"); i++; if (i % 1000 == 0 || bulkRequest.toString().getBytes().length / (1024 * 1024) > 5) { bulkUpload(errors, bulkRequest); bulkRequest = new StringBuilder(); } } if (bulkRequest.length() > 0) { bulkUpload(errors, bulkRequest); } LOGGER.info("Updating status"); refresh(index); updateLatestStatus(index, type, loaddate); status.put("uploaded_docs", i); if (!errors.isEmpty()) status.put("errors", errors); } return status; }  }
ESManager implements Constants { public static Map<String, Object> uploadData(String index, String type, List<Map<String, Object>> docs, String loaddate) { Map<String, Object> status = new LinkedHashMap<>(); List<String> errors = new ArrayList<>(); String actionTemplate = "{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\", \"_id\" : \"%s\" } }%n"; LOGGER.info("*********UPLOADING*** {}" ,type); String keys = ConfigManager.getKeyForType(index, type); String[] _keys = keys.split(","); if (null != docs && !docs.isEmpty()) { LOGGER.info("*********# of docs *** {}" , docs.size()); StringBuilder bulkRequest = new StringBuilder(); int i = 0; for (Map<String, Object> doc : docs) { String id = Util.concatenate(doc, _keys, "_"); StringBuilder _doc = new StringBuilder(createESDoc(doc)); _doc.deleteCharAt(_doc.length() - 1); _doc.append(",\"latest\":true,\"_loaddate\":\"" + loaddate + "\" }"); bulkRequest.append(String.format(actionTemplate, index, type, id)); bulkRequest.append(_doc + "\n"); i++; if (i % 1000 == 0 || bulkRequest.toString().getBytes().length / (1024 * 1024) > 5) { bulkUpload(errors, bulkRequest); bulkRequest = new StringBuilder(); } } if (bulkRequest.length() > 0) { bulkUpload(errors, bulkRequest); } LOGGER.info("Updating status"); refresh(index); updateLatestStatus(index, type, loaddate); status.put("uploaded_docs", i); if (!errors.isEmpty()) status.put("errors", errors); } return status; }  static Map<String, Object> uploadData(String index, String type, List<Map<String, Object>> docs, String loaddate); static void refresh(String index); static void uploadData(String index, String type, List<Map<String, Object>> docs, String idKey,
            boolean refresh); static String createESDoc(Map<String, ?> doc); static Response invokeAPI(String method, String endpoint, String payLoad); static void configureIndexAndTypes(String ds, List<Map<String, String>> errorList); static Map<String, Map<String, String>> getExistingInfo(String indexName, String type,
            List<String> filters); @SuppressWarnings("unchecked") static Map<String, Map<String, Map<String, Object>>> fetchCurrentCountStatsForAssetGroups(String date); static void createIndex(String indexName, List<Map<String, String>> errorList); static void createType(String indexName, String typename, List<Map<String, String>> errorList); static void createType(String index, String type, String parent); static void uploadData(String index, String type, List<Map<String, Object>> docs, String[] parentKey); static void deleteOldDocuments(String index, String type, String field, String value); static long updateLoadDate(String index, String type, String accountId, String region, String loaddate,boolean checkLatest); }
ESManager implements Constants { public static Map<String, Object> uploadData(String index, String type, List<Map<String, Object>> docs, String loaddate) { Map<String, Object> status = new LinkedHashMap<>(); List<String> errors = new ArrayList<>(); String actionTemplate = "{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\", \"_id\" : \"%s\" } }%n"; LOGGER.info("*********UPLOADING*** {}" ,type); String keys = ConfigManager.getKeyForType(index, type); String[] _keys = keys.split(","); if (null != docs && !docs.isEmpty()) { LOGGER.info("*********# of docs *** {}" , docs.size()); StringBuilder bulkRequest = new StringBuilder(); int i = 0; for (Map<String, Object> doc : docs) { String id = Util.concatenate(doc, _keys, "_"); StringBuilder _doc = new StringBuilder(createESDoc(doc)); _doc.deleteCharAt(_doc.length() - 1); _doc.append(",\"latest\":true,\"_loaddate\":\"" + loaddate + "\" }"); bulkRequest.append(String.format(actionTemplate, index, type, id)); bulkRequest.append(_doc + "\n"); i++; if (i % 1000 == 0 || bulkRequest.toString().getBytes().length / (1024 * 1024) > 5) { bulkUpload(errors, bulkRequest); bulkRequest = new StringBuilder(); } } if (bulkRequest.length() > 0) { bulkUpload(errors, bulkRequest); } LOGGER.info("Updating status"); refresh(index); updateLatestStatus(index, type, loaddate); status.put("uploaded_docs", i); if (!errors.isEmpty()) status.put("errors", errors); } return status; }  static Map<String, Object> uploadData(String index, String type, List<Map<String, Object>> docs, String loaddate); static void refresh(String index); static void uploadData(String index, String type, List<Map<String, Object>> docs, String idKey,
            boolean refresh); static String createESDoc(Map<String, ?> doc); static Response invokeAPI(String method, String endpoint, String payLoad); static void configureIndexAndTypes(String ds, List<Map<String, String>> errorList); static Map<String, Map<String, String>> getExistingInfo(String indexName, String type,
            List<String> filters); @SuppressWarnings("unchecked") static Map<String, Map<String, Map<String, Object>>> fetchCurrentCountStatsForAssetGroups(String date); static void createIndex(String indexName, List<Map<String, String>> errorList); static void createType(String indexName, String typename, List<Map<String, String>> errorList); static void createType(String index, String type, String parent); static void uploadData(String index, String type, List<Map<String, Object>> docs, String[] parentKey); static void deleteOldDocuments(String index, String type, String field, String value); static long updateLoadDate(String index, String type, String accountId, String region, String loaddate,boolean checkLatest);  }