文章目录在二手车业务线,现阶段无法实现车辆、人、车商信息的在业务审核流程中的数据查重应用,因此业务方为了达成这一目标,基于数据采集和数据查询,应运而生了关系图谱服务。
- 一、系统架构
- 二、业务概述
- 三、方案设计
- 3.1、数据采集
- 3.1.1、SourceInfoContext
- 3.1.2、AbstractInfoHandler
- 3.1.3、AbstractVehicleInfoHandler
- 3.1.4、AbstractCarDealerInfoHandler
- 3.1.5、AbstractPersonInfoHandler
- 3.1.6、SourceInfoQuerier
- 3.1.7、SubmitEventContext
- 3.1.8、AbstractSubmitListener
- 3.1.9、SubmitEventMulticaster
- 3.1.10、HistoryDataSynchronizer
- 3.1.11、RabbitConsumer
- 3.2、数据查重
- 3.2.1、MultiSearchRequest
- 3.2.2、SearchResultRe
- 3.2.3、SearchController
- 3.2.4、SearchQueryExecutor
- 3.2.5、SearchQueryContext
- 3.2.6、AbstractSearchHandler
- 3.2.7、MultiSearchHandler
- 3.2.8、HitQuerierManager
- 3.2.9、HitQuerierContext
- 3.2.10、AbstractHitQuerier
- 3.2.11、AbstractSearchMode
- 4、扩展部分
- 4.1、查重服务请求参数
- 4.2、数据查重字段配置
- 5、总结
从上述系统架构图我们可以看出:
- 1、关系图谱服务主要提供两种能力,数据采集和数据查重。
- 2、数据采集基于接入消息,消费业务线内部的消息通知,基于元数据进行分析并落库。
- 3、数据查重基于HTTP服务,对业务线提供场景的数据查重服务。
- 4、关系图谱服务内部引擎主要包括,数据采集、数据加工、数据切分、数据存储。
- 5、关系图谱服务一期目标基于当前业务量,基于MySQL数据库存储。
从上述业务流程图我们可以看出:
- 1、数据采集,基于接入MQ消息,然后业务逻辑层分析数据,进行数据加工与切分,并存储到数据库,数据切分成业务表,主要包括(b_person_info和b_vehicle_info)。
- 2、数据查重,基于HTTP接口,外部根据关系图谱协议规范,定义查重参数,并返回命中数据。
上图是根据业务梳理出不同场景阶段可以得到的数据,从上图我们可以得到如下结论
- 1、第一列为人信息的分类,包括:主贷人、配偶、担保人等。
- 2、第二列为数据维度,其中单元格不同颜色表示在不同场景阶段可以拿到数据。
- 3、基于MQ消息通知,我们从消息中可以获取到订单号,然后再查询三方数据获取相关数据项。
数据采集主要完成对接MQ消息通知,以及历史数据的同步(上线时必须),然后对数据加工切分并落库。
上图是整个代码设计的UML类图,从上述图可以看出:
- 1、第一层的"应用场景模块"是两个类,一个是消息消费、一个是数据同步。
- 2、第二层的"事件监听模块",是按照业务场景划分,对于不同业务场景,然后监听器通知对应的处理类处理即可。
- 3、第三层的"数据处理模块",按照数据类型(即主贷人、配偶等)抽象出抽象类,以及派生的子类。
该类主要封装数据处理器上下文数据信息。
@Data @AllArgsConstructor @NoArgsConstructor @Builder public class SourceInfoContext { private String appCode; private Long dealerId; private SceneEnum scene; private List3.1.2、AbstractInfoHandlercarDealers; } public enum SceneEnum implements EnumValue { SECOND_BATCH_SUBMIT(1001,"秒批提交"), ADMITTANCE_SUBMIT(1002,"准入提交"), BEFORE_LOAN_SUBMIT(1003,"贷前提交"), VEHICLE_EVALUATE_REMARK(1004,"车辆评估备注"), MERCHANT_ACCOUNT_SUBMIT(2001,"车商账号提交"), ; private int index; private String value; SceneEnum(int index, String value ){ this.value = value; this.index = index; } @Override public int getIndex() { return index; } @Override public String getName() { return value; } public static SceneEnum getByIndex(int index){ return Stream.of(SceneEnum.values()).filter(each -> each.getIndex() == index).findFirst().get(); } public static String getNameByIndex(int index){ SceneEnum find = Stream.of(SceneEnum.values()).filter(each -> each.getIndex() == index).findFirst().get(); return null == find ? "" : find.getName(); } }
该类主要抽象封装数据处理,是车辆、人、车商信息处理模块的上层抽象类。
public interface SourceInfoHandle { void handle(SourceInfoContext context); } @Slf4j public abstract class AbstractInfoHandler3.1.3、AbstractVehicleInfoHandlerimplements SourceInfoHandle{ protected final static int NORMAL_DATA_STATUS_INDEX = DataStatusEnum.NORMAL.getIndex(); protected final static int DELETeD_DATA_STATUS_INDEX = DataStatusEnum.DELETED.getIndex(); protected final static String EMPTY_VALUE = "-"; protected static final ThreadLocal CONTEXT = new ThreadLocal<>(); protected final SourceTypeEnum sourceType; protected boolean removeDuplicate; @Autowired protected DiamondConfigProxy diamondConfigProxy; @Autowired protected SerialNoGenerator serialNoGenerator; @Autowired protected SourceInfoQuerier sourceInfoQuerier; public AbstractInfoHandler(SourceTypeEnum sourceType,boolean removeDuplicate) { this.sourceType = sourceType; this.removeDuplicate = removeDuplicate; } public void execute(SourceInfoContext ctx){ try { CONTEXT.set(ctx); log.info("[execute]appCode={},Handler={},sourceType={}",ctx.getAppCode(),this.getClass().getSimpleName(),sourceType); handle(ctx); } catch (Exception e) { log.error("[execute]appCode={},Handler={},sourceType={},ex",ctx.getAppCode(),this.getClass().getSimpleName(),sourceType,e); }finally { CONTEXT.remove(); } } protected abstract List queryExistsList(T sourceRecord, SourceInfoContext context); protected abstract void store(T sourceRecord,SourceInfoContext context); protected List getPropertiesFromConfig(){ return getPropertiesFromConfigWithScene(CONTEXT.get().getScene()); } protected List getPropertiesFromConfigWithScene(SceneEnum scene){ Map >> mapping = JSON.parseObject(diamondConfigProxy.configGatherRules(), new TypeReference
该类主要抽象封装车辆信息的数据处理,只有一个派生类VehicleInfoHandler。
@Slf4j public abstract class AbstractVehicleInfoHandler extends AbstractInfoHandler3.1.4、AbstractCarDealerInfoHandler{ @Resource protected VehicleInfoService vehicleInfoService; public AbstractVehicleInfoHandler(SourceTypeEnum sourceType) { super(sourceType,Boolean.FALSE); } @Override protected void store(VehicleInfo record,SourceInfoContext context) { if(Objects.nonNull(record)){ List recordList = queryExistsList(record,context); if(CollectionsTools.isNotEmpty(recordList)){ for (VehicleInfo sourceRecord : recordList) { if(isSameWithAnyFields(sourceRecord.with(),record.with())){ return; } } } record.setDataStatus(NORMAL_DATA_STATUS_INDEX); record.setDataCode(serialNoGenerator.generalVehicleInfoDataCode()); record.setScene(context.getScene().getIndex()); record.setSourceType(sourceType.getIndex()); log.info("[store][insertRecord]appCode={},Handler={},sourceType={}",record.getAppCode(),getHandlerClassName(),sourceType); vehicleInfoService.insertRecord(record.with()); } } @Override protected List queryExistsList(VehicleInfo sourceRecord,SourceInfoContext context) { VehicleInfoForm queryForm = VehicleInfoForm.builder() .appCode(context.getAppCode()) .sourceType(sourceType.getIndex()) //.scene(context.getScene().getIndex()) .dataStatus(NORMAL_DATA_STATUS_INDEX) .build(); return vehicleInfoService.queryList(queryForm); } } @Component @Slf4j public class VehicleInfoHandler extends AbstractVehicleInfoHandler{ public VehicleInfoHandler() { super(SourceTypeEnum.VEHICLE_INFO); } @Override public void handle(SourceInfoContext context) { VehicleInfo sourceRecord = sourceInfoQuerier.queryVehicleInfo(context.getAppCode()); SceneEnum scene = context.getScene(); if(SceneEnum.ADMITTANCE_SUBMIT == scene){ super.store(sourceRecord,context); } if(SceneEnum.BEFORE_LOAN_SUBMIT == scene || SceneEnum.VEHICLE_EVALUATE_REMARK == scene){ List recordList = queryExistsList(sourceRecord,context); Map vinMapping = sourceInfoQuerier.queryVinMapping(context.getAppCode()); log.info("[EvaluateRemark],appCode={},vinMapping={}",context.getAppCode(), JSONObject.toJSONString(vinMapping)); recordList.forEach(each -> { CarEvaluationBo apply = vinMapping.get(each.getVin()); if(Objects.nonNull(apply)){ log.info("[EvaluateRemark][Update],appCode={},vin={},remark={}",context.getAppCode(),each.getVin(),apply.getEvaluationRemarks()); each.setEvaluateRemark(StringTools.isNotEmpty(apply.getEvaluationRemarks()) ? apply.getEvaluationRemarks() : EMPTY_VALUE); each.setModifiedTime(TimeTools.createNowTime()); vehicleInfoService.updateByPrimaryKeySelective(each); } }); } } }
该类主要抽象封装车商信息的数据处理,有三个派生类CarDealerLegalPersonInfoHandler、CarDealerPayeeInfoHandler、CarDealerPrincipalInfoHandler。
@Slf4j public abstract class AbstractCarDealerInfoHandler extends AbstractInfoHandler3.1.5、AbstractPersonInfoHandler{ @Resource DealerInfoService dealerInfoService; public AbstractCarDealerInfoHandler(SourceTypeEnum sourceType) { super(sourceType, Boolean.TRUE); } protected abstract List queryList(Long dealerId); boolean shouldStore(DealerInfo record){ return Objects.nonNull(record) && StringTools.isNotEmpty(record.getName()) && !Objects.equals(record.getName(),EMPTY_VALUE); } @Override protected void store(DealerInfo record,SourceInfoContext context) { if(shouldStore(record)){ //是否删除历史雷同记录 if(removeDuplicate){ List recordList = queryExistsList(record,context); if(CollectionsTools.isNotEmpty(recordList)){ for (DealerInfo sourceRecord : recordList) { if(isSameWithAnyFields(sourceRecord.with(),record.with())){ return; } } } } //存入新纪录 record.setDataCode(serialNoGenerator.generalDealerInfoDataCode()); record.setDataStatus(NORMAL_DATA_STATUS_INDEX); record.setScene(context.getScene().getIndex()); record.setSourceType(sourceType.getIndex()); dealerInfoService.insertRecord(record.with()); }else{ log.debug("[execute]store NullObject,Handler={},sourceType={},context={}", this.getClass().getSimpleName(),sourceType, JSONObject.toJSONString(context)); } } @Override public void handle(SourceInfoContext context) { List listWithoutSameFields = new ArrayList<>(); List list; //倘若是车商账户提交推送的消息,则直接获取无需查询;否则,根据车商ID查询信息。 if(CollectionsTools.isEmpty(context.getCarDealers())){ list = queryList(context.getDealerId()); } else { list = context.getCarDealers().stream().filter(each -> each.getSourceType().intValue() == sourceType.getIndex()) .map( source -> DealerInfo.builder() .externalId(Objects.toString(source.getExternalId())) .appCode(Objects.toString(source.getMerchantId())) .sourceType(source.getSourceType()) .name(source.getName()) .idNo(source.getIdNo()) .primaryMobile(source.getCellphone()) .companyAddressDetail(source.getDealerAddress()) .companyName(source.getDealerName()) .creditCardNo(source.getCreditCardNo()) .build() ).collect(Collectors.toList()); } //本次添加的车商,去除某些字段值雷同的项 for (DealerInfo person : list) { if(listWithoutSameFields.stream().filter(r -> isSameWithAnyFields(r, person)).count() > 0){ continue; } listWithoutSameFields.add(person); } log.info("[storeWithBatch]dealerId={},Handler={},sourceType={},list={}",context.getDealerId(),getHandlerClassName(),sourceType, JSONObject.toJSONString(list)); this.storeWithBatch(listWithoutSameFields,context); } @Override protected List queryExistsList(DealerInfo sourceRecord, SourceInfoContext context) { DealerInfoForm queryForm = DealerInfoForm.builder() .sourceType(sourceType.getIndex()) .scene(context.getScene().getIndex()) .appCode(sourceRecord.getAppCode()) .dataStatus(NORMAL_DATA_STATUS_INDEX) .build(); List recordList = dealerInfoService.queryList(queryForm); return recordList; } protected void storeWithBatch(List recordList,SourceInfoContext context){ if(CollectionsTools.isEmpty(recordList)){ return; } for (DealerInfo record : recordList) { store(record, context); } } } @Component public class CarDealerLegalPersonInfoHandler extends AbstractCarDealerInfoHandler { public CarDealerLegalPersonInfoHandler() { super(SourceTypeEnum.CAR_DEALER_LEGAL_PERSON); } @Override protected List queryList(Long dealerId) { return sourceInfoQuerier.queryCarDealerLegalInfo(dealerId); } } @Component public class CarDealerPayeeInfoHandler extends AbstractCarDealerInfoHandler { public CarDealerPayeeInfoHandler() { super(SourceTypeEnum.CAR_DEALER_PAYEE); } @Override protected List queryList(Long dealerId) { return sourceInfoQuerier.queryCarDealerPayeeInfo(dealerId); } } @Component public class CarDealerPrincipalInfoHandler extends AbstractCarDealerInfoHandler { public CarDealerPrincipalInfoHandler() { super(SourceTypeEnum.CAR_DEALER_PRINCIPAL); } @Override protected List queryList(Long dealerId) { return sourceInfoQuerier.queryCarDealerPrincipalInfo(dealerId); } }
该类主要抽象封装人信息的数据处理,有三个派生类CreditorInfoHandler、MateInfoHandler等。
@Slf4j public abstract class AbstractPersonInfoHandler extends AbstractInfoHandler3.1.6、SourceInfoQuerier{ @Autowired protected PersonInfoService personInfoService; public AbstractPersonInfoHandler(SourceTypeEnum sourceType, boolean removeDuplicate) { super(sourceType, removeDuplicate); } protected boolean shouldStore(PersonInfo record){ return Objects.nonNull(record) && StringTools.isNotEmpty(record.getName()) && !Objects.equals(record.getName(),EMPTY_VALUE); } @Override protected void store(PersonInfo record,SourceInfoContext context) { if(shouldStore(record)){ //是否删除历史雷同记录 if(removeDuplicate){ List recordList = queryExistsList(record,context); if(CollectionsTools.isNotEmpty(recordList)){ for (PersonInfo sourceRecord : recordList) { if(isSameWithAnyFields(sourceRecord.with(),record.with())){ return; } } } } //存入新纪录 record.setDataCode(serialNoGenerator.generalPersonInfoDataCode()); record.setDataStatus(NORMAL_DATA_STATUS_INDEX); record.setScene(context.getScene().getIndex()); record.setSourceType(sourceType.getIndex()); personInfoService.insertRecord(record.with()); }else{ log.debug("[execute]store NullObject,Handler={},sourceType={},context={}", this.getClass().getSimpleName(),sourceType, JSONObject.toJSONString(context)); } } @Override protected List queryExistsList(PersonInfo sourceRecord, SourceInfoContext context) { PersonInfoForm queryForm = PersonInfoForm.builder() .sourceType(sourceType.getIndex()) .scene(context.getScene().getIndex()) .appCode(context.getAppCode()) .dataStatus(NORMAL_DATA_STATUS_INDEX) .build(); List recordList = personInfoService.queryList(queryForm); return recordList; } protected void storeWithBatch(List recordList,SourceInfoContext context){ if(CollectionsTools.isEmpty(recordList)){ return; } recordList.forEach(each -> { log.info("[storeWithBatch]appCode={},Handler={},sourceType={}",each.getAppCode(),getHandlerClassName(),sourceType); each.setDataCode(serialNoGenerator.generalPersonInfoDataCode()); each.setDataStatus(NORMAL_DATA_STATUS_INDEX); each.setSourceType(sourceType.getIndex()); each.setScene(context.getScene().getIndex()); each.with(); }); personInfoService.batchInsert(recordList); } } @Component public class CreditorInfoHandler extends AbstractPersonInfoHandler { public CreditorInfoHandler() { super(SourceTypeEnum.CREDITOR_INFO, Boolean.FALSE); } @Override public void handle(SourceInfoContext context) { //查询新的修改记录 PersonInfo record = sourceInfoQuerier.queryCreditorInfo(context.getAppCode()); if(!shouldStore(record)){ return; } record.with(); //删除【当前场景】【之前场景】的【子集数据】 List propertiesSecond = super.getPropertiesFromConfigWithScene(SceneEnum.SECOND_BATCH_SUBMIT); PersonInfoForm propertiesSecondForm = PersonInfoForm.builder().appCode(context.getAppCode()) .sourceType(sourceType.getIndex()) .scene(SceneEnum.SECOND_BATCH_SUBMIT.getIndex()) .dataStatus(NORMAL_DATA_STATUS_INDEX) .build(); List propertiesAdmittance = super.getPropertiesFromConfigWithScene(SceneEnum.ADMITTANCE_SUBMIT); PersonInfoForm propertiesAdmittanceForm = PersonInfoForm.builder().appCode(context.getAppCode()) .sourceType(sourceType.getIndex()) .scene(SceneEnum.ADMITTANCE_SUBMIT.getIndex()) .dataStatus(NORMAL_DATA_STATUS_INDEX) .build(); List propertiesBeforeLoan = super.getPropertiesFromConfigWithScene(SceneEnum.BEFORE_LOAN_SUBMIT); PersonInfoForm propertiesBeforeLoanForm = PersonInfoForm.builder().appCode(context.getAppCode()) .sourceType(sourceType.getIndex()) .scene(SceneEnum.BEFORE_LOAN_SUBMIT.getIndex()) .dataStatus(NORMAL_DATA_STATUS_INDEX) .build(); switch (context.getScene()){ case SECOND_BATCH_SUBMIT: //软删除【秒批提交】子集 copy(record, propertiesSecondForm, propertiesSecond); personInfoService.updateByQuery(propertiesSecondForm); break; case ADMITTANCE_SUBMIT: //软删除【秒批提交】子集 copy(record, propertiesSecondForm, propertiesSecond); personInfoService.updateByQuery(propertiesSecondForm); //软删除【准入提交】子集 propertiesAdmittance.addAll(propertiesSecond); copy(record, propertiesAdmittanceForm, propertiesAdmittance); personInfoService.updateByQuery(propertiesAdmittanceForm); break; case BEFORE_LOAN_SUBMIT: //软删除【秒批提交】子集 copy(record, propertiesSecondForm, propertiesSecond); personInfoService.updateByQuery(propertiesSecondForm); //软删除【准入提交】子集 propertiesAdmittance.addAll(propertiesSecond); copy(record, propertiesAdmittanceForm, propertiesAdmittance); personInfoService.updateByQuery(propertiesAdmittanceForm); //软删除【贷前提交】子集 propertiesBeforeLoan.addAll(propertiesAdmittance); copy(record, propertiesBeforeLoanForm, propertiesBeforeLoan); personInfoService.updateByQuery(propertiesBeforeLoanForm); break; default: break; } //补录新的修改记录 super.store(record,context); } } @Component public class MateInfoHandler extends AbstractPersonInfoHandler { public MateInfoHandler() { super(SourceTypeEnum.MATE, Boolean.TRUE); } @Override public void handle(SourceInfoContext context) { PersonInfo record = sourceInfoQuerier.queryMateInfo(context.getAppCode()); super.store(record,context); } } @Component public class GuarantorInfoHandler extends AbstractPersonInfoHandler { public GuarantorInfoHandler() { super(SourceTypeEnum.GUARANTOR_INFO, Boolean.TRUE); } @Override public void handle(SourceInfoContext context) { PersonInfo record = sourceInfoQuerier.queryGuarantorInfo(context.getAppCode()); super.store(record,context); } }
该类主要抽象对service查询数据的获取,并封装处理器内部所需的POJO。
@Component @Slf4j public class SourceInfoQuerier { //省略依赖注入Service public PersonInfo queryCreditorInfo(String appCode){ CustomerPersonInfo record = customerPersonInfoService.queryByAppCode(appCode); if(Objects.isNull(record)){ return null; } List3.1.7、SubmitEventContextcustomerCardInfos = customerCardInfoService.queryList(CustomerCardInfoForm.builder().appCode(appCode).build()); CustomerCardInfo customerCardInfo = CollectionsTools.isNotEmpty(customerCardInfos) ? customerCardInfos.get(0) : customerCardInfo_NULL; return PersonInfo.builder() .externalId(Objects.toString(record.getId())) .appCode(appCode) .name(record.getName()) .idNo(record.getIdNumber()) .primaryMobile(record.getMobile()) .secondMobile(record.getMobile2()) .companyName(record.getNowCompany()) .companyAddressOrigin(record.getNowUnitAddress()) .companyAddressProvince(record.getNowUnitProvinceName()) .companyAddressCity(record.getNowUnitCityName()) .companyAddressDistrict(record.getNowUnitDistrictName()) .companyAddressDetail(record.getNowUnitAddress()) .companyTelephone(record.getNowUnitTel()) .censusAddressOrigin(record.getHometownAddress()) .censusAddressProvince(record.getHometownProvinceName()) .censusAddressCity(record.getHometownCityName()) .censusAddressDistrict(record.getHometownDistrictName()) .censusAddressDetail(record.getHometownAddress()) .residenceAddressOrigin(record.getResidenceAddress()) .residenceAddressProvince(record.getResidenceProvinceName()) .residenceAddressCity(record.getResidenceCityName()) .residenceAddressDistrict(record.getResidenceDistrictName()) .residenceAddressDetail(record.getResidenceAddress()) .residenceTelephone(record.getResidenceTel()) .creditCardNo(Objects.nonNull(customerCardInfo) ? customerCardInfo.getRepAccountNo() : EMPTY_VALUE) .build(); } public PersonInfo queryGuarantorInfo(String appCode){ CustomerRelatedInfo record = customerRelatedInfoService.queryByAppCode(appCode); if(Objects.isNull(record)){ return null; } return PersonInfo.builder() .externalId(Objects.toString(record.getId())) .appCode(appCode) .name(record.getDbName()) .idNo(record.getDbIdNo()) .primaryMobile(record.getDbMobile()) .companyName(record.getDbNowCompany()) .companyAddressOrigin(record.getDbNowUnitAddress()) .companyAddressProvince(record.getDbNowUnitProvinceName()) .companyAddressCity(record.getDbNowUnitCityName()) .companyAddressDistrict(record.getDbNowUnitDistrictName()) .companyAddressDetail(record.getDbNowUnitAddress()) //.companyTelphone() .censusAddressOrigin(record.getDbAddress()) .censusAddressProvince(record.getDbProvinceName()) .censusAddressCity(record.getDbCityName()) .censusAddressDistrict(record.getDbDistrictName()) .censusAddressDetail(record.getDbAddress()) .build(); } //省略其他成员方法 }
@Data @AllArgsConstructor @NoArgsConstructor @Builder public class SubmitEventContext { @NotNull private String appCode; @NotNull private Long merchantId; @NotNull private SceneEnum scene; private List3.1.8、AbstractSubmitListenercarDealers; }
监听器抽象类,维护监听器通知的数据处理器成员。
public interface SubmitEventListen { void subscribe(SubmitEventContext context); } @Slf4j public abstract class AbstractSubmitListener implements SubmitEventListen { protected SceneEnum sceneEnum; protected List3.1.9、SubmitEventMulticasterhandlers = Lists.newArrayListWithExpectedSize(20); public AbstractSubmitListener(SceneEnum sceneEnum) { this.sceneEnum = sceneEnum; } @Override public void subscribe(SubmitEventContext context) { String appCode = context.getAppCode(); if(context.getScene() == sceneEnum){ log.info("[SubmitEventListen],appCode={},handlers={}",appCode,handlers.toArray()); handlers.forEach(each -> { SourceInfoContext sourceInfoContext = SourceInfoContext.builder() .appCode(appCode) .dealerId(context.getMerchantId()) .carDealers(context.getCarDealers()) .scene(context.getScene()) .build(); each.execute(sourceInfoContext); }); } } } @Component public class AdmittanceSubmitListener extends AbstractSubmitListener { @Resource CreditorInfoHandler creditorInfoHandler; @Resource MateInfoHandler mateInfoHandler; @Resource GuarantorInfoHandler guarantorInfoHandler; @Resource SellerInfoOfOldCarHandler sellerInfoOfOldCarHandler; @Resource EmergencyContactOneInfoHandler emergencyContactOneInfoHandler; @Resource EmergencyContactTwoInfoHandler emergencyContactTwoInfoHandler; @Resource VehicleInfoHandler vehicleInfoHandler; public AdmittanceSubmitListener() { super(SceneEnum.ADMITTANCE_SUBMIT); } @PostConstruct void init(){ this.handlers.add(creditorInfoHandler); this.handlers.add(mateInfoHandler); this.handlers.add(guarantorInfoHandler); this.handlers.add(sellerInfoOfOldCarHandler); this.handlers.add(emergencyContactOneInfoHandler); this.handlers.add(emergencyContactTwoInfoHandler); this.handlers.add(vehicleInfoHandler); } } @Component public class BeforeLoanSubmitListener extends AbstractSubmitListener { @Resource CreditorInfoHandler creditorInfoHandler; @Resource VehicleInfoHandler vehicleInfoHandler; public BeforeLoanSubmitListener() { super(SceneEnum.BEFORE_LOAN_SUBMIT); } @PostConstruct void init(){ this.handlers.add(creditorInfoHandler); this.handlers.add(vehicleInfoHandler); } } @Component public class MerchantSubmitListener extends AbstractSubmitListener { @Resource CarDealerLegalPersonInfoHandler carDealerLegalPersonInfoHandler; @Resource CarDealerPrincipalInfoHandler carDealerPrincipalInfoHandler; @Resource CarDealerPayeeInfoHandler carDealerPayeeInfoHandler; public MerchantSubmitListener() { super(SceneEnum.MERCHANT_ACCOUNT_SUBMIT); } @PostConstruct void init(){ this.handlers.add(carDealerLegalPersonInfoHandler); this.handlers.add(carDealerPrincipalInfoHandler); this.handlers.add(carDealerPayeeInfoHandler); } } @Component public class SecondBatchSubmitListener extends AbstractSubmitListener { @Resource CreditorInfoHandler creditorInfoHandler; @Resource SellerInfoHandler sellerInfoHandler; @Resource SalesManagerInfoHandler salesManagerInfoHandler; public SecondBatchSubmitListener() { super(SceneEnum.SECOND_BATCH_SUBMIT); } @PostConstruct void init(){ this.handlers.add(creditorInfoHandler); this.handlers.add(sellerInfoHandler); this.handlers.add(salesManagerInfoHandler); } } @Component public class VehicleEvaluateRemarkListener extends AbstractSubmitListener { @Resource VehicleInfoHandler vehicleInfoHandler; public VehicleEvaluateRemarkListener() { super(SceneEnum.VEHICLE_EVALUATE_REMARK); } @PostConstruct void init(){ this.handlers.add(vehicleInfoHandler); } }
该类基于观察者模式,通知所有监听器处理,诸如SecondBatchSubmitListener、AdmittanceSubmitListener等。
@Component @Slf4j public class SubmitEventMulticaster { protected final List3.1.10、HistoryDataSynchronizerlisteners = new ArrayList<>(); @Resource SecondBatchSubmitListener secondBatchSubmitListener; @Resource AdmittanceSubmitListener admittanceSubmitListener; @Resource BeforeLoanSubmitListener beforeLoanSubmitListener; @Resource MerchantSubmitListener merchantAccountSubmitEvent; @Resource VehicleEvaluateRemarkListener vehicleEvaluateRemarkListener; @PostConstruct void init(){ listeners.add(secondBatchSubmitListener); listeners.add(admittanceSubmitListener); listeners.add(beforeLoanSubmitListener); listeners.add(merchantAccountSubmitEvent); listeners.add(vehicleEvaluateRemarkListener); } public void execute(SubmitEventContext context){ log.info("[execute]appCode={},ctx={}",context.getAppCode(),JSONObject.toJSONString(context)); listeners.forEach(event -> event.subscribe(context)); } }
历史数据同步器
@Component @Slf4j public class HistoryDataSynchronizer { @Resource SubmitEventMulticaster submitEventMulticaster; @Resource OrderInfoService orderInfoService; @Resource DealerService dealerService; @Resource ExecutorService synchronizeThreadPoolExecutor; public void synchronizeOrders(Request3.1.11、RabbitConsumerrequest){ boolean byScope = CollectionsTools.isNotEmpty(request.getScope()); List recordList = byScope ? orderInfoService.queryTargets(request.getScope()) : orderInfoService.queryAll(); Map orderMap = recordList.stream() .collect(Collectors.toMap(OrderInfo.SimpleEntity::getAppCode,OrderInfo.SimpleEntity::getStatus)); Set appCodes = orderMap.keySet(); if(request.getLimit() > 0){ appCodes = appCodes.stream().limit(request.getLimit()).collect(Collectors.toSet()); } //订单3种场景 List sceneEnumList; List contextList; SubmitEventContext context; StopWatch stopWatch = new StopWatch(); stopWatch.start("synchronizeOrders"); log.info("[synchronizeOrders]count={}",appCodes.size()); LongAdder adder = new LongAdder(); //遍历订单执行 3种场景场景事件 for(String appCode : appCodes){ Integer status = orderMap.get(appCode); sceneEnumList = OrderScene.getScene(status); if(!CollectionUtils.isEmpty(sceneEnumList)){ contextList = new ArrayList<>(sceneEnumList.size()); for (SceneEnum sceneEnum: sceneEnumList) { context = SubmitEventContext.builder() .appCode(appCode) .scene(sceneEnum) .build(); contextList.add(context); adder.increment(); log.info("[synchronizeOrders]current={},appCode={},context={}",adder.intValue(), appCode, JSON.toJSONString(context)); //asyncCall(context); } asyncCall(contextList); }else { log.warn("[synchronizeOrders][sceneEnum] warning!!! appCode={},status={}", appCode, status); } } stopWatch.stop(); log.info("[synchronizeOrders]finished,total={},duration={}",adder.intValue(),stopWatch.getLastTaskTimeMillis() / 1000); } public void synchronizeDealers(Request request){ boolean byScope = CollectionsTools.isNotEmpty(request.getScope()); List ids = byScope ? dealerService.queryTargets(request.getScope()) : dealerService.queryAllIds(); if(request.getLimit() > 0){ ids = ids.stream().limit(request.getLimit()).collect(Collectors.toList()); } SubmitEventContext context; StopWatch stopWatch = new StopWatch(); stopWatch.start(); log.info("[synchronizeDealers]count={}",ids.size()); LongAdder adder = new LongAdder(); for(Long merchantId : ids){ context = SubmitEventContext.builder() .appCode(Objects.toString(merchantId)) .merchantId(merchantId) .scene(SceneEnum.MERCHANT_ACCOUNT_SUBMIT) .build(); adder.increment(); log.info("[synchronizeDealers]current={},merchantId={},context={}",adder.intValue(),merchantId, JSON.toJSONString(context)); asyncCall(context); } stopWatch.stop(); log.info("[synchronizeDealers]finished,total={},,duration={}",adder.intValue(),stopWatch.getLastTaskTimeMillis() / 1000); } void asyncCall(SubmitEventContext context){ try { synchronizeThreadPoolExecutor.execute(() -> submitEventMulticaster.execute(context)); } catch (Exception e) { log.error("[asyncCall]appCode={},context={}",context.getAppCode(),JSON.toJSONString(context)); } } void asyncCall(List contextList){ try { synchronizeThreadPoolExecutor.execute(() -> { for (SubmitEventContext context : contextList) { submitEventMulticaster.execute(context); } }); } catch (Exception e) { log.error("[asyncCall]size={},contextList={}",contextList.size(),JSON.toJSONString(contextList)); } } enum OrderScene{ SECOND_BATCH_SUBMIT(status -> status.intValue() <= 1200, SceneEnum.SECOND_BATCH_SUBMIT), ADMITTANCE_SUBMIT(status -> status.intValue() > 1200 && status.intValue() <= 2100, SceneEnum.SECOND_BATCH_SUBMIT, SceneEnum.ADMITTANCE_SUBMIT), LOAN_BEFORE_SUBMIT(status -> status.intValue() > 2100 && status.intValue() < 9999, SceneEnum.SECOND_BATCH_SUBMIT, SceneEnum.ADMITTANCE_SUBMIT, SceneEnum.BEFORE_LOAN_SUBMIT); private Predicate predicate; private List sceneEnumList; OrderScene(Predicate predicate, SceneEnum... scenes){ this.predicate = predicate; this.sceneEnumList = Arrays.stream(scenes).collect(Collectors.toList()); } public static List getScene(Integer status){ Optional optional = Stream.of(OrderScene.values()).filter(each -> each.predicate.test(status)).findFirst(); return optional.isPresent() ? optional.get().sceneEnumList : null; } } @Data @NoArgsConstructor public static class Request { @ApiModelProperty(value = "限制条数",dataType = "java.lang.Integer") protected int limit; @ApiModelProperty(value = "指定范围",dataType = "java.lang.List ") protected List scope; } } @Data @Builder @AllArgsConstructor @NoArgsConstructor public class CarEvaluationBo { @ApiModelProperty(value="主键",name="id",dataType="java.lang.Integer") private Integer id; @ApiModelProperty(value="订单号",name="app_code",dataType="java.lang.String") private String appCode; @ApiModelProperty(value="vin码",name="vin",dataType="java.lang.String") private String vin; @ApiModelProperty(value="评估备注",name="evaluation_remarks",dataType="java.lang.String") private String evaluationRemarks; }
@Component @Slf4j public class RabbitConsumer { final static SceneEnum SCENE_MERCHANT_ACCOUNT_SUBMIT = SceneEnum.MERCHANT_ACCOUNT_SUBMIT; static String HOST_ADDRESS; static { try { HOST_ADDRESS = InetAddress.getLocalHost().getHostAddress(); log.info("[RabbitConsumer],HOST_ADDRESS={}",HOST_ADDRESS); } catch (UnknownHostException e) { log.error("[RabbitConsumer],init HOST_ADDRESS exception",e); HOST_ADDRESS = "127.0.0.0"; } } @Autowired ExecutorService commonThreadPoolExecutor; @Autowired SubmitEventMulticaster submitEventMulticaster; @Autowired DiamondConfigProxy diamondConfigProxy; @Autowired RedisService redisService; @Autowired TraceLogFacade traceLogFacade; public void subscribeOrderStatus(String message){ final String LOG_TITLE = "subscribeOrderStatus#orderCenterStatus|ordercenter-key-node-message"; String appCode = null; Integer status = null; try { BusMessage busMessage = JSON.parseObject(message,new TypeReference3.2、数据查重(){}); log.info("{}, params={}", LOG_TITLE, JSON.toJSONString(busMessage)); appCode = busMessage.getAppCode(); status = busMessage.getStatus(); // 2. 查询是否在处理的状态范围 OrderStatusAttention orderStatusAttention = diamondConfigProxy.orderStatusAttention(); List attentionStatusScope = orderStatusAttention.getScope(); if(status != null && !attentionStatusScope.contains(status)) { log.debug("{}, 不需要关注状态 appCode={}, status={}", LOG_TITLE, appCode, status); return; } SceneEnum scene = SceneEnum.valueOf(orderStatusAttention.getMapping().get(status)); SubmitEventContext context = SubmitEventContext.builder() .appCode(appCode) .scene(scene) .build(); log.info("{} appCode={}, SubmitEventContext={}", LOG_TITLE, appCode, JSON.toJSONString(context)); //3、记录图谱记录 if(diamondConfigProxy.switchConfig().subscribeOrderStatus){ submitEventMulticaster.execute(context); } //4、记录链路日志 syncSaveTraceRecord(appCode,message,traceLog -> { traceLog.setUrl("[" + HOST_ADDRESS + "]rabbitConsumer.subscribeOrderStatus"); traceLog.setResponseBody(JSONObject.toJSONString(context)); }); } catch (Exception e) { log.error("[subscribeOrderStatus]异常,appCode={},message={}", appCode,JSONObject.toJSONString(message),e); } } public void subscribeMerchantAccountSubmit(String message){ final String LOG_TITLE = "subscribeOrderStatus#subscribeMerchantAccountSubmit|ordercenter-key-node-message"; Long dealerId = null; try { List carDealerList = JSON.parseArray(message,CarDealerInfoDTO.class); log.info("{}, params={}", LOG_TITLE, JSON.toJSONString(carDealerList)); if(CollectionsTools.isEmpty(carDealerList)){ log.warn("{}, params={}", LOG_TITLE, JSON.toJSONString(carDealerList)); return; } dealerId = carDealerList.get(0).getMerchantId(); SubmitEventContext context = SubmitEventContext.builder() .merchantId(dealerId) .carDealers(carDealerList) .scene(SCENE_MERCHANT_ACCOUNT_SUBMIT) .build(); log.info("{} merchantId={}, SubmitEventContext={}", LOG_TITLE, dealerId, JSON.toJSONString(context)); //1、记录图谱记录 if(diamondConfigProxy.switchConfig().subscribeMerchantAccountSubmit){ submitEventMulticaster.execute(context); } //2、记录链路日志 syncSaveTraceRecord(Objects.toString(dealerId),message,traceLog -> { traceLog.setUrl("[" + HOST_ADDRESS + "]rabbitConsumer.subscribeMerchantAccountSubmit"); traceLog.setResponseBody(JSONObject.toJSONString(context)); }); } catch (Exception e) { log.error("[subscribeMerchantAccountSubmit]异常,dealerId={},message={}", dealerId,message,e); } } void syncSaveTraceRecord(String appCode, String message, Consumer caller){ TraceLog traceLog = TraceLog.builder() .appCode(appCode) .target(this.getClass().getPackage().getName() + "." + this.getClass().getSimpleName()) .requestBody(message) .requestTime(TimeTools.createNowTime()) .responseTime(TimeTools.createNowTime()) .traceType(TraceTypeEnum.RABBIT_CONSUMER.getIndex()).build(); caller.accept(traceLog); commonThreadPoolExecutor.execute(() -> traceLogFacade.saveRecord(traceLog)); } @Data static class BusMessage implements Serializable { private Long messageId; private String messageCode; private String channel; private String appCode; //上一个状态 private Integer lastStatus; //当前状态 private Integer status; private String data; private Date sendTime; //初审增加征信类型 private Integer creditAuthType; } }
上图是整个代码设计的UML类图,从上述图可以看出:
- 1、第一层的"API"层,对外提供HTTP接口。
- 2、第二层的"查询执行器",包装API层的请求,并交由查询处理器处理。
- 3、第三层的"查询处理器",抽象类仅派生一种业务场景多条件查询处理器MultiSearchHandler。
- 3、第四层的"命中查询处理器",按照业务场景派生出三个子类PersonInfoHitQuerier、VehicleInfoHitQuerier、DealerInfoHitQuerier,同时提供了两种数据命中模式ExactSearchMode、SimilarSearchMode。
@Data @Builder @NoArgsConstructor @AllArgsConstructor public class MultiSearchRequest { @ApiModelProperty(value="请求ID(可以使用UUID生成)",dataType="java.lang.String") @NotNull(message = "请求ID[requestId]非空") private String requestId; @ApiModelProperty(value="查询条件",dataType="List3.2.2、SearchResultRe") @NotNull(message = "查询条件[conditions]非空") private List conditions; @Data @Builder @NoArgsConstructor @AllArgsConstructor public static class Condition{ @ApiModelProperty(value="查询字段类型",dataType="SourceTypeEnum") private SourceTypeEnum sourceType; @ApiModelProperty(value="查询字段名称",dataType="java.lang.String") @NotNull(message = "查询字段名称[searchFieldName]不能为空!") @NotEmpty(message = "查询字段名称[searchFieldName]不能为空!") private String searchFieldName; @ApiModelProperty(value="模糊查询分数区间",dataType="java.lang.Double") @Size(max = 2, message = "查询字段名称[scoreRange]列表长度应该0-2") private List scoreRange; @ApiModelProperty(value="查询字段输入值",dataType="java.lang.String") @NotNull(message = "查询字段输入值[searchFieldValue]不能为空!") @NotEmpty(message = "查询字段输入值[searchFieldValue]不能为空!") private String searchFieldValue; @ApiModelProperty(value="查询字段描述",dataType="java.lang.String") @NotNull(message = "查询字段描述[searchFieldDesc]不能为空!") @NotEmpty(message = "查询字段描述[searchFieldDesc]不能为空!") private String searchFieldDesc; } } public enum SourceTypeEnum implements EnumValue { CREDITOR_INFO(1,"主贷人"), SELLER_INFO(2,"销售"), SALES_MANAGER_INFO(3,"销售主管"), MATE(4,"配偶"), GUARANTOR_INFO(5,"担保人"), SELLER_INFO_OF_OLD_CAR(6,"二手车卖方"), EMERGENCY_CONTACT_ONE(7,"紧急联系人1"), EMERGENCY_CONTACT_TWO(8,"紧急联系人2"), CAR_DEALER_LEGAL_PERSON(9,"车商法人"), CAR_DEALER_PRINCIPAL(10,"车商负责人"), CAR_DEALER_PAYEE(11,"车商收款人"), VEHICLE_INFO(12,"车辆"), ; private int index; private String value; SourceTypeEnum(int index, String value ){ this.value = value; this.index = index; } @Override public int getIndex() { return index; } @Override public String getName() { return value; } public static SourceTypeEnum getByIndex(int index){ return Stream.of(SourceTypeEnum.values()).filter(each -> each.getIndex() == index).findFirst().get(); } public static String getNameByIndex(int index){ SourceTypeEnum find = Stream.of(SourceTypeEnum.values()).filter(each -> each.getIndex() == index).findFirst().get(); return null == find ? "" : find.getName(); } }
@Data @Builder @NoArgsConstructor @AllArgsConstructor public class SearchResultRe3.2.3、SearchController{ @ApiModelProperty(value="提示信息",dataType="java.lang.String") private String message; @ApiModelProperty(value="查询字段名称",dataType="java.lang.String") private String searchFieldName; @ApiModelProperty(value="查询字段输入值",dataType="java.lang.String") private String searchFieldValue; @ApiModelProperty(value="查询字段描述",dataType="java.lang.String") private String searchFieldDesc; @ApiModelProperty(value="命中条数",dataType="java.lang.Integer") private int hitCount; @ApiModelProperty(value="命中记录",dataType="List ") private List hitRecords; public synchronized void increaseHitCount(int delta){ this.hitCount += delta; } public synchronized void addHits(List hits){ if(!isEmpty(hits)){ if(isEmpty(this.hitRecords)){ this.hitRecords = new ArrayList<>(hits.size()); } this.hitRecords.addAll(hits); } } boolean isEmpty(Collection> collection){ return null == collection || collection.isEmpty(); } @Data @SuperBuilder @NoArgsConstructor @AllArgsConstructor public static class Record{ @ApiModelProperty(value="主键ID",dataType="java.lang.Integer") private Integer id; @ApiModelProperty(value="外部数据id",dataType="java.lang.String") private String externalId; @ApiModelProperty(value="业务单号",dataType="java.lang.String") private String appCode; @ApiModelProperty(value="数据单号",dataType="java.lang.String") private String dataCode; @ApiModelProperty(value="场景",dataType="java.lang.Integer") private Integer scene; private String sceneDesc; @ApiModelProperty(value="来源类型",dataType="java.lang.Integer") private Integer sourceType; private String sourceTypeDesc; } @Data @SuperBuilder @NoArgsConstructor public static class PersonInfoRecord extends Record{ @ApiModelProperty(value="姓名",dataType="java.lang.String") private String name; @ApiModelProperty(value="身份证号",dataType="java.lang.String") private String idNo; @ApiModelProperty(value="手机号",dataType="java.lang.String") private String mobile; @ApiModelProperty(value="银行卡号",dataType="java.lang.String") private String creditCardNo; @ApiModelProperty(value="省",dataType="java.lang.String") private String provinceAddress; @ApiModelProperty(value="市",dataType="java.lang.String") private String cityAddress; @ApiModelProperty(value="区",dataType="java.lang.String") private String districtAddress; @ApiModelProperty(value="详细地址",dataType="java.lang.String") private String detailAddress; } @Data @SuperBuilder @NoArgsConstructor public static class VehicleInfoRecord extends Record{ @ApiModelProperty(value="VIN码",dataType="java.lang.String") private String vin; @ApiModelProperty(value="公里数",dataType="java.lang.Integer") private Integer mileage; @ApiModelProperty(value="评估备注",dataType="java.lang.String") private String evaluateRemark; } }
@RestController @Api(description = "查重数据", tags = "查重数据") @RequestMapping("/search") public class SearchController { @Autowired SearchQueryExecutor searchQueryExecutor; @PostMapping("/multiQuery") @ApiOperation(value = "多条件查询", notes = "多条件查询") @NoAuthRequired @OvalValidator(value = "多条件查询[multiQuery]") public Result3.2.4、SearchQueryExecutor> multiQuery(@RequestBody MultiSearchRequest request) { return searchQueryExecutor.execute(SearchQueryExecutor.Type.MULTI,request); } }
@Component public class SearchQueryExecutor { final Map3.2.5、SearchQueryContextHANDLER_MAP = Maps.newHashMap(); @Resource MultiSearchHandler multiSearchHandler; @PostConstruct void init(){ HANDLER_MAP.put(Type.MULTI, multiSearchHandler); } public enum Type{ SIMPLE, MULTI } public Result > execute(Type type, T request){ if(Type.MULTI == type){ SearchQueryContext
context = SearchQueryContext. builder().param((MultiSearchRequest)request).build(); HANDLER_MAP.get(type).execute(context); return Result.suc(context.getResults()); } return Result.suc(); } }
@Data @Builder @AllArgsConstructor @NoArgsConstructor public class SearchQueryContext3.2.6、AbstractSearchHandler{ private String requestId; private T param; private List results; private boolean success; private String message; public SearchQueryContext withSuccess(boolean success,String message){ this.setSuccess(success); this.setMessage(message); return this; } }
@Slf4j public abstract class AbstractSearchHandler3.2.7、MultiSearchHandler{ final ThreadLocal > CONTEXT = new ThreadLocal<>(); @Resource DiamondConfigProxy diamondConfigProxy; @Resource HitQuerierManager hitQuerierManager; @Resource ExecutorService searchQueryPoolExecutor; public void execute(SearchQueryContext context){ try { CONTEXT.set(context); doQuery(context); } catch (Exception e) { log.error("[execute],requestId={},ctx={}",context.getRequestId(),JSONObject.toJSONString(context),e); }finally { CONTEXT.remove(); } } abstract void doQuery(SearchQueryContext context); SearchResultRe buildExceptionSearchResult(MultiSearchRequest.Condition condition,String message){ return SearchResultRe.builder() .searchFieldName(condition.getSearchFieldName()) .searchFieldValue(condition.getSearchFieldValue()) .searchFieldDesc(condition.getSearchFieldDesc()) .message(message) .hitRecords(Collections.emptyList()) .build(); } }
多条件查询处理器,这里采用多线程,以条件维度,以子线程处理,主线程阻塞等待所有子线程处理,并把命中数据结果统一封装返回。
@Slf4j @Component public class MultiSearchHandler extends AbstractSearchHandler3.2.8、HitQuerierManager{ @Override void doQuery(SearchQueryContext context) { context.setRequestId(context.getParam().getRequestId()); String requestId = context.getRequestId(); //条件列表 List conditions = context.getParam().getConditions(); //结果列表 List results = Lists.newArrayListWithExpectedSize(conditions.size()); //Future任务列表 List > futureList = new ArrayList<>(conditions.size()); for(MultiSearchRequest.Condition condition : conditions){ //参数验证 if(!checkCondition(condition)){ results.add(super.buildExceptionSearchResult(condition, MessageFormat.format("查询异常|参数非法,field={0}",condition.getSearchFieldName()))); continue; } Future future = searchQueryPoolExecutor.submit(() -> { HitQuerierContext hitQuerierContext = HitQuerierContext.builder() .requestId(requestId) .condition(condition) .build(); hitQuerierManager.execute(hitQuerierContext); return hitQuerierContext; }); futureList.add(future); } int count = 0; for (Future f : futureList) { try { HitQuerierContext hitQuerierContext = f.get(); results.add(hitQuerierContext.getHitResult()); } catch (InterruptedException | ExecutionException e) { log.error("[doQuery][InterruptedException|ExecutionException],requestId={},context={}",requestId, JSONObject.toJSON(context),e); results.add(super.buildExceptionSearchResult(conditions.get(count), MessageFormat.format("查询异常|运行时异常,message={0}",e.getMessage()))); continue; } count++; } context.setResults(results); context.withSuccess(Boolean.TRUE,"查询成功"); } boolean checkCondition(MultiSearchRequest.Condition condition){ Set configFields = diamondConfigProxy.searchFieldConfig().keySet(); log.info("[checkCondition],configFields={}", configFields.toArray()); String searchFieldName = condition.getSearchFieldName(); return configFields.contains(searchFieldName); } }
命中查询管理器,由于一个查询条件会去多张表进行数据查询,然后再数据合并,因此这里吧查询器统一注册给hitList,每个请求则统一调用execute方法循环迭代所有查询器处理。
@Component public class HitQuerierManager { static final List3.2.9、HitQuerierContexthitList = new ArrayList<>(); @Resource PersonInfoHitQuerier personInfoHitQuerier; @Resource DealerInfoHitQuerier dealerInfoHitQuerier; @Resource VehicleInfoHitQuerier vehicleInfoHitQuerier; @PostConstruct void init(){ hitList.add(personInfoHitQuerier); hitList.add(dealerInfoHitQuerier); hitList.add(vehicleInfoHitQuerier); } public void execute(HitQuerierContext context){ MultiSearchRequest.Condition condition = context.getCondition(); SearchResultRe hitResult = SearchResultRe.builder() .searchFieldValue(condition.getSearchFieldValue()) .searchFieldName(condition.getSearchFieldName()) .searchFieldDesc(condition.getSearchFieldDesc()) .message("查询成功") .build(); for(AbstractHitQuerier querier : hitList){ SearchResultRe hit = querier.execute(context); hitResult.increaseHitCount(hit.getHitCount()); hitResult.addHits(hit.getHitRecords()); } context.setHitResult(hitResult); } }
@Data @Builder @AllArgsConstructor @NoArgsConstructor public class HitQuerierContext { private String requestId; private MultiSearchRequest.Condition condition; private SearchResultRe hitResult; }3.2.10、AbstractHitQuerier
@Slf4j public abstract class AbstractHitQuerier3.2.11、AbstractSearchMode{ protected final SearchFieldConfig.Table table; @Resource DiamondConfigProxy diamondConfigProxy; public AbstractHitQuerier(SearchFieldConfig.Table table) { this.table = table; } abstract SearchResultRe doQuery(HitQuerierContext context); public SearchResultRe execute(HitQuerierContext context){ SearchResultRe searchResultRe; try { if(!executeCurrent(context)){ searchResultRe = buildDefaultSearchResult(context.getCondition()); return searchResultRe; } searchResultRe = doQuery(context); } catch (Exception e) { log.error("[execute],requestId={},ctx={}",context.getRequestId(),JSONObject.toJSONString(context),e); searchResultRe = buildDefaultSearchResult(context.getCondition()); searchResultRe.setMessage("查询异常|message=" + e.getMessage()); } return searchResultRe; } protected boolean executeCurrent(HitQuerierContext context){ return getSearchFieldConfig(context.getCondition().getSearchFieldName()).getTables().contains(table); } SearchResultRe buildDefaultSearchResult(MultiSearchRequest.Condition condition){ String fieldName = condition.getSearchFieldName(); String fieldValue = condition.getSearchFieldValue(); SearchResultRe hitResult = SearchResultRe.builder() .searchFieldName(fieldName) .searchFieldValue(fieldValue) .hitCount(0) .hitRecords(Collections.emptyList()) .build(); return hitResult; } SearchFieldConfig getSearchFieldConfig(String fieldName){ Map mapping = diamondConfigProxy.searchFieldConfig(); SearchFieldConfig searchFieldConfig = mapping.get(fieldName); log.info("[getSearchFieldConfig],fieldName={},config={}",fieldName, JSONObject.toJSONString(searchFieldConfig)); return searchFieldConfig; } List hitRecords(HitQuerierContext context, Object queryForm, BiFunction > recordsCaller){ MultiSearchRequest.Condition condition = context.getCondition(); SearchFieldConfig searchFieldConfig = getSearchFieldConfig(condition.getSearchFieldName()); SearchFieldConfig.SearchMode searchMode = searchFieldConfig.getSearchMode(); return AbstractSearchMode.getAbstractSearchMode(searchMode).execute(searchFieldConfig, context, queryForm, recordsCaller); } } @Slf4j @Component public class DealerInfoHitQuerier extends AbstractHitQuerier { public DealerInfoHitQuerier() { super(SearchFieldConfig.Table.DEALER_INFO); } @Resource DealerInfoService dealerInfoService; @Override SearchResultRe doQuery(HitQuerierContext context) { MultiSearchRequest.Condition condition = context.getCondition(); SearchResultRe hitResult = super.buildDefaultSearchResult(condition); DealerInfoForm queryForm = DealerInfoForm.builder().dataStatus(0).build(); List hitRecords = super.hitRecords(context,queryForm,(String fieldName,Object searchForm) -> { List recordList = dealerInfoService.queryList((DealerInfoForm)searchForm); SearchFieldConfig config = getSearchFieldConfig(condition.getSearchFieldName()); if(SearchFieldConfig.SearchMode.SIMILAR == config.getSearchMode() && condition.getSearchFieldName().contains("Address")){ String provinceAddressFiledName = config.getMapping().get(config.getMapping().size()-1); String cityAddressFiledName = config.getMapping().get(config.getMapping().size()-1); String districtAddressFiledName = config.getMapping().get(config.getMapping().size()-1); String detailAddressFiledName = config.getMapping().get(config.getMapping().size()-1); return BeanConverter.convertFromDealer(recordList, each -> BeanTool.getObjectValue(each, provinceAddressFiledName), each -> BeanTool.getObjectValue(each, cityAddressFiledName), each -> BeanTool.getObjectValue(each, districtAddressFiledName), each -> BeanTool.getObjectValue(each, detailAddressFiledName)); } return BeanConverter.convertFromDealer(recordList, null, null, null,null); }); hitResult.setHitRecords(hitRecords); hitResult.setHitCount(hitRecords.size()); return hitResult; } } @Slf4j @Component public class PersonInfoHitQuerier extends AbstractHitQuerier { static final String MOBILE_FIELD = "mobile"; static final String ADDRESS_FIELD_SUFFIX = "Address"; public PersonInfoHitQuerier() { super(SearchFieldConfig.Table.PERSON_INFO); } @Resource PersonInfoService personInfoService; @Override SearchResultRe doQuery(HitQuerierContext context) { MultiSearchRequest.Condition condition = context.getCondition(); SearchResultRe hitResult = super.buildDefaultSearchResult(condition); PersonInfoForm queryForm = PersonInfoForm.builder().dataStatus(0).build(); List hitRecords = super.hitRecords(context,queryForm,(String fieldName,Object searchForm) -> { List recordList = personInfoService.queryList((PersonInfoForm)searchForm); if(MOBILE_FIELD.equals(condition.getSearchFieldName())){ return BeanConverter.convertFromPerson(recordList, each -> BeanTool.getObjectValue(each, fieldName), null, null, null, null); } SearchFieldConfig config = getSearchFieldConfig(condition.getSearchFieldName()); if(SearchFieldConfig.SearchMode.SIMILAR == config.getSearchMode() && condition.getSearchFieldName().contains(ADDRESS_FIELD_SUFFIX)){ String provinceAddressFiledName = config.getMapping().get(0); String cityAddressFiledName = config.getMapping().get(1); String districtAddressFiledName = config.getMapping().get(2); String detailAddressFiledName = config.getMapping().get(3); return BeanConverter.convertFromPerson(recordList, null, each -> BeanTool.getObjectValue(each, provinceAddressFiledName), each -> BeanTool.getObjectValue(each, cityAddressFiledName), each -> BeanTool.getObjectValue(each, districtAddressFiledName), each -> BeanTool.getObjectValue(each, detailAddressFiledName)); } return BeanConverter.convertFromPerson(recordList); }); hitResult.setHitRecords(hitRecords); hitResult.setHitCount(hitRecords.size()); return hitResult; } } @Slf4j @Component public class VehicleInfoHitQuerier extends AbstractHitQuerier { public VehicleInfoHitQuerier() { super(SearchFieldConfig.Table.VEHICLE_INFO); } @Resource VehicleInfoService vehicleInfoService; @Override SearchResultRe doQuery(HitQuerierContext context) { MultiSearchRequest.Condition condition = context.getCondition(); SearchResultRe hitResult = super.buildDefaultSearchResult(condition); VehicleInfoForm queryForm = VehicleInfoForm.builder().dataStatus(0).build(); List hitRecords = super.hitRecords(context,queryForm,(String fieldName,Object searchForm) -> { List recordList = vehicleInfoService.queryList((VehicleInfoForm)searchForm); return BeanConverter.convertFromVehicle(recordList); }); hitResult.setHitRecords(hitRecords); hitResult.setHitCount(hitRecords.size()); return hitResult; } }
@Slf4j public abstract class AbstractSearchMode { public static Map4、扩展部分 4.1、查重服务请求参数abstractSearchModeMap = new HashMap<>(); public static AbstractSearchMode getAbstractSearchMode(SearchFieldConfig.SearchMode modeEnum){ return abstractSearchModeMap.get(modeEnum); } public abstract List execute(SearchFieldConfig searchFieldConfig, HitQuerierContext context, Object queryForm, BiFunction > recordsCaller); public void copyExclude (SearchFieldConfig searchFieldConfig, Object queryForm){ Map exclude = searchFieldConfig.getExclude(); if(MapUtils.isNotEmpty(exclude)){ BeanTool.copyFromOneMap(exclude,queryForm); } } } @Slf4j @Component public class ExactSearchMode extends AbstractSearchMode { { abstractSearchModeMap.put(SearchFieldConfig.SearchMode.EXACT, this); } @Override public List execute(SearchFieldConfig searchFieldConfig, HitQuerierContext context, Object queryForm, BiFunction > recordsCaller) { //拷贝配置的扩展参数到实体类 copyExclude (searchFieldConfig, queryForm); //入参条件 String requestId = context.getRequestId(); String fieldName = context.getCondition().getSearchFieldName(); String fieldValue = context.getCondition().getSearchFieldValue(); //结果集 List hitRecords; //有效查询参数 Map sourceValues = Maps.newHashMap(); //入参一个参数映射成查询两个参数(入参mobile对应数据库primaryMobile,SecondMobile) List mapping = searchFieldConfig.getMapping(); if(null != mapping && !mapping.isEmpty()){ hitRecords = Lists.newArrayList(); for(String fieldNameAlias : mapping){ sourceValues.put(fieldNameAlias,fieldValue); //map键值对拷贝到实体类 BeanTool.copyFromOneMap(sourceValues,queryForm); log.info("[execute][hitRecords],requestId={},queryForm={}",requestId, JSONObject.toJSONString(queryForm)); List hitRecordsTemp = recordsCaller.apply(fieldNameAlias,queryForm); log.info("[execute][hitRecords],requestId={},hitRecordsFromDB={}",requestId,hitRecordsTemp.size()); if(CollectionsTools.isNotEmpty(hitRecordsTemp)){ hitRecords.addAll(hitRecordsTemp); } //抹掉本次参数(本次参数设置为null,并拷贝到查询实体类) sourceValues.put(fieldNameAlias,null); } }else{ sourceValues.put(fieldName,fieldValue); BeanTool.copyFromOneMap(sourceValues,queryForm); log.info("[execute][hitRecords],requestId={},queryForm={}",requestId,JSONObject.toJSONString(queryForm)); hitRecords = recordsCaller.apply(fieldName,queryForm); log.info("[execute][hitRecords],requestId={},hitRecordsFromDB={}",requestId,hitRecords.size()); } return hitRecords; } } @Slf4j @Component public class SimilarSearchMode extends AbstractSearchMode { final String SIMILAR_DETAIL_FIELD = "detail"; final String OUT_MODEL_PROVINCE_ADDRESS = "provinceAddress"; final String OUT_MODEL_CITY_ADDRESS = "cityAddress"; final String OUT_MODEL_DISTRICT_ADDRESS = "districtAddress"; final String OUT_MODEL_DETAIL_ADDRESS = "detailAddress"; final String SIMILAR_TYPE = "m:organization.organization.name"; { abstractSearchModeMap.put(SearchFieldConfig.SearchMode.SIMILAR, this); } @Override public List execute(SearchFieldConfig searchFieldConfig, HitQuerierContext context, Object queryForm, BiFunction > recordsCaller) { //拷贝配置的扩展参数到实体类 copyExclude (searchFieldConfig, queryForm); //入参条件 String requestId = context.getRequestId(); String fieldName = context.getCondition().getSearchFieldName(); String fieldValue = context.getCondition().getSearchFieldValue(); //入参条件拆分(地址拆分为省、市、区、详细) List mapping = searchFieldConfig.getMapping(); String[] fieldValues = fieldValue.split("\|"); //需要分词比较的详细地址(具体字段名称) String similarField = mapping.get(mapping.size()-1); //用户传过来的详细地址 String addressDetail = fieldValues[fieldValues.length-1]; HsmmAddressNormalizer anm = new HsmmAddressNormalizer(); String addressDetailFormat = fieldValues[0] + fieldValues[1] + fieldValues[2] + ( (HashMap )anm.splitAddress(addressDetail) ).get(SIMILAR_DETAIL_FIELD); //有效查询参数 Map sourceValues; //结果集 List hitRecords = Collections.emptyList(); if(CollectionUtils.isEmpty(mapping)){ log.warn("[execute]diamond mapping is null,requestId={},fieldName={}",requestId,fieldName); return hitRecords; } sourceValues = Maps.newHashMap(); String[] mappingValues = mapping.toArray(new String[mapping.size()]); //最后一项不作为查询条件 for (int i = 0; i < mappingValues.length - 1; i++) { sourceValues.put(mappingValues[i],fieldValues[i]); } BeanTool.copyFromOneMap(sourceValues,queryForm); log.info("[execute][hitRecords],requestId={},queryForm={}",requestId, JSONObject.toJSONString(queryForm)); List hitRecordsTemp = recordsCaller.apply(fieldName,queryForm); log.debug("[execute][hitRecords],requestId={},hitRecordsFromDB={}",requestId,hitRecordsTemp.size()); if(CollectionsTools.isEmpty(hitRecordsTemp)){ return hitRecords; } //匹配度分数区间,长度限制0-2【分数区间为空-没有分数要求;分数区间长度为1-最低分要求;分数区间长度为2-分数区间要求】 List scoreRangeList = context.getCondition().getScoreRange(); Double[] scoreRange = CollectionUtils.isEmpty(scoreRangeList) ? new Double[0] : scoreRangeList.toArray(new Double[scoreRangeList.size()]); if(scoreRange.length == 0){ //分数区间为空-没有分数要求 return hitRecordsTemp; }else{ String provinceAddressFromDb; String cityAddressFromDb; String districtAddressFromDb; String detailAddressFromDb; String detailAddressFromDbFormat; Double score; hitRecords = Lists.newArrayListWithExpectedSize(hitRecordsTemp.size()); for (E item : hitRecordsTemp) { //相似分数符合条件的添加到结果集 provinceAddressFromDb = BeanTool.getObjectValue(item, OUT_MODEL_PROVINCE_ADDRESS); cityAddressFromDb = BeanTool.getObjectValue(item, OUT_MODEL_CITY_ADDRESS); districtAddressFromDb = BeanTool.getObjectValue(item, OUT_MODEL_DISTRICT_ADDRESS); detailAddressFromDb = BeanTool.getObjectValue(item, OUT_MODEL_DETAIL_ADDRESS); detailAddressFromDbFormat = provinceAddressFromDb + cityAddressFromDb + districtAddressFromDb + ( (HashMap )anm.splitAddress(detailAddressFromDb) ).get(SIMILAR_DETAIL_FIELD); score = NLPUtil.getUtil().similarity(SIMILAR_TYPE, detailAddressFromDbFormat, addressDetailFormat); log.info("[execute][hitRecords][similarScore],requestId={},addressDetailFormat={},addressDetailFromDb={},score={}", requestId, addressDetailFormat, detailAddressFromDbFormat, score); //分数区间长度为1-最低分要求 if(scoreRange.length == 1 && score >= scoreRange[0]){ hitRecords.add(item); } //分数区间长度为2-分数区间要求 if(scoreRange.length == 2 && score >= scoreRange[0] && score <= scoreRange[1]){ hitRecords.add(item); } } } return hitRecords; } }
请求参数示例
{ "requestId":"1", "conditions": [{ "sourceType": "CREDITOR_INFO", "searchFieldDesc": "主贷人身份证号", "searchFieldName": "idNo", "searchFieldValue": "350****8114118" }, { "sourceType": "CREDITOR_INFO", "searchFieldDesc": "主贷人手机号", "searchFieldName": "mobile", "searchFieldValue": "186****2901" }, { "sourceType": "CREDITOR_INFO", "searchFieldDesc": "销售手机号", "searchFieldName": "mobile", "searchFieldValue": "182****4023" }, { "sourceType": "CREDITOR_INFO", "searchFieldDesc": "二手车卖方身份证号", "searchFieldName": "idNo", "searchFieldValue": "3522****2138" }, { "sourceType": "CREDITOR_INFO", "searchFieldDesc": "紧急联系人1手机号", "searchFieldName": "mobile", "searchFieldValue": "139****603" }, { "sourceType": "CREDITOR_INFO", "searchFieldDesc": "紧急联系人2手机号", "searchFieldName": "mobile", "searchFieldValue": "18****637" }, { "sourceType": "CREDITOR_INFO", "searchFieldDesc": "担保人身份证号", "searchFieldName": "idNo", "searchFieldValue": "350****38" }, { "sourceType": "CREDITOR_INFO", "searchFieldDesc": "担保人手机号", "searchFieldName": "mobile", "searchFieldValue": "15****020" }, { "sourceType": "VEHICLE_INFO", "searchFieldDesc": "车辆VIN", "searchFieldName": "vin", "searchFieldValue": "LFV****3721" }] }
响应结果示例
部分数据脱敏展示了,比如手机号、银行卡号、身份证号。
{ "code": 0, "data": [ { "hitCount": 1, "hitRecords": [ { "appCode": "F2009111915000180101", "creditCardNo": "-", "dataCode": "P20121700551635", "externalId": "100021309", "id": 243677, "idNo": "350122**4118", "mobile": "186**01", "name": "郑**", "scene": 1003, "sourceType": 1 } ], "message": "查询成功", "searchFieldDesc": "主贷人身份证号", "searchFieldName": "idNo", "searchFieldValue": "3501**14118" }, { "hitCount": 2, "hitRecords": [ { "appCode": "F2009151915000180101", "creditCardNo": "-", "dataCode": "P20121700538028", "externalId": "100022351", "id": 219723, "idNo": "-", "mobile": "186**901", "name": "郑**", "scene": 1002, "sourceType": 8 }, { "appCode": "F2009111915000180101", "creditCardNo": "-", "dataCode": "P20121700551635", "externalId": "100021309", "id": 243677, "idNo": "3501**118", "mobile": "186**01", "name": "郑**", "scene": 1003, "sourceType": 1 } ], "message": "查询成功", "searchFieldDesc": "主贷人手机号", "searchFieldName": "mobile", "searchFieldValue": "186**2901" }, { "hitCount": 1, "hitRecords": [ { "appCode": "F2009151915000180106", "creditCardNo": "-", "dataCode": "P20121700537447", "externalId": "100022605", "id": 218697, "idNo": "3505**5550", "mobile": "182**4023", "name": "欧**", "scene": 1003, "sourceType": 1 } ], "message": "查询成功", "searchFieldDesc": "销售手机号", "searchFieldName": "mobile", "searchFieldValue": "182**023" }, { "hitCount": 6, "hitRecords": [ { "appCode": "F2011091915000180104", "creditCardNo": "-", "dataCode": "P20121700417419", "externalId": "100038555", "id": 7575, "idNo": "35223**38", "mobile": "-", "name": "阮**", "scene": 1002, "sourceType": 6 }, //省略其他 ], "message": "查询成功", "searchFieldDesc": "二手车卖方身份证号", "searchFieldName": "idNo", "searchFieldValue": "3522**32138" }, { "hitCount": 3, "hitRecords": [ { "appCode": "F2010191915000180107", "creditCardNo": "-", "dataCode": "P20121700481894", "externalId": "100032121", "id": 121031, "idNo": "-", "mobile": "139**603", "name": "陈**", "scene": 1002, "sourceType": 7 }, //省略其他 ], "message": "查询成功", "searchFieldDesc": "紧急联系人1手机号", "searchFieldName": "mobile", "searchFieldValue": "139**603" }, { "hitCount": 3, "hitRecords": [ { "appCode": "F2007281915000180103", "creditCardNo": "-", "dataCode": "P20121700447417", "externalId": "100010141", "id": 60345, "idNo": "-", "mobile": "1810**37", "name": "林**", "scene": 1002, "sourceType": 8 }, //省略其他 ], "message": "查询成功", "searchFieldDesc": "紧急联系人2手机号", "searchFieldName": "mobile", "searchFieldValue": "181**637" }, { "hitCount": 3, "hitRecords": [ { "appCode": "F2009281915000180104", "creditCardNo": "-", "dataCode": "P20121700442684", "externalId": "100026661", "id": 52033, "idNo": "35012**938", "mobile": "152**20", "name": "许**", "scene": 1003, "sourceType": 1 }, //省略其他 ], "message": "查询成功", "searchFieldDesc": "担保人身份证号", "searchFieldName": "idNo", "searchFieldValue": "35012****195938" }, { "hitCount": 4, "hitRecords": [ { "appCode": "F2009281915000180104", "creditCardNo": "-", "dataCode": "P20121700442684", "externalId": "100026661", "id": 52033, "idNo": "3501**8", "mobile": "152050**", "name": "许**", "scene": 1003, "sourceType": 1 }, //省略其他 ], "message": "查询成功", "searchFieldDesc": "担保人手机号", "searchFieldName": "mobile", "searchFieldValue": "152**" }, { "hitCount": 6, "hitRecords": [ { "appCode": "F2011091915000180104", "dataCode": "V20121700417457", "evaluateRemark": "正常。1.备胎槽照片重新拍摄,要求完整清晰。n2.补充左右后叶子板流水槽照片n3.补充左右前纵梁照片n4.补充主副驾驶座椅滑轨照片n5.有补领记录,补充车架拓印号照片(铁皮上的)", "externalId": "100038555", "id": 1045, "mileage": 136029, "scene": 1002, "sourceType": 12, "vin": "LFV4A24F7A30837**" }, //省略其他 ], "message": "查询成功", "searchFieldDesc": "车辆VIN", "searchFieldName": "vin", "searchFieldValue": "LFV4A24F7A3083**" } ], "msg": "操作成功", "success": true }4.2、数据查重字段配置
为了提高数据查重接口的扩展性,基于配置化的元数据配置。
value包含如下参数
- desc:参数描述,无业务逻辑;仅仅作为字段说明使用。
- searchMode:查询方式,目前仅支持两种 EXACT(精准查询)、SIMILAR(相似度查询)。
- tables:json字符串数组,适用于该域查询的表,目前表共三个(PERSON_INFO、DEALER_INFO、VEHICLE_INFO)。
- mapping:查询字段映射,字符串数据,查询形式以或作为条件,结果集会进行合并。
- exclude:查询过滤条件,查询结果集以该配置参数作为过滤条件。字段key作为查询条件field,value作为条件。
{ "idNo": { "desc": "身份证号", "searchMode": "EXACT", "mapping": [], "tables": [ "PERSON_INFO", "DEALER_INFO" ] }, "name": { "desc": "姓名", "searchMode": "EXACT", "mapping": [], "tables": [ "PERSON_INFO", "DEALER_INFO" ] }, "mobile": { "desc": "手机号(primaryMobile,SecondMobile)", "searchMode": "EXACT", "mapping": [ "primaryMobile", "secondMobile" ], "tables": [ "PERSON_INFO", "DEALER_INFO" ], "exclude": { "sourceTypeScopeExclude": [ 2, 3 ] } }, "creditCardNo": { "desc": "银行卡号", "searchMode": "EXACT", "mapping": [], "tables": [ "PERSON_INFO", "DEALER_INFO" ] }, "companyAddress": { "desc": "单位地址", "searchMode": "SIMILAR", "mapping": [ "companyAddressProvince", "companyAddressCity", "companyAddressDistrict", "companyAddressDetail" ], "tables": [ "PERSON_INFO", "DEALER_INFO" ] }, "censusAddress": { "desc": "户籍地址", "searchMode": "SIMILAR", "mapping": [ "censusAddressProvince", "censusAddressCity", "censusAddressDistrict", "censusAddressDetail" ], "tables": [ "PERSON_INFO" ] }, "residenceAddress": { "desc": "居住地址", "searchMode": "SIMILAR", "mapping": [ "residenceAddressProvince", "residenceAddressCity", "residenceAddressDistrict", "residenceAddressDetail" ], "tables": [ "PERSON_INFO" ] }, "vin": { "desc": "车辆VIN", "searchMode": "EXACT", "mapping": [], "tables": [ "VEHICLE_INFO" ] } }5、总结
总体设计上运用了相关设计模式,并分成了多个模块,每个模块负责各自的业务逻辑职责。其中在数据查重接口设计上,考虑查询数据量比较多,基于输入的多个条件,运用并行处理,并把多个处理器的查询结果再进行合并,从而提高接口的性能。