File Processing - Process Execution Solution
1 like533 views
This is a Software Architecture Solution to the Processing of batch files and online requests. Solution with Apache Camel.
More Related Content
What's hot (14)
Viewers also liked (20)
Similar to File Processing - Process Execution Solution (20)
More from Abimael Desales López (18)
Recently uploaded (20)
File Processing - Process Execution Solution
- 1. PROCESS EXECUTION AUTOMATICS, INDIVIDUALS AND MASSIVES Abimael Desales López 22/10/2015
- 4. Automatic processing it will carry out through a component whose mission is verify execution times by the means of scheduler that it will have maped the time and frecuency of each process in data base, and further it will have the ability to build the expression that feed to quartz to make the scheduling of task being performed. It plans split the automatic processing in two coarse components that it will execute each one their corresponding function, the (fisrt) Controller it will verify every input of the process and it will decide if it going to execute, fulfilling its pre-conditions, defining the commit frecuency and number of threads in parallel to execute. The (second) Executor is responsible to execute the process business logic. Automatic Process
- 5. Individual Processing Process Bootstrap Individual SubtypesofProcess F6s f5ds9f f9sf55 f6sd5f f6sd5lm ñlk F5d f5d6xf d5f6d 5fd6s f6ds5f6 wa Result DB Getparameters VerifiyConvivenceMatrix ExecuteBusinessRules MakeCalculations Typing Processing DB
- 6. In the individual processing typing the data in screen and accepting it; to begin the processing the system it will create a ticket, will assign it a ticket number to the request (if and only if the validation rules have been successes and the required precondition is ok), it will get the process parameters, if applies their execution this will be carried out, in other way the ticket is automaticly canceled. The business rules will be get and applyed in predefined order for the process. If all processing was executed successfully and the business rules are ok then it begin the dump of information to database, inserting and/or updating registries in the respective tables of the process, and finally updating the ticket status to PROCESSED. Individual Processing
- 7. Massive Processing Segment Bootstrap Massive SubtypesofProcesses CAMEL TXT XLS CSV F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5 F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65d Dsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f5 85fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2 Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf2 6df5s6df6ds5f5sd6f5s65f6sd5f65ds6f 65ds56sd5f6f5d6g5df5gdf6df6h5gfh5 Df65f5d65gd6fg56df5gdf6gdf5gd6fgf 5dgffd6g5df6g5df5gdf5g651df5g6f5d F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5 F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65d Dsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f5 85fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2 Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf2 6df5s6df6ds5f5sd6f5s65f6sd5f65ds6f 65ds56sd5f6f5d6g5df5gdf6df6h5gfh5 Df65f5d65gd6fg56df5gdf6gdf5gd6fgf 5dgffd6g5df6g5df5gdf5g65df5g62f5d F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5 F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65d Dsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f5 85fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2 Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf2 6df5s6df6ds5f5sd6f5s65f6sd5f65ds6f 65ds56sd5f6f5d6g5df5gdf6df6h5gfh5 Df65f5d65gd6fg56df5gdf6gdf5gd6fgf 5dgffd6g5df6g5df5gdf51g65df5g6f5d OK File FAILED File OK File FAILED File OK File FAILED File CAMEL Result of Processing DB Processing Export Export Export Controller Executor
- 8. Massive Processing Confirm F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5 F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65d Dsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f5 85fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2 Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf2 6df5s6df6ds5f5sd6f5s65f6sd5f65ds6f 65ds56sd5f6f5d6g5df5gdf6df6h5gfh5 Df65f5d65gd6fg56df5gdf6gdf5gd6fgf 5dgffd6g5df6g5df5gdf5g651df5g6f5d OK_File CAMEL Processing Controller Executor Calculationofthreads ThrownThreads Definecommits GettheFile Creategenericlayout DB Thread 2 Thread 3 Thread 4 Thread n-1 Thread 5 Thread n Thread 1 Commit x Thread and saved of processing data to keep the trace to reprocess the file in case of failure Updating to ARCH_PROCESSED Here it will execute final insert
- 9. Massive Processing In the files massive processing highlight three generales steps: • File load and format recognition. • File processing, upon their format, and dump in database the summary of the process. • Dump in database of each file records (or lines) processed. To carry out the first two steps it is intended to exploit the features of the framework Apache Camel, and apply the procedure made in the proof of concept.
- 10. The tasks that will be carry out through Camel in the first stage of the massive processing are below: • Routing with dependence of file format • Get the file registries for the processing • Validation of structure of the file • Locate the suitable place to apply the business rules. • Depending on the result, on the rules/validations enforcement decide wheter the registry will be write in a OK file or in a FAILED file. • Routing the outcoming files to a specified route. Massive Processing
- 11. The task to be carry out by Camel in the second phase of the massive processing (since that is already integrated the OK file) will be the following: • Obtaining the OK file • Receiving as a parameter the number of threads to create. • Do the parcial Commits towards database by each thread. • Saved trail processed and not processed blocks • Positioning in the appropiate place in the file to run only a block of the same, or no secuential blocks as effect the start of processing of failed registries. • Writing to database the summary of failed records of the file. • Summary of cuantity by columns when is required. Massive Processing
- 12. Structure of the Processes Manager The Process Manager it will be a component composed of another 3 main components, the Scheduler, the Controller and the Executor.
- 13. 13 Design Guidelines 10/22/2015 Solution Design Processing Quartz Job (Modulo) Web App Controller ExecutorJob Business Logic Business Tables Processes Manager Components Application ComponentsApplication Java Classes (thread) Java Classes (Process) B/L Java Classes Call Call Repository Business Application DB Shell Script Specific Tasks Call Shell Unix ►Run Web App Web App Restringed Admin SuperAdmin OS JVM OS JVM Execution engine Execution engine OS Controller Executor Trámite 1 Trámite 2 Trámite 3 Trámite 4 Assign Job ID and register to queue Get and execute Job Business DB Block 1 Block 2 Block 3
- 14. Flow from loading the file Web Sytem UploadFileBeanuploadFile(…) Upload File to Server Create File Structure Create Ticket Get the tickets to Process Camel class move file to processing path and route it to right method ProcessedFile insert() Get the Shedule Put the OK and FAILED files ProcessedFile Service update() TicketService update() Process registries by Camel (apply Business Rules) UploadFileBean
- 15. Structure of the Processes Manager Scheduler Responsabilities: • Schedule the execution times of the process, if are there some window for modify de execution times, have a funcionality that is capable to traslate date-time to Quartz expression and back again. • Create a Job Group by process. • Start each process to the time of day that is scheduled in automatic case, or at once in massive e individual cases, this is to pass the control flow to the Controller.
- 16. Approach of the Scheduler We look that a Job is scheduled, which is only responsible to invoke to one Manager, although this could change if necessary invoking something more. Manager is responsible for orchestrating their services and these ones to their corresponding Daos.
- 17. Structure of the Processes Manager Responsabilities of the Scheduler: • It will be created so many Job classes as different types of services you have to run, namely, services will can be grouped in a single Job so that these conform transactional units or database access, each Job must implement the org.quartz.Job interface. • The package structure will be as follows: – For the orchestator classes com.axxi.gestor.procesos.planificador – For the Job classes: com.axxi.gestor.procesos.planificador.job – For the job launchers (classes with main method) if required: com.axxi.gestor.procesos.planificador.launcher
- 18. Structure of the Processes Manager package com.axxi.quartz; import org.quartz.Job; import org.quartz.JobDataMap; import org.quartz.JobExecutionContext; import org.quartz.JobExecutionException; // Otros imports public class ControllerJob implements Job { // Services injected by Spring should be injected by setter injection private IProcessXManager processXManager; public void setProcessXManager(IProcessXManager processXManager){ this.processXManager=processXManager; } public void execute(JobExecutionContext arg0) throws JobExecutionException { //Get JobDataMap from the context JobDataMap dataMap = arg0.getMergedJobDataMap(); processX.orchest(datamap.getWrappedMap()); } }
- 19. Structure of the Processes Manager Responsabilities of the Scheduler: • Inside each Job in the execute(…) method it should get the context’s JobDataMap parameters and set them to the service. JobDataMap dataMap = arg0.getMergedJobDataMap(); Solicitud solicitud = (Solicitud) dataMap.get("solicitud"); ArchivoProcesado archProcesado = (ArchivoProcesado) dataMap.get("archProcesado"); • In the Job class it will be injected the service by Spring through setter injection. • The Manager, Service and Dao clases still work the same way as in the web project, injecting the instances in the Job, Manager and Service classes respectively, by Spring.
- 20. Manager Example public class TramiteManager implements ITramiteManager { // Get the instance by Spring private ISolicitudService solicitudService; private IArchivoProcesadoService fileProcessedService; private IVerificadorPrecondicionesService verifyPreconditions ; //Setters … public void orchest(Map<String, Object> parameters) { Solicitud solicitud = (Solicitud) parameters.get("solicitud"); FileProcessed fileProcessed = (FileProcessed) parameters.get(“fileProcessed"); Integer xParameter = (Integer) parametros.get("xParameter"); solicitudService.createSolicitud(solicitud); fileProcessedService.createFileProcessed(fileProcessed); verifyPreconditions.verifyPreconditionsAfiliacion(xParameter); } }
- 21. Structure of the Processes Manager Responsabilities of the Scheduler: • It will be created one Scheduler class for massive processing, one for individual processing and one for automatic processing, each one must have as minimum one method by each process type. – MassiveScheduler.java – IndividualScheduler.java – AutomaticScheduler.java • Each class must implement one specific interface with the required methods for the functionality.
- 22. Structure of the Processes Manager Responsabilities of the Scheduler: • It must create one properties file with the mapping of groups with the processes name. #Mapping of execution groups with processes Grupo1=Affiliation Grupo2=Collection Grupo3=ABCAdministrative Grupo4=Unification Grupo5=Withdraw Grupo6=ReinvestmentResources Grupo7=RGPE Grupo8=Yield Grupo9=Comission Grupo10=Loan
- 23. Structure of the Processes Manager Responsabilities of the Scheduler: • Must have one orchestrator class by each process type (massive, automatic, individual) by way of pseudo-workflow to execute the flows. • Each one of the orchestrator class must be within of the next package: com.xxx.manager.processes.scheduler.launcher • In order to maintain there should be one service to build a request and one processed file of each process type since each process has specific values for their request.
- 24. Structure of the Processes Manager Responsabilities of the Controller: • Verify the preconditions to start the process. • Determine if the process goes forward in function of the matrix of coexistence. • Get the parameters to start the process, this is, the layout, and maybe some specific execution parameters, as the ID of the process, etc. • Give way to the Executor so it get the file and begin their processing, passing it the entire set of parameters obtained. • When applicable, depending of the process, make the calculations of threads, and launching threads. • Control the threads processing and commits, saving the track of the processed registries, when applicable the processing for threads. • Having ability to restart.
- 25. Structure of the Processes Manager Responsabilities of the Controller: For the phase of final processing: • Getting file to be processed with the corresponding parameters (i.e. enterprise, product, process y subtype), ticket number. • Invocation to generic layout maker class. • Determination of the number of commits. • Calculation of execution threads –if is staticly assgned, this step is not necessary. • Launching and control of threads, unexecuted blocks recovery. • Allow recovery of certain blocks of the file.
- 26. Structure of the Processes Manager Responsabilities of the Executor: For the phase of final processing: • Receive the parameters get in the Controller. • Start the flow of Camel processing. • In case of to be interrupted the flow of Camel, send one flag to the Controller that the process was interrupted in certain point or in certain block that includes since the X registry to the Y registry (may be writing to database), so that this can reprocess the file. • Notify the conclusión of the processing to the Controller • Get a generic service that records and updates the request and the table of processed files.
- 27. Structure of the Processes Manager Responsabilities of the Executor: For the phase of final processing: • Receiv as parameters the generic layout, the ticket file number, the frecuency of commits. • Invoke to the class of each process so that it executes the insert to the database. • Return the control of the flow to the Controller so that this register the block of processed information. • Invoke to the generic service so that it updates the process state.
- 28. Structure of the Processes Manager Responsabilities of the Executor: For the phase of final processing: • It must be created one transactional method for each process to insert to database the lists of particular layouts corresponding to each process, each one of this methods must be developed by the owner responsible of that process logic, can’t be a generic functionality. The method will receiv one list of Map from the Camel class to get registries. Within this method it will be invoked one method that assemble the generic layouts. • The method must return a Map holding three data: – 1.- Position of initial registry – 2.- Position of final registry – Flag indicating if it can to do the insert (True or False)
- 29. Structure of the Processes Manager Responsabilities of the Executor: For the phase of final processing: • It must be created one method for the assembly of the generic layout that will receiv one Map conteining the field ID - value of the field, and the last data of the Map will be the number (position) of the registry in the original file to track what is being inserted, and we can be aware not inserted records to try the rework/restart. This method must return one bean with the received data of the specific layout, but conteining all possible generic fields.
- 30. Structure of the Processes Manager Responsabilities of the Executor: For the phase of final processing: • The required methods will have the next suggested signs: LayoutGenericoRecaudacion armarLayoutRecaudacion(Map<String, String> layGenerico); @Transactional Map <String, Object> insertRecaudacion(List<Map<String, String>> listaLayouts){ LayoutGenericoRecaudacion layGenerico; for(Map<String, String> map : listaLayouts){ layGenerico = armarLayoutGenerico(map); //Processing … } }
- 31. THANKS Abimael Desales López Software Architect Ibermática México a.desales@ibermatica.com It.adesales@gmail.com Feel free to contact me!