CN109101440B - Method for processing trace data concurrent request based on JVM (Java virtual machine) cache - Google Patents
Method for processing trace data concurrent request based on JVM (Java virtual machine) cache Download PDFInfo
- Publication number
- CN109101440B CN109101440B CN201810863056.8A CN201810863056A CN109101440B CN 109101440 B CN109101440 B CN 109101440B CN 201810863056 A CN201810863056 A CN 201810863056A CN 109101440 B CN109101440 B CN 109101440B
- Authority
- CN
- China
- Prior art keywords
- data
- interface
- processing
- writing
- signature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 230000004044 response Effects 0.000 claims description 13
- 230000002688 persistence Effects 0.000 claims description 9
- 230000005540 biological transmission Effects 0.000 claims description 6
- 230000003139 buffering effect Effects 0.000 claims description 6
- 238000007726 management method Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 4
- 238000013480 data collection Methods 0.000 description 3
- 235000013372 meat Nutrition 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000001125 extrusion Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
- G06F12/084—Multiuser, multiprocessor or multiprocessing cache systems with a shared cache
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
- G06F12/0842—Multiuser, multiprocessor or multiprocessing cache systems for multiprocessing or multitasking
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method for processing traceable data concurrent requests based on JVM (Java virtual machine) cache, belonging to the technical field of application software in the Internet industry. The method for processing the trace data concurrent request based on the JVM cache comprises the steps of setting a unified interface protocol standard and realizing internal processing. The method for processing the trace data concurrency request based on the JVM cache can provide a convenient and quick interface protocol for the outside and a stable and efficient concurrency storage processing capability for the inside, and has good popularization and application values.
Description
Technical Field
The invention relates to the technical field of application software in the Internet industry, and particularly provides a method for processing a trace data concurrent request based on JVM (Java virtual machine) caching.
Background
The data collection interface service of the original meat dish traceability city platform adopts a mode of WebService+XML, the WebService protocol is heavy, the XML readability is not high, the analysis rule is complex, and the transmission efficiency is low.
The original meat dish traces back the data collection interface service of the city platform, from receiving the request to processing, storing the data and returning the response result, the same thread sequential processing mode is adopted, the processing speed is low, the concurrency capability is not high, and the high concurrency data can not be loaded.
Disclosure of Invention
The technical task of the invention is to provide a method for processing trace data concurrent requests based on JVM cache, which can provide an interface protocol for external providing convenience and rapidness and provide stable and efficient concurrent storage processing capacity for internal.
In order to achieve the above purpose, the present invention provides the following technical solutions:
a method for processing trace back data concurrent request based on JVM buffer memory includes setting unified interface protocol standard and realizing internal processing.
The method provides a convenient and quick interface protocol for the outside and provides stable and efficient concurrent storage processing capacity for the inside.
Preferably, the setting unified interface protocol standard specifically includes setting a transmission protocol, a data format, a character set, a digest algorithm, a date format, a server time, a picture format, a request packet header, a response packet header and a signature.
The transmission protocol is HTTP POST.
The data format is application/json.
The character set is UTF-8.
The digest algorithm is SHA-256.
The date format is yyyy-MM-dd HH: MM: ss.
The server time is Beijing time, 24 hours. The system time with the dockee allows errors within 30 minutes.
The picture format is BASE64 encoding string.
The request header includes a name, meaning, type, mandatory entry, and description.
The response header includes a name, meaning, type, mandatory entry, and description.
The signature algorithm is SHA256 (json + key), which is provided by the authenticator.
Preferably, the signature adopts SHA-256 to sign the complete json character string, and the signature result is attached to the tail part of the json character string and sent together.
Preferably, the implementation internal processing includes interface entry, interface processing, trace back data assembly and write-out, thread pool management, data buffering, and data batch persistence.
Preferably, the interface entry is implemented by a CollectServlet class, which includes obtaining request information, splitting json body and signature part, resolving json body, checking request code, checking authentication and signature, checking request time, calling interface processing class to execute interface processing, obtaining processing result, and writing out corresponding result.
Preferably, the interface process calls an interface implementation method according to the interface type, and the interface implementation method comprises the steps of verifying service parameters, generating a traceability code, writing traceability data and returning a response result.
Preferably, in the interface implementation method, a rule checking module is called to perform service parameter verification, a tracing code generating module is called to generate a tracing code according to a tracing code generating rule, and tracing data is called to write out implementation class asynchronous writing-out tracing data.
Preferably, in the thread pool management, the thread pool manager allocates a thread pool for each data reporting unit, and isolates different reporting units, so that concurrent requests among the reporting units do not affect each other.
Preferably, in the data buffering, a common buffer pool is provided for concurrent data storage of a reporting unit by the data buffer pool, and the peaks are resolved and written in batches.
Preferably, the batch persistence of the data is realized by a similar TraceDB, the TraceDB receives a buffer pool data set through a construction method, submits a write-out task through a thread pool of a single thread which is held in a static state, and realizes the writing of trace-back data into a database.
Compared with the prior art, the method for processing the trace back data concurrency request based on the JVM cache has the following outstanding beneficial effects: the method for processing the concurrent request of the traceable data based on the JVM cache can effectively solve the problems of high concurrent request and large data storage of an important product traceable platform, provides stable and efficient support for data collection of the important product traceable platform, simultaneously provides a convenient and fast interface protocol for the outside, provides stable and efficient concurrent storage processing capacity for the inside, and has good popularization and application values.
Detailed Description
The method for processing the trace back data concurrent request based on the JVM cache according to the invention is further described in detail below with reference to the embodiments.
Examples
The method for processing the trace data concurrent request based on the JVM cache comprises the steps of setting a unified interface protocol standard and realizing internal processing.
The unified interface protocol standard is specifically set up by setting up transmission protocol, data format, character set, abstract algorithm, date format, server time, picture format, request packet header, response packet header and signature.
Wherein the transmission protocol is HTTP POST.
The data format is application/json.
The character set is UTF-8.
The digest algorithm is SHA-256.
The date format is yyyy-MM-dd HH: MM: ss.
The server time is Beijing time, 24 hours. The system time with the dockee allows errors within 30 minutes.
The picture format is BASE64 encoding string.
The request packet header includes a name, meaning, type, mandatory entry and description, see table 1 in particular.
Table 1 request packet header
Interface example:
the response header includes name, meaning, type, mandatory entry and description, see in particular table 2. Table 2 response header
| Name of the name | Meaning of | Type(s) | Must be filled with | Description of the invention |
| BIZ_TYPE | Service type | varchar(32) | Is that | Such as "traceDataResponse" |
| RESP_TIME | Response time | char(19) | Is that | yyyy-MM-dd HH24:MI:SS |
| RESP_ID | Response encoding | varchar(19) | Is that | Unique code generated by a responder |
| REQ_ID | Request encoding | varchar(32) | Is that | Unique code generated by a requestor |
| RESULT | Responsive to data parameters | clob | Is that | Request data parameters |
Interface example:
the signature adopts SHA-256 to sign the complete json character string, and the signature result is attached to the tail part of the json character string and sent together. The signature algorithm is SHA256 (json + key), where the key is provided by the authenticator.
And secondly, realizing internal processing, namely interface entry, interface processing, trace data assembly and writing-out, thread pool management, data buffering and data batch persistence.
(1) Interface entry
The interface entry is implemented by a CollectServlet class, and the basic logic steps include: obtaining a request message, splitting a json body and a signature part, analyzing the json body, checking a request code, checking authentication and signature, checking the request time, calling an interface processing class to execute interface processing, obtaining a processing result and writing out a response result.
(2) Interface processing
According to the interface TYPE BIZ_TYPE, obtaining an instance of an interface processing class TraceDataColllect from an interface processing factory, and calling an interface implementation method process, wherein the basic logic steps of the interface implementation method comprise: and calling a rule checking module to perform necessary service parameter verification, calling a tracing code generating module to generate a tracing code according to a tracing code generating rule, calling tracing data to write out realization class asynchronous writing-out tracing data, and returning a response result.
(3) Trace-back data write-out
Trace back data write out calling code: new (New)
TraceData((String)reqMap.get("REQ_ID"),
(String)reqMap.get("AUTH_ID"),paramMap,traceIdList,
traceStatus).write()；
Construction method TraceData receives the necessary service parameters.
The write method obtains a corresponding thread pool from a thread pool manager according to the authentication code, and calls the object to asynchronously write out the thread pool:
public void write(){
writeManager.getExecutorService(authId).execute(this)；
}
the thread Chi Yibu calls the run method of the object, organizes the service parameters into trace-back data objects to be stored in the run method, and adds the data objects to the buffer pool TraceBuffer.
(4) Thread pool management
A thread pool manager in the thread Chi Guan distributes a thread pool for each data reporting unit, plays a role in isolating the reporting units, ensures that concurrent requests among the reporting units are not affected, and is performed simultaneously; and concurrent requests in the same reporting unit can be performed simultaneously when the maximum concurrent thread number of the thread pool is not exceeded, and the task queue needs to be entered for execution when the maximum concurrent thread number is exceeded, so that thread extrusion caused by excessive concurrency is prevented from influencing system stability.
The core code of the thread pool manager is as follows:
(5) Data buffering
The data buffer pool provides a common buffer pool for concurrent data storage of all reporting units, and plays a role in resolving peaks and writing in batches.
After the data object is added into the buffer pool, judging whether the buffer pool is full or not, or reaching the limited buffer time, if one of two conditions is met, giving the buffer pool data set to the data persistence type traceDB, and writing the data set into the database in batches. And simultaneously, the buffer pool is emptied, and the buffer time is refreshed.
A timer is arranged in the data buffer pool, whether the data in the buffer pool reaches the limited buffer time is detected every fixed frequency, if the condition is met, the data set in the buffer pool is delivered to the data persistence type traceDB, and batch writing is carried out to the database. And simultaneously, the buffer pool is emptied, and the buffer time is refreshed.
The timer is used for solving the problem that the data in the buffer pool cannot be written into the database for a long time because the buffer pool is not full due to the fact that the subsequent data does not arrive for a long time.
The core code of the data buffer pool is as follows:
(6) Data batch persistence
The data batch persistence is implemented by a class TraceDB. Calling codes: new (New)
TraceDB(bufferList).write()；
The traceDB receives the data set of the buffer pool through the construction method, submits a writing-out task through a single-thread pool held in an internal static state, asynchronously calls a run method, and realizes the writing of trace data into the database.
The traceDB asynchronous operation is used as a secondary asynchronous in the flow, and the function of the traceDB asynchronous operation is to solve the problem that a buffer pool cannot end a call in time due to long writing time of a data set, so that new data cannot be received and enter the buffer pool.
The above examples are only preferred embodiments of the present invention, and it is intended that all changes and substitutions that would be within the scope of the present invention are within the ordinary skill in the art.
Claims (1)
1. A method for processing trace back data concurrent request based on JVM buffer storage is characterized in that: the method comprises the steps of setting a unified interface protocol standard and realizing internal processing, wherein the setting of the unified interface protocol standard specifically comprises setting a transmission protocol, a data format, a character set, a summary algorithm, a date format, server time, a picture format, a request packet header, a response packet header and a signature; the signature adopts SHA-256 to sign the complete json character string, and the signature result is attached to the tail part of the json character string and sent together;
the implementation of internal processing comprises interface entry, interface processing, trace data assembly and writing-out, thread pool management, data buffering and data batch persistence;
the interface entry is realized by a CollectServlet class and comprises the steps of obtaining request information, splitting json bodies and signature parts, analyzing json bodies, checking request codes, checking authentication and signature, checking request time, calling interface processing classes to execute interface processing, obtaining processing results and writing out corresponding results;
the interface processing calls an interface implementation method according to the interface type, and the interface implementation method comprises the steps of verifying service parameters, generating a traceability code, writing out traceability data and returning a response result;
in the interface realization method, a rule checking module is called to verify service parameters, a tracing code generating module is called to generate tracing codes according to tracing code generating rules, and tracing data writing-out realization class asynchronous writing-out tracing data is called;
in the thread pool management, a thread pool manager allocates a thread pool for each data reporting unit, and isolates different reporting units so that concurrent requests among the reporting units are not affected; in the data buffering, a common buffer pool is provided for concurrent data storage of a reporting unit by the data buffer pool, and wave crests are resolved and written in batches;
the batch persistence of the data is realized by a similar TraceDB, the TraceDB receives a buffer pool data set through a construction method, submits a write-out task through a single-thread pool which is statically held inside, and realizes the writing of trace-back data into a database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810863056.8A CN109101440B (en) | 2018-08-01 | 2018-08-01 | Method for processing trace data concurrent request based on JVM (Java virtual machine) cache |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810863056.8A CN109101440B (en) | 2018-08-01 | 2018-08-01 | Method for processing trace data concurrent request based on JVM (Java virtual machine) cache |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109101440A CN109101440A (en) | 2018-12-28 |
| CN109101440B true CN109101440B (en) | 2023-08-04 |
Family
ID=64848135
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810863056.8A Active CN109101440B (en) | 2018-08-01 | 2018-08-01 | Method for processing trace data concurrent request based on JVM (Java virtual machine) cache |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109101440B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6766349B1 (en) * | 1999-09-24 | 2004-07-20 | Sun Microsystems, Inc. | Mechanism for obtaining a thread from, and returning a thread to, a thread pool without attaching and detaching |
| JP2005044205A (en) * | 2003-07-24 | 2005-02-17 | Sony Corp | Electronic equipment, format method and format instructing method |
| CN1822545A (en) * | 2005-02-14 | 2006-08-23 | 耶德托存取公司 | Method of controlling communication between a head-end system and a plurality of client systems |
| CN102025606A (en) * | 2009-09-23 | 2011-04-20 | 中兴通讯股份有限公司 | Data transmission method and system |
| CN103366471A (en) * | 2013-06-26 | 2013-10-23 | 福建联迪商用设备有限公司 | Online service concurrent processing method, system and server |
| CN105635298A (en) * | 2016-01-12 | 2016-06-01 | 安徽科力信息产业有限责任公司 | Service isolation principle based data acquisition equipment unified access system |
| WO2016082371A1 (en) * | 2014-11-25 | 2016-06-02 | 中国科学院声学研究所 | Ssh protocol-based session parsing method and system |
| CN105868412A (en) * | 2016-04-28 | 2016-08-17 | 焦点科技股份有限公司 | Multithread data grabbing method based on B2B (business to business) platform |
| CN106790763A (en) * | 2017-01-18 | 2017-05-31 | 北京鑫通运科信息技术有限公司 | A kind of Internet of Things unifies the implementation and system of trace platform |
| CN106991196A (en) * | 2017-05-02 | 2017-07-28 | 山东浪潮通软信息科技有限公司 | A kind of general data traceability method and system |
| CN107360035A (en) * | 2017-07-27 | 2017-11-17 | 北京市天元网络技术股份有限公司 | A kind of data processing method and system |
-
2018
- 2018-08-01 CN CN201810863056.8A patent/CN109101440B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6766349B1 (en) * | 1999-09-24 | 2004-07-20 | Sun Microsystems, Inc. | Mechanism for obtaining a thread from, and returning a thread to, a thread pool without attaching and detaching |
| JP2005044205A (en) * | 2003-07-24 | 2005-02-17 | Sony Corp | Electronic equipment, format method and format instructing method |
| CN1822545A (en) * | 2005-02-14 | 2006-08-23 | 耶德托存取公司 | Method of controlling communication between a head-end system and a plurality of client systems |
| CN102025606A (en) * | 2009-09-23 | 2011-04-20 | 中兴通讯股份有限公司 | Data transmission method and system |
| CN103366471A (en) * | 2013-06-26 | 2013-10-23 | 福建联迪商用设备有限公司 | Online service concurrent processing method, system and server |
| WO2016082371A1 (en) * | 2014-11-25 | 2016-06-02 | 中国科学院声学研究所 | Ssh protocol-based session parsing method and system |
| CN105635298A (en) * | 2016-01-12 | 2016-06-01 | 安徽科力信息产业有限责任公司 | Service isolation principle based data acquisition equipment unified access system |
| CN105868412A (en) * | 2016-04-28 | 2016-08-17 | 焦点科技股份有限公司 | Multithread data grabbing method based on B2B (business to business) platform |
| CN106790763A (en) * | 2017-01-18 | 2017-05-31 | 北京鑫通运科信息技术有限公司 | A kind of Internet of Things unifies the implementation and system of trace platform |
| CN106991196A (en) * | 2017-05-02 | 2017-07-28 | 山东浪潮通软信息科技有限公司 | A kind of general data traceability method and system |
| CN107360035A (en) * | 2017-07-27 | 2017-11-17 | 北京市天元网络技术股份有限公司 | A kind of data processing method and system |
Non-Patent Citations (1)
| Title |
|---|
| 面向高并发复杂民航业务的服务器架构设计;李永进;田丰;倪兆阳;;计算机应用与软件(05);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109101440A (en) | 2018-12-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109344172B (en) | High-concurrency data processing method and device and client server | |
| US9639444B2 (en) | Architecture for end-to-end testing of long-running, multi-stage asynchronous data processing services | |
| CN111352902A (en) | Log processing method and device, terminal equipment and storage medium | |
| CN112134909B (en) | Time sequence data processing method, device, system, server and readable storage medium | |
| US10698962B2 (en) | Analysis of data utilization | |
| CN111628859B (en) | Method and device for auditing short message text based on distributed high concurrency condition | |
| CN107247777A (en) | A kind of form methods of exhibiting inversely counted based on database and system | |
| CN114281803A (en) | Data migration method, device, equipment, medium and program product | |
| CN109101440B (en) | Method for processing trace data concurrent request based on JVM (Java virtual machine) cache | |
| CN112148705A (en) | Data migration method and device | |
| CN113742364B (en) | Data access method, device, electronic equipment, storage medium and program product | |
| CN110781647B (en) | Method for realizing data format verification based on Flink | |
| CN114168607A (en) | Global serial number generation method, device, equipment, medium and product | |
| US9172729B2 (en) | Managing message distribution in a networked environment | |
| CN110705935B (en) | Logistics document processing method and device | |
| CN107995301B (en) | Rapid data receiving and transmitting method based on Internet | |
| CN113076331B (en) | Method, device, equipment, storage medium and program product for processing middle-stage data | |
| US20220108005A1 (en) | Protecting software supply chain using secure log generated in a trusted build environment | |
| CN112650915B (en) | Data interaction method and device based on real-time query | |
| CN110134691B (en) | Data verification method, device, equipment and medium | |
| CN117278659A (en) | Equipment operation data simulation generation method and device | |
| CN115858587A (en) | Request data encryption and decryption method, device, medium and electronic equipment | |
| CN117435618A (en) | Data query method, data query system, electronic device and storage medium | |
| CN111563091A (en) | Non-round-trip batch MongoDB updating method and system | |
| CN115617839A (en) | Method, device, electronic equipment and computer readable medium for inquiring verification data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |