CN116910104A

CN116910104A - Construction industry construction safety intelligent log recording method based on large language model

Info

Publication number: CN116910104A
Application number: CN202310976512.0A
Authority: CN
Inventors: 方东平; 杨乐; 古博韬; 黄玥诚; 郭红领
Original assignee: Tsinghua University
Current assignee: Tsinghua University
Priority date: 2023-08-04
Filing date: 2023-08-04
Publication date: 2023-10-20
Anticipated expiration: 2043-08-04
Also published as: CN116910104B

Abstract

The embodiment of the invention discloses a building industry construction safety intelligent log recording method based on a large language model, which comprises the following steps: acquiring construction safety inspection information, wherein the inspection information comprises inspection text, inspection pictures and inspection audio; converting the inspection picture and the inspection audio into a description text; inputting the inspection text and the description text into a trained large language model to obtain inspection records, risk identification records and safety summary, and inserting the inspection records, the risk identification records and the safety summary into a structured database; and automatically generating a construction safety log by utilizing the structured database. The embodiment realizes the automatic generation of the security log.

Description

Construction industry construction safety intelligent log recording method based on large language model

Technical Field

The embodiment of the invention relates to the technical field of intelligent writing of safety logs, in particular to a construction industry construction safety intelligent log recording method based on a large language model.

Background

With the development of the construction industry, the importance of construction safety is increasing. Construction safety logs serve as an important means for safety personnel to find and rectify problems, and play an important role in construction safety in the construction industry.

In the prior art, security logs are recorded electronically or handwritten every day by security personnel. The security log contains the following parts: basic information (date, time, temperature, etc.), attendance conditions, construction contents, site meetings, technical quality requirements, quality inspection, security inspection, etc. However, due to uneven quality of the safety officer, the hands are insufficient, the construction site is complex and changeable in safety risk, and the safety officer is difficult to complete a safety log on site by using high-quality text.

Although the safety officer can select the problems and conditions by listing the preset options, the input quantity of the text is reduced, and the filling burden is increased. Meanwhile, options can be greatly changed in different stages of construction, and applicability of option modes is limited.

Patent CN116362448A discloses an engineering management sharing platform and method, patent CN115375146a discloses a digital construction integrated platform, and no description is given of how to implement automation of construction safety logs.

Disclosure of Invention

The embodiment of the invention provides a large language model-based construction industry construction safety intelligent log recording method, which realizes automatic generation of a safety log.

In a first aspect, an embodiment of the present invention provides a method for recording a building industry construction security intelligent log based on a large language model, including:

acquiring construction safety inspection information, wherein the inspection information comprises inspection text, inspection pictures and inspection audio;

converting the inspection picture and the inspection audio into a description text;

inputting the inspection text and the description text into a trained large language model to obtain inspection records, risk identification records and safety summary, and inserting the inspection records, the risk identification records and the safety summary into a structured database;

and automatically generating a construction safety log by utilizing the structured database.

In a second aspect, an embodiment of the present invention provides a construction industry construction safety intelligent log recording system based on a large language model, which is characterized by comprising: the system comprises an interaction module, a storage module, a log production module and a large language model module; wherein,

in each log period, the interaction module is used for receiving and processing the inspection text, the inspection picture and the inspection audio of the security personnel and transmitting the inspection text, the inspection picture and the inspection audio to the storage module;

when each log period is finished, the storage module is used for transmitting the received data of the current period to the log production module; the log production module is used for calling the big language model module and generating a security log according to the processed patrol text, patrol pictures and patrol audio; the storage module is also used for transmitting the security log to the interaction module for display to a security officer and modifying information according to user feedback.

In a third aspect, an embodiment of the present invention further provides an electronic device, including:

one or more processors;

a memory for storing one or more programs,

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the large language model based construction safety intelligent logging method of any embodiment.

In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where a computer program is stored, where the program when executed by a processor implements the method for recording the intelligent log of construction safety of building industry based on the large language model according to any embodiment.

According to the automatic recording method for the security log, the unstructured records uploaded by a security officer in a log period are collected, the inspection information of a text type, a picture type and a video type is uniformly converted into a text format, a trained large language model is input, the structured inspection record, the structured risk identification record and the structured security summary are gradually extracted, the structured information is mapped into a security log template, and the construction security log is automatically generated. The method helps the security officer to use simple text description in the inspection, and assisted by other format data such as audio, pictures and the like, so that the security officer can automatically generate a high-quality security log, the cost of filling in a report is reduced, and the security officer is more concentrated on problem discovery and recording, but not on text writing.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a construction industry construction safety intelligent log recording method based on a large language model provided by an embodiment of the invention;

FIG. 2 is a schematic diagram of a security log provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a construction safety intelligent log recording system based on a large language model in the construction industry according to the embodiment of the invention;

fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the invention, are within the scope of the invention.

In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.

In the description of the present invention, it should also be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.

FIG. 1 is a flow chart of a construction industry construction safety intelligent log recording method based on a large language model provided by an embodiment of the invention. The method is suitable for the situation that a large amount of log information exists and the update frequency is high, and is executed by the electronic equipment. As shown in fig. 1, the method specifically includes:

s110, acquiring construction safety inspection information, wherein the inspection information comprises inspection text, inspection pictures and inspection audio.

In the embodiment, the field inspection is used as an information source of the safety log, and the inspection information uploaded by the inspection personnel can be periodically obtained through the interaction module. Optionally, when the security officer performs daily inspection, the security officer needs to record the current occurrence, such as the potential safety hazard, the current work performed, and the like, and input a short text through a mobile phone APP, a mobile phone webpage and a WeChat dialogue, or shoot a mobile phone photo in real time, or input a section of audio through a front-end audio interface. The data is uniformly transmitted to the electronic equipment and used as a data source for generating the security log. For example, the received inspection information may be denoted as input, each input comprising: the method comprises the steps of sending date and time date, a sender pid, an information type and source information msg, wherein the information type comprises a patrol text, a patrol picture and a patrol audio, and the source information refers to collected original data.

S120, converting the inspection picture and the inspection audio into descriptive text.

The step converts the collected inspection pictures and inspection audio into descriptive text, and is convenient for data processing and storage based on unified text form in subsequent operation. Optionally, after the inspection text, the inspection picture and the inspection audio are acquired, the inspection information is classified according to the input/pid of the sender, and the inspection information is archived according to the building project to which the sender belongs; and classifying the inspection information according to the information type input.type, performing data screening and preprocessing, and converting the data into text data.

Further, the process of processing the inspection information into text information according to the different information types includes the following optional embodiments:

in a first alternative embodiment, the type of the inspection information is text, and is expressed as textmsg, and then text=textmsg directly exists, so that additional processing is not needed.

In a second alternative embodiment, the type of the inspection information is a picture, which is denoted as figmsg, and the picture needs to be converted into a corresponding natural language description text. Optionally, the embodiment first uses CNN (Convolutional Neural Networks, convolutional neural network) to extract image features, and introduces an attention mechanism to dynamically select different feature dimensions of the image for description, so as to better capture detailed information of the image; then, when each word is generated, different dimensions of the image are adaptively selected for focusing, and the generated words form a figmsg descriptive text.

Specifically, the process of converting the inspection picture into the descriptive text specifically comprises the following steps:

step one, inputting the inspection picture into an image encoder, and extracting image features with different dimensions. The image encoder can convert the inspection picture I into a group of characteristic vectors through a convolutional neural networkWherein I represents a patrol picture, < >>Representing the output image features, the image feature vector v being defined by a plurality of dimensions _i Composition (S)/(S)>And representing the operation corresponding to the CNN neural network model. The image dimensions herein may be different types of target objects, such as person objects, building objects, etc.

Initializing the model state of the sequence model of the preset structure in the current time step. The sequence model is used for generating word sequences, and can adopt a network structure such as a cyclic neural network (Recurrent Neural Network, RNN) or a variant model (such as a long-short-term memory network). The input to the sequence model is the output of the previous time step and the image feature vector selected by the current attention mechanism. At each time step, the sequence model generates an output vectorThe vector represents the word generated at the current time step. The initial state of the sequence model can be initialized to an all zero vector +. >. The output of the sequence model can be converted into a probability distribution using a softmax function, representing the current time stepGenerating a probability for each word:

（1）

wherein ,representing a sequence of words generated in a previous time step, +.> and />The weight matrix and the bias vector of the sequence model, respectively.

And thirdly, weighting the attention of the image features of each dimension according to the correlation between the image features of each dimension and the model state of the current time step. This step uses an attention mechanism to dynamically select feature vectors of different dimensions in the inspection picture to generate a textual description related to the current time step. Alternatively, the process can be regarded as a weighted summation process in which the weight of each image feature vector is determined by the model state of the current time stepAnd correlation decisions for corresponding feature dimensions in the image.

In a specific embodiment, first, a correlation score between a model state of a sequence model at a current time step and feature vectors of each dimension of a patrol picture is calculated through a FNN (Feedforward Neural Network ):

（2）

wherein ,model state representing the current time step +.>Inspection picture->Correlation score between feature vectors of individual dimensions, < - >Representing the operation of the feed forward neural network.

Then, calculating a correlation score between the current state of the sequence model and feature vectors of each dimension of the inspection picture:

（3）

wherein ,model state representing the current time step +.>And the correlation score between the feature vector of the ith dimension of the inspection picture and the feature vector, wherein N represents the number of the feature dimensions of the inspection image.

Finally, the feature vectors of all dimensions of the inspection picture are weighted according to the relevance score to be used as the output of an attention mechanism:

（4）

wherein ,representing weighted image features,/->A feature vector representing the ith dimension of the inspection image.

And step four, fusing the weighted image features with the output of the sequence model in the current time step to obtain the final output of the current time step, wherein the final output represents the word finally generated in the current time step. This stepThe combination of the attention mechanism and the sequence model is realized: image feature vector for selecting attention mechanismAnd the output vector of the sequence model->In combination, the final output vector for the current time step is generated +.>：

（5）

wherein ,the representation will-> and />Spliced together, are added with> and />Respectively representing the weight matrix and the bias vector of the bonding layer, and tanh represents the hyperbolic tangent function operation.

Step five, according to the model state of the current time stepAnd final output +.>Updating the model state of the next time step, taking the next time step as a new current time step, returning to the attention weighting operation of the image features, and repeating the steps in a circulating way until a termination condition is set. Illustratively, the termination condition may beTo be a final output as a certain stop symbol, such as a period.

And step six, forming a word sequence by the final output of each time step, and taking the word sequence as a description text of the inspection picture.

The above-mentioned processes from step one to step six can be expressed as the following formula:

：

in a third alternative embodiment, the type of the inspection information is audio, and is represented as audio, two steps of noise reduction and speech recognition are required to be performed, and the inspection audio is converted into a corresponding natural language description text. Due to the complexity of the construction site, a great amount of noise may exist, and the inspection audio is first noise reduced to eliminate noise interference therein, so as to improve the audio quality and the intelligibility.

In one embodiment, the noise reduction process includes the steps of:

step (a)1. And carrying out frequency domain low-pass filtering on the audio to filter low-power spectral noise. The method converts the inspection audio to the frequency domain, filters out noise frequency components through a filter, and finally converts the frequency domain signal back to the time domain. Illustratively, the frequency domain filter may be a band reject filter, a band pass filter, or the like. Specifically, the patrol audio is recorded as The corresponding noise signal is +.>The observed noise contaminated patrol audio can be expressed as:

（6）

where n represents the length of the audio sequence. For a pair ofDiscrete fourier transform (Discrete Fourier Transform, DFT) is performed to obtain:

（7）

wherein k represents a pair ofSequence length after DFT conversion, +.>Representation->DFT, & gt>Representation ofIs a DFT of (d). Since the power spectrum of noise is low, it is possible to use one in the frequency domainAnd a low pass filter to remove noise. Let the frequency response of the filter be +.>The output of the filter is:

（8）

will bePerforming an inverse discrete Fourier transform (Inverse Discrete Fourier Transform, IDFT) to obtain a noise-reduced audio signal +.>The method comprises the following steps:

（9）

where M is the number of sampling points of the audio signal. Finally obtained noise-reduced audio signalCan be used for subsequent audio processing.

And step two, performing time domain filtering on the signals after frequency filtering to realize secondary noise reduction. The filtering algorithm in step one has good noise effects for fixed frequencies (consistent with fixed equipment noise in certain construction sites), but poor noise effects for non-linearities and time-variations. Therefore, the step performs a noise reduction algorithm based on time domain filtering, which directly filters the audio signal in the time domain, and removes noise by designing different filters. Illustratively, the time domain filtering algorithm may employ median filtering, kalman filtering, wavelet noise reduction, and the like. Time-domain filtering can effectively remove non-linear and time-varying noise (consistent with intermittent noise in certain construction sites), but has poor noise effects for fixed frequencies. Noise signals with various characteristics can be removed through the combination of time domain filtering and frequency domain filtering, and the filtering quality is improved.

And after the secondary noise reduction, carrying out audio recognition on the inspection audio with the noise removed, and obtaining the description text of the inspection audio. For convenience of distinction and description, the description text of the patrol picture is referred to as a first description text, and the description text of the patrol audio is referred to as a second description text. Optionally, the relationship between the audio signal and the text is modeled by a deep learning model such as DNN (Deep Neural Networks, deep neural network) or RNN, so as to implement speech recognition. This approach tends to be superior in performance to the traditional hidden Markov model approach.

S130, inputting the inspection text and the description text into a trained large language model to obtain inspection records, risk identification records and security summaries, and inserting the inspection records, the risk identification records and the security summaries into a structured database.

The patrol records correspond to source information uploaded by the security officer, each patrol record corresponds to one piece of source information, and the patrol records comprise risk basic information related to each piece of source information. The risk identification records correspond to patrol records, each risk identification record corresponds to one patrol record, and the risk identification records comprise risk level judgment and countermeasures made according to the risk basic information. The safety summary represents a periodic construction summary made based on the inspection records and the risk identification records.

After each log period is finished (for example, after daily work is finished), the inspection text and the description text in the period are organized into set data forms, and are input into a trained large language model, so that inspection records, risk identification records and safety summary are sequentially obtained and used for providing contents for the safety construction log. Meanwhile, the three types of information are respectively inserted into the corresponding structured databases, so that the information is convenient to read and utilize in subsequent operation.

To illustrate the above procedure, a large language model will be first described. The large language model (Large Language Model, LLM) is a deep learning model for obtaining output words from input words, and can generate natural language text or understand the meaning of language text by training a large amount of text data. Traditional language models are often oriented to a certain type of natural language task, such as text classification, translation, question-answering and the like, and LLM enlarges the model scale and displays stronger natural language processing capability (such as learning through context). Illustratively, LLM of the present embodiment employs T0, chatGLM, alpaca, GPT (generating Pre-Trained Transformer), chatGPT, etc. The ChatGPT is changed into a dialogue form on the basis of LLM, and one LLM is used for completing multiple tasks in a mode of giving prompt instructions in the interaction process. More specifically, the language model may be regarded as a black box that accepts a token string as input (where the token may be a Chinese character, or an English word, etc.), and outputs a probability that the token string is a normal human sentence (or fragment). Mathematical formalization is as follows: given a sequence of tokens (u 1, u2,..un), the language model outputs a probability p (u 1, u2,.., un) that indicates the probability that the tokens compose a sentence (or fragment) in order. The following formula expresses the language model described above, expanding this probability into the form of a conditional probability: p (u 1, u2,) un) =p (u 1) pi p (ui|u1, u2,) ui-1. The language model can complete the task of text generation: giving a plurality of generated words in front, calculating the next word with the maximum sequence probability, and outputting the word as a prediction result; the model then adds the predicted word to the given sequence and repeats the process described above, continuing to predict the next word until the next word is predicted to be an end symbol or a desired length is reached.

In a specific embodiment, assuming that the text input into the large language model is text, the final output result is obtained after the following steps.

Step one, input pretreatment. Preprocessing the input text, including word segmentation, stop word removal, part-of-speech tagging and other operations, so as to obtain a preprocessed text sequence. Assuming that the input text is text, the text sequence obtained after preprocessing is token, wherein each token represents a word or a symbol, as shown in table 1:

TABLE 1

And step two, inputting codes. The input preprocessed text sequence is encoded into a vector of values for input into a neural network for computation. Each word may be mapped to a real vector using word embedding techniques and then the entire text sequence represented as a matrix, as shown in table 2.

TABLE 2

And thirdly, inputting the coded text sequence into a large language model neural network for calculation by model calculation to obtain an output vector, wherein the output vector is shown in a table 3. For example, a large language model may be calculated using a recurrent neural network or variant Transformer model, where model parameters have been trained through a large amount of text data during the training phase.

TABLE 3 Table 3

And step four, outputting codes. And decoding the output vector obtained by the model calculation to obtain the final output word output_encoding. Illustratively, decoding may use an output layer to map the output vector into words or characters in a vocabulary, and then combine the words or characters into a segment of text output, as shown in Table 4.

TABLE 4 Table 4

And fifthly, outputting post-treatment. Post-processing the output text, including removing redundant spaces, punctuation, etc., and further text processing and analysis as needed, is shown in table 5.

TABLE 5

Based on the above large language model, the inputting the inspection text and the description text into the trained large language model to obtain an inspection record, a risk identification record and a security summary, and inserting the inspection record, the risk identification record and the security summary into a structured database may include the following steps:

step one, inputting the patrol text and the description text into a trained large language model to obtain a patrol record, and inserting the patrol record into a patrol record database. IN a specific embodiment, first, after each log period is finished, a patrol text and a description text obtained by patrol information processing are organized into first input data IN1, and are input into a trained large language model, a first form result OUT1 is obtained, and each row IN an OUT1 form corresponds to basic information about risks of a patrol record. Optionally, multiple pieces of routing inspection information can be packaged and processed, so that the LLM receives multiple pieces of items, thereby saving the calling times and the total number of tokens. An example of IN1 and OUT1 is given below, where each row of OUT1 information includes time, risk, measure, note that several items can be inserted into the structured database.

Example input 1 IN1

You are a security guard of the construction site, today { time date. Day }, the following section records the situation of the site, please expand it to a complete section, make it simplified, and meet the requirement of security log record:

- { entry 1 time date. Time, text }

- { entry 2 time date. Time, text }

...

- { entry n time date. Time, text }

Example output 1 OUT1

Entry time risk measure remark

xxx|xxx| xxx|xxx| xxx// corresponding entry 1

xxx|xxx| xxx|xxx| xxx// corresponding entry 2

...

xxx|xxx| xxx|xxx| xxx// corresponding entry n

After the patrol record OUT1 is obtained, the SQL database is used for storing the specific information of the patrol record. Optionally, the inspection record database is implemented using a table named injection, which includes the following:

id: record id, integer type, primary key;

date: date, date type;

risk: risk hidden danger and character string type;

measurement: measures are taken, string type.

Accordingly, an injection table may be created first using the following SQL statement:

SQL

CREATE TABLE inspection (

id INTEGER PRIMARY KEY,

date DATE,

risk VARCHAR(255),

measure VARCHAR(255),

...);

then, the information in the inspection record OUT1 is automatically inserted into the injection table, so that the structured storage of the inspection record is completed. The SQL statement is as follows:

Plaintext

INSERT INTO inspection (id, date, risk, measure, ...)

VALUES (1, '2023-05-20', '3 constructors in C1 area do not wear a helmet', 'alert workers to wear a helmet',.

And secondly, adding risk types and countermeasures before and after the risk basic information, inputting a trained large language model together, obtaining a risk identification record and inserting the risk identification record into a risk identification database. IN a specific embodiment, first, additional information such as "judging risk type" and "measure to be taken" is added before and after the result OUT1 obtained IN the first step to obtain second input data IN2; and inputting the IN2 into the trained large language model to obtain a table result OUT2, wherein each row IN the OUT2 table comprises risk identification information of a patrol record. An example of IN2 and OUT2 is given below, wherein each row of information of the OUT2 table contains several items of risk type risk-type, risk level risk-level, measure-token, follow-up recommendation, which can be inserted into the structured database.

Example input 2 IN2

The risk of the project comprises four categories, namely a safety inspection category, a worker safety training education category, a safety product and mechanical entrance condition and a safety summarization category. Wherein, XXX class risk generally describes XXX, grade XX, with the risk of XXX, and countermeasures include XXX. ...

The risk of an item in the current period is given in the form of the following table:

entry time risk measure remark// "example output 1 OUT1"

xxx|xxx| xxx|xxx| xxx// corresponding entry 1

xxx|xxx| xxx|xxx| xxx// corresponding entry 2

...

xxx|xxx| xxx|xxx| xxx// corresponding entry n

Please judge the summary and output it in form of a table.

Example output 2 OUT2

Entry risk type level countermeasure follow-up advice

xxx|xxx| xxx|xxx| xxx// corresponding entry 1

xxx|xxx| xxx|xxx| xxx// corresponding entry 2

...

xxx|xxx| xxx|xxx| xxx// corresponding entry n

After the risk identification record OUT2 is obtained, the SQL database is used for storing specific information of the risk identification record. Alternatively, the risk identification record database is implemented using a table named risk_document, which includes the following:

id: evaluating id, integer type, primary key;

risk-type: risk type, character string type;

risk-level: risk level, integer type.

Accordingly, the following SQL statement may be first used to create the task_document table:

SQL

CREATE TABLE risk_assessment (

id INTEGER PRIMARY KEY,

risk-type VARCHAR(255),

risk-level INTEGER,

...);

Then, the information in the risk identification record OUT2 is automatically inserted into the task_assessment table, and structured storage of the risk identification record is completed. The SQL statement is as follows:

Plaintext

INSERT INTO task_Assemblem (id, task-type, task-level,) value (1, 'worker personal guard,', 3,) is provided.

And thirdly, inputting the patrol record and the risk identification record into a trained large language model together, generating a safety summary and inserting the safety summary into a safety summary database. IN a specific embodiment, first, the inspection record obtained IN the first step and the risk identification record obtained IN the second step are combined to obtain third input data IN3, and the third input data IN3 is input into a trained large language model to generate a security summary OUT3 IN a current log period. An example of IN3 and OUT3 is given below.

Example input 3 IN3

The patrol record in the current period is given in the form of the following table:

entry time risk measure remark

xxx|xxx| xxx|xxx| xxx// corresponding entry 1

xxx|xxx| xxx|xxx| xxx// corresponding entry 2

...

xxx|xxx| xxx|xxx| xxx// corresponding entry n

The corresponding risk identification record is given in the form of a table as follows:

Entry risk type level countermeasure follow-up advice

xxx|xxx| xxx|xxx| xxx// corresponding entry 1

xxx|xxx| xxx|xxx| xxx// corresponding entry 2

...

xxx|xxx| xxx|xxx| xxx// corresponding entry n

Example output 3 OUT3

We completed two routine security rounds and performed specific supervision of the C1 and C3 regions, with no major security issues found in general. However, there is a need to be noted that at 10:35, we find that there is a vehicle in the C3 region that is driven away without a signal, which is a potential safety risk. In the aspect of personnel safety education, a worker in the C1 area is found to wear no safety helmet at 15:47, and the worker is reminded immediately. Regarding the equipment entry, we received two tower crane foundation pens with number 10023 at 10:31 for acceptance certification, the place of origin was Zhejiang, and the certification was already well stored and recorded. Overall, we have maintained good control over safety management except that the behavior of the vehicle in the C3 region requires further attention and handling. In the future, we will continue to strictly adhere to the safety regulations and ensure the safety of the construction site.

After the security summary OUT3 is obtained, the SQL database is used for storing the specific content of the security summary. Alternatively, the security summary database is implemented using a table named day_summary, which includes the following:

id: record id, integer type, primary key;

surmmary: summarizing the content, the string type.

Accordingly, a day_summary table may be first created using the following SQL statement:

SQL

CREATE TABLE daily_summary (

id INTEGER PRIMARY KEY,

summary VARCHAR(255));

and then, automatically inserting the security summary OUT3 into a day_summary table to finish the structured storage of the security summary. The SQL statement is as follows:

Plaintext

INSERT INTO daily_summary (id, summary)

VALUES (1, 'we completed two routine security rounds and were specially supervised for the C1 and C3 regions.,').

Further, after the security summary OUT3 is obtained, a specific evaluation method can be used for evaluating the quality of the generated text so as to ensure the correctness, fluency and integrity of the generated text. The fluency is evaluated by using an automatic evaluation index method, and the quality of the generated text is evaluated by calculating the similarity between the generated text and the reference text. Illustratively, the similarity is calculated by comparing the n-gram overlap of the generated text and the reference text using BLEU (Bilingual Evaluation Understudy, bilingual evaluation alternative) as an automatic evaluation index. Correctness can use natural language reasoning techniques to detect logical errors and contradictions in generating text. The integrity can help to determine the key content for generating the abstract through keyword extraction, and ensure that the abstract contains all key information in the original text. If the text intrinsic quantity evaluation index is lower than a certain threshold value, the system reminds a safety officer to adjust and supplement the input text of the IN1, and the data processing process of the large language model is repeated until the evaluation index meets the requirement.

Furthermore, the large language model is first trained before running the large language model. In order to obtain a large language model suitable for construction log generation, a training sample library can be constructed through construction safety data formatting and/or construction safety field human question-answering; and performing instruction fine adjustment and weight fine adjustment on a preset large language model by using the training sample library to obtain a trained large language model. Specifically, first, existing data may be formatted, or questions of a human may be collected, and corresponding answers given to construct training samples. Then, adopting an instruction fine tuning method to receive training samples in a plurality of instruction formats as input, wherein the samples are associated with specific tasks; then, the internal parameters of the LLM model are adjusted by a fine tuning algorithm so that the output of the LLM model is more matched with the training samples. The instructions and tasks herein are each constructed from the inputs IN and OUT that are expected by the large language model. Specifically, the "instruction" is an IN and OUT pair IN the log generation process, for example, an IN1 format input may be configured for an actual scene, and an OUT1 format output may be generated, so as to form a set of training samples (where OUT may be manually written or may be generated by a machine). For the three tasks (IN 1, IN2, IN 3), several training samples can be constructed and input into a large language model for fine tuning.

And S140, automatically generating a construction safety log by utilizing the structured database.

The embodiment provides a model of a construction safety log, which comprises a patrol record part, a risk assessment record part and a safety summary part, and corresponds to the patrol record database, the risk identification record database and the safety summary database constructed in S130 one by one. Optionally, the required data content is read from the inspection record database, the risk identification database and the safety summary data respectively, and the construction safety log template is embedded according to the mapping rule, so that the construction safety log can be automatically generated.

In one embodiment, the specific format and automatic generation process of the three parts of the security log paradigm are as follows:

the inspection recording part consists of a plurality of 'risk records' and 'daily work records', and each record corresponds to inspection text, inspection picture information or inspection audio submitted to the system by a security officer one by one. Illustratively, each record contains an item as shown in Table 6; and reading out the structured information from the inspection database according to the line, and obtaining the content of the inspection record part in the security log. The SQL statement for querying the inspection database is as follows:

Plaintext

SELECT FROM injection; # query all records in the injection table

TABLE 6

The risk assessment record is composed of a plurality of records, and each record corresponds to one existing risk hidden trouble, and corresponds to the type, grade and countermeasure. Illustratively, the risk assessment record is as shown in table 7; and reading out the structured information from the risk identification database according to the rows, so that the data of the risk evaluation record part in the security log can be obtained. The SQL statement for querying the risk identification database is as follows:

Plaintext

SELECT FROM task_estimate; # queries all records in the task_estimate table

TABLE 7

The security summary part includes a section of 300-500 words, so that the security condition of the engineering in the current log period is summarized, and the text content is read from the security summary database and inserted into the template. The resulting security log is shown in fig. 2. The SQL statement for querying the security summary database is as follows:

Plaintext

SELECT FROM day_summary; # query all records in the day_summary table

It should be noted that fig. 2 only shows an optional display template of the security log, in practical application, part of the content may be read from the database according to the need and displayed in the security log, and the display format and the appearance picture of the security log may be flexibly set, which is not limited in this embodiment.

According to the automatic recording method for the security log, unstructured records uploaded by a security officer in a log period are collected, inspection information of a text type, a picture type and a video type is uniformly converted into a text format, a trained large language model is input, structured inspection records, structured risk identification records and structured security summaries are gradually extracted, the structured information is mapped into a security log template, and a construction security log is automatically generated. The method helps the security officer to use simple text description in the inspection, and assisted by other format data such as audio, pictures and the like, so that the security officer can automatically generate a high-quality security log, the cost of filling in a report is reduced, and the security officer is more concentrated on problem discovery and recording, but not on text writing.

FIG. 3 is a schematic diagram of a construction safety intelligent log recording system based on a large language model in the construction industry according to the embodiment of the invention. The system is based on a large language model, text recognition, audio recognition and database technology, and combines hardware such as a smart phone and a high-video-memory workstation to realize automatic production of a security log after the security personnel patrols and examines natural description risks on site. As shown in FIG. 3, the system comprises four technical modules, namely an interaction module, a storage module, a log production module and a large language model module.

Based on this system, the present embodiment divides the workflow into four parts per log period (e.g., daily): during operation, each period ends, the iteration is modified and the log generation presentation is performed.

The first part, during operation, the interactive module can receive the inspection text, the inspection picture and the inspection audio of the security officer, and preprocess and clean the data and transmit the data to the storage module.

The second part, when each log period is finished, the storage module transmits the record received in the current period to the log production module to generate a log; the log production module calls a large language model module to convert unstructured text information into structured information; this information is then transferred back to the storage module, updating the recorded items according to the security log template.

The third part, the storage module transmits the structured information to the interaction module; the interaction module displays for the safety officer, receives feedback information of the user, and updates corresponding content in the storage module.

And a fourth part, log generation and display. And reading the recorded content from the storage module, and generating a report in PDF/Word/Excel and other formats according to the security log template. These reports are transmitted to the interactive module for eventual presentation to the security officer.

Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 4, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device may be one or more, one processor 60 being taken as an example in fig. 4; the processor 60, the memory 61, the input means 62 and the output means 63 in the device may be connected by a bus or other means, in fig. 4 by way of example.

The memory 61 is used as a computer readable storage medium for storing software programs, computer executable programs and modules, such as program instructions/modules corresponding to the intelligent logging method for construction safety of building industry based on a large language model in the embodiment of the present invention. The processor 60 executes various functional applications of the apparatus and data processing by running software programs, instructions and modules stored in the memory 61, i.e., implements the above-described construction safety intelligent logging method based on the large language model.

The memory 61 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application program required for functions; the storage data area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some examples, memory 61 may further comprise memory remotely located relative to processor 60, which may be connected to the device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input means 62 may be used to receive entered numeric or character information and to generate key signal inputs related to user settings and function control of the device. The output 63 may comprise a display device such as a display screen.

The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the program is executed by a processor to realize the building industry construction safety intelligent log recording method based on the large language model of any embodiment.

The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the C-programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. The intelligent log recording method for construction safety of building industry based on large language model is characterized by comprising the following steps:

2. The method of claim 1, wherein the converting the inspection picture and inspection audio to descriptive text comprises:

Inputting the inspection picture into an image encoder, and extracting image features with different dimensions;

initializing the model state of the sequence model in the current time step;

according to the correlation between the image characteristics of each dimension and the model state of the current time step, carrying out attention weighting on the image characteristics of each dimension;

fusing the weighted image features with the output of the sequence model at the current time step to obtain the final output of the current time step, wherein the output and the final output of the sequence model represent words generated at the current time step;

updating the model state of the next time step according to the model state and the final output of the current time step, taking the next time step as a new current time step, and returning to the attention weighting operation of the image features until a termination condition is set;

and forming a word sequence by the final output of each time step, and taking the word sequence as a first description text of the inspection picture.

3. The method of claim 1, wherein the converting the inspection picture and inspection audio to descriptive text comprises:

carrying out frequency domain low-pass filtering on the inspection audio to filter low-power spectrum noise;

Performing time domain filtering on the frequency filtered signal to realize secondary noise reduction;

and carrying out audio recognition on the signal after the secondary noise reduction to obtain a second description text of the patrol audio.

4. The method of claim 1, further comprising, prior to said entering said inspection text and description text into a trained large language model, obtaining an inspection record, a risk identification record, and a security summary and inserting into a structured database:

building a training sample library through construction safety data formatting and/or construction safety field human question-answering;

and performing instruction fine adjustment and weight fine adjustment on the large language model by using the training sample library to obtain a trained large language model.

5. The method of claim 1, wherein said entering the inspection text and description text into a trained large language model results in inspection records, risk identification records, and security summaries and inserting into a structured database, comprising:

inputting the inspection text and the description text into a trained large language model to generate an inspection record, wherein the inspection record comprises risk basic information;

adding risk types and countermeasures before and after the risk basic information, and inputting a trained large language model together to generate a risk identification record;

And inputting the patrol record and the risk identification record into a trained large language model together to generate a safety summary.

6. The method of claim 5, further comprising, after said co-entering said inspection record and risk identification record into a trained large language model to generate a security summary:

comparing the security summary with the n-gram overlap of the reference text to verify the fluency of the security summary;

detecting logical errors and contradictions of the security summary by using a natural language reasoning technology to verify the correctness of the security summary;

and determining key contents of the security summary through keyword extraction so as to verify the integrity of the security summary.

7. The method of claim 1, wherein said automatically generating a construction safety log using said structured database comprises:

and reading required data content from each structured database according to rows, embedding a construction safety log template according to the mapping rule, and automatically generating a construction safety log.

8. A large language model based construction safety intelligent log recording system for construction, comprising: the system comprises an interaction module, a storage module, a log production module and a large language model module; wherein,

9. An electronic device, comprising:

one or more processors;

a memory for storing one or more programs,

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the large language model based construction safety intelligent logging method of any one of claims 1-7.

10. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, implements the large language model based construction safety intelligent logging method of any one of claims 1 to 7.