CN112148205A - Data management method and device - Google Patents
Data management method and device Download PDFInfo
- Publication number
- CN112148205A CN112148205A CN201910576111.XA CN201910576111A CN112148205A CN 112148205 A CN112148205 A CN 112148205A CN 201910576111 A CN201910576111 A CN 201910576111A CN 112148205 A CN112148205 A CN 112148205A
- Authority
- CN
- China
- Prior art keywords
- data
- deep learning
- user
- memory
- group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Human Computer Interaction (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data management method and device, and belongs to the field of data processing. The method comprises the following steps: the deep learning platform reads data of the data set through the first memory; the deep learning platform reads and writes deep learning task data through the second memory, the deep learning task is run, and the first memory and the second memory are physically isolated. Therefore, the data set data and the deep learning task data can be stored and read and written separately, reading and management of the data set data and the deep learning task data are facilitated, and data reading efficiency in the deep learning process is improved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data management method, apparatus, and storage medium.
Background
With the development of Artificial Intelligence (AI) technology, various deep learning platforms for providing training and reasoning services for deep learning models have emerged. In general, a large amount of data set data is needed in the deep learning process, and a large amount of deep learning task data is also generated in the process of using the deep learning platform by a user.
Disclosure of Invention
The embodiment of the application provides a data management method and a data management device, and the technical scheme is as follows:
in one aspect, a data management method is provided, and the method includes:
the deep learning platform reads data of the data set through the first memory;
the deep learning platform reads and writes deep learning task data through a second memory and runs a deep learning task, and the first memory and the second memory are physically isolated.
Optionally, the method further comprises:
and if the user grade of the reference user account is greater than a grade threshold value or the reference user account is a user account of a preset user, configuring a data writing authority of the first memory for the reference user account, wherein the data writing authority of the first memory is used for allowing data set data to be written in or deleted from the first memory.
Optionally, the user accounts of the deep learning platform are divided into a plurality of user groups, each user group includes at least one user account, and the method includes:
allocating a first storage resource for storing data set data for each of the plurality of user groups based on available storage resources of the first storage, the first storage resource of each user group being shared by user accounts in each user group; and/or the presence of a gas in the gas,
and allocating a second storage resource for storing deep learning task data for each user group in the plurality of user groups based on the available storage resource of the second storage, wherein the second storage resource of each user group is shared and used by the user accounts in each user group.
Optionally, a reference user group is allocated with a third storage resource, the third storage resource includes the first storage resource and/or the second storage resource, the reference user group is any user group in the plurality of user groups, and the method further includes:
and allocating the use range of the third storage resource for any user account in the reference user group based on the storage resource allocation request of the group primary account of the reference user group.
Optionally, the user accounts of the deep learning platform are divided into a plurality of user groups, each user group includes at least one user account, and the method further includes:
based on a data set data sharing request of a group primary account of a reference user group, sharing data set data of the reference user group to all user accounts or part of user accounts in the reference user group, or sharing data set data of the reference user group to all user accounts or part of user accounts in other reference user groups, wherein the reference user group is any one of the plurality of user groups.
Optionally, the sharee of the data set data has read-only right for the shared data set data, and does not have right to share the shared data to other user accounts.
Optionally, the method further comprises:
based on a deep learning task data sharing request of a reference user account, the deep learning task data of the reference user account is shared to other user accounts, and the reference user is any user account in the deep learning platform.
Optionally, the sharee of the deep learning task data has at least one of read permission and write permission for the shared deep learning task data.
In one aspect, a data management method is provided, and the method includes:
displaying a web page provided by a deep learning platform;
when a data browsing operation is detected based on the web page, a file or a directory corresponding to deep learning data is displayed on the web page, wherein the deep learning data comprises data set data and/or deep learning task data.
Optionally, the deep learning data includes deep learning data of a current user account and/or deep learning data shared by other user accounts.
Optionally, the current user account has a read-only permission for data set data in the deep learning data, and has a read permission and a write permission for deep learning task data in the deep learning data.
Optionally, a file in a reference file format is displayed on the web page, and the method further includes:
and when the trigger operation on the file with the reference file format is detected, displaying the file content of the file with the reference file format.
In one aspect, a data management method is provided, and the method includes:
accessing deep learning data stored in a memory through a file directory mounted on an operating system of user equipment by the memory, wherein the deep learning data comprises data set data or deep learning task data; alternatively, the first and second electrodes may be,
the deep learning data stored in the memory is remotely accessed through a secure shell protocol SSH.
In one aspect, a data management apparatus is provided, which is applied to a deep learning platform, and the apparatus includes:
the reading module is used for reading the data set data through the first memory;
and the reading and writing module is used for reading and writing the deep learning task data through a second memory and running the deep learning task, and the first memory and the second memory are physically isolated.
Optionally, the apparatus further comprises:
the configuration module is configured to configure a data writing permission of the first storage for a reference user account if a user grade of the reference user account is greater than a grade threshold or the reference user account is a user account of a preset user, where the data writing permission of the first storage is used to allow data set data to be written or deleted in the first storage.
Optionally, the user accounts of the deep learning platform are divided into a plurality of user groups, each user group includes at least one user account, and the apparatus includes:
a first allocation module, configured to allocate, for each user group in the plurality of user groups, a first storage resource for storing data set data based on an available storage resource of the first memory, where the first storage resource of each user group is shared by user accounts in each user group; and/or the presence of a gas in the gas,
a second allocating module, configured to allocate, for each user group in the plurality of user groups, a second storage resource for storing deep learning task data based on an available storage resource of the second memory, where the second storage resource of each user group is shared and used by user accounts in each user group.
Optionally, a reference user group is allocated with a third storage resource, the third storage resource includes the first storage resource and/or the second storage resource, the reference user group is any user group in the plurality of user groups, and the apparatus further includes:
and the third allocation module is used for allocating the use range of the third storage resource to any user account in the reference user group based on the storage resource allocation request of the group primary account of the reference user group.
Optionally, the user accounts of the deep learning platform are divided into a plurality of user groups, each user group includes at least one user account, and the apparatus further includes:
the first sharing module is configured to share data set data of a reference user group to all user accounts or a part of user accounts in the reference user group, or to all user accounts or a part of user accounts in other reference user groups, based on a data set data sharing request of a group primary account of the reference user group, where the reference user group is any one of the plurality of user groups.
Optionally, the sharee of the data set data has read-only right for the shared data set data, and does not have right to share the shared data to other user accounts.
Optionally, the apparatus further comprises:
the second sharing module is used for sharing the deep learning task data of the reference user account to other user accounts based on a deep learning task data sharing request of the reference user account, wherein the reference user is any user account in the deep learning platform.
Optionally, the sharee of the deep learning task data has at least one of read permission and write permission for the shared deep learning task data.
In one aspect, a data management apparatus is provided, where the apparatus includes:
the first display module is used for displaying a web page provided by the deep learning platform;
and the second display module is used for displaying a file or a directory corresponding to deep learning data on the web page when a data browsing operation is detected based on the web page, wherein the deep learning data comprises data set data and/or deep learning task data.
Optionally, the deep learning data includes deep learning data of a current user account and/or deep learning data shared by other user accounts.
Optionally, the current user account has a read-only permission for data set data in the deep learning data, and has a read permission and a write permission for deep learning task data in the deep learning data.
Optionally, a file in a reference file format is displayed on the web page, and the apparatus further includes:
and the third display module is used for displaying the file content of the file in the reference file format when the trigger operation on the file in the reference file format is detected.
In one aspect, a data management apparatus is provided, where the apparatus includes:
the device comprises a first access module, a second access module and a third access module, wherein the first access module is used for accessing deep learning data stored in a memory through a file directory mounted on an operating system of user equipment by the memory, and the deep learning data comprises data set data or deep learning task data; alternatively, the first and second electrodes may be,
and the second access module is used for remotely accessing the deep learning data stored in the memory through a secure shell protocol SSH.
In one aspect, a data management apparatus is provided, the apparatus including:
one or more processors;
one or more memories for storing the one or more processor-executable instructions;
wherein the one or more processors are configured to perform any of the data management methods described above.
In one aspect, a non-transitory computer-readable storage medium is provided, wherein instructions in the storage medium, when executed by a processor, enable the processor to perform any of the above-described data management methods.
The technical scheme provided by the embodiment of the application has the following beneficial effects:
in the embodiment of the application, the deep learning platform can read data set data through the first storage, read and write deep learning task data through the second storage, and the first storage and the second storage are physically isolated, so that the data set data and the deep learning task data are stored and read and written separately, reading and management of the data set data and the deep learning task data are facilitated, and data reading efficiency in a deep learning process is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic illustration of an implementation environment to which embodiments of the present application relate;
FIG. 2 is a schematic diagram of a storage cluster provided in an embodiment of the present application;
FIG. 3 is a schematic illustration of another implementation environment to which embodiments of the present application relate;
FIG. 4 is a schematic diagram illustrating an operation of deep learning data according to an embodiment of the present disclosure;
fig. 5 is a flowchart of a data management method provided in an embodiment of the present application;
FIG. 6 is a schematic diagram of a shared data provided in an embodiment of the present application;
FIG. 7 is a flow chart of another data management method provided by an embodiment of the application;
FIG. 8 is a flow chart of another data management method provided by an embodiment of the present application;
fig. 9 is a block diagram of a data management apparatus according to an embodiment of the present application;
FIG. 10 is a block diagram of another data management device provided in an embodiment of the present application;
FIG. 11 is a block diagram of another data management apparatus provided in an embodiment of the present application;
fig. 12 is a schematic structural diagram of a data management apparatus according to an embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
Before explaining the embodiments of the present application in detail, an application scenario of the embodiments of the present application will be described.
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
Before explaining the embodiments of the present application in detail, an application scenario of the embodiments of the present application will be described.
At present, deep learning models are widely applied to various industries. For example, in the field of intelligent transportation, vehicle images acquired by monitoring equipment can be identified and classified through a deep learning model. For another example, in the field of security, a face image acquired by an image acquisition device may be identified through a deep learning model, and the like.
Before image recognition is performed by using a deep learning model, the deep learning model needs to be trained firstly. Based on the above, various deep learning platforms for deep learning training calculation are currently available, and users can use platform functions of the deep learning platforms as required to train and reason models. In the deep learning process, a large amount of data set data is needed. Moreover, a great amount of deep learning task data, such as a user log or data generated during execution of a deep learning task, may also be generated during use of the deep learning platform by a user. In order to guarantee the operation performance of the deep learning platform, the data needs to be managed effectively.
Next, an implementation environment related to the embodiments of the present application will be described.
Fig. 1 is a schematic diagram of an implementation environment related to an embodiment of the present application, and as shown in fig. 1, the implementation environment includes a deep learning platform 10 and a memory 20, the memory 20 includes a first memory 21 and a second memory 22, and the first memory 21 and the second memory 22 are two different memories that are physically separated. The first memory 21 is used for storing data set data, and the second memory 22 is used for storing deep learning task data, so that the deep learning platform 10 can read the data set data through the first memory 21 and read the deep learning task data through the second memory 22.
As one example, as shown in FIG. 2, the memory 20 may be a memory cluster that includes a plurality of servers. In the embodiment of the present application, the storage cluster may be divided into two sub-storage clusters, where one of the two sub-storage clusters serves as the first storage 11, and the other one serves as the second storage 12. The first memory 11 comprises at least one server for storing data set data; the second memory 12 includes at least one server for storing deep learning task data. Furthermore, the first memory 11 and the second memory are physically isolated.
In addition, the deep learning platform is also provided with a data sharing function, and users can share the individual deep learning data to other users through the sharing function, for example, the individual deep learning task data is shared to other users.
Fig. 3 is a schematic diagram of another implementation environment related to an embodiment of the present application, and as shown in fig. 3, the implementation environment includes a user device 30, a deep learning platform 10, and a memory 20, and the user device 20 may access deep learning data stored in the memory 20 through the deep learning platform. The user equipment 30 may be a terminal or a personal server. The memory 20 may include a first memory for storing data set data and a second memory for storing deep learning task data, such that the data set data and the deep learning task data are stored separately, although in another embodiment, the memory 20 may store data set data and deep learning task data in a mixed manner.
In the embodiment of the present application, the user device 30 may browse the deep learning data through a web page provided by the deep learning platform, or may access the deep learning data stored in the memory 20 through an operating system such as Windows or Iinux supported by the user device 30. The deep learning data may be data set data or deep learning task data.
As an example, referring to fig. 4, the operation mode of the deep learning data may be as shown in fig. 4, that is, a user may browse the deep learning data through a web page, open the deep learning data through the web page, access the deep learning data through an operating system such as Windows or Iinux, and read and write the deep learning data during a task execution process of the user, so that a plurality of modes for operating the deep learning data are provided for the user, which facilitates the operation of the user and improves the usability of the deep learning platform.
Next, a detailed description is given of a data management method provided in an embodiment of the present application.
Fig. 5 is a flowchart of a data management method provided in an embodiment of the present application, where the method is applied to a deep learning platform, and as shown in fig. 5, the method includes the following steps:
step 501: the deep learning platform reads the data set data through the first memory.
The data set data refers to sample data set data used for deep learning, such as an image set or a sentence set. The first memory is used for storing data set data and is used for reading operation of a user.
As one example, a storage cluster of a deep learning platform may be physically isolated into two sub-storage clusters, one as a first storage and the other as a second storage. The first memory is used for storing data set data, and the second memory is used for storing deep learning task data.
As an example, to avoid frequent reading and writing of data set data, in the deep learning platform, a general user account only has a reading permission of the data set data, but does not have a writing permission of the data set data, and only users with higher permission levels, such as a group leader or a system operation and maintenance user, have the reading permission of the data set data.
For example, if the user level of the reference user account is greater than the level threshold, or the reference user account is a user account of a preset user, the data writing permission of the first memory is configured for the reference user account, and the data writing permission of the first memory is used for allowing data set data to be written or deleted in the first data system.
Wherein, the grade threshold value and the preset user can be preset. The preset user can be a group leader or a system operation and maintenance user. For example, the user level may be used to indicate a hierarchical position of the user account in the organizational structure, and if the user account is higher in the organizational structure, the user level of the user account is higher. For example, the data writing authority of the first storage is configured for the user account numbers of the group leader and above the group leader level in the organizational structure, so that only the personnel of the group leader and above the group leader level have the data writing authority of the first storage, and the ordinary user only has the data reading authority of the first storage and does not have the data writing authority of the first storage.
Step 502: the deep learning platform reads and writes deep learning task data through the second memory, the deep learning task is run, and the first memory and the second memory are physically isolated.
The deep learning task data refers to data involved in a deep learning task execution process, such as deep learning codes generated in playground processes of encoding, debugging, compiling and the like performed by a user, or user information such as account information and authority information of the user. In addition, deep learning tasks can be executed in the second memory, and threads for encoding, debugging and compiling operations by a user can be implemented in the second memory.
The second memory is used for storing deep learning task data and is used for frequent read-write operation of a user. Because the deep learning task data needs to be frequently changed, the reading permission and the writing permission of the deep learning task data can be distributed to the user account number capable of accessing the deep learning task data, so that the user can frequently read and write the deep learning task data.
As one example, the user accounts of the deep learning platform may be divided into a plurality of user groups, each user group including at least one user account. For example, users of the deep learning platform may be divided into groups according to an organizational scheme, each group having a group length. As one example, storage resources may be allocated for a group in units of groups.
For example, a first storage resource for storing data set data may be allocated for each of a plurality of user groups based on available storage resources of a first memory. The first storage resource of each user group is shared by the user accounts in each user group, that is, the members in the group can share the first storage resource of the group.
For another example, a second storage resource for storing deep learning task data may be allocated for each of the plurality of user groups based on available storage resources of the second memory. The second storage resource of each user group is shared by the user accounts in each user group, that is, the members in the group can share the second storage resource of the group.
As an example, if the reference user group is allocated with the third storage resource, the usage scope of the third storage resource may also be allocated for any user account in the reference user group. The third storage resource comprises a first storage resource used for storing data set data and/or a second storage resource used for storing deep learning task data, and the reference user group is any one of a plurality of user groups. For example, a range of accessible data sets, or an upper limit of use of a second storage resource that can be used, may be configured for a certain user account.
The use range of the third storage resource of any user account in the reference user group may be allocated by default by the deep learning platform, or may be allocated by the group primary account of the reference user group.
For example, the usage scope of the third storage resource may be allocated to any user account in the reference user group based on the storage resource allocation request of the group primary account of the reference user group. The group primary account of the reference user group refers to a user account of a group administrator of the reference user group, that is, a user account of a group leader. That is, the use range of the storage resource of each group member can be set by the group leader.
As an example, when the storage resource usage of the reference user group is over-supported, the reference user group cannot perform the task submitting operation.
As one example, the user accounts of the deep learning platform may also share data set data or deep learning task data.
In one possible embodiment, if the user accounts of the deep learning platform are divided into a plurality of user groups, each user group including at least one user account, the data set data of the reference user group may be shared based on the data set data sharing request of the group primary account of the reference user group. For example, the data set data of the reference user group may be shared to all user accounts or a part of user accounts in the reference user group, or shared to all user accounts or a part of user accounts in other reference user groups. The reference user group is any one of the plurality of user groups.
For example, the group leader may manage the data set data of the group, and may share the data set data of the group with the group members or other group members of the group.
As one example, sharees of data set data have read-only rights to the shared data set data. That is, the sharee can only perform read operations on the shared data set data and cannot perform write operations, i.e., cannot make changes to the shared data set data.
As one example, the sharees of the data set data do not have the authority to share the shared data set data to other user accounts. That is, the sharee cannot perform the secondary sharing operation on the shared data set data. For example, if a shares data set data to B, B can no longer share the data set data to C.
In another possible embodiment, the deep learning task data of the reference user account can be shared to other user accounts based on the deep learning task data sharing request of the reference user account, and the reference user is any user account in the deep learning platform. That is, each user of the deep learning platform may share the personal deep learning task data to other users.
As one example, the sharee of the deep learning task data has at least one of read rights and write rights to the shared deep learning task data. That is, the sharee can perform not only a read operation but also a write operation on the shared deep learning task data, that is, can change the shared deep learning task data.
As an example, when sharing data set data or deep learning task data, the data set data or deep learning task data may be shared with other users by sharing access paths or access permissions of the data set data or deep learning task data with other users, so that multiple users may share the same stored data, and frequent backup of the data is avoided.
For example, as shown in fig. 6, the group leader may share the data set data of the group to the group members, may share the data of the individual deep learning task to the group members, and may set the use range of the storage resources of the group members. In addition, the group leader may also share the data set data of the group or the deep learning task data of the individual to the group leader or the group member of other groups. The members in the same group can share the data of the individual deep learning tasks, and the members in different groups can share the data of the individual deep learning tasks. In addition, the shared data set data only provides read-only permission, and the shared deep learning task data can provide read permission and write permission.
In the embodiment of the application, the deep learning platform can read data set data through the first storage, and read and write deep learning task data through the second storage, and the first storage and the second storage are physically isolated, so that the data set data and the deep learning task data are stored and read and written separately, reading and management of the data set data and the deep learning task data are facilitated, and data reading efficiency in a deep learning process is improved. In addition, by means of storage separation and read-write separation of the data set data and the deep learning task data, data deletion operation of data set storage hardware can be reduced, and the service life of a storage hard disk of a server where the data set is stored is effectively prolonged.
Fig. 7 is a flowchart of another data management method provided in an embodiment of the present application, where the method is applied to a user equipment, where the user equipment may be a terminal or a personal server, as shown in fig. 7, and the method includes the following steps:
step 701: and displaying the web page provided by the deep learning platform.
In the embodiment of the application, the deep learning platform can provide a web page for a user, the web page has a visual browsing function, and the user can browse deep learning data of the deep learning platform through the web page. The deep learning data comprises data set data and/or deep learning task data
Step 702: when a data browsing operation is detected based on the web page, a file or a directory corresponding to the deep learning data is displayed on the web page.
The deep learning data may be deep learning data of a current user account, or deep learning data shared by other user accounts. The deep learning data shared by other user accounts refers to the deep learning data shared by other user accounts to the user account.
For example, the deep learning data may include deep learning task data for the current user account, as well as deep learning task data shared by other user accounts.
As an example, the current user account has read-only permission for the data set data in the deep learning data, and has read permission and write permission for the deep learning task data in the deep learning data. Of course, if the user level of the current user account is higher or is the user account of the preset user, the current user account may also have the write permission for the data set data in the deep learning data.
As an example, if a file in a reference file format is displayed on a web page, when a trigger operation on the file in the reference file format is detected, the file content of the file in the reference file format may also be displayed. The reference file format is a preset file format, such as txt, jpg or html format.
That is, the user may also open a file in a reference file format, such as a txt, jpg, or html file, on the web page provided by the deep learning platform.
In the embodiment of the application, the web page provided by the deep learning platform has a data browsing function, a user can conveniently browse and open deep learning data through the web page, the data operation mode is simple and convenient, and the usability of the deep learning platform is improved.
Fig. 8 is a flowchart of another data management method provided in an embodiment of the present application, where the method is applied to a user equipment, where the user equipment may be a terminal or a personal server, as shown in fig. 8, and the method includes the following steps:
step 801: and accessing the deep learning data stored in the memory through a file directory mounted on an operating system of the user equipment by the memory, wherein the deep learning data is data set data or deep learning task data.
The memory may be the first memory or the second memory, or may be a memory capable of storing both the data set data and the deep learning task data. The operating system of the user equipment can be a Windows operating system or a Linux operating system.
By mounting the memory partition of the memory on a file directory of an operating system of the user equipment, the user equipment can directly access the memory through the file directory.
For example, the user device may access the data set data stored by the first memory through a file directory mounted on an operating system of the user device by the first memory, or access the deep learning task data stored by the second memory through a file directory mounted on an operating system of the user device by the second memory.
As an example, when accessing the data set data stored in the first memory, the user only has read permission, and when accessing the deep learning task data stored in the second memory, the user has read permission and write permission. Of course, if the user level of the user is higher, the user may have write permission when accessing the data set data stored in the first storage.
As an example, the user device may access the deep learning data of the deep learning platform through FTP (File Transfer Protocol) or spft (Secure File Transfer Protocol).
In another embodiment, the user equipment may also remotely access the deep learning data stored in the memory through SSH (Secure Shell, Secure Shell protocol).
That is, the deep learning platform may also provide an SSH function, and the user equipment may remotely access the deep learning data stored in the memory through the SSH function provided by the deep learning platform.
In the embodiment of the application, a user can access deep learning data of the deep learning platform through personal user equipment, so that the user can conveniently use a corresponding Integrated Development Environment (IDE) to perform coding, debugging or compiling operations on the personal user equipment.
Fig. 9 is a block diagram of a data management apparatus provided in an embodiment of the present application, where the apparatus may be a deep learning platform, and the apparatus includes:
a reading module 901, configured to read data of a data set through a first memory;
and a reading and writing module 902, configured to read and write deep learning task data through a second memory, and run the deep learning task, where the first memory and the second memory are physically isolated.
Optionally, the apparatus further comprises:
and the configuration module is used for configuring the data writing authority of the first memory for the reference user account if the user grade of the reference user account is greater than the grade threshold or the reference user account is a user account of a preset user, wherein the data writing authority of the first memory is used for allowing data set data to be written or deleted in the first memory.
Optionally, the user accounts of the deep learning platform are divided into a plurality of user groups, each user group includes at least one user account, and the apparatus further includes:
a first allocation module, configured to allocate, for each user group in the multiple user groups, a first storage resource for storing data set data based on an available storage resource of the first memory, where the first storage resource of each user group is shared by user accounts in each user group; and/or the presence of a gas in the gas,
and the second allocating module is used for allocating a second storage resource for storing deep learning task data for each user group in the plurality of user groups based on the available storage resource of the second storage, and the second storage resource of each user group is shared and used by the user accounts in each user group.
Optionally, a reference user group is allocated with a third storage resource, where the third storage resource includes the first storage resource and/or the second storage resource, and the reference user group is any one of the plurality of user groups, and the apparatus further includes:
and the third allocation module is used for allocating the use range of the third storage resource for any user account in the reference user group based on the storage resource allocation request of the group primary account of the reference user group.
Optionally, the user accounts of the deep learning platform are divided into a plurality of user groups, each user group includes at least one user account, and the apparatus further includes:
the first sharing module is configured to share data set data of a reference user group to all user accounts or a part of user accounts in the reference user group, or to all user accounts or a part of user accounts in other reference user groups, based on a data set data sharing request of a group primary account of the reference user group, where the reference user group is any one of the plurality of user groups.
Optionally, the sharee of the data set data has read-only right for the shared data set data, and does not have right to share the shared data to other user accounts.
Optionally, the apparatus further comprises:
and the second sharing module is used for sharing the deep learning task data of the reference user account to other user accounts based on the deep learning task data sharing request of the reference user account, wherein the reference user is any user account in the deep learning platform.
Optionally, the sharee of the deep learning task data has at least one of read permission and write permission for the shared deep learning task data.
In the embodiment of the application, the deep learning platform can read data set data through the first storage, and read and write deep learning task data through the second storage, and the first storage and the second storage are physically isolated, so that the data set data and the deep learning task data are stored and read and written separately, reading and management of the data set data and the deep learning task data are facilitated, and data reading efficiency in a deep learning process is improved. In addition, by means of storage separation and read-write separation of the data set data and the deep learning task data, data deletion operation of data set storage hardware can be reduced, and the service life of a storage hard disk of a server where the data set is stored is effectively prolonged.
Fig. 10 is a block diagram of another data management apparatus provided in an embodiment of the present application, where the apparatus may be a user equipment, and the apparatus includes:
a first display module 1001, configured to display a web page provided by a deep learning platform;
a second display module 1002, configured to display a file or a directory corresponding to deep learning data on the web page when a data browsing operation is detected based on the web page, where the deep learning data includes data set data and/or deep learning task data.
Optionally, the deep learning data includes deep learning data of the current user account and/or deep learning data shared by other user accounts.
Optionally, the current user account has a read-only permission for data set data in the deep learning data, and has a read permission and a write permission for deep learning task data in the deep learning data.
Optionally, a file in a reference file format is displayed on the web page, and the apparatus further includes:
and the third display module is used for displaying the file content of the file with the reference file format when the trigger operation on the file with the reference file format is detected.
In the embodiment of the application, the web page provided by the deep learning platform has a data browsing function, a user can conveniently browse and open deep learning data through the web page, the data operation mode is simple and convenient, and the usability of the deep learning platform is improved.
Fig. 11 is a block diagram of another data management apparatus provided in an embodiment of the present application, where the apparatus may be a user equipment, and the apparatus includes:
a first accessing module 1101, configured to access deep learning data stored in a memory through a file directory mounted on an operating system of a user device, where the deep learning data includes data set data or deep learning task data; alternatively, the first and second electrodes may be,
a second accessing module 1102, configured to remotely access the deep learning data stored in the memory through a secure shell protocol SSH.
In the embodiment of the application, a user can access deep learning data of the deep learning platform through personal user equipment, and the user can conveniently use a corresponding IDE to perform coding, debugging or compiling operation on the user equipment.
In this embodiment of the application, the deep learning task may include a training task of a neural network model, may also include a task that performs each function of target recognition (face recognition, human body recognition, vehicle recognition, license plate recognition, etc.), behavior recognition, target tracking, voice recognition, etc. by using a neural network, may also include other tasks related to deep learning, and is not limited herein.
It should be noted that: in the data management device provided in the above embodiment, when performing data management, only the division of the above functional modules is exemplified, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the data management apparatus and the data management method provided in the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments and are not described herein again.
Fig. 12 is a schematic structural diagram of a data management apparatus 1200 according to an embodiment of the present application, where the data management apparatus 1200 may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 1201 and one or more memories 1202, where the memory 1202 stores at least one instruction, and the at least one instruction is loaded and executed by the processors 1201 to implement the data management methods provided by the above method embodiments. Of course, the data management apparatus 1200 may also have components such as a wired or wireless network interface, a keyboard, and an input/output interface, so as to perform input/output, and the data management apparatus 1200 may also include other components for implementing device functions, which are not described herein again. The data management device may be a deep learning platform or a user equipment.
In another embodiment, a non-transitory computer readable storage medium is provided, in which instructions, when executed by a processor, enable the processor to perform the data management method described in the above embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (26)
1. A method for managing data, the method comprising:
the deep learning platform reads data of the data set through the first memory;
the deep learning platform reads and writes deep learning task data through a second memory and runs a deep learning task, and the first memory and the second memory are physically isolated.
2. The method of claim 1, wherein the method further comprises:
and if the user grade of the reference user account is greater than a grade threshold value or the reference user account is a user account of a preset user, configuring a data writing authority of the first memory for the reference user account, wherein the data writing authority of the first memory is used for allowing data set data to be written in or deleted from the first memory.
3. The method of claim 1, wherein the user accounts of the deep learning platform are divided into a plurality of user groups, each user group including at least one user account, the method further comprising:
allocating a first storage resource for storing data set data for each of the plurality of user groups based on available storage resources of the first storage, the first storage resource of each user group being shared by user accounts in each user group; and/or the presence of a gas in the gas,
and allocating a second storage resource for storing deep learning task data for each user group in the plurality of user groups based on the available storage resource of the second storage, wherein the second storage resource of each user group is shared and used by the user accounts in each user group.
4. The method of claim 3, wherein a reference user group is allocated a third storage resource, the third storage resource comprising the first storage resource and/or the second storage resource, the reference user group being any one of the plurality of user groups, the method further comprising:
and allocating the use range of the third storage resource for any user account in the reference user group based on the storage resource allocation request of the group primary account of the reference user group.
5. The method of claim 1, wherein the user accounts of the deep learning platform are divided into a plurality of user groups, each user group including at least one user account, the method further comprising:
based on a data set data sharing request of a group primary account of a reference user group, sharing data set data of the reference user group to all user accounts or part of user accounts in the reference user group, or sharing data set data of the reference user group to all user accounts or part of user accounts in other reference user groups, wherein the reference user group is any one of the plurality of user groups.
6. The method of claim 5, wherein sharees of the data set data have read-only rights to the shared data set data and do not have rights to share the shared data set data to other user accounts.
7. The method of claim 1, wherein the method further comprises:
based on a deep learning task data sharing request of a reference user account, the deep learning task data of the reference user account is shared to other user accounts, and the reference user is any user account in the deep learning platform.
8. The method of claim 7, wherein sharees of deep learning task data have at least one of read and write rights to the shared deep learning task data.
9. A method for managing data, the method comprising:
displaying a web page provided by a deep learning platform;
when a data browsing operation is detected based on the web page, a file or a directory corresponding to deep learning data is displayed on the web page, wherein the deep learning data comprises data set data and/or deep learning task data.
10. The method of claim 9, wherein the deep learning data comprises deep learning data for a current user account and/or deep learning data shared by other user accounts.
11. The method of claim 9, wherein a current user account has read-only rights to data set data in the deep learning data and read and write rights to deep learning task data in the deep learning data.
12. The method of claim 9, wherein a file in a reference file format is displayed on the web page, the method further comprising:
and when the trigger operation on the file with the reference file format is detected, displaying the file content of the file with the reference file format.
13. A method for managing data, the method comprising:
accessing deep learning data stored in a memory through a file directory mounted on an operating system of user equipment by the memory, wherein the deep learning data comprises data set data or deep learning task data; alternatively, the first and second electrodes may be,
the deep learning data stored in the memory is remotely accessed through a secure shell protocol SSH.
14. A data management device, which is applied to a deep learning platform, the device comprises:
the reading module is used for reading the data set data through the first memory;
and the reading and writing module is used for reading and writing the deep learning task data through a second memory and running the deep learning task, and the first memory and the second memory are physically isolated.
15. The apparatus of claim 14, wherein the apparatus further comprises:
the configuration module is configured to configure a data writing permission of the first storage for a reference user account if a user grade of the reference user account is greater than a grade threshold or the reference user account is a user account of a preset user, where the data writing permission of the first storage is used to allow data set data to be written or deleted in the first storage.
16. The apparatus of claim 14, wherein the user accounts of the deep learning platform are divided into a plurality of user groups, each user group including at least one user account, the apparatus further comprising:
a first allocation module, configured to allocate, for each user group in the plurality of user groups, a first storage resource for storing data set data based on an available storage resource of the first memory, where the first storage resource of each user group is shared by user accounts in each user group; and/or the presence of a gas in the gas,
a second allocating module, configured to allocate, for each user group in the plurality of user groups, a second storage resource for storing deep learning task data based on an available storage resource of the second memory, where the second storage resource of each user group is shared and used by user accounts in each user group.
17. The apparatus of claim 16, wherein a reference user group is allocated a third storage resource, the third storage resource comprising the first storage resource and/or the second storage resource, the reference user group being any one of the plurality of user groups, the apparatus further comprising:
and the third allocation module is used for allocating the use range of the third storage resource to any user account in the reference user group based on the storage resource allocation request of the group primary account of the reference user group.
18. The apparatus of claim 14, wherein the user accounts of the deep learning platform are divided into a plurality of user groups, each user group including at least one user account, the apparatus further comprising:
the first sharing module is configured to share data set data of a reference user group to all user accounts or a part of user accounts in the reference user group, or to all user accounts or a part of user accounts in other reference user groups, based on a data set data sharing request of a group primary account of the reference user group, where the reference user group is any one of the plurality of user groups.
19. The apparatus of claim 18, wherein sharees of the data set data have read-only rights to the shared data set data and do not have rights to share the shared data set data to other user accounts.
20. The apparatus of claim 14, wherein the apparatus further comprises:
the second sharing module is used for sharing the deep learning task data of the reference user account to other user accounts based on a deep learning task data sharing request of the reference user account, wherein the reference user is any user account in the deep learning platform.
21. The apparatus of claim 20, wherein sharees of deep learning task data have at least one of read and write rights to the shared deep learning task data.
22. A data management apparatus, characterized in that the apparatus comprises:
the first display module is used for displaying a web page provided by the deep learning platform;
and the second display module is used for displaying a file or a directory corresponding to deep learning data on the web page when a data browsing operation is detected based on the web page, wherein the deep learning data comprises data set data and/or deep learning task data.
23. The apparatus of claim 22, in which the deep learning data comprises deep learning data for a current user account and/or deep learning data shared by other user accounts.
24. The apparatus of claim 22, wherein a current user account has read-only rights to data set data in the deep learning data and read-and-write rights to deep learning task data in the deep learning data.
25. The apparatus of claim 22, wherein a file in a reference file format is displayed on the web page, the apparatus further comprising:
and the third display module is used for displaying the file content of the file in the reference file format when the trigger operation on the file in the reference file format is detected.
26. A data management apparatus, characterized in that the apparatus comprises:
the device comprises a first access module, a second access module and a third access module, wherein the first access module is used for accessing deep learning data stored in a memory through a file directory mounted on an operating system of user equipment by the memory, and the deep learning data comprises data set data or deep learning task data; alternatively, the first and second electrodes may be,
and the second access module is used for remotely accessing the deep learning data stored in the memory through a secure shell protocol SSH.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910576111.XA CN112148205A (en) | 2019-06-28 | 2019-06-28 | Data management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910576111.XA CN112148205A (en) | 2019-06-28 | 2019-06-28 | Data management method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112148205A true CN112148205A (en) | 2020-12-29 |
Family
ID=73869383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910576111.XA Pending CN112148205A (en) | 2019-06-28 | 2019-06-28 | Data management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112148205A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411833A (en) * | 2011-08-02 | 2012-04-11 | 杭州威威网络科技有限公司 | Networking alarm apparatus based on audio identification |
| CN104765572A (en) * | 2015-03-25 | 2015-07-08 | 华中科技大学 | Energy-saving virtual storage server system and scheduling method |
| JP2015201053A (en) * | 2014-04-08 | 2015-11-12 | キヤノン株式会社 | Image forming apparatus and control method of the same |
| CN106022007A (en) * | 2016-06-14 | 2016-10-12 | 中国科学院北京基因组研究所 | Cloud platform system and method oriented to biological omics big data calculation |
| CN106484330A (en) * | 2016-09-27 | 2017-03-08 | 郑州云海信息技术有限公司 | A kind of hybrid magnetic disc individual-layer data optimization method and device |
| CN107145395A (en) * | 2017-07-04 | 2017-09-08 | 北京百度网讯科技有限公司 | Method and apparatus for handling task |
| US20190050683A1 (en) * | 2018-09-28 | 2019-02-14 | Intel Corporation | Edge devices utilizing personalized machine learning and methods of operating the same |
| CN109447248A (en) * | 2018-11-09 | 2019-03-08 | 深圳增强现实技术有限公司 | Deep learning platform and method |
| CN109840591A (en) * | 2017-11-29 | 2019-06-04 | 华为技术有限公司 | Model training systems, method and storage medium |
-
2019
- 2019-06-28 CN CN201910576111.XA patent/CN112148205A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411833A (en) * | 2011-08-02 | 2012-04-11 | 杭州威威网络科技有限公司 | Networking alarm apparatus based on audio identification |
| JP2015201053A (en) * | 2014-04-08 | 2015-11-12 | キヤノン株式会社 | Image forming apparatus and control method of the same |
| CN104765572A (en) * | 2015-03-25 | 2015-07-08 | 华中科技大学 | Energy-saving virtual storage server system and scheduling method |
| CN106022007A (en) * | 2016-06-14 | 2016-10-12 | 中国科学院北京基因组研究所 | Cloud platform system and method oriented to biological omics big data calculation |
| CN106484330A (en) * | 2016-09-27 | 2017-03-08 | 郑州云海信息技术有限公司 | A kind of hybrid magnetic disc individual-layer data optimization method and device |
| CN107145395A (en) * | 2017-07-04 | 2017-09-08 | 北京百度网讯科技有限公司 | Method and apparatus for handling task |
| CN109840591A (en) * | 2017-11-29 | 2019-06-04 | 华为技术有限公司 | Model training systems, method and storage medium |
| US20190050683A1 (en) * | 2018-09-28 | 2019-02-14 | Intel Corporation | Edge devices utilizing personalized machine learning and methods of operating the same |
| CN109447248A (en) * | 2018-11-09 | 2019-03-08 | 深圳增强现实技术有限公司 | Deep learning platform and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108287669B (en) | Date storage method, device and storage medium | |
| CN109902255B (en) | Page mixed browsing record generation method, device, equipment and storage medium | |
| CN102938784A (en) | Method and system used for data storage and used in distributed storage system | |
| CN113760180A (en) | Storage resource management method, device, equipment and computer readable storage medium | |
| US20140282530A1 (en) | Refreshing memory topology in virtual machine operating systems | |
| KR102502569B1 (en) | Method and apparuts for system resource managemnet | |
| CN106919494A (en) | The implementation method and device of Android application daily record | |
| CN115543548B (en) | Configuration method, device and equipment of container group and readable storage medium | |
| CN113535087B (en) | Data processing method, server and storage system in data migration process | |
| CN110780821A (en) | Optimization method and device of distributed storage system, server and storage medium | |
| CN113204425A (en) | Method and device for process management internal thread, electronic equipment and storage medium | |
| CN110347546B (en) | Dynamic adjustment method, device, medium and electronic equipment for monitoring task | |
| CN109558082B (en) | Distributed file system | |
| CN111930684A (en) | Small file processing method, device and equipment based on HDFS (Hadoop distributed File System) and storage medium | |
| CN113127430A (en) | Mirror image information processing method and device, computer readable medium and electronic equipment | |
| CN112149139A (en) | Authority management method and device | |
| JP2021513137A (en) | Data migration in a tiered storage management system | |
| CN112148205A (en) | Data management method and device | |
| CN1645856B (en) | Deterministic rule-based dispatch of objects to code | |
| CN111488218B (en) | All-in-one machine computing system and method | |
| Kishani et al. | Padsa: Priority-aware block data storage architecture for edge cloud serving autonomous vehicles | |
| CN103685359A (en) | Data processing method and device | |
| CN112148461A (en) | Application scheduling method and device | |
| Shabeera et al. | A novel approach for improving data locality of mapreduce applications in cloud environment through intelligent data placement | |
| CN111782834A (en) | Image retrieval method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |