CN108170663A - Term vector processing method, device and equipment based on cluster - Google Patents

Abstract

The embodiment of this specification discloses a cluster-based word vector processing method, device, and equipment. The scheme includes: the cluster includes a server cluster and a working machine cluster; each working machine in the working machine cluster reads part of the corpus, and reads the Extract words and their context words, obtain corresponding word vectors from servers in the server cluster and calculate gradients, and update the gradients to the server asynchronously; the server updates the word vectors of words and their context words according to the gradients.

Description

Cluster-based word vector processing method, device and equipment

technical field

This description relates to the technical field of computer software, in particular to a cluster-based word vector processing method, device and equipment.

Background technique

Most of today's natural language processing solutions use a neural network-based architecture, and an important basic technology under this architecture is word vectors. A word vector is a vector that maps a word to a fixed dimension, which represents the semantic information of the word.

In the prior art, common algorithms for generating word vectors, such as Google's word vector algorithm and Microsoft's deep neural network algorithm, often run on a single machine.

Based on the existing technology, an efficient large-scale word vector training scheme is needed.

Contents of the invention

The embodiments of this specification provide a cluster-based word vector processing method, device, and equipment to solve the following technical problem: an efficient large-scale word vector training solution is required.

In order to solve the above-mentioned technical problems, the embodiments of this specification are implemented as follows:

The embodiment of this specification provides a cluster-based word vector processing method, the cluster includes multiple working machines and servers, and the method includes:

Each of the working machines executes respectively:

Obtain words extracted from part of the corpus and their context words;

Obtain word vectors of the word and its context words;

Calculate the gradient according to the word and its context word, and the corresponding word vector;

asynchronously updating the gradient to the server;

The server updates word vectors of the word and its context words according to the gradient.

The embodiment of this specification provides a cluster-based word vector processing device, the cluster includes multiple working machines and servers, the device is located in the cluster, and includes a first acquisition module and a second acquisition module located in the working machines module, a gradient calculation module, an asynchronous update module, and a word vector update module located at the server;

Each working machine executes respectively through corresponding modules:

The first acquisition module acquires words extracted from part of the corpus and context words thereof;

The second obtaining module obtains the word vector of the word and its context word;

The gradient calculation module calculates the gradient according to the word and its context words, and the corresponding word vector;

The asynchronous update module asynchronously updates the gradient to the server;

The word vector updating module of the server updates the word vectors of the word and its context words according to the gradient.

A cluster-based word vector processing device provided in an embodiment of this specification, the device belonging to the cluster, includes:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to:

Obtain words extracted from part of the corpus and their context words;

Obtain word vectors of the word and its context words;

Calculate the gradient according to the word and its context word, and the corresponding word vector;

updating the gradient asynchronously;

The word vectors of the word and its context words are updated according to the asynchronously updated gradient.

The above-mentioned at least one technical solution adopted in the embodiment of this specification can achieve the following beneficial effects: in the training process, each working machine does not need to wait for each other, and asynchronously updates the gradient calculated for each word to the server, and then the server updates each word according to the gradient. Therefore, it is beneficial to improve the convergence speed of word vector training, coupled with the distributed processing capability of the cluster, making this scheme suitable for large-scale word vector training with high efficiency.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments described in this specification. Those skilled in the art can also obtain other drawings based on these drawings without any creative effort.

FIG. 1 is a schematic diagram of an overall architecture involved in a practical application scenario of the solution in this specification;

Fig. 2 is a schematic flow chart of a cluster-based word vector processing method provided by an embodiment of this specification;

FIG. 3 is a schematic diagram of the principle of a cluster-based word vector processing method in an actual application scenario provided by an embodiment of this specification;

Fig. 4 is a detailed flowchart of a cluster-based word vector processing method corresponding to Fig. 3 provided by the embodiment of this specification;

FIG. 5 is a schematic structural diagram of a cluster-based word vector processing device corresponding to FIG. 2 provided by the embodiment of this specification.

Detailed ways

The embodiments of this specification provide a cluster-based word vector processing method, device and equipment.

In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described The embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments of this specification, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

The solution in this manual is suitable for clusters, and the processing efficiency of large-scale word vectors is higher in clusters. Specifically: the training corpus can be split, and multiple working machines in the cluster are distributed according to the split part of the corpus. One or more servers train the word vectors corresponding to the part of the corpus. During the training process, each working machine is responsible for calculating the gradient corresponding to each word, and asynchronously updates to the server, and the server is responsible for updating the word vector according to the gradient.

There can be one or more clusters involved in the solution. Taking Figure 1 as an example, two clusters are involved.

FIG. 1 is a schematic diagram of an overall architecture involved in the solution of this specification in a practical application scenario. The overall architecture mainly involves three parts: a server cluster including multiple servers, a working machine cluster including multiple working machines, and a database. The database saves the corpus for training, which is read by the worker cluster, and the server cluster saves the original word vectors. The worker cluster and the server cluster cooperate to realize the training of word vectors by updating the gradient asynchronously.

The architecture in Figure 1 is exemplary and not exclusive. For example, the solution may only involve one cluster, and the cluster includes at least one scheduler and multiple worker machines, and the scheduler completes the work of the above server cluster; another example, the solution may also involve a worker cluster and a server; etc. Wait.

Based on the architecture in FIG. 1 , the solution of this specification will be described in detail below.

FIG. 2 is a schematic flowchart of a cluster-based word vector processing method provided by an embodiment of this specification, and the cluster includes a working machine cluster and a server cluster. Each step in Figure 2 is executed by at least one machine (or program on the machine) in the cluster, and the execution subjects of different steps can be different. The process in Figure 2 can be executed in multiple rounds, and each round can use different groups of corpus. for training word vectors.

The flow in Figure 2 includes the following steps:

S202: Each working machine included in the working machine cluster respectively obtains words extracted from part of the corpus and context words thereof.

S204: The working machine acquires word vectors of the word and its context words.

S206: The working machine calculates a gradient according to the word, its context word, and the corresponding word vector.

S208: The working machine asynchronously updates the gradient to the servers included in the server cluster.

S210: The server updates word vectors of the word and its context words according to the gradient.

In the embodiment of this specification, each working machine can perform steps S202-S208 in a distributed manner, and the part of the corpus corresponding to each working machine is usually different, so that large-scale training corpus can be efficiently used, and the training efficiency of word vectors can also be improved. . For example, for the corpus currently used for training word vectors, the corpus can be split into multiple parts, and each working machine can read a part, and then perform steps S202-S208 based on the part of the corpus it reads.

For ease of description, for steps S202 to S208, the following embodiments are mainly described from the perspective of a certain working machine.

In the embodiment of this specification, if the current round of process is the first round of process, the word vector acquired in step S204 may be obtained by initialization. For example, the word vector of each word and the word vector of the context word of each word can be initialized randomly or according to a specified probability distribution, and the specified probability distribution is, for example, 0-1 distribution. If the current round of the process is not the first round of the process, the word vector obtained in step S204 may be the word vector updated and saved after the execution of the previous round of the process.

In the embodiment of this specification, the process of training word vectors mainly includes calculating gradients and updating vectors according to the gradients, which are respectively executed by worker clusters and server clusters. During the training process, after the calculation of the working computer is completed, the result needs to be synchronized to the server. There are usually two modes: synchronous update and asynchronous update. Synchronous update means that each working machine averages the model in a certain way and then updates it to the server (generally, different averaging strategies will result in different results, and the strategy design of model averaging is an important part of synchronous update). The asynchronous update means that any worker computer updates the data to the server immediately after the calculation is completed, without waiting for other workers and model averaging. In terms of the final effect, asynchronous update does not need to wait for other working computers to complete the calculation, so the training convergence speed is often faster. The solution in this manual is mainly based on the asynchronous update method. The specific asynchronous update data includes the data calculated by the working computer. The gradient corresponding to each word.

In the embodiment of this specification, step S208 is executed by the server cluster, and the updated word vectors are also stored in the server cluster for use in the next round of processes. Certainly, in other architectures than those shown in FIG. 1 , step S208 may also be executed by a scheduler or server belonging to the same cluster as the worker.

By analogy, after performing multiple rounds of the process until all the training corpora of all groups are used up, the server cluster can write out the finally updated word vectors to the database for use in various scenarios that require word vectors.

Through the method in Figure 2, during the training process, each working machine does not need to wait for each other, and asynchronously updates the gradient calculated for each word to the server, and then the server updates the word vector of each word according to the gradient. Therefore, it is beneficial to improve the word vector The speed of training convergence, coupled with the distributed processing capability of the cluster, makes this scheme suitable for large-scale word vector training with high efficiency.

Based on the method in FIG. 2 , the embodiment of this specification also provides some specific implementation solutions and extension solutions of the method, and the description will continue based on the architecture in FIG. 1 .

In the embodiment of this specification, the extraction of words and their context words from the corpus may be performed by a working machine, or may be performed in advance by other devices. Taking the previous method as an example, for step S202, before the acquisition of the words extracted from the partial corpus and their context words, it may also be performed: each of the working machines reads and obtains the partial corpus in a distributed manner. If the corpus is saved in the database, it can be read from the database.

In the embodiment of this specification, the acquisition of words extracted from part of the corpus and their context words may specifically include: establishing corresponding word pairs based on the corpus that you have read, and the word pairs include the current word and its context words. Upper and lower words. For example, you can scan the words in the corpus you have read. The currently scanned word is recorded as w, and a sliding window containing w is determined according to the set sliding window distance, and each other in the sliding window is Words are respectively used as a context word of w, denoted as c, thus forming a word pair {w,c}.

Further, it is assumed that word vectors are stored on multiple servers included in the server cluster. Then for step S204, the acquisition of the word vector of the word and its context word may specifically include: extracting the current word set and the context word set according to each of the word pairs established by oneself; obtaining the word vector from the server Word vectors of the words contained in the current word set and the context word set. Of course, this is not the only implementation mode. For example, when scanning the corpus, the word vector of the currently scanned word can be obtained from the server synchronously without necessarily relying on the established word pair, and so on.

In the embodiment of this specification, the gradient corresponding to each word can be calculated according to the specified loss function, each word pair created by itself, and the word vector of the word and its context word.

In order to obtain better training effect and faster convergence, the specified negative sample words can also be introduced as the contrast calculation gradient of the context words. The negative sample words are regarded as: compared with the context words, they are related to the corresponding current words Generally speaking, several words can be randomly selected from all words. In this case, for step S206, the calculation of the gradient according to the word and its context word, and the corresponding word vector may specifically include: according to the specified loss function, negative sample words, and each established The predicate pair, and the word vectors of the word and its context word, calculate the gradient corresponding to each word.

The current word and each of its negative sample words can also form a word pair (called a negative sample word pair), c' is used to represent a negative sample word, and the negative sample word pair is denoted as {w,c'}, assuming There are λ negative sample words, and the corresponding λ negative sample word pairs can be recorded as {w,c' ₁ }, {w,c' ₂ }, ..., {w,c' _λ }, for the convenience of description, The negative example word pair and the above context word pair (the word pair formed by the current word and its context word) are collectively recorded as {w,c}, and are distinguished by y. For the context word pair, y=1, for the negative example word pair, y=0.

In the embodiment of the present invention, the above-mentioned loss function can have various forms, generally including at least two items, one item reflects the similarity between the current word and its context, and the other item reflects the similarity between the current word and its negative sample word Similarity, where the similarity can be measured by vector dot product, or can be measured in other ways. Taking a practical application scenario as an example, for example, use the following formula to calculate the gradient ▽ corresponding to the current word:

in, Represents the word vector of w, Represents the word vector of c, σ is the activation function, assumed to be a Sigmoid function, then

Furthermore, one or more threads on each worker machine can calculate gradients in an asynchronous and update-free manner. Therefore, each thread in the working machine can also calculate the gradient in parallel without interfering with each other, which can further improve the calculation efficiency.

In the embodiment of this specification, for step S208, the working machine asynchronously updates the gradient to the server, which may specifically include: after the working machine calculates the gradient, sends the gradient to the server , wherein the execution of the sending action does not need to wait for other working machines to send gradients to the server.

In the embodiment of this specification, after the server obtains the gradient updated asynchronously by the working machine, it can use the gradient to update the corresponding word vector of the current word. Not only that, the server can also use the gradient to update the context word of the current word and the word vector of the negative sample word. The specific update method can refer to the gradient descent method.

For example, for step S210, the server updates the word vectors of the word and its context words according to the gradient, which may specifically include:

According to the following formula, the word and its context word, and the word vector of the negative sample word are iteratively updated:

in, w represents the current word, c represents the context word of w, c' represents the negative sample word, Represents the word vector of w, Represents the word vector of c, and Represents the tth update on the server, B _k represents the kth group of corpus on the working machine, Γ(w) represents the set of context words and negative sample words of w, α represents the learning rate, and σ is, for example, Sigmoid function.

According to the above description, the embodiment of this specification also provides a schematic diagram of the principle of a cluster-based word vector processing method in an actual application scenario, as shown in Figure 3. Further, the embodiment of this specification also provides A detailed flowchart of a cluster-based word vector processing method, as shown in Figure 4.

In Fig. 3, the working machines 0-2 and the servers 0-2 are exemplarily shown, and the description mainly focuses on the working machine 0, while the working machines 1 and 2 are briefly shown, and the working mode is the same as that of the working machine 0. of. "wid" and "cid" are identifiers, representing the current word and the context word respectively, and "wid list" and "cid list" are identifier lists, representing the current word set and the context word set respectively. The simplified workflow in Figure 3 includes: each working machine reads the corpus in a distributed manner and creates word pairs; each working machine obtains the corresponding word vector from the server cluster; each working machine calculates the gradient using the read corpus and updates it to the server asynchronously Cluster; the server cluster updates word vectors according to the gradient.

A more detailed process is shown in Figure 4, which mainly includes the following steps:

S402: Each working machine reads part of the corpus in a distributed manner, creates word pairs {w, c}, and extracts wid list and cid list from the word pairs, as shown by working machine 0 in FIG. 4 .

S404: The working machine pulls the corresponding word vector from the server according to the wid list and the cid list, and the server sends the corresponding word vector to the working machine.

S406: The working machine calculates the gradient according to the word pairs and the corresponding word vectors, specifically using the above formula 1 for calculation.

S408: Each thread of the working machine calculates the gradient in an asynchronous manner without locking and updating. After completing the gradient calculation, it does not wait for other working machines, and directly transmits the calculated gradients corresponding to all the words on the working machine to the server. .

S410: The server cluster updates the word vector according to the gradient, specifically using the above formula 2 and formula 3 for calculation.

Based on the same idea, the embodiment of this specification also provides a corresponding device of the above method, as shown in FIG. 5 .

Fig. 5 is a schematic structural diagram of a cluster-based word vector processing device corresponding to Fig. 2 provided by the embodiment of this specification. The cluster includes a plurality of working machines and servers, and the device is located in the cluster, including the The first acquisition module 501 of the working machine, the second acquisition module 502, the gradient calculation module 503, the asynchronous update module 504, and the word vector update module 505 located at the server;

Each working machine executes respectively through corresponding modules:

The first obtaining module 501 obtains words and context words thereof extracted from part of the corpus;

The second obtaining module 502 obtains the word vector of the word and its context word;

The gradient calculation module 503 calculates the gradient according to the word and its context words, and the corresponding word vector;

The asynchronous update module 504 asynchronously updates the gradient to the server;

The word vector updating module 505 of the server updates the word vectors of the word and its context words according to the gradient.

Optionally, before the first acquisition module 501 acquires the words extracted from the partial corpus and their context words, read the partial corpus in a distributed manner;

The first acquiring module 501 acquires words and context words thereof extracted from part of the corpus, specifically including:

The first acquisition module 501 creates corresponding word pairs according to the corpus it has read, and the word pairs include the current word and its context words.

Optionally, the second acquiring module 502 acquires word vectors of the word and its context words, specifically including:

The second acquisition module 502 extracts the current word set and the context word set according to each of the word pairs established by the first acquisition module 501;

Obtain word vectors of words contained in the current word set and the context word set from the server.

Optionally, the gradient calculation module 503 calculates the gradient according to the word and its context words, and the corresponding word vector, specifically including:

The gradient calculation module 503 calculates the gradient corresponding to each word according to the specified loss function, negative sample words, each word pair created by itself, and the word vector of the word and its context word.

Optionally, the gradient calculation module 503 calculates the gradient, specifically including:

One or more threads of the gradient calculation module 503 calculate gradients in a manner of asynchronous calculation and update without locking.

Optionally, the asynchronous update module 504 asynchronously updates the gradient to the server, specifically including:

The asynchronous update module 504 sends the gradient to the server after the gradient calculation module 503 calculates the gradient, wherein the execution of the sending action does not need to wait for the asynchronous update module 504 of other working machines to send the gradient to the server. The server sends gradients.

Optionally, the word vector update module 505 updates the word vectors of the word and its context words according to the gradient, specifically including:

The word vector update module 505 iteratively updates the word and its context words, and the word vector of the negative example word according to the following formula:

in, w represents the current word, c represents the context of w

word, c' represents a negative sample word, Represents the word vector of w, Represents the word vector of c, and express

In the tth update on the server, B _k represents the kth group of corpus on the working machine, Γ(w) table

Shows the set of context words and negative sample words of w, α represents the learning rate, and σ is the Sigmoid function.

Based on the same idea, the embodiment of this specification also provides a cluster-based word vector processing device corresponding to Figure 2, which belongs to the cluster, including:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to:

Obtain words extracted from part of the corpus and their context words;

Obtain word vectors of the word and its context words;

Calculate the gradient according to the word and its context word, and the corresponding word vector;

updating the gradient asynchronously;

The word vectors of the word and its context words are updated according to the asynchronously updated gradient.

Based on the same idea, the embodiment of this specification also provides a non-volatile computer storage medium corresponding to FIG. 2, which stores computer-executable instructions, and the computer-executable instructions are set to:

Obtain words extracted from part of the corpus and their context words;

Obtain word vectors of the word and its context words;

Calculate the gradient according to the word and its context word, and the corresponding word vector;

updating the gradient asynchronously;

The word vectors of the word and its context words are updated according to the asynchronously updated gradient.

The foregoing describes specific embodiments of this specification. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiments.

The device, device, and non-volatile computer storage medium provided in the embodiments of this specification correspond to the method. Therefore, the device, device, and non-volatile computer storage medium also have beneficial technical effects similar to those of the corresponding method. Since the above The beneficial technical effects of the method are described in detail, therefore, the beneficial technical effects of the corresponding devices, equipment, and non-volatile computer storage media will not be repeated here.

In the 1990s, the improvement of a technology can be clearly distinguished as an improvement in hardware (for example, improvements in circuit structures such as diodes, transistors, and switches) or improvements in software (improvement in method flow). However, with the development of technology, the improvement of many current method flows can be regarded as the direct improvement of the hardware circuit structure. Designers almost always get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (Programmable Logic Device, PLD) (such as a Field Programmable Gate Array (Field Programmable Gate Array, FPGA)) is such an integrated circuit, and its logic function is determined by programming the device by a user. It is programmed by the designer to "integrate" a digital system on a PLD, instead of asking a chip manufacturer to design and make a dedicated integrated circuit chip. Moreover, nowadays, instead of making integrated circuit chips by hand, this kind of programming is mostly realized by "logic compiler (logic compiler)" software, which is similar to the software compiler used when writing programs. The original code of the computer must also be written in a specific programming language, which is called a hardware description language (Hardware Description Language, HDL), and there is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language) , AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., currently the most commonly used is VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should also be clear to those skilled in the art that only a little logical programming of the method flow in the above-mentioned hardware description languages and programming into an integrated circuit can easily obtain a hardware circuit for realizing the logic method flow.

The controller may be implemented in any suitable way, for example the controller may take the form of a microprocessor or processor and a computer readable medium storing computer readable program code (such as software or firmware) executable by the (micro)processor , logic gates, switches, Application Specific Integrated Circuit (ASIC), programmable logic controllers, and embedded microcontrollers, examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the memory's control logic. Those skilled in the art also know that, in addition to realizing the controller in a purely computer-readable program code mode, it is entirely possible to make the controller use logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded The same function can be realized in the form of a microcontroller or the like. Therefore, such a controller can be regarded as a hardware component, and the devices included in it for realizing various functions can also be regarded as structures within the hardware component. Or even, means for realizing various functions can be regarded as a structure within both a software module realizing a method and a hardware component.

The systems, devices, modules, or units described in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or Combinations of any of these devices.

For the convenience of description, when describing the above devices, functions are divided into various units and described separately. Of course, when implementing this specification, the functions of each unit can be implemented in one or more pieces of software and/or hardware.

Those skilled in the art should understand that the embodiments of this specification may be provided as methods, systems, or computer program products. Accordingly, the embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The specification is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the specification. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. Memory is an example of computer readable media.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present description may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, refer to part of the description of the method embodiment.

The above descriptions are only examples of the present specification, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (15)

1. a cluster-based word vector processing method, the cluster includes a plurality of work machines and servers, the method comprising: Each of the working machines executes respectively: Obtain words extracted from part of the corpus and their context words; Obtain word vectors of the word and its context words; Calculate the gradient according to the word and its context word, and the corresponding word vector; asynchronously updating the gradient to the server; The server updates word vectors of the word and its context words according to the gradient. 2. The method according to claim 1, before the described acquisition is extracted from part of the corpus and its context word, the method also includes: Each of the working machines reads in a distributed manner to obtain part of the corpus; The acquisition of words and context words extracted from part of the corpus specifically includes: According to the corpus read by oneself, corresponding word pairs are established, and the word pairs include the current word and its upper and lower words. 3. The method according to claim 2, the word vectors of the described word and its context word are obtained, specifically comprising: Extract the current word set and the context word set according to each of the word pairs established by oneself; Obtain word vectors of words contained in the current word set and the context word set from the server. 4. the method for claim 2, described according to described word and context word thereof, and corresponding word vector, calculate gradient, specifically comprise: Calculate the gradient corresponding to each word according to the specified loss function, negative sample words, each word pair created by yourself, and the word vector of the word and its context word. 5. The method according to claim 1, said calculating the gradient, specifically comprising: One or more threads on the working machine calculate gradients in a manner of asynchronous calculation and update without locking. 6. The method according to claim 1, wherein the working machine asynchronously updates the gradient to the server, specifically comprising: After the working computer calculates the gradient, it sends the gradient to the server, wherein the sending action does not need to wait for other working computers to send the gradient to the server. 7. The method according to claim 4, wherein the server updates the word vectors of the word and its context word according to the gradient, specifically comprising: According to the following formula, the word and its context word, and the word vector of the negative sample word are iteratively updated: in, w represents the current word, c represents the context word of w, c' represents the negative sample word, Represents the word vector of w, Represents the word vector of c, and Represents the tth update on the server, B _k represents the kth group of corpus on the working machine, Γ(w) represents the set of context words and negative sample words of w, α represents the learning rate, and σ is the Sigmoid function . 8. A cluster-based word vector processing device, the cluster includes a plurality of working machines and servers, the device is located in the cluster, including a first acquisition module, a second acquisition module, and a gradient calculation located in the working machine module, an asynchronous update module, a word vector update module located at the server; Each working machine executes respectively through corresponding modules: The first acquisition module acquires words extracted from part of the corpus and context words thereof; The second obtaining module obtains the word vector of the word and its context word; The gradient calculation module calculates the gradient according to the word and its context words, and the corresponding word vector; The asynchronous update module asynchronously updates the gradient to the server; The word vector updating module of the server updates the word vectors of the word and its context words according to the gradient. 9. The device according to claim 8, before the first acquisition module obtains the word extracted from the partial corpus and its context word, the partial corpus is read in a distributed manner; The first acquisition module acquires words extracted from part of the corpus and context words thereof, specifically including: The first acquisition module creates corresponding word pairs according to the corpus it has read, and the word pairs include the current word and its context words. 10. The device according to claim 9, the second acquisition module acquires the word vector of the word and its context word, specifically comprising: The second acquisition module extracts the current word set and the context word set according to the word pairs established by the first acquisition module; Obtain word vectors of words contained in the current word set and the context word set from the server. 11. The device according to claim 9, the gradient calculation module calculates the gradient according to the word and its context words, and the corresponding word vector, specifically comprising: The gradient calculation module calculates the gradient corresponding to each word according to the specified loss function, negative sample words, each word pair created by itself, and the word vector of the word and its context word. 12. The device according to claim 8, the gradient calculation module calculates the gradient, specifically comprising: One or more threads of the gradient calculation module calculate gradients in a manner of asynchronous calculation and update without locking. 13. The device according to claim 8, wherein the asynchronous update module asynchronously updates the gradient to the server, specifically comprising: The asynchronous update module sends the gradient to the server after the gradient calculation module calculates the gradient, wherein the execution of the sending action does not need to wait for the asynchronous update module of other working machines to send the gradient to the server. gradient. 14. The device according to claim 11, the word vector updating module updates the word vector of the word and its context word according to the gradient, specifically comprising: The word vector update module performs iterative updates to the word and its context words, and the word vector of the negative example word according to the following formula: in, w represents the current word, c represents the context word of w, c' represents the negative sample word, Represents the word vector of w, Represents the word vector of c, and Represents the tth update on the server, B _k represents the kth group of corpus on the working machine, Γ(w) represents the set of context words and negative sample words of w, α represents the learning rate, and σ is the Sigmoid function . 15. A cluster-based word vector processing device, the device belonging to the cluster, comprising: at least one processor; and, a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to: Obtain words extracted from part of the corpus and their context words; Obtain word vectors of the word and its context words; Calculate the gradient according to the word and its context word, and the corresponding word vector; updating the gradient asynchronously; The word vectors of the word and its context words are updated according to the asynchronously updated gradient.