US20210042940A1 - Digital twin monitoring systems and methods - Google Patents
Digital twin monitoring systems and methods Download PDFInfo
- Publication number
- US20210042940A1 US20210042940A1 US16/534,938 US201916534938A US2021042940A1 US 20210042940 A1 US20210042940 A1 US 20210042940A1 US 201916534938 A US201916534938 A US 201916534938A US 2021042940 A1 US2021042940 A1 US 2021042940A1
- Authority
- US
- United States
- Prior art keywords
- zone
- computer
- interest
- surveilled area
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000012544 monitoring process Methods 0.000 title abstract description 24
- 238000003384 imaging method Methods 0.000 claims abstract description 63
- 230000005019 pattern of movement Effects 0.000 claims abstract description 21
- 230000004048 modification Effects 0.000 claims abstract description 11
- 238000012986 modification Methods 0.000 claims abstract description 11
- 230000033001 locomotion Effects 0.000 claims description 16
- 238000001514 detection method Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 description 16
- 238000012545 processing Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000008450 motivation Effects 0.000 description 1
- 239000010813 municipal solid waste Substances 0.000 description 1
- 229920001690 polydopamine Polymers 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/292—Multi-camera tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/167—Position within a video image, e.g. region of interest [ROI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/268—Signal distribution or switching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30236—Traffic on road, railway or crossing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H04N5/247—
Definitions
- This disclosure generally relates to monitoring systems, and more particularly relates to monitoring systems employing a digital twin.
- Monitoring systems may be implemented by using various kinds of devices, systems, and procedures. Some monitoring systems include imaging devices such as video cameras and infrared cameras for various purposes such as for securing a property, for detecting suspicious activity, or for monitoring traffic. Processing circuitry used for processing images captured by an imaging device may vary from one monitoring system to another depending on the nature and purpose of the monitoring. For example, a sophisticated image processing system may be used for processing images in a monitoring system that employs a high-speed imaging camera, whereas a relatively less sophisticated image processing system may be employed to process images in a monitoring system that employs a less-expensive camera for capturing images containing objects that remain stationary for extended periods of time.
- imaging devices such as video cameras and infrared cameras
- Processing circuitry used for processing images captured by an imaging device may vary from one monitoring system to another depending on the nature and purpose of the monitoring. For example, a sophisticated image processing system may be used for processing images in a monitoring system that employs a high-speed imaging camera, whereas a relatively less sophisticated image processing system may
- the sophisticated image processing system may be found suitable for use at a first location where a monitored area contains a few fast moving objects but may be unsuitable for use at a second location where the monitoring area may be too large for detecting fast moving objects and/or may include too many fast moving objects that cannot be monitored effectively.
- FIG. 1 illustrates a first exemplary monitoring system that includes a digital twin displayed on a computer in accordance with an embodiment the disclosure.
- FIG. 2 illustrates a first exemplary operational configuration that may be used in the digital twin shown in FIG. 1 .
- FIG. 3 illustrates a second exemplary operational configuration that may be used in the digital twin shown in FIG. 1 .
- FIG. 4 illustrates a third exemplary operational configuration that may be used in the digital twin shown in FIG. 1 .
- FIG. 5 illustrates a second exemplary monitoring system incorporating a digital twin displayed on a computer in accordance with another embodiment the disclosure.
- FIG. 6 shows a flowchart of a method of operating the second exemplary monitoring system shown in FIG. 5 .
- FIG. 7 shows a first exemplary traffic condition displayed in a digital twin in accordance with the disclosure.
- FIG. 8 illustrates predictive travel routes that may be taken by a vehicle shown in FIG. 7 .
- FIG. 9 shows the vehicle having taken one of the predictive travel routes shown in FIG. 8 .
- FIG. 10 shows some exemplary components of a computer that may be used to operate a digital twin in accordance with an exemplary embodiment of the disclosure.
- digital twin refers to a digital replica of a physical asset, process, people, place, system, and/or device.
- one or more images captured by an imaging apparatus may be used to define a first portion of a surveilled area as a zone of primary interest and a second portion of the surveilled area as a zone of secondary interest.
- the digital twin may be utilized to primarily focus on the zone of primary interest such as for determining a pattern of movement of one or more objects in the zone of primary interest.
- One or both of the zone of primary interest and the zone of secondary interest may be dynamically redefined upon receiving additional images that indicate a modification in the pattern of movement of one or more objects in the surveilled area.
- any of the functionality described with respect to a particular component such as a first processor in a first computer may be performed by another component such as a second processor in another computer.
- a particular component such as a first processor in a first computer
- another component such as a second processor in another computer.
- embodiments of the disclosure may relate to numerous other device characteristics.
- embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the embodiments.
- words and terms are used herein solely for convenience and such words and terms should be interpreted as referring to various objects and actions that are generally understood in various forms and equivalencies by persons of ordinary skill in the art.
- words such as “object” and “vehicle” may be used interchangeably and should be understood in the context of the disclosure.
- Words such as “data” “signals,” “indication,” and “information” may be used interchangeably and should be understood in the context of the disclosure.
- Words such as “display,” “images,” “imagery,” “snapshots,” “video,” and “video frames,” may be used herein in an interchangeable manner and should be understood to refer to sources of information for monitoring purposes.
- the word “information” as used herein may refer to various items such as digital data, analog data, audio content, video content, and/or messages. These items may be operated upon by a computer containing a processor. Words such as “monitoring” and “surveillance” may be used interchangeably and should be understood in the context of the disclosure. Phrases such as “an object is displayed” or “based on an (xxx factor)” as used herein should not be construed exclusively in a singular sense. These phrases should be understood to also encompass multiple objects that may be displayed and based on multiple factors. It should also be understood that the word “example” as used herein is intended to be non-exclusionary and non-limiting in nature. More particularly, the word “exemplary” as used herein indicates one among several examples, and it should be understood that no undue emphasis or preference is being directed to the particular example being described.
- FIG. 1 illustrates an exemplary monitoring system 100 that includes a digital twin 107 displayed on a screen 106 of a computer 105 in accordance with an embodiment the disclosure.
- the computer 105 may be coupled via a network 110 to an imaging apparatus 115 that is configured to capture one or more images of a surveilled area 120 .
- the network 110 may include any one or a combination of various networks such as a data network, a telephone network, a cellular network, a cable network, a wireless network, a private network, a public network, a local area network (LAN), a wide area network (WAN), and the Internet.
- the network 110 may support communication technologies such as Bluetooth, cellular, near-field communication (NFC), Wi-Fi, and/or Wi-Fi direct.
- the computer 105 may be directly coupled to the imaging apparatus 115 via an electrical cable or a fiber optic cable and communicatively coupled via the network 110 to other components such as a server system 135 .
- the imaging apparatus 115 can include various types of imaging devices such as, for example, a digital camera configured to take snapshots of the surveilled area 120 (on a cyclic basis, an intermittent basis, or as-needed basis), a video camera configured to generate video files based on video surveillance of the surveilled area 120 , or an infrared camera configured to generate infrared imagery (snapshots and/or video) of the surveilled area 120 .
- the imaging apparatus 115 may also include components such as a motion detector or a speed sensor.
- the motion detector may be used in some applications to trigger image capture (or video capture) in the imaging apparatus 115 .
- the speed detector may be used to obtain motion information associated with one or more moving objects in the surveilled area 120 .
- the motion information may be a numerical speed value of a moving object in the surveilled area 120 (65 miles per hour, for example).
- the numerical speed value may be propagated to the computer 105 from the imaging apparatus 115 via the network 110 .
- the numerical speed value may further include a time-stamp that is indicative of a time at which the speed of the moving object was captured.
- the computer 105 may determine a speed of an object by processing multiple snapshots or video frames captured by the imaging apparatus 115 and propagated to the computer 105 via the network 110 .
- the surveilled area 120 includes an exemplary road 122 , bordered by an exemplary sidewalk 121 on one side.
- any of various other types of areas may be placed under surveillance.
- a few examples of such areas may include a factory floor, a warehouse, a parking lot, a prison, and a company location. Each of these areas may include a combination of moving objects and stationary objects over one or more periods of time.
- the surveilled area 120 may include one or more moving objects such as, for example, a vehicle 130 travelling east on the road 122 , and one or more stationary objects such as a dumpster 125 that is stationed upon a section of the road 122 adjacent to the sidewalk 121 .
- the surveilled area 120 may further include moving objects such as a bicyclist (not shown) riding a bicycle down the road 122 , and/or a pedestrian (not shown) walking on the sidewalk 121 .
- the surveilled area 120 may also include stationary objects such as an unattended vehicle, a wrecked vehicle involved in an accident, a light pole, a traffic light, a construction barrier, trash, and/or debris.
- the moving objects may include workers and/or vehicles and the stationary objects may include machinery, packages, and building structures (pillars, poles, racks, etc.).
- the images captured by the imaging apparatus 115 may be received by the computer 105 , processed for obtaining information about the surveilled area 120 , and used for generating the digital twin 107 that is displayed on the screen 106 .
- the digital twin 107 may be updated in real-time for rendering a real-time digital replication of the surveilled area 120 .
- any changes in the surveilled area 120 such as for example, the vehicle 130 moving out of the surveilled area 120 or the dumpster 125 being relocated on to the sidewalk 121 , can be detected by the computer 105 and used for various purposes.
- the computer 105 may be configured to detect the relocation of a stationary object such as the dumpster 125 on the sidewalk 121 . Upon detecting the relocation, the computer 105 may obtain data from a server system 135 to verify whether such an action is permissible under laws governing the surveilled area 120 .
- the server system 135 can be a cloud-based system that includes one or more computers containing components such as processors and memory devices.
- the server system 135 can include at least one computer having at least a processor and a memory.
- the memory which is an example of a non-transitory computer-readable medium, may be used to store various types of information such as laws and rules governing the surveilled area 120 , traffic rules, municipality codes, vehicle registration data, and registration information (permits, licenses, etc.).
- the server system 135 may respond to a request received from a computer such as the computer 105 , by transmitting the requested information to the computer via the network 110 .
- the information received from the server system 135 may be used by the computer 105 to process imagery received from the imaging apparatus 115 and operate the digital twin 107 in accordance with one or more embodiments of the disclosure.
- the methods for processing imagery and/or operating the digital twin 107 may provide various advantages such as minimizing the amount of time for initializing the monitoring system 100 to operate in various environments and various applications, optimizing information obtained from processing imagery, generating desired information based on high-speed processing of selected portions of a surveilled area, rapidly and dynamically modifying monitored zones inside a surveilled area, and system scalability so as to allow multiple imaging systems to cooperate efficiently for generating desired information about a surveilled area.
- FIG. 2 illustrates a first exemplary operational configuration that may be used by the computer 105 to process images received from the imaging apparatus 115 and generate the digital twin 107 .
- the surveilled area 120 may be partitioned into two or more zones of interest based on various criteria.
- the surveilled area 120 may be partitioned into a zone of primary interest 205 and a zone of secondary interest 210 based on a motivation to monitor moving objects with a higher level of priority than stationary objects that may be monitored with a relatively lower level of priority.
- the surveilled area 120 may be an alleyway and the road 122 may be a one-way street in which vehicles are permitted to only travel eastwards.
- the monitoring system 100 may be used to capture images of the surveilled area 120 that provide information such as an occurrence of one or more infractions.
- the sidewalk 121 in this example may be a narrow sidewalk, and an exemplary infraction may involve parking of the dumpster 125 upon the sidewalk 121 thereby compelling pedestrians walking on the sidewalk 121 to step on the road 122 in order to get around the dumpster 125 .
- an infraction local residents and/or business people in the area may have noticed that some vehicles disregard traffic rules several times during a day and drive the wrong way in an opposite direction.
- the imaging apparatus 115 to capture images of the entire surveilled area 120 at a high capture rate (an image every second, for example) and use the computer 105 to process a large amount of image content generated by such a high capture rate may be sub-optimal in at least some cases such as when the dumpster 125 is moved. Consequently, in accordance with the disclosure, the computer 105 may be configured to operate in a manner that minimizes and/or simplifies image processing operations for generating the digital twin 107 .
- the zone of primary interest 205 may be defined on the basis of factors such as a pattern of movement of one or more objects in the surveilled area 120 and the availability of a drivable portion of the road 122 .
- the computer 105 may be configured to process a first image of the surveilled area 120 that shows the dumpster 125 parked on the road 122 and a vehicle 130 moving on the road 122 at the time the image was captured by the imaging apparatus 115 , followed by additional images that may be processed by the computer 105 to validate or modify the information derived from the first image.
- the information derived from the first image and/or additional images may be used by the computer 105 to define a perimeter of the zone of primary interest 205 .
- the perimeter of the zone of the primary interest 205 which precludes a portion of the road 122 on which the dumpster 125 is located, may be based on one or more factors such as a pattern of movement of the vehicle 130 and/or the dumpster 125 .
- the pattern of movement of the vehicle 130 and the dumpster 125 may be monitored over a first period of time (a day, for example) and used to classify the vehicle 130 as a moving object for purposes of surveillance.
- the pattern of movement of the dumpster 125 may be monitored over a second period of time (a month, for example) to determine that the dumpster 125 is moved occasionally from the road 122 to the sidewalk 121 , or is never moved from the road 122 .
- the relatively slow occurrence of movement, or the lack of movement, of the dumpster 125 may be used by the computer 105 to classify the dumpster as a stationary object for purposes of surveillance.
- each of the first period of time and the second period of time may be determined by an operator of the computer 105 after evaluating one or more characteristics of the various objects in the surveilled area 120 .
- the computer 105 may define a perimeter of the zone of secondary interest 210 based in part on classifying the dumpster 125 as a stationary object.
- the zone of secondary interest 210 may encompass the sidewalk 121 and a portion of the road 122 on which the dumpster 125 is located.
- the computer 105 may process image content in the zone of primary interest 205 with a higher level of priority than that in the zone of secondary interest 210 .
- the imaging apparatus 115 is configured to capture images of the zone of primary interest 205 more frequently and/or for a longer period of time than images of the zone of secondary interest 210 .
- a first set of images containing image content exclusively from the zone of primary interest 205 may be captured for a long period of time (an hour, for example) and a second set of images containing image content from the zone of secondary interest 210 may be captured for a shorter period of time (10 minutes, for example).
- Such a procedure reduces the amount of image data processed by the computer 105 in comparison to an amount of data that would be generated by the imaging apparatus 115 upon scanning both the zone of primary interest 205 and the zone of secondary interest 210 in a concurrent manner all the time.
- the imaging apparatus 115 may be configured to capture a first set of images by scanning the zone of primary interest 205 at a fast scanning rate ( 20 scans per hour, for example) and to capture a second set of images by scanning the zone of secondary interest 210 at a relatively slower scanning rate ( 2 scans per hour, for example).
- This exemplary procedure also reduces the amount of image data processed by the computer 105 in comparison to an amount of data that would be generated by the imaging apparatus 115 upon using the same scanning rate for scanning both the zone of primary interest 205 and the zone of secondary interest 210 .
- FIG. 3 illustrates a second exemplary operational configuration that may be used by the computer 105 to process images received from the imaging apparatus 115 and generate the digital twin 107 .
- the computer 105 may process one or more images captured by the imaging apparatus 115 and detect a vehicle 315 that is parked on the road 122 for an extended period of time (several days, for example).
- the computer 105 may then classify the vehicle 315 as a stationary object that has to be treated as an obstruction by moving objects such as the vehicle 130 .
- a perimeter of the zone of secondary interest 210 may be redefined by the computer 105 in accordance with the classification of the vehicle 315 as a stationary object so as to encompass a portion of the road 122 on which the vehicle 315 is parked.
- the perimeter of the zone of primary interest 205 is also redefined in this case so as to preclude the portion of the road 122 on which the vehicle 315 is parked.
- Redefining one or both of the zone of primary interest 205 and the zone of secondary interest 210 may be carried out dynamically in accordance with the disclosure whenever one or more objects move into the surveilled area 120 or an object that is already present in the surveilled area 120 undergoes a modification in a pattern of movement.
- FIG. 4 illustrates a third exemplary operational configuration that may be used by the computer 105 to process images received from the imaging apparatus 115 and generate the digital twin 107 .
- the computer 105 may process one or more images captured by the imaging apparatus 115 and detect a pedestrian 405 walking on the sidewalk 121 .
- the computer 105 may classify the pedestrian 405 as a moving object and may redefine a perimeter of the zone of primary interest 205 based on this classification.
- the perimeter of the zone of secondary interest 210 is also redefined to preclude the portion of the sidewalk 121 on which the pedestrian 405 is walking.
- Redefining one or both of the zone of primary interest 205 and the zone of secondary interest 210 may be carried out dynamically in accordance with the disclosure, whenever one or more objects such as the pedestrian 405 move into the surveilled area 120 or an object that is already present in the surveilled area 120 undergoes a modification in a pattern of movement.
- the computer 105 may also be configured to process image content from both the zone of primary interest 205 and the zone of secondary interest 210 in conformance with one or more laws that preclude collection of personal information of people (such as the pedestrian 405 ).
- FIG. 5 illustrates a second exemplary monitoring system 500 that can include two or more imaging apparatuses in accordance with an embodiment the disclosure.
- a first imaging apparatus 530 provides an image capture area 505 that may be defined by a line-of-sight 532 and a line-of-sight 531 of the first imaging apparatus 530 .
- Images captured by the first imaging apparatus 530 may be transmitted to the computer 105 via a network such as the network 110 shown in FIG. 1 .
- the computer 105 may process these images for various purposes such as for displaying a digital twin 550 on the display screen 106 that includes objects present in the image capture area 505 .
- a second imaging apparatus 540 provides an image capture area 515 that may be defined by a line-of-sight 541 and a line-of-sight 542 of the second imaging apparatus 540 . Images captured by the second imaging apparatus 540 may be transmitted to the computer 105 via a network such as the network 110 . The computer 105 may process these images for various purposes such as for configuring the digital twin 550 to include objects present in the image capture area 515 in addition to the objects present in the image capture area 505 .
- the digital twin 550 may be provided in the form of a concatenated display that incorporates images captured by the second imaging apparatus 540 as well as images captured by the first imaging apparatus 530 .
- the digital twin 550 can be provided in the form of two separate windows—a first window containing images captured by the first imaging apparatus 530 and a second window containing images captured by the second imaging apparatus 540 .
- the image capture area 505 overlaps the image capture area 515 in an overlap area 506 .
- the overlap area 506 allows for seamless tracking of a vehicle such as the vehicle 130 when the vehicle is moving on the road 122 .
- the seamless tracking allows the computer 105 to eliminate blind spots by providing the computer 105 , the ability to detect stoppage of the vehicle 130 at a boundary portion between the image capture area 505 and the image capture area 515 .
- a third imaging apparatus 535 provides an image capture area 520 that may be defined by a line-of-sight 536 and a line-of-sight 537 of the third imaging apparatus 535 .
- An overlap area 507 allows for seamless tracking of a vehicle such as the vehicle 130 , through the image capture area 515 and the image capture area 520 .
- the seamless tracking allows the computer 105 to eliminate blind spots by providing the computer 105 , the ability to detect stoppage of the vehicle 130 at a boundary portion between the image capture area 515 and the image capture area 520 .
- a fourth imaging apparatus 545 provides an image capture area 525 that may be defined by a line-of-sight 546 and a line-of-sight 547 of the fourth imaging apparatus 545 .
- An overlap area 508 allows for seamless tracking of a vehicle such as the vehicle 130 , through the image capture area 520 and the image capture area 525 .
- the seamless tracking allows the computer 105 to eliminate blind spots by providing the computer 105 , the ability to detect stoppage of the vehicle 130 at a boundary portion between the image capture area 520 and the image capture area 525 .
- the digital twin 550 may be provided in the form of a concatenated display that incorporates images captured by the first imaging apparatus 530 , the second imaging apparatus 540 , the third imaging apparatus 535 , and the fourth imaging apparatus 545 .
- the digital twin 550 can be provided in the form of four separate windows—a first window containing images captured by the first imaging apparatus 530 , a second window containing images captured by the second imaging apparatus 540 , a third window containing images captured by the third imaging apparatus 535 , and a fourth window containing images captured by the fourth imaging apparatus 545 .
- some or all of the image capture areas can be modified in size for various reasons such as for defining one or more zones of primary interest and/or one or more zones of secondary interest in accordance with the disclosure.
- FIG. 6 shows an exemplary flowchart 600 of a method of operating the monitoring system 500 shown in FIG. 5 .
- the exemplary flowchart 600 illustrates a sequence of operations that can be implemented in hardware, software, or a combination thereof.
- the operations represent computer-executable instructions stored on one or more non-transitory computer-readable media, that, when executed by one or more processors, perform the recited operations.
- computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types.
- an object such as the vehicle 130 shown in FIG. 1 , enters a first zone (the image capture area 505 , for example).
- the first imaging apparatus 530 captures an image of the first zone and transmits the image to the computer 105 .
- the computer 105 processes the image and detects the entry of the vehicle 130 into the first zone.
- the detection may be made with “x” % confidence level.
- the image may have high resolution and the detection of the vehicle 130 (based on distinguishing marks on the vehicle 130 , for example) may be carried out with a 99% confidence level.
- the image may have poor quality and the detection may be carried out with a 60% confidence level.
- the vehicle passes through the first zone and enters a second zone (the image capture area 515 , for example).
- the second imaging apparatus 540 captures an image of the second zone and transmits the image to the computer 105 .
- the computer 105 processes the image and detects the entry of the vehicle 130 into the second zone.
- the detection may be made with “y” % confidence level.
- the image may have high resolution and the detection of the vehicle 130 (based on distinguishing marks on the vehicle 130 , for example) may be carried out with an 80% confidence level.
- the image may have poor quality and the detection may be carried out with a 40% confidence level.
- tracking of the vehicle 130 through additional zones may be carried out by using other imaging apparatuses such as the third imaging apparatus 535 and the fourth imaging apparatus 545 .
- one or more zones of primary interest may be determined or redefined and applied so as to recursively implement the operations described above.
- the determination indicates no overlap (for example, in an implementation where the overlap area 506 does not exist)
- tracking of the vehicle 130 through additional zones may be carried out as described above.
- FIG. 7 shows an exemplary traffic condition that may be displayed in the digital twin 107 at a first instant in time (t 0 ) in accordance with the disclosure.
- the vehicle 130 is traveling eastwards along a path 705 .
- the vehicle 130 has not yet reached a spot 710 .
- the spot 710 which is aligned with a distal end 715 of the dumpster 125 in this example.
- FIG. 8 illustrates a few exemplary predictive travel paths that the vehicle 130 may take after reaching the spot 710 .
- the exemplary predictive travel paths may be identified by the computer 105 by executing a prediction procedure in accordance with the disclosure that may include processing one or more images provided to the computer 105 by the imaging apparatus 115 at the first instant in time (t 0 ).
- the vehicle 130 may take one of a first predictive travel path 805 or a second predictive travel path 810 .
- the prediction procedure used by the computer 105 can include various algorithms, models, and/or techniques such as Markov chains, neural networks, artificial intelligence (AI), and finite state machines.
- AI artificial intelligence
- the prediction procedure may be executed by the computer 105 independently, or in conjunction with other procedures such as image processing procedures or simulation procedures, for defining and/or redefining a perimeter of a zone of primary interest and/or a zone of secondary interest.
- the defining or redefining may be carried out by using a finite state machine (FSM) to simulate various conditions (states).
- FSM finite state machine
- the various states may change as a result of various types of inputs. For example, a state of the dumpster 125 may change if the dumpster 125 is moved after having been left undisturbed for a long period of time.
- a perimeter of a zone of primary interest which may be used as one of the states in an FSM, may change based on an object entering the zone of primary interest.
- FIG. 9 shows the vehicle 130 having reached the spot 710 at a second instant in time (t 1 ) and having opted to travel along the first predictive travel path 805 .
- the action taken by the vehicle 130 to travel along the first predictive travel path 805 may be identified by the computer 105 by executing a verification procedure in accordance with the disclosure.
- the verification procedure may include processing one or more images provided to the computer 105 by the imaging apparatus 115 at the second instant in time (t 1 ).
- the prediction procedure and the evaluation procedure described above may be used for various purposes such as for dynamically modifying one or more zones of primary interest and/or for dynamically modifying one or more zones of secondary interest, for pre-empting traffic congestion on the road 122 , for issuing movement violation citations, and/or issuing parking citations.
- FIG. 10 shows some exemplary components of the computer 105 in accordance with an embodiment of the disclosure.
- the computer 105 can include various components such as a processor 11 , a communications module 12 , and a memory 13 .
- the memory 13 which is one example of a non-transitory computer-readable medium, may be used to store an operating system (OS) 16 and various other code modules such as a digital twin operating module 14 and a database 15 .
- OS operating system
- FIG. 10 shows some exemplary components of the computer 105 in accordance with an embodiment of the disclosure.
- the computer 105 can include various components such as a processor 11 , a communications module 12 , and a memory 13 .
- the memory 13 which is one example of a non-transitory computer-readable medium, may be used to store an operating system (OS) 16 and various other code modules such as a digital twin operating module 14 and a database 15 .
- OS operating system
- the communications module 12 which may include an input/output interface for communicatively coupling the computer 105 to the network 110 , allows the computer 105 to communicate with the server system 135 and with various imaging apparatuses such as the imaging apparatus 115 shown in FIG. 1 .
- the digital twin operating module 14 may be executed by the processor 11 for implementing various operations described herein in accordance with the disclosure.
- the memory 13 can include any one memory element or a combination of volatile memory elements (e.g., random-access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, hard drive, tape, CD ROM, etc.).
- volatile memory elements e.g., random-access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)
- non-volatile memory elements e.g., ROM, hard drive, tape, CD ROM, etc.
- the memory device may incorporate electronic, magnetic, optical, and/or other types of storage media.
- a “non-transitory computer-readable medium” can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device.
- the computer-readable medium would include the following: a portable computer diskette (magnetic), a random-access memory (RAM) (electronic), a read-only memory (ROM) (electronic), an erasable programmable read-only memory (EPROM, EEPROM, or Flash memory) (electronic), and a portable compact disc read-only memory (CD ROM) (optical).
- a portable computer diskette magnetic
- RAM random-access memory
- ROM read-only memory
- EPROM erasable programmable read-only memory
- EPROM erasable programmable read-only memory
- CD ROM portable compact disc read-only memory
- the computer-readable medium could even be paper or another suitable medium upon which the program is printed, since the program can be electronically captured, for instance, via optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
- the operations described herein with respect to the computer 105 may be implemented by computer-executable instructions stored on one or more non-transitory computer-readable media such as the memory 13 , that, when executed by one or more processors such as the processor 11 , perform the recited operations.
- computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types.
- Implementations of the systems, apparatuses, devices, and methods disclosed herein may comprise or utilize one or more devices that include hardware, such as, for example, one or more processors and system memory, as discussed herein.
- An implementation of the devices, systems, and methods disclosed herein may communicate over a computer network.
- a “network” and a “bus” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices.
- Transmission media can include a network and/or data links, which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of non-transitory computer-readable media.
- Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause the processor to perform a certain function or group of functions.
- the computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code.
- the present disclosure may be practiced in network computing environments with many types of computer system configurations, including in-dash vehicle computers, personal computers, desktop computers, laptop computers, message processors, handheld devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like.
- the disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by any combination of hardwired and wireless data links) through a network, both perform tasks.
- program modules may be located in both the local and remote memory storage devices.
- ASICs application specific integrated circuits
- a sensor may include computer code configured to be executed in one or more processors and may include hardware logic/electrical circuitry controlled by the computer code.
- processors may include hardware logic/electrical circuitry controlled by the computer code.
- At least some embodiments of the present disclosure have been directed to computer program products comprising such logic (e.g., in the form of software) stored on any computer-usable medium.
- Such software when executed in one or more data processing devices, causes a device to operate as described herein.
- any or all of the aforementioned alternate implementations may be used in any combination desired to form additional hybrid implementations of the present disclosure.
- any of the functionality described with respect to a particular device or component may be performed by another device or component.
- embodiments of the disclosure may relate to numerous other device characteristics.
- embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the embodiments.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
Description
- This disclosure generally relates to monitoring systems, and more particularly relates to monitoring systems employing a digital twin.
- Monitoring systems may be implemented by using various kinds of devices, systems, and procedures. Some monitoring systems include imaging devices such as video cameras and infrared cameras for various purposes such as for securing a property, for detecting suspicious activity, or for monitoring traffic. Processing circuitry used for processing images captured by an imaging device may vary from one monitoring system to another depending on the nature and purpose of the monitoring. For example, a sophisticated image processing system may be used for processing images in a monitoring system that employs a high-speed imaging camera, whereas a relatively less sophisticated image processing system may be employed to process images in a monitoring system that employs a less-expensive camera for capturing images containing objects that remain stationary for extended periods of time.
- In at least some applications, the sophisticated image processing system may be found suitable for use at a first location where a monitored area contains a few fast moving objects but may be unsuitable for use at a second location where the monitoring area may be too large for detecting fast moving objects and/or may include too many fast moving objects that cannot be monitored effectively.
- It is therefore desirable to provide monitoring systems and methods that offer desirable attributes such as scalability, transportability, and cost-effectiveness for use in various applications at various locations.
- A detailed description is set forth below with reference to the accompanying drawings. The use of the same reference numerals may indicate similar or identical items. Various embodiments may utilize elements and/or components other than those illustrated in the drawings, and some elements and/or components may not be present in various embodiments. Elements and/or components in the figures are not necessarily drawn to scale. Throughout this disclosure, depending on the context, singular and plural terminology may be used interchangeably.
-
FIG. 1 illustrates a first exemplary monitoring system that includes a digital twin displayed on a computer in accordance with an embodiment the disclosure. -
FIG. 2 illustrates a first exemplary operational configuration that may be used in the digital twin shown inFIG. 1 . -
FIG. 3 illustrates a second exemplary operational configuration that may be used in the digital twin shown inFIG. 1 . -
FIG. 4 illustrates a third exemplary operational configuration that may be used in the digital twin shown inFIG. 1 . -
FIG. 5 illustrates a second exemplary monitoring system incorporating a digital twin displayed on a computer in accordance with another embodiment the disclosure. -
FIG. 6 shows a flowchart of a method of operating the second exemplary monitoring system shown inFIG. 5 . -
FIG. 7 shows a first exemplary traffic condition displayed in a digital twin in accordance with the disclosure. -
FIG. 8 illustrates predictive travel routes that may be taken by a vehicle shown inFIG. 7 . -
FIG. 9 shows the vehicle having taken one of the predictive travel routes shown inFIG. 8 . -
FIG. 10 shows some exemplary components of a computer that may be used to operate a digital twin in accordance with an exemplary embodiment of the disclosure. - In terms of a general overview, various exemplary embodiments described in this disclosure are directed to systems and methods for operating a monitoring system incorporating a digital twin. In some instances, digital twin refers to a digital replica of a physical asset, process, people, place, system, and/or device. In one exemplary method of operating the digital twin, one or more images captured by an imaging apparatus may be used to define a first portion of a surveilled area as a zone of primary interest and a second portion of the surveilled area as a zone of secondary interest. The digital twin may be utilized to primarily focus on the zone of primary interest such as for determining a pattern of movement of one or more objects in the zone of primary interest. One or both of the zone of primary interest and the zone of secondary interest may be dynamically redefined upon receiving additional images that indicate a modification in the pattern of movement of one or more objects in the surveilled area.
- The disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the disclosure are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made to various embodiments without departing from the spirit and scope of the present disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments but should be defined only in accordance with the following claims and their equivalents. The description below has been presented for the purposes of illustration and is not intended to be exhaustive or to be limited to the precise form disclosed. It should be understood that alternate implementations may be used in any combination desired to form additional hybrid implementations of the present disclosure. For example, any of the functionality described with respect to a particular component such as a first processor in a first computer may be performed by another component such as a second processor in another computer. Furthermore, while specific device characteristics have been described, embodiments of the disclosure may relate to numerous other device characteristics. Further, although embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the embodiments.
- Certain words and terms are used herein solely for convenience and such words and terms should be interpreted as referring to various objects and actions that are generally understood in various forms and equivalencies by persons of ordinary skill in the art. For example, words such as “object” and “vehicle” may be used interchangeably and should be understood in the context of the disclosure. Words such as “data” “signals,” “indication,” and “information” may be used interchangeably and should be understood in the context of the disclosure. Words such as “display,” “images,” “imagery,” “snapshots,” “video,” and “video frames,” may be used herein in an interchangeable manner and should be understood to refer to sources of information for monitoring purposes. Furthermore, the word “information” as used herein may refer to various items such as digital data, analog data, audio content, video content, and/or messages. These items may be operated upon by a computer containing a processor. Words such as “monitoring” and “surveillance” may be used interchangeably and should be understood in the context of the disclosure. Phrases such as “an object is displayed” or “based on an (xxx factor)” as used herein should not be construed exclusively in a singular sense. These phrases should be understood to also encompass multiple objects that may be displayed and based on multiple factors. It should also be understood that the word “example” as used herein is intended to be non-exclusionary and non-limiting in nature. More particularly, the word “exemplary” as used herein indicates one among several examples, and it should be understood that no undue emphasis or preference is being directed to the particular example being described.
-
FIG. 1 illustrates anexemplary monitoring system 100 that includes adigital twin 107 displayed on ascreen 106 of acomputer 105 in accordance with an embodiment the disclosure. Thecomputer 105 may be coupled via anetwork 110 to animaging apparatus 115 that is configured to capture one or more images of asurveilled area 120. Thenetwork 110 may include any one or a combination of various networks such as a data network, a telephone network, a cellular network, a cable network, a wireless network, a private network, a public network, a local area network (LAN), a wide area network (WAN), and the Internet. In some instances, thenetwork 110 may support communication technologies such as Bluetooth, cellular, near-field communication (NFC), Wi-Fi, and/or Wi-Fi direct. In another exemplary embodiment, thecomputer 105 may be directly coupled to theimaging apparatus 115 via an electrical cable or a fiber optic cable and communicatively coupled via thenetwork 110 to other components such as aserver system 135. - The
imaging apparatus 115 can include various types of imaging devices such as, for example, a digital camera configured to take snapshots of the surveilled area 120 (on a cyclic basis, an intermittent basis, or as-needed basis), a video camera configured to generate video files based on video surveillance of thesurveilled area 120, or an infrared camera configured to generate infrared imagery (snapshots and/or video) of thesurveilled area 120. Theimaging apparatus 115 may also include components such as a motion detector or a speed sensor. The motion detector may be used in some applications to trigger image capture (or video capture) in theimaging apparatus 115. The speed detector may be used to obtain motion information associated with one or more moving objects in thesurveilled area 120. In an exemplary implementation, the motion information may be a numerical speed value of a moving object in the surveilled area 120 (65 miles per hour, for example). The numerical speed value may be propagated to thecomputer 105 from theimaging apparatus 115 via thenetwork 110. The numerical speed value may further include a time-stamp that is indicative of a time at which the speed of the moving object was captured. In another implementation, thecomputer 105 may determine a speed of an object by processing multiple snapshots or video frames captured by theimaging apparatus 115 and propagated to thecomputer 105 via thenetwork 110. - In the exemplary embodiment shown in
FIG. 1 , the surveilledarea 120 includes anexemplary road 122, bordered by anexemplary sidewalk 121 on one side. In other embodiments in accordance with the disclosure, any of various other types of areas may be placed under surveillance. A few examples of such areas may include a factory floor, a warehouse, a parking lot, a prison, and a company location. Each of these areas may include a combination of moving objects and stationary objects over one or more periods of time. - In this exemplary embodiment, the surveilled
area 120 may include one or more moving objects such as, for example, avehicle 130 travelling east on theroad 122, and one or more stationary objects such as adumpster 125 that is stationed upon a section of theroad 122 adjacent to thesidewalk 121. The surveilledarea 120 may further include moving objects such as a bicyclist (not shown) riding a bicycle down theroad 122, and/or a pedestrian (not shown) walking on thesidewalk 121. The surveilledarea 120 may also include stationary objects such as an unattended vehicle, a wrecked vehicle involved in an accident, a light pole, a traffic light, a construction barrier, trash, and/or debris. When the surveilledarea 120 is a factory floor, the moving objects may include workers and/or vehicles and the stationary objects may include machinery, packages, and building structures (pillars, poles, racks, etc.). - The images captured by the
imaging apparatus 115 may be received by thecomputer 105, processed for obtaining information about the surveilledarea 120, and used for generating thedigital twin 107 that is displayed on thescreen 106. In at least some implementations where theimaging apparatus 115 is configured to transmit real-time images of the surveilledarea 120, thedigital twin 107 may be updated in real-time for rendering a real-time digital replication of the surveilledarea 120. When rendered in real-time, any changes in the surveilledarea 120, such as for example, thevehicle 130 moving out of the surveilledarea 120 or thedumpster 125 being relocated on to thesidewalk 121, can be detected by thecomputer 105 and used for various purposes. - In one exemplary application, the
computer 105 may be configured to detect the relocation of a stationary object such as thedumpster 125 on thesidewalk 121. Upon detecting the relocation, thecomputer 105 may obtain data from aserver system 135 to verify whether such an action is permissible under laws governing the surveilledarea 120. In general, theserver system 135 can be a cloud-based system that includes one or more computers containing components such as processors and memory devices. In the illustrative example shown inFIG. 1 , theserver system 135 can include at least one computer having at least a processor and a memory. The memory, which is an example of a non-transitory computer-readable medium, may be used to store various types of information such as laws and rules governing the surveilledarea 120, traffic rules, municipality codes, vehicle registration data, and registration information (permits, licenses, etc.). Theserver system 135 may respond to a request received from a computer such as thecomputer 105, by transmitting the requested information to the computer via thenetwork 110. - The information received from the
server system 135 may be used by thecomputer 105 to process imagery received from theimaging apparatus 115 and operate thedigital twin 107 in accordance with one or more embodiments of the disclosure. The methods for processing imagery and/or operating thedigital twin 107 may provide various advantages such as minimizing the amount of time for initializing themonitoring system 100 to operate in various environments and various applications, optimizing information obtained from processing imagery, generating desired information based on high-speed processing of selected portions of a surveilled area, rapidly and dynamically modifying monitored zones inside a surveilled area, and system scalability so as to allow multiple imaging systems to cooperate efficiently for generating desired information about a surveilled area. Some of these aspects are described below in more detail using other figures. -
FIG. 2 illustrates a first exemplary operational configuration that may be used by thecomputer 105 to process images received from theimaging apparatus 115 and generate thedigital twin 107. In this first exemplary operational configuration, the surveilledarea 120 may be partitioned into two or more zones of interest based on various criteria. For example, the surveilledarea 120 may be partitioned into a zone ofprimary interest 205 and a zone ofsecondary interest 210 based on a motivation to monitor moving objects with a higher level of priority than stationary objects that may be monitored with a relatively lower level of priority. - In one exemplary scenario, the surveilled
area 120 may be an alleyway and theroad 122 may be a one-way street in which vehicles are permitted to only travel eastwards. Themonitoring system 100 may be used to capture images of the surveilledarea 120 that provide information such as an occurrence of one or more infractions. Thesidewalk 121 in this example, may be a narrow sidewalk, and an exemplary infraction may involve parking of thedumpster 125 upon thesidewalk 121 thereby compelling pedestrians walking on thesidewalk 121 to step on theroad 122 in order to get around thedumpster 125. As another example of an infraction, local residents and/or business people in the area may have noticed that some vehicles disregard traffic rules several times during a day and drive the wrong way in an opposite direction. - Using the
imaging apparatus 115 to capture images of the entire surveilledarea 120 at a high capture rate (an image every second, for example) and use thecomputer 105 to process a large amount of image content generated by such a high capture rate may be sub-optimal in at least some cases such as when thedumpster 125 is moved. Consequently, in accordance with the disclosure, thecomputer 105 may be configured to operate in a manner that minimizes and/or simplifies image processing operations for generating thedigital twin 107. - In the first exemplary operational configuration shown in
FIG. 2 , the zone ofprimary interest 205 may be defined on the basis of factors such as a pattern of movement of one or more objects in the surveilledarea 120 and the availability of a drivable portion of theroad 122. Towards this end, thecomputer 105 may be configured to process a first image of the surveilledarea 120 that shows thedumpster 125 parked on theroad 122 and avehicle 130 moving on theroad 122 at the time the image was captured by theimaging apparatus 115, followed by additional images that may be processed by thecomputer 105 to validate or modify the information derived from the first image. - The information derived from the first image and/or additional images may be used by the
computer 105 to define a perimeter of the zone ofprimary interest 205. The perimeter of the zone of theprimary interest 205, which precludes a portion of theroad 122 on which thedumpster 125 is located, may be based on one or more factors such as a pattern of movement of thevehicle 130 and/or thedumpster 125. The pattern of movement of thevehicle 130 and thedumpster 125 may be monitored over a first period of time (a day, for example) and used to classify thevehicle 130 as a moving object for purposes of surveillance. The pattern of movement of thedumpster 125 may be monitored over a second period of time (a month, for example) to determine that thedumpster 125 is moved occasionally from theroad 122 to thesidewalk 121, or is never moved from theroad 122. The relatively slow occurrence of movement, or the lack of movement, of thedumpster 125 may be used by thecomputer 105 to classify the dumpster as a stationary object for purposes of surveillance. In one exemplary implementation, each of the first period of time and the second period of time may be determined by an operator of thecomputer 105 after evaluating one or more characteristics of the various objects in the surveilledarea 120. - The
computer 105 may define a perimeter of the zone ofsecondary interest 210 based in part on classifying thedumpster 125 as a stationary object. The zone ofsecondary interest 210 may encompass thesidewalk 121 and a portion of theroad 122 on which thedumpster 125 is located. - The
computer 105 may process image content in the zone ofprimary interest 205 with a higher level of priority than that in the zone ofsecondary interest 210. In one exemplary procedure in accordance with the disclosure, theimaging apparatus 115 is configured to capture images of the zone ofprimary interest 205 more frequently and/or for a longer period of time than images of the zone ofsecondary interest 210. For example, a first set of images containing image content exclusively from the zone ofprimary interest 205 may be captured for a long period of time (an hour, for example) and a second set of images containing image content from the zone ofsecondary interest 210 may be captured for a shorter period of time (10 minutes, for example). Such a procedure reduces the amount of image data processed by thecomputer 105 in comparison to an amount of data that would be generated by theimaging apparatus 115 upon scanning both the zone ofprimary interest 205 and the zone ofsecondary interest 210 in a concurrent manner all the time. - In another exemplary procedure, the
imaging apparatus 115 may be configured to capture a first set of images by scanning the zone ofprimary interest 205 at a fast scanning rate (20 scans per hour, for example) and to capture a second set of images by scanning the zone ofsecondary interest 210 at a relatively slower scanning rate (2 scans per hour, for example). This exemplary procedure also reduces the amount of image data processed by thecomputer 105 in comparison to an amount of data that would be generated by theimaging apparatus 115 upon using the same scanning rate for scanning both the zone ofprimary interest 205 and the zone ofsecondary interest 210. -
FIG. 3 illustrates a second exemplary operational configuration that may be used by thecomputer 105 to process images received from theimaging apparatus 115 and generate thedigital twin 107. In this second operational configuration, thecomputer 105 may process one or more images captured by theimaging apparatus 115 and detect avehicle 315 that is parked on theroad 122 for an extended period of time (several days, for example). Thecomputer 105 may then classify thevehicle 315 as a stationary object that has to be treated as an obstruction by moving objects such as thevehicle 130. In this case, a perimeter of the zone ofsecondary interest 210 may be redefined by thecomputer 105 in accordance with the classification of thevehicle 315 as a stationary object so as to encompass a portion of theroad 122 on which thevehicle 315 is parked. The perimeter of the zone ofprimary interest 205 is also redefined in this case so as to preclude the portion of theroad 122 on which thevehicle 315 is parked. Redefining one or both of the zone ofprimary interest 205 and the zone ofsecondary interest 210 may be carried out dynamically in accordance with the disclosure whenever one or more objects move into the surveilledarea 120 or an object that is already present in the surveilledarea 120 undergoes a modification in a pattern of movement. -
FIG. 4 illustrates a third exemplary operational configuration that may be used by thecomputer 105 to process images received from theimaging apparatus 115 and generate thedigital twin 107. In this third operational configuration, thecomputer 105 may process one or more images captured by theimaging apparatus 115 and detect apedestrian 405 walking on thesidewalk 121. Thecomputer 105 may classify thepedestrian 405 as a moving object and may redefine a perimeter of the zone ofprimary interest 205 based on this classification. The perimeter of the zone ofsecondary interest 210 is also redefined to preclude the portion of thesidewalk 121 on which thepedestrian 405 is walking. Redefining one or both of the zone ofprimary interest 205 and the zone ofsecondary interest 210 may be carried out dynamically in accordance with the disclosure, whenever one or more objects such as thepedestrian 405 move into the surveilledarea 120 or an object that is already present in the surveilledarea 120 undergoes a modification in a pattern of movement. Thecomputer 105 may also be configured to process image content from both the zone ofprimary interest 205 and the zone ofsecondary interest 210 in conformance with one or more laws that preclude collection of personal information of people (such as the pedestrian 405). -
FIG. 5 illustrates a secondexemplary monitoring system 500 that can include two or more imaging apparatuses in accordance with an embodiment the disclosure. In the exemplary illustration shown inFIG. 5 , afirst imaging apparatus 530 provides animage capture area 505 that may be defined by a line-of-sight 532 and a line-of-sight 531 of thefirst imaging apparatus 530. Images captured by thefirst imaging apparatus 530 may be transmitted to thecomputer 105 via a network such as thenetwork 110 shown inFIG. 1 . Thecomputer 105 may process these images for various purposes such as for displaying adigital twin 550 on thedisplay screen 106 that includes objects present in theimage capture area 505. - A
second imaging apparatus 540 provides animage capture area 515 that may be defined by a line-of-sight 541 and a line-of-sight 542 of thesecond imaging apparatus 540. Images captured by thesecond imaging apparatus 540 may be transmitted to thecomputer 105 via a network such as thenetwork 110. Thecomputer 105 may process these images for various purposes such as for configuring thedigital twin 550 to include objects present in theimage capture area 515 in addition to the objects present in theimage capture area 505. - In one exemplary implementation, the
digital twin 550 may be provided in the form of a concatenated display that incorporates images captured by thesecond imaging apparatus 540 as well as images captured by thefirst imaging apparatus 530. In another implementation, thedigital twin 550 can be provided in the form of two separate windows—a first window containing images captured by thefirst imaging apparatus 530 and a second window containing images captured by thesecond imaging apparatus 540. - In the example embodiment illustrated in
FIG. 5 , theimage capture area 505 overlaps theimage capture area 515 in anoverlap area 506. Theoverlap area 506 allows for seamless tracking of a vehicle such as thevehicle 130 when the vehicle is moving on theroad 122. The seamless tracking allows thecomputer 105 to eliminate blind spots by providing thecomputer 105, the ability to detect stoppage of thevehicle 130 at a boundary portion between theimage capture area 505 and theimage capture area 515. - A
third imaging apparatus 535 provides animage capture area 520 that may be defined by a line-of-sight 536 and a line-of-sight 537 of thethird imaging apparatus 535. Anoverlap area 507 allows for seamless tracking of a vehicle such as thevehicle 130, through theimage capture area 515 and theimage capture area 520. The seamless tracking allows thecomputer 105 to eliminate blind spots by providing thecomputer 105, the ability to detect stoppage of thevehicle 130 at a boundary portion between theimage capture area 515 and theimage capture area 520. - A
fourth imaging apparatus 545 provides animage capture area 525 that may be defined by a line-of-sight 546 and a line-of-sight 547 of thefourth imaging apparatus 545. Anoverlap area 508 allows for seamless tracking of a vehicle such as thevehicle 130, through theimage capture area 520 and theimage capture area 525. The seamless tracking allows thecomputer 105 to eliminate blind spots by providing thecomputer 105, the ability to detect stoppage of thevehicle 130 at a boundary portion between theimage capture area 520 and theimage capture area 525. - In one exemplary implementation, the
digital twin 550 may be provided in the form of a concatenated display that incorporates images captured by thefirst imaging apparatus 530, thesecond imaging apparatus 540, thethird imaging apparatus 535, and thefourth imaging apparatus 545. In another implementation, thedigital twin 550 can be provided in the form of four separate windows—a first window containing images captured by thefirst imaging apparatus 530, a second window containing images captured by thesecond imaging apparatus 540, a third window containing images captured by thethird imaging apparatus 535, and a fourth window containing images captured by thefourth imaging apparatus 545. - In another exemplary implementation, some or all of the image capture areas can be modified in size for various reasons such as for defining one or more zones of primary interest and/or one or more zones of secondary interest in accordance with the disclosure.
-
FIG. 6 shows anexemplary flowchart 600 of a method of operating themonitoring system 500 shown inFIG. 5 . Theexemplary flowchart 600 illustrates a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable media, that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be carried out in a different order, omitted, combined in any order, and/or carried out in parallel. Some or all of the operations described in theexemplary flowchart 600 may be carried out by using an application stored in a memory provided in thecomputer 105 and executed by a processor in thecomputer 105. - At
block 605, an object such as thevehicle 130 shown inFIG. 1 , enters a first zone (theimage capture area 505, for example). Atblock 610, thefirst imaging apparatus 530 captures an image of the first zone and transmits the image to thecomputer 105. Thecomputer 105 processes the image and detects the entry of thevehicle 130 into the first zone. The detection may be made with “x” % confidence level. For example, in one case, the image may have high resolution and the detection of the vehicle 130 (based on distinguishing marks on thevehicle 130, for example) may be carried out with a 99% confidence level. In another case, the image may have poor quality and the detection may be carried out with a 60% confidence level. - At
block 615, the vehicle passes through the first zone and enters a second zone (theimage capture area 515, for example). Atblock 620, thesecond imaging apparatus 540 captures an image of the second zone and transmits the image to thecomputer 105. Thecomputer 105 processes the image and detects the entry of thevehicle 130 into the second zone. The detection may be made with “y” % confidence level. For example, in one case, the image may have high resolution and the detection of the vehicle 130 (based on distinguishing marks on thevehicle 130, for example) may be carried out with an 80% confidence level. In another case, the image may have poor quality and the detection may be carried out with a 40% confidence level. - At
block 625, a determination is made whether the first zone overlaps the second zone. If an overlap exists (such as in the overlap area 506), atblock 630, a detection result obtained from the first zone (having the “x” % confidence level) may be validated by using a detection result obtained from the second zone (having the “y” % confidence level). - At
block 635, tracking of thevehicle 130 through additional zones may be carried out by using other imaging apparatuses such as thethird imaging apparatus 535 and thefourth imaging apparatus 545. Atblock 640, one or more zones of primary interest may be determined or redefined and applied so as to recursively implement the operations described above. - If at
block 625, the determination indicates no overlap (for example, in an implementation where theoverlap area 506 does not exist), atblock 635, tracking of thevehicle 130 through additional zones may be carried out as described above. -
FIG. 7 shows an exemplary traffic condition that may be displayed in thedigital twin 107 at a first instant in time (t0) in accordance with the disclosure. Thevehicle 130 is traveling eastwards along apath 705. At the first instant in time (t0), thevehicle 130 has not yet reached aspot 710. Thespot 710, which is aligned with adistal end 715 of thedumpster 125 in this example. -
FIG. 8 illustrates a few exemplary predictive travel paths that thevehicle 130 may take after reaching thespot 710. The exemplary predictive travel paths may be identified by thecomputer 105 by executing a prediction procedure in accordance with the disclosure that may include processing one or more images provided to thecomputer 105 by theimaging apparatus 115 at the first instant in time (t0). In this example, thevehicle 130 may take one of a firstpredictive travel path 805 or a secondpredictive travel path 810. The prediction procedure used by thecomputer 105 can include various algorithms, models, and/or techniques such as Markov chains, neural networks, artificial intelligence (AI), and finite state machines. - The prediction procedure may be executed by the
computer 105 independently, or in conjunction with other procedures such as image processing procedures or simulation procedures, for defining and/or redefining a perimeter of a zone of primary interest and/or a zone of secondary interest. In one exemplary case, the defining or redefining may be carried out by using a finite state machine (FSM) to simulate various conditions (states). The various states may change as a result of various types of inputs. For example, a state of thedumpster 125 may change if thedumpster 125 is moved after having been left undisturbed for a long period of time. As another example, a perimeter of a zone of primary interest, which may be used as one of the states in an FSM, may change based on an object entering the zone of primary interest. -
FIG. 9 shows thevehicle 130 having reached thespot 710 at a second instant in time (t1) and having opted to travel along the firstpredictive travel path 805. The action taken by thevehicle 130 to travel along the first predictive travel path 805 (rather than the second predictive travel path 810) may be identified by thecomputer 105 by executing a verification procedure in accordance with the disclosure. The verification procedure may include processing one or more images provided to thecomputer 105 by theimaging apparatus 115 at the second instant in time (t1). - The prediction procedure and the evaluation procedure described above may be used for various purposes such as for dynamically modifying one or more zones of primary interest and/or for dynamically modifying one or more zones of secondary interest, for pre-empting traffic congestion on the
road 122, for issuing movement violation citations, and/or issuing parking citations. -
FIG. 10 shows some exemplary components of thecomputer 105 in accordance with an embodiment of the disclosure. Thecomputer 105 can include various components such as aprocessor 11, acommunications module 12, and amemory 13. Thememory 13, which is one example of a non-transitory computer-readable medium, may be used to store an operating system (OS) 16 and various other code modules such as a digitaltwin operating module 14 and adatabase 15. - The
communications module 12, which may include an input/output interface for communicatively coupling thecomputer 105 to thenetwork 110, allows thecomputer 105 to communicate with theserver system 135 and with various imaging apparatuses such as theimaging apparatus 115 shown inFIG. 1 . The digitaltwin operating module 14 may be executed by theprocessor 11 for implementing various operations described herein in accordance with the disclosure. - The
memory 13 can include any one memory element or a combination of volatile memory elements (e.g., random-access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, hard drive, tape, CD ROM, etc.). Moreover, the memory device may incorporate electronic, magnetic, optical, and/or other types of storage media. In the context of this document, a “non-transitory computer-readable medium” can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: a portable computer diskette (magnetic), a random-access memory (RAM) (electronic), a read-only memory (ROM) (electronic), an erasable programmable read-only memory (EPROM, EEPROM, or Flash memory) (electronic), and a portable compact disc read-only memory (CD ROM) (optical). Note that the computer-readable medium could even be paper or another suitable medium upon which the program is printed, since the program can be electronically captured, for instance, via optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory. - In the context of software, the operations described herein with respect to the
computer 105 may be implemented by computer-executable instructions stored on one or more non-transitory computer-readable media such as thememory 13, that, when executed by one or more processors such as theprocessor 11, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. - In the above disclosure, reference has been made to the accompanying drawings, which form a part hereof, which illustrate specific implementations in which the present disclosure may be practiced. It is understood that other implementations may be utilized, and structural changes may be made without departing from the scope of the present disclosure. References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, one skilled in the art will recognize such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
- Implementations of the systems, apparatuses, devices, and methods disclosed herein may comprise or utilize one or more devices that include hardware, such as, for example, one or more processors and system memory, as discussed herein.
- An implementation of the devices, systems, and methods disclosed herein may communicate over a computer network. A “network” and a “bus” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network, a bus, or another communications connection (either hardwired, wireless, or any combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmission media can include a network and/or data links, which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of non-transitory computer-readable media.
- Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause the processor to perform a certain function or group of functions. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.
- Those skilled in the art will appreciate that the present disclosure may be practiced in network computing environments with many types of computer system configurations, including in-dash vehicle computers, personal computers, desktop computers, laptop computers, message processors, handheld devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by any combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both the local and remote memory storage devices.
- Further, where appropriate, the functions described herein can be performed in one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the description and claims refer to particular system components. As one skilled in the art will appreciate, components may be referred to by different names. This document does not intend to distinguish between components that differ in name, but not function.
- It should be noted that the sensor embodiments discussed above may comprise computer hardware, software, firmware, or any combination thereof to perform at least a portion of their functions. For example, a sensor may include computer code configured to be executed in one or more processors and may include hardware logic/electrical circuitry controlled by the computer code. These example devices are provided herein for purposes of illustration and are not intended to be limiting. Embodiments of the present disclosure may be implemented in further types of devices, as would be known to persons skilled in the relevant art(s).
- At least some embodiments of the present disclosure have been directed to computer program products comprising such logic (e.g., in the form of software) stored on any computer-usable medium. Such software, when executed in one or more data processing devices, causes a device to operate as described herein.
- While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the present disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments but should be defined only in accordance with the following claims and their equivalents. The foregoing description has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. Further, it should be noted that any or all of the aforementioned alternate implementations may be used in any combination desired to form additional hybrid implementations of the present disclosure. For example, any of the functionality described with respect to a particular device or component may be performed by another device or component. Further, while specific device characteristics have been described, embodiments of the disclosure may relate to numerous other device characteristics. Further, although embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the embodiments. Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments could include, while other embodiments may not include, certain features, elements, and/or steps. Thus, such conditional language is not generally intended to imply that features, elements, and/or steps are in any way required for one or more embodiments. Although certain aspects of various embodiments may have been described using a singular word or phrase (such as “a signal” or “a processor”) it should be understood that the description may be equally applicable to plural words or phrases (such as “signals” and “processors”).
Claims (20)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/534,938 US10922826B1 (en) | 2019-08-07 | 2019-08-07 | Digital twin monitoring systems and methods |
CN202010789307.XA CN112351190A (en) | 2019-08-07 | 2020-08-07 | Digital twin monitoring system and method |
DE102020120953.3A DE102020120953A1 (en) | 2019-08-07 | 2020-08-07 | DIGITAL TWIN SURVEILLANCE SYSTEMS AND METHODS |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/534,938 US10922826B1 (en) | 2019-08-07 | 2019-08-07 | Digital twin monitoring systems and methods |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210042940A1 true US20210042940A1 (en) | 2021-02-11 |
US10922826B1 US10922826B1 (en) | 2021-02-16 |
Family
ID=74188398
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/534,938 Active US10922826B1 (en) | 2019-08-07 | 2019-08-07 | Digital twin monitoring systems and methods |
Country Status (3)
Country | Link |
---|---|
US (1) | US10922826B1 (en) |
CN (1) | CN112351190A (en) |
DE (1) | DE102020120953A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10997802B2 (en) * | 2019-02-14 | 2021-05-04 | Oshkosh Corporation | Systems and methods for a virtual refuse vehicle |
CN114707209A (en) * | 2022-03-23 | 2022-07-05 | 南京工业大学 | Pavement detection method and system based on digital twins and construction method thereof |
US11683224B1 (en) * | 2022-03-16 | 2023-06-20 | International Business Machines Corporation | Device introduction into an existing ecosystem |
US12045753B2 (en) * | 2022-07-01 | 2024-07-23 | Corners Co., Ltd. | System for detecting accident risk in working place |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112991742B (en) * | 2021-04-21 | 2021-08-20 | 四川见山科技有限责任公司 | Visual simulation method and system for real-time traffic data |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160180669A1 (en) * | 2014-12-18 | 2016-06-23 | Checkpoint Systems, Inc. | Dual mode security tags |
US20170313332A1 (en) * | 2002-06-04 | 2017-11-02 | General Electric Company | Autonomous vehicle system and method |
US20190057647A1 (en) * | 2017-08-18 | 2019-02-21 | Universal Display Corporation | Low latency low video bandwidth oled display architecture |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9711050B2 (en) | 2015-06-05 | 2017-07-18 | Bao Tran | Smart vehicle |
EP3433131B1 (en) | 2016-03-23 | 2023-07-12 | Netradyne, Inc. | Advanced path prediction |
EP3598074B1 (en) | 2016-06-27 | 2021-10-20 | Mobileye Vision Technologies Ltd. | Controlling host vehicle based on the detection of a vehicle moving towards the host vehicle's lane. |
US20190050732A1 (en) | 2018-08-28 | 2019-02-14 | Intel Corporation | Dynamic responsiveness prediction |
-
2019
- 2019-08-07 US US16/534,938 patent/US10922826B1/en active Active
-
2020
- 2020-08-07 DE DE102020120953.3A patent/DE102020120953A1/en active Pending
- 2020-08-07 CN CN202010789307.XA patent/CN112351190A/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170313332A1 (en) * | 2002-06-04 | 2017-11-02 | General Electric Company | Autonomous vehicle system and method |
US20160180669A1 (en) * | 2014-12-18 | 2016-06-23 | Checkpoint Systems, Inc. | Dual mode security tags |
US20190057647A1 (en) * | 2017-08-18 | 2019-02-21 | Universal Display Corporation | Low latency low video bandwidth oled display architecture |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10997802B2 (en) * | 2019-02-14 | 2021-05-04 | Oshkosh Corporation | Systems and methods for a virtual refuse vehicle |
US11380145B2 (en) * | 2019-02-14 | 2022-07-05 | Oshkosh Corporation | Systems and methods for a virtual refuse vehicle |
US11710356B2 (en) | 2019-02-14 | 2023-07-25 | Oshkosh Corporation | Systems and methods for a virtual refuse vehicle |
US11769354B2 (en) | 2019-02-14 | 2023-09-26 | Oshkosh Corporation | Systems and methods for a virtual vehicle |
US11683224B1 (en) * | 2022-03-16 | 2023-06-20 | International Business Machines Corporation | Device introduction into an existing ecosystem |
CN114707209A (en) * | 2022-03-23 | 2022-07-05 | 南京工业大学 | Pavement detection method and system based on digital twins and construction method thereof |
US12045753B2 (en) * | 2022-07-01 | 2024-07-23 | Corners Co., Ltd. | System for detecting accident risk in working place |
Also Published As
Publication number | Publication date |
---|---|
CN112351190A (en) | 2021-02-09 |
US10922826B1 (en) | 2021-02-16 |
DE102020120953A1 (en) | 2021-02-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10922826B1 (en) | Digital twin monitoring systems and methods | |
US11562020B2 (en) | Short-term and long-term memory on an edge device | |
US10599929B2 (en) | Event monitoring with object detection systems | |
US10678240B2 (en) | Sensor modification based on an annotated environmental model | |
US9275547B2 (en) | Prediction of free parking spaces in a parking area | |
US10971003B2 (en) | Systems and methods for predicting pedestrian behavior | |
CN113409587B (en) | Abnormal vehicle detection method, device, equipment and storage medium | |
US20220303738A1 (en) | On-board machine vision device for activating vehicular messages from traffic signs | |
KR20140055175A (en) | Method and apparatus for sharing picture in the system | |
Nithya et al. | A smart parking system: an IoT based computer vision approach for free parking spot detection using faster R-CNN with YOLOv3 method | |
KR101954719B1 (en) | Apparatus for Detecting Event and Driving Method Thereof | |
Senthil et al. | An IoT-enabled smart network traffic signal assistant system for emergency vehicles using computer vision | |
KR20180068462A (en) | Traffic Light Control System and Method | |
CN115174889A (en) | Position deviation detection method for camera, electronic device, and storage medium | |
Choi et al. | State Machine and Downhill Simplex Approach for Vision‐Based Nighttime Vehicle Detection | |
KR102248658B1 (en) | Traffic managing system using artificial intelligence | |
Kedwan | Traffic light timer adjustment system using threshold edge detection for image processing: A software engineering approach | |
JP2018192844A (en) | Monitoring device, monitoring system, monitoring program, and storage medium | |
Sukhinskiy et al. | Developing a parking monitoring system based on the analysis of images from an outdoor surveillance camera | |
Rindt | Situational Awareness for Transportation Management: Automated Video Incident Detection and Other Machine Learning Technologies for the Traffic Management Center | |
KR102483250B1 (en) | Apparatus for safe and management employing image artificial intelligence for controlling to infra of ocean city | |
Africa | International Journal of Emerging Trends in Engineering Research | |
Thakare et al. | Advanced Traffic Clearance System for Emergency Vehicles | |
US20220036098A1 (en) | Vehicle sensor data sharing | |
Gupta et al. | A Novel Trespassing Detection System Using Deep Networks |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: FORD GLOBAL TECHNOLOGIES, LLC, MICHIGAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MITRA, PRAMITA;RAO, PRASHANT;WINGFIELD, ERIC H.;AND OTHERS;SIGNING DATES FROM 20190607 TO 20190611;REEL/FRAME:052627/0641 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |