Data diversity boosts machine learning

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory and its Laboratory for Information and Decision Systems have developed an algorithm that makes the selection of diverse subsets much more practical.

12/22/2016


Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory and its Laboratory for Information and Decision Systems have designed a new algorithm that makes it much more practical to select diverse subsets from a much larger dataset. CoWhen data sets get too big, sometimes the only way to do anything useful with them is to extract much smaller subsets and analyze those instead.

Those subsets have to preserve certain properties of the full sets, however, and one property that's useful in a wide range of applications is diversity. If, for instance, you're using your data to train a machine-learning system, you want to make sure that the subset you select represents the full range of cases that the system will have to confront.

Researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and its Laboratory for Information and Decision Systems presented a new algorithm that makes the selection of diverse subsets much more practical.

Whereas the running times of earlier subset-selection algorithms depended on the number of data points in the complete data set, the running time of the new algorithm depends on the number of data points in the subset. That means that if the goal is to winnow a data set with 1 million points down to one with 1,000, the new algorithm is 1 billion times faster than its predecessors.

"We want to pick sets that are diverse," said Stefanie Jegelka, the X-Window Consortium career development assistant professor in MIT's Department of Electrical Engineering and Computer Science and senior author on the new paper.

"Why is this useful? One example is recommendation. If you recommend books or movies to someone, you maybe want to have a diverse set of items, rather than 10 little variations on the same thing," Jegelka said. "Or if you search for, say, the word 'Washington.' There's many different meanings that this word can have, and you maybe want to show a few different ones. Or if you have a large data set and you want to explore—say, a large collection of images or health records—and you want a brief synopsis of your data, you want something that is diverse, that captures all the directions of variation of the data.

"The other application where we actually use this thing is in large-scale learning. You have a large data set again, and you want to pick a small part of it from which you can learn very well," Jegelka said. 

Thinking small

Traditionally, if you want to extract a diverse subset from a large data set, the first step is to create a similarity matrix - a huge table that maps every point in the data set against every other point. The intersection of the row representing one data item and the column representing another contains the points' similarity score on some standard measure.

There are several standard methods to extract diverse subsets, but they all involve operations performed on the matrix as a whole. With a data set with a million data points—and a million-by-million similarity matrix—this is prohibitively time consuming.

The MIT researchers' algorithm begins, instead, with a small subset of the data, chosen at random. Then it picks one point inside the subset and one point outside it and randomly selects one of three simple operations: swapping the points, adding the point outside the subset to the subset, or deleting the point inside the subset.

The probability with which the algorithm selects one of those operations depends on both the size of the full data set and the size of the subset, so it changes slightly with every addition or deletion. But the algorithm doesn't necessarily perform the operation it selects.

The decision to perform the operation or not is probabilistic, but here the probability depends on the improvement in diversity that the operation affords. For additions and deletions, the decision also depends on the size of the subset relative to that of the original data set. That is, as the subset grows, it becomes harder to add new points unless they improve diversity dramatically.

This process repeats until the diversity of the subset reflects that of the full set. Since the diversity of the full set is never calculated, however, the question is how many repetitions are enough. The researchers' chief results are a way to answer that question and a proof that the answer will be reasonable.

Massachusetts Institute of Technology (MIT)

www.mit.edu 

- Edited by Chris Vavra, production editor, Control Engineering, CFE Media, cvavra@cfemedia.com. See more Control Engineering asset management stories.



The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers. Vote now (if qualified)!
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
HMI effectiveness; Distributed I/O; Engineers' Choice Award finalists; System Integrator advice; Inside Machines
Women in engineering; Engineering Leaders Under 40; PID benefits and drawbacks; Ladder logic; Cloud computing
Robotic integration and cloud connections; SCADA and cybersecurity; Motor efficiency standards; Open- and closed-loop control; Augmented reality
Programmable logic controllers (PLCs) represent the logic (decision) part of the control loop of sense, decide, and actuate. As we know, PLCs aren’t the only option for making decisions in a control loop, but they are likely why you’re here.
This digital report explains how motion control advances and solutions can help with machine control, automated control on assembly lines, integration of robotics and automation, and machine safety.
This article collection contains several articles on how advancements in vision system designs, computing power, algorithms, optics, and communications are making machine vision more cost effective than ever before.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Control room technology innovation; Practical approaches to corrosion protection; Pipeline regulator revises quality programs
Cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers. Vote now (if qualified)!
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
HMI effectiveness; Distributed I/O; Engineers' Choice Award finalists; System Integrator advice; Inside Machines
Women in engineering; Engineering Leaders Under 40; PID benefits and drawbacks; Ladder logic; Cloud computing
Robotic integration and cloud connections; SCADA and cybersecurity; Motor efficiency standards; Open- and closed-loop control; Augmented reality
Programmable logic controllers (PLCs) represent the logic (decision) part of the control loop of sense, decide, and actuate. As we know, PLCs aren’t the only option for making decisions in a control loop, but they are likely why you’re here.
This digital report explains how motion control advances and solutions can help with machine control, automated control on assembly lines, integration of robotics and automation, and machine safety.
This article collection contains several articles on how advancements in vision system designs, computing power, algorithms, optics, and communications are making machine vision more cost effective than ever before.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Control room technology innovation; Practical approaches to corrosion protection; Pipeline regulator revises quality programs
Cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by Control Engineering subscribers. Vote now (if qualified)!
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
HMI effectiveness; Distributed I/O; Engineers' Choice Award finalists; System Integrator advice; Inside Machines
Women in engineering; Engineering Leaders Under 40; PID benefits and drawbacks; Ladder logic; Cloud computing
Robotic integration and cloud connections; SCADA and cybersecurity; Motor efficiency standards; Open- and closed-loop control; Augmented reality
Programmable logic controllers (PLCs) represent the logic (decision) part of the control loop of sense, decide, and actuate. As we know, PLCs aren’t the only option for making decisions in a control loop, but they are likely why you’re here.
This digital report explains how motion control advances and solutions can help with machine control, automated control on assembly lines, integration of robotics and automation, and machine safety.
This article collection contains several articles on how advancements in vision system designs, computing power, algorithms, optics, and communications are making machine vision more cost effective than ever before.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Control room technology innovation; Practical approaches to corrosion protection; Pipeline regulator revises quality programs
Cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me