MIT News - Low-density parity check codes
https://news.mit.edu/topic/mitlow-density-parity-check-codes-rss.xml
MIT news feed about: Low-density parity check codesenThu, 04 Aug 2011 04:00:01 -0400Perfect communication with imperfect chips
https://news.mit.edu/2011/imperfect-circuits-0804
Error-correcting codes discovered at MIT can still guarantee reliable communication, even in cellphones with failure-prone low-power chips.Thu, 04 Aug 2011 04:00:01 -0400https://news.mit.edu/2011/imperfect-circuits-0804Larry Hardesty, MIT News OfficeOne of the triumphs of the information age is the idea of error-correcting codes, which ensure that data carried by electromagnetic signals — traveling through the air, or through cables or optical fibers — can be reconstructed flawlessly at the receiving end, even when they’ve been corrupted by electrical interference or other sources of what engineers call “noise.”<br /><br />For more than 60 years, the analysis of error-correcting codes has assumed that, however corrupted a signal may be, the circuits that decode it are error-free. In the next 10 years, however, that assumption may have to change. In order to extend the battery life of portable computing devices, manufacturers may soon turn to low-power signal-processing circuits that are themselves susceptible to noise, meaning that errors sometimes creep into their computations.<br /><br />Fortunately, <a href="http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5895097" target="_blank">in the July issue</a> of <i>IEEE Transactions on Information Theory</i>, Lav Varshney PhD ‘10, a research affiliate at MIT’s Research Laboratory of Electronics, demonstrates that some of the most commonly used codes in telecommunications can still ensure faithful transmission of information, even when the decoders themselves are noisy. The same analysis, which is adapted from his MIT thesis, also demonstrates that memory chips, which present the same trade-off between energy efficiency and reliability that signal-processing chips do, can preserve data indefinitely even when their circuits sometimes fail.<br /><br />According to the semiconductor industry’s 15-year projections, both memory and computational circuits “will become smaller and lower-power,” Varshney explains. “As you make circuits smaller and lower-power, they’re subject to noise. So these effects are starting to come into play.”<br /><br /><strong>Playing the odds</strong><br /><br />The theory of error-correcting codes <a href="/newsoffice/2010/explained-shannon-0115.html" target="_self">was established</a> by Claude Shannon — who taught at MIT for 22 years — in a groundbreaking 1948 paper. Shannon envisioned a message sent through a communications medium as a sequence of bits — 0s and 1s. Noise in the channel might cause some of the bits to flip or become indeterminate. An error-correcting code would consist of additional bits tacked on to the message bits and containing information about them. If message bits became corrupted, the extra bits would help describe what their values were supposed to be.<br /><br />The longer the error-correcting code, the less efficient the transmission of information, since more total bits are required for a given number of message bits. To date, the most efficient codes known are those <a href="/newsoffice/2010/gallager-codes-0121.html" target="_self">discovered in 1960</a> by MIT professor emeritus Robert Gallager, which are called low-density parity-check codes — or sometimes, more succinctly, Gallager codes. Those are the codes that Varshney analyzed.<br /><br />The key to his new analysis, Varshney explains, was not to attempt to quantify the performance of particular codes and decoders but rather to look at the statistical properties of whole classes of them. Once he was able to show that, on average, a set of noisy decoders could guarantee faithful reconstruction of corrupted data, he was then able to identify a single decoder within that set that met the average-performance standard.<br /><br /><strong>The noisy brain</strong><br /><br />Today, updated and optimized versions of Gallager’s 1960 codes are used for error correction by many cellphone carriers. Those codes would have to be slightly modified to guarantee optimal performance with noisy circuits, but “you use essentially the same decoding methodologies that Gallager did,” Varshney says. And since the codes have to correct for errors in both transmission and decoding, they would also yield lower transmission rates (or require higher-power transmitters).<br /><br />Shashi Chilappagari, an engineer at Marvell Semiconductor, which designs signal-processing chips, says that like Gallager’s codes, the question of whether noisy circuits can correct transmission errors dates back to the 1960s, when it attracted the attention of computing pioneer John von Neumann. “This is kind of a surprising result,” Chilappagari says. “It’s not very intuitive to say that this kind of scheme can work.” Chilappagari points out, however, that like most analyses of error-correcting codes, Varshney’s draws conclusions by considering what happens as the length of the encoded messages approaches infinity. Chipmakers would be reluctant to adopt the coding scheme Varshney proposes, Chilappagari says, without “time to test it and see how it works on a given-length code.”<br /><br />While researching his thesis, Varshney noticed that the decoder for Gallager’s codes — which in fact passes data back and forth between several different decoders, gradually refining its reconstruction of the original message — has a similar structure to ensembles of neurons in the brain’s cortex. In ongoing work, he’s trying to determine whether his analysis of error-correcting codes can be adapted to characterize information processing in the brain. “It’s pretty well established that neural things are noisy — neurons fire randomly … and there’s other forms of noise as well,” Varshney says.<br /><br />Graphic: Christine DaniloffInformation theoryLow-density parity check codesResearch Laboratory of ElectronicsCoding theoryError correctionFault-tolerant computingExplained: Gallager codes
https://news.mit.edu/2010/gallager-codes-0121
In 1993, scientists achieved the maximum rate for data transmission — only to find they’d been scooped 30 years earlier by an MIT grad student.Thu, 21 Jan 2010 05:01:00 -0500https://news.mit.edu/2010/gallager-codes-0121Larry Hardesty, MIT News Office<i>This is the second part of a two-part Explained about information theory. The first part, on the <a target="_blank" mce_href="/newsoffice/2010/explained-shannon-0115.html" href="/newsoffice/2010/explained-shannon-0115.html">Shannon limit</a>, appeared on Tuesday.</i><br /><br />In the 1948 paper that created the field of information theory, MIT grad (and future professor) Claude Shannon threw down the gauntlet to future generations of researchers. In those predigital days, communications channels — such as phone lines or radio bands — were particularly susceptible to the electrical or electromagnetic disruptions known as “noise.” Shannon proved the counterintuitive result that no matter how noisy a channel, information could be sent over it error free. All you needed was a way to add enough redundancy to the information so that errors could be corrected. He also demonstrated that there was a hard limit on how efficient those error-correcting codes could be — a minimum amount of extra information that would guarantee near-zero error. Since longer codes take longer to send, a minimum code length implied a maximum transmission rate — <a target="_blank" mce_href="/newsoffice/2010/explained-shannon-0115.html" href="/newsoffice/2010/explained-shannon-0115.html">the Shannon limit</a>. Finally, Shannon proved that codes approaching that limit must exist. But he didn’t show how to find them.<br /><br />For the next 45 years, researchers sought those codes. Along the way, there were improvements of the kind that helped increase modem speeds from 9.6 kilobits per second to 14.4 kilobits per second in the early 1980s. But according to Muriel Médard, a professor of computer science and electrical engineering at MIT, proposed codes tended to run up against a limit called the computational cutoff rate. That rate varied according to the transmission power and noise characteristics of a channel, but in practical communications systems, it might be only halfway to the Shannon limit.<br /><br />Then, in 1993, at the Institute of Electrical and Electronics Engineers’ International Communications Conference, Alain Glavieux and Claude Berrou of the École Nationale Supérieure des Télécommunications de Bretagne presented a new set of codes that they claimed came very close to the Shannon limit. “People almost laughed them out of the room,” Médard says, “especially because they were not coming from the coding side; they were coming from the electronics side.” The researchers had developed their codes — dubbed “turbo codes” — largely through trial and error and had no elegant formal explanation for why they worked so well. Nonetheless, subsequent investigation quickly confirmed their results.<br /><br />Turbo codes are so-called iterative codes, which means that the decoder makes a series of guesses about what the encoded message is supposed to be. Each successive guess is fed back into the decoder, and the result is a more refined guess. Ideally, repeating the process over and over will get the error rate as low as you want.<br /><br />The startling performance of turbo codes mobilized researchers to try to explain why they worked so well. Within a few years, investigation of iterative coding schemes had yielded a perhaps even more surprising result: a set of codes that worked at least as well as turbo codes had been around since 1960, when they were introduced in the MIT doctoral thesis of Robert Gallager.<br /><br /><b>Quiet revolution</b><br /><br />The power of Gallager’s codes went unappreciated for so long because the decoding process he proposed was simply too complicated for 1960s-era technology. Which is ironic, since simplifying the decoding process was his motive in creating the codes. “The crux of the whole thing was, How do you design a good decoding algorithm?” says Gallager, who taught at MIT from 1960 to 2001 and still supervises graduate students as a professor emeritus. “And then given that idea for how to do that, how do you generate codes that you can actually decode in this way?” At the time, however, research on new coding schemes frequently depended on statistical claims about the performance of hypothetical ideal decoders. For researchers like Gallager, who were trying to develop codes that approached the Shannon limit, specifying a concrete decoding algorithm at all was already an uncommon step in the direction of practical deployment.<br /><br />Gallager’s codes use so-called parity bits — extra bits that contain information about message bits. One parity bit might indicate, say, whether the sum of message bits 1, 2, and 4 is even or odd; the next parity bit might do the same for message bits 3, 4, and 6; and so on, through successive triplets of bits. Reliable information about any two bits in a triplet conveys reliable information about the third. “Iterative techniques involve making a first guess of what a received bit might be and giving it a weight according to how reliable it was,” says David Forney, an adjunct professor in MIT’s Laboratory for Information and Decision Systems. “Then maybe you get more information about it because it’s involved in parity checks with other bits, and so that gives you an improved estimate of its reliability — might go the same way, might go the opposite way — and through a series of computations like this, hopefully the thing will converge to where all the bits are known highly reliably.” Problems arise, Forney says, “if you begin to confuse yourself because you’re just feeding back reliabilities that you’ve already used in the same computation, so you get a false positive increase in reliability. It’s like a rumor mill. If you keep hearing the same rumor from the same people again and again, you can all begin to think it’s true, when it’s really just a closed circuit.” The trick to the design of Gallager’s codes, Forney says, was to minimize the likelihood of such closed loops. “It should take a long time for the telephone chain to go all around the world before it gets back to you again,” he says.<br /><br />To date, Gallager’s codes have enabled the closest approaches to the Shannon limit for a given communications channel — closer even than turbo codes. They’ve been integrated into standards for wireless data transmissions, and computer chips dedicated to decoding Gallager’s codes can be found in commercial cell phones. During their long eclipse, did Gallager have any inkling of how good they were?<br /><br />“I had a little bit of an inkling, but I also had a suspicion that they well might not be,” Gallager says. “And I spent a long time trying to resolve whether they were or weren’t.” His conclusion was equivocal: “What I showed is that with different classes of these codes, you could achieve positive [transmission] rates. As you change the class to make it more complicated, the rate would continue to increase. If you made it complicated enough, you could reach capacity — but you would probably never decode it. What’s happened since is that people have found ways of somewhat streamlining the way you choose the codes to make them better codes.”<br /><br /><br />Professor Emeritus Robert GallagerChannel codingExplainedInformation theoryLow-density parity check codesTurbo codes