ENGG1150 Reflectoin
on!the!randomness!of!the!event!was!quite!astonishing.!The!next!part!of!our!discussion!was!related!to!source!coding.!Source!coding!is!the!science!of!representing!information!as!binary!numbers.!The!main!purpose!of!source!coding!is!data!compression.!It!amazed!me!how!this!coding!can!compress!such!large!amounts!of!information!into!binary!numbers!and!transmit!it.!After!this!I!understood!how!the!zip.!file/compress!function!worked!on!my!computer.!Also,!it!was!very!interesting!to!know!how!this!code!was!used!in!the!fax!machine!to!remove!all!the!irregularities!and!to!compress!the!file!so!that!it!could!be!sent!over.!The!major!advantage!of!source!coding!is!that!it!reduces!the!amount!of!space!taken!up!by!the!information!and!decreases!the!amount!of!resources!used,!thus!increasing!the!efficiency.!Therefore,!with!an!increase!in!randomness!or!entropy!in!this!world!we!have!more!information!that!can!easily!be!stored!and!transmitted!using!source!coding.!!A!small!disadvantage!of!source!coding!can!be!that!when!the!information!is!decompressed,!it!will!take!the!same!amount!of!storage!and!thus!requiring!resources.!But!we!have!to!keep!in!mind!how!source!coding!can!be!used!to!send!multiple!files!at!a!single!time,!which!allows!the!efficient!utilization!of!resources.!According!to!me,!source!coding!makes!the!process!of!storage!and!transmission!very!efficient!and!reduces!the!amount!of!resources!used.!!According!to!the!lecture!notes,!channel!is!an!abstraction!of!anything!that!takes!in!and!outputs!information,!with!the!intention!of!keeping!the!information!unchanged.!This!channel!can!be!a!transmission!system!like!a!copper!wire,!radio!or!it!can!be!a!storage!device!like!a!CD!Rom.!But!in!every!channel!due!to!the!noise,!errors!will!occur.!Channel!coding!is!to!add!redundancy!to!information!so!that!
on!the!randomness!of!the!event!was!quite!astonishing.!The!next!part!of!our!discussion!was!related!to!source!coding.!Source!coding!is!the!science!of!representing!information!as!binary!numbers.!The!main!purpose!of!source!coding!is!data!compression.!It!amazed!me!how!this!coding!can!compress!such!large!amounts!of!information!into!binary!numbers!and!transmit!it.!After!this!I!understood!how!the!zip.!file/compress!function!worked!on!my!computer.!Also,!it!was!very!interesting!to!know!how!this!code!was!used!in!the!fax!machine!to!remove!all!the!irregularities!and!to!compress!the!file!so!that!it!could!be!sent!over.!The!major!advantage!of!source!coding!is!that!it!reduces!the!amount!of!space!taken!up!by!the!information!and!decreases!the!amount!of!resources!used,!thus!increasing!the!efficiency.!Therefore,!with!an!increase!in!randomness!or!entropy!in!this!world!we!have!more!information!that!can!easily!be!stored!and!transmitted!using!source!coding.!!A!small!disadvantage!of!source!coding!can!be!that!when!the!information!is!decompressed,!it!will!take!the!same!amount!of!storage!and!thus!requiring!resources.!But!we!have!to!keep!in!mind!how!source!coding!can!be!used!to!send!multiple!files!at!a!single!time,!which!allows!the!efficient!utilization!of!resources.!According!to!me,!source!coding!makes!the!process!of!storage!and!transmission!very!efficient!and!reduces!the!amount!of!resources!used.!!According!to!the!lecture!notes,!channel!is!an!abstraction!of!anything!that!takes!in!and!outputs!information,!with!the!intention!of!keeping!the!information!unchanged.!This!channel!can!be!a!transmission!system!like!a!copper!wire,!radio!or!it!can!be!a!storage!device!like!a!CD!Rom.!But!in!every!channel!due!to!the!noise,!errors!will!occur.!Channel!coding!is!to!add!redundancy!to!information!so!that!
on!the!randomness!of!the!event!was!quite!astonishing.!
The!next!part!of!our!discussion!was!related!to!source!coding.!Source!coding!is!the!
science!of!representing!information!as!binary!numbers.!The!main!purpose!of!
source!coding!is!data!compression.!It!amazed!me!how!this!coding!can!compress!
such!large!amounts!of!information!into!binary!numbers!and!transmit!it.!After!
this!I!understood!how!the!zip.!file/compress!function!worked!on!my!computer.!
Also,!it!was!very!interesting!to!know!how!this!code!was!used!in!the!fax!machine!
to!remove!all!the!irregularities!and!to!compress!the!file!so!that!it!could!be!sent!
over.!The!major!advantage!of!source!coding!is!that!it!reduces!the!amount!of!
space!taken!up!by!the!information!and!decreases!the!amount!of!resources!used,!
thus!increasing!the!efficiency.!Therefore,!with!an!increase!in!randomness!or!
entropy!in!this!world!we!have!more!information!that!can!easily!be!stored!and!
transmitted!using!source!coding.!!A!small!disadvantage!of!source!coding!can!be!
that!when!the!information!is!decompressed,!it!will!take!the!same!amount!of!
storage!and!thus!requiring!resources.!But!we!have!to!keep!in!mind!how!source!
coding!can!be!used!to!send!multiple!files!at!a!single!time,!which!allows!the!
efficient!utilization!of!resources.!According!to!me,!source!coding!makes!the!
process!of!storage!and!transmission!very!efficient!and!reduces!the!amount!of!
resources!used.!!
According!to!the!lecture!notes,!channel!is!an!abstraction!of!anything!that!takes!in!
and!outputs!information,!with!the!intention!of!keeping!the!information!
unchanged.!This!channel!can!be!a!transmission!system!like!a!copper!wire,!radio!
or!it!can!be!a!storage!device!like!a!CD!Rom.!But!in!every!channel!due!to!the!noise,!
errors!will!occur.!Channel!coding!is!to!add!redundancy!to!information!so!that!
Reflective)statement.)Week)3)
!
Entropy!is!the!measure!of!randomness!in!the!event.!However,!in!information!
theory,!entropy!is!the!average!amount!of!information!contained!in!each!message!
received.!It!is!very!interesting!to!know!that!amount!of!information!that!is!
received!will!depend!on!the!probability!of!the!appearance!of!the!required!
symbol.!!
Based!on!this,!we!can!find!a!relation!between!the!amount!of!information!and!the!
randomness!in!the!symbols.!If!the!randomness!in!the!symbols!will!increase!then!
the!amount!of!information!that!one!will!have!to!ask!to!predict!the!required!
symbol!will!also!be!more.!Thus!an!increase!in!randomness!or!entropy!will!result!
in!an!increase!in!the!amount!of!information!received.!!Therefore,!we!can!say!that!
randomness!and!the!amount!of!information!received!are!directly!proportional!to!
each!other.!According!to!Claude!Shannon,!an!event!will!have!the!greatest!entropy!
or!will!gather!the!maximum!information!when!all!possible!outcomes!are!equally!
likely.!!
For!example,!lets!take!the!weather!of!a!place!over!a!period!of!31!days.!If!the!
weather!is!random!and!unpredictable!then!we!will!have!to!store!the!entire!
information!or!we!will!have!to!ask!more!information!in!order!to!get!the!right!
result.!On!the!other!hand,!if!we!take!a!place!where!rain!is!rare,!we!would!need!
the!same!information!as!compared!to!the!previous!place!but!this!information!will!
be!compact!and!less!as!compared!to!above!because!the!chances!of!rain!are!less!
and!the!information!can!simply!be!put!into!a!pattern.!This!shows!how!
randomness!increases!the!amount!of!information!required!to!reach!a!particular!
symbol.!Getting!to!know!how!the!amount!of!information!received!was!dependent!
errors!can!either!be!detected!or!corrected.!Thus,!channel!coding!will!ensure!a!more!reliable!connection!between!the!transmitter!and!receiver!and!will!make!sure!that!the!right!information!is!transmitted.!There!are!two!types!of!channel!codes,!error!correction!code,!which!will!detect!the!errors!and!correct!them!and!error!detection!code,!which!will!discover!the!error!and!sends!a!resend!request!to!the!transmitter!to!transmit!another!wave.!These!two!are!generally!used!in!combination!to!get!the!best!results!out!of!channel!coding.!Examples!of!error!detection!codes!are!parity!check,!check!sum!etc.!These!are!used!in!various!places!like!credit!cards,!HKID,!ISBN!code!behind!the!book!etc.!The!last!number!of!the!10Qdigit!ISBN!code!behind!a!book!is!set!in!such!a!manner,!that!after!following!certain!calculations!the!sum!is!a!multiple!of!11.!If!this!not!the!case!then!there!is!an!error!in!the!code.!This!will!only!DETECT!the!error!and!will!not!be!able!to!correct!it.!!On!the!other!hand,!some!examples!of!the!error!correction!code!are!forward!error!correction!(FEC),!hamming!code!etc.!This!can!be!seen!in!CD!Roms,!hard!drive!etc.!Now!days!we!have!stopped!using!CDs!but!we!all!have!gone!through!a!phase!when!we!got!our!favorite!music!onto!a!CD!and!heard!it!through!a!Walkman.!Sometimes,!these!CDs!got!some!scratches!on!it!and!still!worked,!whereas!sometime!they!stopped!working.!!I!was!shocked!to!know!that!this!is!because!of!the!error!correction!code!used!in!CDs.!When!there!are!a!few!scratches!on!the!CD,!it!means!that!there!is!information!missing!on!the!CD!due!to!these!scratches.!But!it!still!works!as!it!gets!the!information!using!the!channel!codes!embedded!in!it.!The!channel!code!detects!what!information!is!missing!and!completes!it!in!order!to!make!the!CD!work.!But!when!there!are!too!many!scratches!on!the!CD,!a!lot!of!information!is!missing!and!the!channel!code!is!not!able!to!retrieve!the!information!causing!the!CD!to!dysfunction.!!So!as!the!number!of!scratches!will!increase!the!channel!code!will!be!able!to!detect!the!error!but!wont!be!able!to!correct!the!same!causing!the!CD!not!to!work.!!Another!important!thing!that!I!learned!during!the!discussion!with!the!professor!was!that!the!channel!capacity!plays!a!very!important!role!in!keeping!the!information!together.!If!the!information!sent!through!a!channel!will!exceed!its!capacity!then!the!information!that!is!being!transmitted!will!be!lost.!This!is!where!source!coding!can!be!used!to!reduce!the!size!of!the!information!and!transmit!it!easily!without!losing!the!information.!!When!I!look!back!before!this!class!and!think!about!these!basic!things,!I!realize!how!technology!has!advanced!over!the!period!and!how!it!is!being!efficiently!used!in!every!possible!manner.!There!is!still!a!long!way!to!go!but!what!we!have!achieved!is!remarkable.!I!was!always!confused!about!how!a!CD!can!work!if!it!has!scratches!on!it.!Channel!coding!was!the!answer!I!was!looking!for.!I!always!imagined!how!my!dad!got!information!from!across!the!world!through!a!fax!machine.!The!answer!was!source!coding.!When!I!think!about!all!these!things!and!look!around,!it!makes!me!realize!why!the!21st!century!is!called!the!age!of!technology.!Learning!all!this!has!a!very!positive!effect!on!me!as!it!indulges!me!to!find!out!more!about!how!technology!has!advanced!and!how!it!is!changing!basic!things.!Even!though,!we!have!reached!a!remarkable!level,!according!to!me,!we!have!a!long!way!to!go.!We!might!reach!a!point!where!all!of!the!above!gets!obsolete!and!we!start!depending!on!some!new!technology.!We!now!dont!use!CDs!as!we!have!everything!available!over!the!net.!We!now!have!pen!drives!and!
errors!can!either!be!detected!or!corrected.!Thus,!channel!coding!will!ensure!a!more!reliable!connection!between!the!transmitter!and!receiver!and!will!make!sure!that!the!right!information!is!transmitted.!There!are!two!types!of!channel!codes,!error!correction!code,!which!will!detect!the!errors!and!correct!them!and!error!detection!code,!which!will!discover!the!error!and!sends!a!resend!request!to!the!transmitter!to!transmit!another!wave.!These!two!are!generally!used!in!combination!to!get!the!best!results!out!of!channel!coding.!Examples!of!error!detection!codes!are!parity!check,!check!sum!etc.!These!are!used!in!various!places!like!credit!cards,!HKID,!ISBN!code!behind!the!book!etc.!The!last!number!of!the!10Qdigit!ISBN!code!behind!a!book!is!set!in!such!a!manner,!that!after!following!certain!calculations!the!sum!is!a!multiple!of!11.!If!this!not!the!case!then!there!is!an!error!in!the!code.!This!will!only!DETECT!the!error!and!will!not!be!able!to!correct!it.!!On!the!other!hand,!some!examples!of!the!error!correction!code!are!forward!error!correction!(FEC),!hamming!code!etc.!This!can!be!seen!in!CD!Roms,!hard!drive!etc.!Now!days!we!have!stopped!using!CDs!but!we!all!have!gone!through!a!phase!when!we!got!our!favorite!music!onto!a!CD!and!heard!it!through!a!Walkman.!Sometimes,!these!CDs!got!some!scratches!on!it!and!still!worked,!whereas!sometime!they!stopped!working.!!I!was!shocked!to!know!that!this!is!because!of!the!error!correction!code!used!in!CDs.!When!there!are!a!few!scratches!on!the!CD,!it!means!that!there!is!information!missing!on!the!CD!due!to!these!scratches.!But!it!still!works!as!it!gets!the!information!using!the!channel!codes!embedded!in!it.!The!channel!code!detects!what!information!is!missing!and!completes!it!in!order!to!make!the!CD!work.!But!when!there!are!too!many!scratches!on!the!CD,!a!lot!of!information!is!missing!and!the!channel!code!is!not!able!to!retrieve!the!information!causing!the!CD!to!dysfunction.!!So!as!the!number!of!scratches!will!increase!the!channel!code!will!be!able!to!detect!the!error!but!wont!be!able!to!correct!the!same!causing!the!CD!not!to!work.!!Another!important!thing!that!I!learned!during!the!discussion!with!the!professor!was!that!the!channel!capacity!plays!a!very!important!role!in!keeping!the!information!together.!If!the!information!sent!through!a!channel!will!exceed!its!capacity!then!the!information!that!is!being!transmitted!will!be!lost.!This!is!where!source!coding!can!be!used!to!reduce!the!size!of!the!information!and!transmit!it!easily!without!losing!the!information.!!When!I!look!back!before!this!class!and!think!about!these!basic!things,!I!realize!how!technology!has!advanced!over!the!period!and!how!it!is!being!efficiently!used!in!every!possible!manner.!There!is!still!a!long!way!to!go!but!what!we!have!achieved!is!remarkable.!I!was!always!confused!about!how!a!CD!can!work!if!it!has!scratches!on!it.!Channel!coding!was!the!answer!I!was!looking!for.!I!always!imagined!how!my!dad!got!information!from!across!the!world!through!a!fax!machine.!The!answer!was!source!coding.!When!I!think!about!all!these!things!and!look!around,!it!makes!me!realize!why!the!21st!century!is!called!the!age!of!technology.!Learning!all!this!has!a!very!positive!effect!on!me!as!it!indulges!me!to!find!out!more!about!how!technology!has!advanced!and!how!it!is!changing!basic!things.!Even!though,!we!have!reached!a!remarkable!level,!according!to!me,!we!have!a!long!way!to!go.!We!might!reach!a!point!where!all!of!the!above!gets!obsolete!and!we!start!depending!on!some!new!technology.!We!now!dont!use!CDs!as!we!have!everything!available!over!the!net.!We!now!have!pen!drives!and!
errors!can!either!be!detected!or!corrected.!Thus,!channel!coding!will!ensure!a!
more!reliable!connection!between!the!transmitter!and!receiver!and!will!make!
sure!that!the!right!information!is!transmitted.!There!are!two!types!of!channel!
codes,!error!correction!code,!which!will!detect!the!errors!and!correct!them!and!
error!detection!code,!which!will!discover!the!error!and!sends!a!resend!request!to!
the!transmitter!to!transmit!another!wave.!These!two!are!generally!used!in!
combination!to!get!the!best!results!out!of!channel!coding.!Examples!of!error!
detection!codes!are!parity!check,!check!sum!etc.!These!are!used!in!various!places!
like!credit!cards,!HKID,!ISBN!code!behind!the!book!etc.!The!last!number!of!the!
10Qdigit!ISBN!code!behind!a!book!is!set!in!such!a!manner,!that!after!following!
certain!calculations!the!sum!is!a!multiple!of!11.!If!this!not!the!case!then!there!is!
an!error!in!the!code.!This!will!only!DETECT!the!error!and!will!not!be!able!to!
correct!it.!!
On!the!other!hand,!some!examples!of!the!error!correction!code!are!forward!error!
correction!(FEC),!hamming!code!etc.!This!can!be!seen!in!CD!Roms,!hard!drive!
etc.!Now!days!we!have!stopped!using!CDs!but!we!all!have!gone!through!a!phase!
when!we!got!our!favorite!music!onto!a!CD!and!heard!it!through!a!Walkman.!
Sometimes,!these!CDs!got!some!scratches!on!it!and!still!worked,!whereas!
sometime!they!stopped!working.!!I!was!shocked!to!know!that!this!is!because!of!
the!error!correction!code!used!in!CDs.!When!there!are!a!few!scratches!on!the!CD,!
it!means!that!there!is!information!missing!on!the!CD!due!to!these!scratches.!But!
it!still!works!as!it!gets!the!information!using!the!channel!codes!embedded!in!it.!
The!channel!code!detects!what!information!is!missing!and!completes!it!in!order!
to!make!the!CD!work.!But!when!there!are!too!many!scratches!on!the!CD,!a!lot!of!
information!is!missing!and!the!channel!code!is!not!able!to!retrieve!the!
information!causing!the!CD!to!dysfunction.!!So!as!the!number!of!scratches!will!
increase!the!channel!code!will!be!able!to!detect!the!error!but!wont!be!able!to!
correct!the!same!causing!the!CD!not!to!work.!!
Another!important!thing!that!I!learned!during!the!discussion!with!the!professor!
was!that!the!channel!capacity!plays!a!very!important!role!in!keeping!the!
information!together.!If!the!information!sent!through!a!channel!will!exceed!its!
capacity!then!the!information!that!is!being!transmitted!will!be!lost.!This!is!where!
source!coding!can!be!used!to!reduce!the!size!of!the!information!and!transmit!it!
easily!without!losing!the!information.!!
When!I!look!back!before!this!class!and!think!about!these!basic!things,!I!realize!
how!technology!has!advanced!over!the!period!and!how!it!is!being!efficiently!used!
in!every!possible!manner.!There!is!still!a!long!way!to!go!but!what!we!have!
achieved!is!remarkable.!I!was!always!confused!about!how!a!CD!can!work!if!it!has!
scratches!on!it.!Channel!coding!was!the!answer!I!was!looking!for.!I!always!
imagined!how!my!dad!got!information!from!across!the!world!through!a!fax!
machine.!The!answer!was!source!coding.!When!I!think!about!all!these!things!and!
look!around,!it!makes!me!realize!why!the!21st!century!is!called!the!age!of!
technology.!Learning!all!this!has!a!very!positive!effect!on!me!as!it!indulges!me!to!
find!out!more!about!how!technology!has!advanced!and!how!it!is!changing!basic!
things.!Even!though,!we!have!reached!a!remarkable!level,!according!to!me,!we!
have!a!long!way!to!go.!We!might!reach!a!point!where!all!of!the!above!gets!
obsolete!and!we!start!depending!on!some!new!technology.!We!now!dont!use!
CDs!as!we!have!everything!available!over!the!net.!We!now!have!pen!drives!and!
SD!cards!to!cover!the!need!for!CDs.!I!think!that!there!are!more!changes!waiting!for!us!ahead!which!makes!me!more!inquisitive.!!!!!!References:!!!1- Lecture notes on information theory 2- http://en.wikipedia.org/wiki/Entropy_(information_theory) 3- https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/information-entropy 4- http://users.df.uba.ar/giribet/f4/computation.pdf 5- http://www.wisegeek.com/what-is-channel-coding.htm 6- http://en.wikipedia.org/wiki/Error_detection_and_correction#Implementation 7- https://www.youtube.com/watch?v=sMb00lz-IfE 8- https://www.youtube.com/watch?v=sPFWfAxIiwg
SD!cards!to!cover!the!need!for!CDs.!I!think!that!there!are!more!changes!waiting!for!us!ahead!which!makes!me!more!inquisitive.!!!!!!References:!!!1- Lecture notes on information theory 2- http://en.wikipedia.org/wiki/Entropy_(information_theory) 3- https://www.khanacademy.org/computing/computer-science/informationtheory/moderninfotheory/v/information-entropy 4- http://users.df.uba.ar/giribet/f4/computation.pdf 5- http://www.wisegeek.com/what-is-channel-coding.htm 6- http://en.wikipedia.org/wiki/Error_detection_and_correction#Implementation 7- https://www.youtube.com/watch?v=sMb00lz-IfE 8- https://www.youtube.com/watch?v=sPFWfAxIiwg
SD!cards!to!cover!the!need!for!CDs.!I!think!that!there!are!more!changes!waiting!
for!us!ahead!which!makes!me!more!inquisitive.!!
!
!
!
!
References:!!
!
1- Lecture notes on information theory
2- http://en.wikipedia.org/wiki/Entropy_(information_theory)
3- https://www.khanacademy.org/computing/computer-
science/informationtheory/moderninfotheory/v/information-entropy