Explain the season one climax (technical) of Silicon Valley

You're really asking three sub-questions here, so to take each on their own:

During the TechCrunch event what kind of modifications the protagonist made to his compressing software Pied Piper which helped his team to win the event?

The two types of compression Richard describes on show as being currently used are Shannon codes and Huffman codes. To quote from an article discussing the show's compression software:

Shannon codes model data in a treelike structure, working from the roots to the leaves. Huffman codes, used today in media files like MP3 and JPEG, perform better by working from leaves to roots...

This is a really basic overview of how both types of compression works - imagine one starting from the top and working down, and the other starting from the bottom and working up. Richard's algorithm is supposed to start from the middle and effectively bounce outwards to find any hidden structure and compress it at the same time.

No such algorithm actually exists, but it's certainly possible it could do in the future.

I'd suggest reading the article, as it contains lots of information from Vinith Misra, the PhD student who was asked to effectively flesh out the details of the fictional algorithm by Tsachy Weissman, a Stanford professor who was approached by the HBO show creators to tackle the problem.

Why the protagonist initially become gloomy while compressing a 3D data file which size was greater than 100 GB (from judges) to mere 20 GB?

When Richard did his own tests, he recorded a Weissman score (another fictional score) of 3.8. When he performs the compression in front of a live audience, the final file size (slightly bigger than 20GB) is much smaller than it should be if his score of 3.8 is correct. This suggests one thing - the compression algorithm skipped some of the data and thus isn't working properly, meaning he's just presented a false algorithm to an entire room of people.

Why he became happy when the software showed no error?

Once no errors showed, he realised his algorithm had worked perfectly and was in fact running even better than his initial test suggested. Instead of a score of 3.8, beating the previous score of 2.9, he recorded a 5.2 - obliterating the previous score. No errors basically means that the compressed file had everything that was supposed to be there, i.e. nothing had been skipped or missed. In other words, his algorithm worked perfectly and he won the day. Hurrah!