Shoutbox

Log Viewer is slow with large chat logs - Printable Version

-Shoutbox (https://shoutbox.menthix.net)
+-- Forum: MsgHelp Archive (/forumdisplay.php?fid=58)
+--- Forum: Messenger Plus! for Live Messenger (/forumdisplay.php?fid=4)
+---- Forum: WLM Plus! General (/forumdisplay.php?fid=23)
+----- Thread: Log Viewer is slow with large chat logs (/showthread.php?tid=81528)

Log Viewer is slow with large chat logs by garreh on 02-08-2008 at 10:05 PM

Decrypting a log file which is 2.75MB takes an agonising 26 seconds. It's the 8th -- and I dare think what the log file size will be by the end of the month, and how long the LogViewer will take to show me the log!

Is there anyway chance of improving the speed Patchou?


RE: Log Viewer is slow with large chat logs by Patchou on 02-08-2008 at 10:11 PM

well, it shouldn't take that long... it depends on your system specs really. All I can suggest is to decrease the encryption key size (preferences), that's what will make the most difference (for future generated log files).


RE: Log Viewer is slow with large chat logs by garreh on 02-08-2008 at 10:24 PM

Thanks for your reply Patchou.

My system spec isn't too flashy... AMD AthlonXP @ 2.21GHz.

Encryption key size is 640.

Does the Log Viewer decrypt and write to the HTML file as its going along? Or does it decrypt, then generate a HTML file?


RE: Log Viewer is slow with large chat logs by Matti on 02-09-2008 at 09:27 AM

Basically, encrypted log files are HTML files which were encrypted by Plus! Live. When Plus! opens the log file, it checks the given password by decrypting a known signature in the log file and if that result matches the original signature, it decrypts the whole log file and stores it in a temporary location. Then, it loads that file in the Log Viewer window. In fact, you can encrypt any type of file with the Plus! encryption, and it'll work perfectly fine.

In my opinion, 2.75 MB for a log file is quite huge, assuming that they're just HTML files. HTML is just text, the images (smilies, wink thumbnails and the like) are stored in a subdirectory in the logs folder. So, I think that 2.75 MB of logged chat messages is quite big for a period of 8 days, and I doubt you have lots of other huge log files. Maybe you're just chatting too much? :grin:


RE: Log Viewer is slow with large chat logs by garreh on 02-09-2008 at 02:52 PM

Thanks for your reply Mattike.

Yes maybe I should tell her to
stop
writing
like
this...

And spamming me with nudges :P

But on a serious note.

Is there any chance of future versions of Messenger Plus! to incorporate 7zip to --

Output: Generate HTML File --> 7zip (via command line, compress and output to file) --> Encrypt log --> Output to .ple
Input: Decrypt log --> 7zip decompress and output to HTML file --> Show log in viewer

Compressing a 2.75MB HTML file with 7zip outputs, approximately, a 114KB file.

Imagine the speed difference in encrypting and decrypting a 114KB file over a 2.75MB file!

As quoted from http://www.7-zip.org/license.txt

quote:
Note:
    You can use 7-Zip on any computer, including a computer in a commercial
    organization. You don't need to register or pay for 7-Zip.


RE: Log Viewer is slow with large chat logs by Patchou on 02-13-2008 at 06:29 PM

Ziping before encrypting could be a good idea, the problem however is that logs, by nature, are ever growing files. Compressing each block in the file could be prone to error and cause many differnt kinds of problems. Still, I'll keep that in mind.


RE: Log Viewer is slow with large chat logs by garreh on 02-13-2008 at 10:06 PM

Yes, Patchou -- This is the problem with zipping. However, even now, without much change, filesizes could be reduced by over half in most cases...

code:
  <tr><th><span class="time">(16:06)</span> Gary: </th><td style="font-family:&quot;Georgia&quot;;color:#000080;font-weight:bold;">almost got parking ticket</td></tr>
166 bytes


This line could be reduced to simply...

code:
<tr><th><i>(16:06)</i> Gary: </th><td class="a">almost got parking ticket</td></tr>
83 bytes -- 50% Reduction

So what did I do?

1: Replaced <span class="time"> to <i> and also replaced the CSS entry     .mplsession table th span.time {color:#949494;font-weight:normal;} with...

code:
    .mplsession table tr i {font-style:normal;color:#949494;font-weight:normal;}

2: Replaced style="X" with class="a" ... How this works is by adding an entry to CSS like so...

code:
    td.a { font-family:Georgia;color:#000080;font-weight:bold; }
    td.b { font-family:"Microsoft Sans Serif";color:#FF00FF; }

a relates to one persons colour, and b for another, more can be added obviously like c, d, e, f, .... you get the picture.

A simple function could be made AddCssColour(<css code>) and simply use base 26 to increment. Then GetCurrentCssColour(<contact id>) to get the CSS class name.

I also removed trailing spaces and newlines, which removes a further 4 bytes for every line, which would amount to 23KB of pure junk over 6000 lines, why not?

-------------

Using these simple changes:

Original filesize: 2.13MB
New, reduced filesize: 1.16MB - 54% reduction!

Not only would files load twice as quick, it would reduce amount of data in memory, amount being written to disk and increase encrypting and decrypting speed.

These changes wouldn't affect old log files, but I think could easily be incorporated to improve future versions of Plus!
RE: Log Viewer is slow with large chat logs by Patchou on 02-13-2008 at 10:52 PM

you just can't add this kind of thing in html logs... remember that log files grow, the header, once written in the file, cannot be changed.


RE: Log Viewer is slow with large chat logs by ahmetgns on 02-29-2008 at 12:01 AM

By the way, I found an annoying thing which may grow html files rapidly.

When people use Plus! formatting tags, if they write like [b]hel[/b][b]lo[/b], then in log file it looks like

code:
<span font-weight:bold>hel</span><<span font-weight:bold>lo</span>
As you see, there are two bold information when not needed. I know, noone inserts tags like that. However the important thing is if user give a gradient effect to his messages with other formatting like bold or italic, then these bold and italic informations are written in html files for each letter.For example, if user sends a message like[b][c=4]Hello my contact[/c=8][/b] then the result in html file is
code:
<span font-weight:bold; color=fffff>   H   </span>
<span font-weight:bold; color=ffff00>   e   </span>
...
..
.
<span font-weight:bold; color=fcfc32>   t   </span>
This problems arises in nicknames mostly, because people can give formats other than gradient effect to their messages from original Messenger's font option (and this will not cause this duplicate entry problem for each letter) but for display names if people make their name bold&italic&underlined and give a gradient color, then for each letter, the bold&italic&underlined infos are written seperately. And this is unnecessary I think. If possible, please optimize logging of texts formatted like that.