Latest Blog Entries on ChaosCoding

Digg and Facebook Partner - One Step Closer to a Semantic Web

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Two web giants, Digg.com and Facebook have announced a partnership that will hopefully set a precedent for other websites to follow. Users of Facebook can login to Digg with their Facebook username and password combinations and use Diggs features - the stories they digg will even be published to their Facebook mini-feeds. The full write-up can be found over on the digg blog.

The seamless connectivity of the two websites is a move in the right direction for social networks, the more semantic they become, the better. Perhaps in the future OpenID will catch on more and all accounts across the internet will become seamless.

Web API Authentication

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

This problem has been bugging me for the past few weeks - should web application APIs, such as the ones that Twitter, Pownce, Digg, etc. employ, require the programmer to authenticate by some means, or should it simply be open access? There are very good arguments for both, but the choice lies within the type of API it is. If the API provides sensitive data, such as the Twitter API, some sort of authentication should probably be used. If the data is public anyway, such as Digg, then no authentication is needed.

No Authentication: Easiest on the programmer side, hardest on the service that is offering the API. It allows those using the API to dive right in - no need to sign up for a project id or a username and password combination to authenticate with. The programmer can start playing right away; however, the downside is that the web service offering the API find it much harder, if not impossible, to track any API abuse. The service provider must weigh the benefit of allowing programmers to dive head first into the API with how crucial tracking is.

Username/Password Authentication: The opposite of open access is of course closed access - the programmer must provide a username or password to access any data from the API. Now the web service can easily track who and when a program/script requested information from the API; though, the downside is that the programmer cannot start playing with the API immediately.

APIs that provide the programmer with sensitive data should use some sort of authentication method. For example, using the Twitter API a programmer can request information from a "timeline" of user activity - the programmer then needs to supply authentication for that user. That way data sensitivity is not compromised.

Digg however employs a method somewhere in between these two methods - a username and password is not required but an application ID is. When requesting data from the Digg API the programmer must provide an application ID that he created. This proves to be a happy median because the programmer can start playing with the API right away and Digg can also track when the application ID is used, the only problem arising when a two programmers choose the same application ID for their project.

Once again, the type of method is dependent on the project. If sensitive data is being requested, of course authentication is needed. If the data is publicly viewable to begin with then no authentication is needed, but you might want to consider requiring an application ID so the program/script can be tracked.

Unix Timestamp In Javascript

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Using a Unix time stamp, or POSIX time stamp is a quick and fast way to represent a point in them. A Unix time stamp returns the number of seconds since the Unix epoch, January 1, 1970; however, it does not count leap seconds.

To create a Unix time stamp function exactly like the one in PHP, you first have to realize that JavaScript calculates to the millisecond instead of just the second. Obviously, you can either divide by 1000 or take the first ten characters of the substring.

The following function excepts an optional date object as a parameter and returns a Unix time stamp equivalent to the one PHPs time() function would return:

1.function getUnixTimestamp(timeObj)
2. {
3.  var dateObj = null;
4.
5.  if (timeObj != null)
6.   {
7.    dateObj = new Date(timeObj);
8.   }
9.
10.  else
11.   {
12.    dateObj = new Date();
13.   }
14.
15.  return (parseInt(dateObj.getTime().toString().substring(0, 10)));
16.  // alternatively, the following can be used:
17.  // return (parseInt(dateObj.getTime() / 1000));
18. }

This function, while extremely basic, will prove to be useful in the next blog post where I will show you how to calculate relative times similar to Twitter and Digg. ex: "1 day 23 hours ago" or "10 minutes 3 seconds ago".

Digg Algorithm vs. RIMA (My) Algorithm

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Re-post from DiggMatch

Digg released its recommended user/story algorithm last week - this algorithm had taken them over a year and a half to develop and I must say, I am very, very disappointed. As a social news website first, and a social networking website second, the focus of digg should always be on the stories. Sure, it is nice to find users that share my digging tastes, just like it is nice to find people to share my listening tastes over at last.fm, but I do not go to digg to meet new friends - I go there for the content.

And as for the amount of recommended stories, digg offers me three to look at. Three stories. Thats it. Out of the thousands and thousands of submissions per day, there are only three that match my digging tastes. Once again, I stress the point that while social networking is nice, social bookmarking is what digg.com is all about.

The difference between the digg recommended story/user algorithm and the RIMA powered algorithm is simple:

Digg - Long Term Relationship: Long term relationships are hit or miss. Sure, you may be happy for the rest of your life, waking up next to the same algorithm everyday, eating the same thing for breakfast, and driving your matching Prius to work. But there is an equal chance that the relationship could end badly. You could catch the algorithm in bed with another byte of code - the entire relationship, built up to one tragic end. Using the social networking road builds a list of friends with (somewhat) similar digging tastes, which in turn gives you a posse of diggers to promote any story your submit. It may work, or it may not. It is hit or miss for every user.

RIMA - One Night Stand: Couples fight, couples bicker, and couples who have been married for a long time never have sex. What is the fun in that? Sometimes I just want to get my digging rocks off to a story that interests me - I do not have the time to meet a bunch of friends and "network" myself around. I go to digg for the stories and thats what I want recommended to me first, users second. In the end I may not have as many friends as I want to, but at least I will be satisfied.

Which method works best for you? Let me know, I am curious.

RIMA - Rosebrocks Index-Match Algorithm

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

My pride and joy - well so far at least. I finished up a specification on the algorithm I used to write DiggMatch. The algorithm is used for information retrieval, specifically taking a set of data, indexing it, then comparing other entities of data against the indexed set to see how related the entity and the table are too each other.

For example, say you were to visit the New York Times website every day for a week and save the text from stories that interested you. The indexing portion of RIMA could be applied to the saved text, creating a table of data. Then for every subsequent day, any given story on the NYT website could be compared to the table using the match portion of RIMA. A score would be returned representing the relevancy between the story and the table - the higher the score, the higher the relation. Using RIMA, a programmer can create a "recommended set of data/text" for a website, an essential part of any new web application.

RIMA can be downloaded in pdf format or viewed online in html.

WebComic Law #4: Let Your Users Have A Voice

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

So you have followed the rules so far - the comic is above the fold as much as possible, the dialogue of the comic is stored in a database so users can search for comics, and you have formatted the URLs cleanly and efficiently so search engines such as Google and Yahoo can index your website better. All this improves the usability of the website but has nothing to do with user involvement - a key component of any website looking to ensure returning visitors.

MegaTokyo is a webcomic that seems to do everything right; however, simultaneously does everything completely wrong. Fred Gallagher has the comic at the top of the page, a search featured implemented, and the correct formating of the comic URL along with a descriptive TITLE HTML element - and thats it. There is no social interaction. Sure, one can argue that MegaTokyo, along with many other popular webcomics have forums where users can converse, but that feels almost like a slap in the face. Why can I not post my opinion and impressions directly on the comic page? Are my thoughts and comments not worthy enough to be featured with the glorious and illustrious art of Fred Gallagher? Can I not craft poetic language that forms images in the readers heads more vivid than any painting or drawing could ever hope to do? Probably not. And even if I could, there is no guarantee that the vast amount of internet users could either. Some users may post non-sensical comments, others may provide insight, while many will criticize the piece. Overall, the problem resides with the artist - can he sacrifice the potential defilement in presentation of his artwork for more user interaction?

Hopefully the answer is yes. As long as the comments are kept below the comic, the reader is forced to form his own opinion by viewing the comic first, prior to reading any other user comments; that way any notions of impressionability are dismissed.

WebComic Law #4: Let Your Users Have A Voice: User interaction is key to building an online community. Sure forums are great, but they are also very detached from the original content on the website. Let users comment on your comic, similar to how a user comments on a blog post; this gives them incentive to come back and read other users replies. Plus, it helps them feel closer to the creator of it all, you, the artist.

Fred Gallagher could learn a lot from this conjecture - while his artwork is remarkably stunning and splendid, the website as a whole feels phlegmatic and sterile. Users have no way to interact with him and his artwork, forcing them to take a back seat to the webcomic ride.

Think of it this way: first there were books, words on pages that told a story. Then came movies, moving pictures and sound that told a story linearly. Finally, video games were released. A gamer was now part of the movie, giving him control of movie and altering altering the story line. The same can be applied to the evolution of the web. First there was simple text. Then came images and movies. Now we are seeing a conglomeration of them all with an added twist - user interaction. The roles of the user and website administrator have switched. Users are now creating the content, promoting it, and deeming whether it is important or not. Administrators provide the service to do so and enjoy the ride, an online community at its best. There is no reason why webcomics cannot become social as well.

Law #1: WebComic Law #1: Front and Center
Law #2: WebComic Law #2: Build Bridges, Not Mountains
Law #3: WebComic Law #3: You Name the Comic for a Reason

LinkShare.com is a Stain on the Internet

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Without a doubt, LinkShare.com is a large, monumental stain on the internet. One that has set in and left an unsightly shade of murky yellow on my new beige carpet. If only Resolve or Spot Shot made a product to remove websites from the internet. A quick spray, wait ten seconds, and wipe clean - good as new. Unfortunately, without such a mythical product, I am left to writing a blog entry detailing my immense aversion, animosity, and antipathy to this company.

But before I begin barraging the war zone, I suppose a little backstory is in order. For those of you who do not know, I am the head developer/designer of norodo.com. Norodo is an affiliate based online mall where users can quickly and easily find stores that interest them. To populate the website with stores, LinkShare was used as a service to get in contact with companies such as Wal-Mart, Apple, NetFlix, etc. If a user were to click on a link to a store from Norodo and buy an item, part of the sale would go to LinkShare and another part to me. To create links to the stores, all I had to do was login into my LinkShare account, generate a URL that would track where the user was coming from and insert it into the Norodo database. Simple and easy.

Or so I thought. Three weeks ago I received a plethora of emails from my affiliates that all stated the same thing - there is a problem on the LinkShare side and not all the click-throughs (when a user clicks on a store link) are being tracked; therefore, not all the sales are being recorded as belonging to your website; however, a fix was set to go in place on the 19th of June and all clock-throughs would be tracked perfectly thereafter.

To start the general ambiguity of the email startled me at first. What do you mean "not all"? How much is "not all"? A handful? A smattering? All of them? After looking up the number of click-throughs norodo recorded and the number of click-throughs LinkShare recorded I deduced that a staggering 75% of them were not being tracked. Any sales that were made in that lost 75% Norodo earns no profit from, and to only add more fervor to the matter, since LinkShare does not know who to pay for the sale, they just get to keep it all. Great. Now they are getting rewarded for hiring completely incompetent programmers.

Robert, the founder of Norodo, and I decided to work LinkShare from two angles. I would try their email tech support while he wrang the necks of the customer support of the phone. I went the normal route, starting at their website and filling out a request, and in quite detail I might add, for technical support. A few minutes later I checked my inbox and found an email stating to check the FAQ. I was not really surprised. Most companies go the automated response route first. I replied to the email, telling them that the FAQ was of no relevance and to please check my help for submission. They replied stating they had no record of my help form. Apparently, their help form does not record the data to the database. It simply generates a FAQ email. Obviously vexed, I re-typed my question, asking why the click-throughs were not being tracked and how the June 19th fix was going to mend the problem. All I got back was an email stating to copy and paste my HTML so they could check the validity of the store links - they did not assess my question at all, and I sure as hell told them so. Again, I received an email telling me to copy and paste the store links. Remember at the end of Shaun of the Dead the zombies were used to do repetitive and ultimately meaningless tasks? I suppose LinkShare hired some of them.

Roberts story is equally troubling. He called the helpline and the customer service person was even less adept than the zombies. He thought the patch was supposed to go live on July 19th, not June 19th. Strike one. Robert then asked what the problem was with the link tracking - a consistent flow of "umms" and "uhhs" followed. Strike two. When asked how to do we fix the problem, the man replied you will have to regenerate all the links for your affiliates. All 240 stores. LinkShare screws up, takes whatever profit would have been made in that 75% and then informs me that I have to re-generate 240 links so that they will be tracked correctly? I think the use of the words "bend over" are quite legitimate.

And who is to say that this "problem" will not happen again in the next six months? Will the LinkShare programmers find another bug in the code rendering the click-throughs untracked? I would not be surprised at all if LinkShare ended up being one very elaborate scam.

Finally, to the LinkShare company: I am thoroughly disgusted with my experience with you. Your service is questionable, your customer service appalling and your programmers incompetent. I may not be able to wipe your stain off the internet, but I sure can wipe it from my website.

My Web Design/Development Portfolio

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Earlier this morning, I finished up my web design and development portfolio. At the very least, it gives a snapshot of what I have done thus far, and if you are interested in hiring me for a job, it goes into detail about my expertise and skills as well.

Fear Hillary - It Goes Both Ways

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

"Fear Hillary". These two words stared back at me from the rear end of a baby blue Honda Accord at an interminably long red light last Tuesday morning. What did it mean? Confused, I scratched my heading, pondering what stance the driver of the Accord trying to take. Did the driver deem Hillary to be a socialist left wing radical, "feared" as a threat to the American people? Or did this person revere Hillary as an illustrious, glorious savior, in which case the word "fear" would be used to strike anxiety and apprehension into the hearts her opponents?

Analyzing the sticker itself, the background color is a navy blue, the color representing the Democratic party. The bold font, while slightly crunched together shows no jagged edges, no offensive stance would prove to be consistent with an outwards attack - they seem to contradict each other.

In this case, I would say the font trumps the color and the sticker is stating that Hillary should be seen as a threat and a menace, but there is no way to be sure. In politics everything is about appearance - the candidates image has to be absolutely pristine. Why bother with ambiguities? Graphic designers have a tremendous responsibility to get their messages across, thats why true artists adhere to the old adage "a picture paints a thousand words". This designer has painted a graphic worth a thousand words; unfortunately, all one thousand of these words form sentences left with question marks.

Web 2.0 Go Away - True Innovation

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Thats it. It has been too long. I am coming clean - I have an addiction, one that has been consuming my life for years. Night after night I lay awake, craving, yearning, every single impulse of my body irrevocably and immutably lamenting for my keyboard. To design, to code, to take a single idea and build it, mold it, shape it into the next greatest web application. To me, this addiction is the greatest thing that could ever happen to a coder/designer.

While some may call this disease a vice, I call it a remedy for modern design. The internet is teeming with buzzwords such as, "web 2.0", "semantic web", "AJAX", "Ruby on Rails", and "blogosphere". Practically every designer is hopping on the bandwagon of this Web 2.0 phase calling themselves innovative and original; however, in reality they are degrading the very art of web design. True innovation takes an already established concept and introduces new methods or ideas to accomplish the task. Web 2.0 concepts are innovative, but only when compared to the lack of social networking in the Web 1.0 era. Likewise, Web 2.0 design is innovative as well, but only when compared to the lack of browser friendly styling or clean-cut templating.

If a designer/coder truly wants to call himself/herself innovative, they must first understand the inner workings of Web 2.0, take this understanding, and craft something undeniably and inexorably different. For instance, aspiring artists may travel to Venice, Italy to study the work of the great renaissance painters. A writer may venture to England and study the works of Shakespeare among the worlds most elite writers. The same applies to designers and programmers.

To fathom true innovation, first, study the greats, the distinguished and illustrious. Take the understanding gained from them and apply it to your own ideas - even if your ideas clash and differ. True innovation does not allow for standing on the shoulders of giants, only becoming a giant yourself.

Who knows how long Web 2.0 will stick around for. It could only be another couple of months. More likely another few years. Perhaps even a decade. But eventually it will be unthroned, a new era of design will reign. Will you be the one to start this new, true period of innovation?

Digg - Still No Recommended Story Feature

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

I was browsing digg a few minutes ago and came across a story where Kevin Rose and Jay Adelson were discussing the The Future of Digg. When asked what new features digg will be putting out, Kevin replied by describing an algorithm used to recommend stories and users with similar digging tastes. My question to digg: why is this taking so long to do? I have had an algorithm up for over a month for recommended digg stories. Granted digg has much more to implement, but the premise remains the same - it has been entirely too long.

WebComic Law #3: You Name the Comic for a Reason

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

This week, instead of lambasting a webomic for missing some essential feature, I am going to commend PvPonline for their superlative use of the TITLE HTML tag and URL formation.

When webcomics first started long ago, it was considered normal to use a .html page for each and every comic. Upon the advent of dynamic web coding practices, languages like PHP, Perl, and Python were used to manage large comic websites, but left the query part of a URL ugly and unappealing. Instead of 060808.html, the URL of webcomic archives looked like comic_id=130.

At first glance, there appears to be nothing wrong with this practice. As long as the content gets displayed to the user correctly with no dead URL links what is the problem? SEO (Search Engine Optimization) is the problem. Having a URL like www.pvponline.com/archive.php?comic_id=130 may deliver the comic with id of 130 to the browser, but that gives no information regarding the content of the comic. To make matters worse, many webcomics, such as reallifecomics.com keep the same webpage title for every archive page: "Real Life Comics - The Online Comic ©1999-2008 Greg Dean". Any search engine optimizer will tell you, the TITLE tag is the most key element to pay attention to. Each TITLE should be crafted to fit each page perfectly.

How does this come into play with webcomics? The comic title of course. Instead of using the same static title for each and every page, include the date the comic was published online and the title of the comic. While Scott Kurtz of PvP does not include the date within the TITLE tag the title of the comic is. This way search engines can distinguish between different pages, allowing them to bring more accurate results to people searching their site - in the long run this brings more conversions, or viewers, your way.

WebComic Law #3: You Name the Comic for a Reason: Include at least the name of the comic in the TITLE tag for each comic page. (See an example from PvP for reference) Notice how Scott also includes the comic title in the URL as well along with the date - this is considered absolutely pertinent information and should be added in as well. No more 060808.html or comic_id=130 URLs.

WebComic Law #2: Build Bridges, Not Mountains

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Last week, RealLife Comics ran a scavenger hunt on their website to celebrate their 2000th comic - quite an impressive achievement. The scavenger hunt gave users a handful of clues which would point them to ten comics in the archive. Users then scoured the website looking for these precious comics, loading page after page after page. In the end, 48 entries were entered into the contest, with only 12 having all of the clues correct.

Now RealLife is another very well known webcomic with a very committed fan base reading each comic everyday since 1999. Greg Dean, the artist behind ReaLife Comics even travels to conventions, gives tutorials in Illustrator, and sells swag like a crack dealer. Considering this, why did only 48 of his devout fans enter the contest? Was it because of the one week time restraint on the scavenger hunt? Perhaps. Though I believe there is a much larger culprit behind the scenes. I am willing to bet that there were hundreds of people who knew the comic Greg was referring to in the scavenger hunt clues but these readers did not have a clue where the comic was located in the archive. One week is a short time to forage around 2000 comics looking for ten golden gems however it certainly can be done. Once again, I am willing to venture a postulation that of those hundreds that potentially knew the comics off the top of their head they simply became frustrated after rummaging through the first few hundred comics. In the end, only the truly devoted or the people with immense time on their hands came out the victors.

Imagine if RealLife had a search feature where users could search for dialogue, props or even abstract concepts referenced in the comics. Would that defeat the purpose of a scavenger hunt? Most likely. The clue could easily be deciphered with a few keystrokes and a press of the mouse using the search feature. Instead of having 48 entries into the contest, there could potentially be thousands.

While a search feature may ravish any hopes of a scavenger hunt, it certainly would help viewers of a webcomic find content that interests them. When a user arrives at the homepage of RealLife he/she may find a comic about D&D and want to find other comics about D&D, but a las, there is no search feature. Do you really expect a viewer to go through each and every comic with a URL query slightly deviating from the form 050308.html or 05/03/08? If you do, you have obviously set your expectation bar to high. This brings us to the second law of webcomics.

WebComic Law #2: Build Bridges, Not Mountains: WebComics are extremely content rich websites; many even fill a certain niche for their viewers, however this contents value is severely degraded for each and every step a user has to take to find it. Artists looking to start a webcomic or upgrade a pre-existing one should implement a search feature that indexes not only the dialogue of the comic, but who is saying it as well. If needed, props and even concepts should be included in the search.

And if you really need to hold a scavenger hunt, either remove the indexed entries for any given comic page or open up Photoshop/Illustrator and add some sort of symbol, emblem or fire breathing techno dragon. Be creative. That is your role as an artist, no?

Law #1: WebComic Law #1: Front and Center

WebComic Law #1: Front and Center

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

This is a new series of blog entries I am going to do every week or so, critiquing popular webcomics that are lagging behind in the ever growing social web, while creating a list of "laws" for anyone looking to start or improve their own webcomic.

WebComics serve a great medium for communities, many in fact have online forums; though, web forums are very old and dated. The new trend in websites is to allow users to interact with your content as closely as possible - forums are a far cry from this.

These entries will eventually lead up to the launch of my friends webcomic which I am doing the design and coding for. While the final website may be far from a perfect Web 2.0 community, hopefully it will gain some attention and propel webcomic authors forward and inspire them to adopt different concepts.

penny-arcade.com. One of my own personal favorites and perhaps one of the most well known webcomics ever - it was even mentioned in Playstation magazine many times. The point is, this webcomic is top tier - they have an extremely dedicated and loyal fan base, their comics take hysterical, perverted, twists on the gaming world and leave the user rushing to read their archives, and their forums are lively; yet, their website seems dated like countless other comic websites.

To start, why is their comic not on the homepage? Somehow a little blog post from Gabe or Tycho trumps the importance of the latest comic. While Gabe and Tycho have a lot of weight in the video game world, the text spewed from their keyboards in no way matches the precedence of the comic genius crafted in Illustrator. Users come to PA to read the comic first and then start clicking around the website. Why does PA make their users click a separate link or append "/comic" to the URL when they are typing it into their web browser?

WebComic Law #1: Front and Center: Put the latest comic on the homepage. If you designed your website with the latest comic on a separate page you designed it wrong. There are thousands of webcomics online and you are fighting for very limited number of search results and niches - do not risk losing valuable viewers because of a negligence. Show them what you have to offer and sell it to them right on the homepage. If you need to post something, either put it under the comic of create a blog page.

In fact, the only thing about PA that saves it from being a Web 1.0 abysmal is its RSS feed, though the actual comic image is not even displayed in the feed. The user still has to click another link to bring them to the website.

Do not be fooled into believing the old, arcane adage of "build it and they will come". The website needs to be built correctly the first time, especially in the world of webcomics where the first impressions can be the difference between a daily reader and a viewer never to return.

Up next week is another one of my personal favorites, RealLife Comics. Remember, I am not saying these webcomics are in anyway bad or do not deserve their devoted fan base, I am just making the point that webcomics are lagging behind in todays web standards.

The Three Types of Blog Spam

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

To an owner of a blog, spam is a cancer - if left unchecked it can end the life of your website. Users that come to your site are discouraged by the comment spam; your site seems less professional, especially if the spam references pornography or the promotion of certain pharmaceutical drugs. When search engine spiders come to index your website is degraded in two ways. First, if you are lucky, the spider notices all the spam links and quickly lowers your ranking on the search engine, removing any relevancy searchers may use to find your website. Notice I said if you lucky. If you are not so lucky, the spider could potentially assume the spam is meaningful within your website - now all the pornographic links, enlargement pills and scam artists become associated with your website.

To combat comment spam, you need to be able to recognize what kind of spam it is.

1. Little Kid With A Grudge: In this case, you probably made some little fanboy kid mad and he has set off on his own personal vendetta to make you pay. He will post malicious comments all over your blog until he tires himself out or one of his nettlesome friends sends him a link over AIM to the next blog to flame.

The good news. They tire quickly. And to make them tire even faster, implement a comment system that does not allow a person to comment more than one time per minute or two. While this kid developed a grudge quickly and decided to spam your site, the discouragement of not being able to flame fast enough will make him cry into his Ovaltine and find another site to spam.

2. Automated Spam Bot: Do I dare say that this spam might actually be beneficial? Well, it is not; though, if you are new to the blogosphere, take heart, your blog is getting noticed! Search spiders crawl your website so it can be indexed on their search engine - similarly, spam robots crawl your website looking for your comment form. Once they find it, they read the ACTION attribute, and use the URL to construct POST data. The data is submitted to your website, masking as a browser, and a las, a new spam comment is added.

However, you too can do your own masking. Since these spam bots have many, many other websites to crawl and spam, using some JavaScript to encrypt the ACTION attribute until the user presses submit will confuse the spam spiders - they will give up and return to their nest.

Note: This type of spam is normally not meant to be malicious, just a website trying to get some links back to it so it can make a little coin. Immoral? Yes. Malicious? No.

3. Beelzebub Is Hungry: Either you really pushed him over the edge or your website is just that popular, Beelzebub is out for you. Even though you may has masked your ACTION attribute in the FORM tag, Beelzebub can easily fire up a program like Wireshark and see where his packets are going. By reading through his data sent he can find what URL you are using to post your comments. Remember, JavaScript is client side. JavaScript decodes your encrypted URL and then changes the ACTION attribute to the URL. The URL is still visible to someone monitoring their data transfer, and in fact, always will be. Now that Beelzebub has the URL, he can craft his own PHP/Python/Perl/whatever script to post data to the URL and start a hellacious spam parade. There is nothing you do to stop it, but you can through a few holy barriers in his way.

The first is to use CAPTCHA images. They require a commenter to enter in a series of letters and numbers that are built right into the image. The server side comment script then checks the letters and numbers that the user entered in and compares them to what was within the image. (CAPTCHA images are normally created by a server side script, so the combination can easily be stored in a variable) If the two values are equal, the commenter is not a spammer.

While this method works well and is in fact the de facto standard for preventing spam, I personally find it old and cumbersome. CAPTCHA images are normally ugly and do not fit will within a design. Even worse, they can create server accessibility issues that may drive users away from your blog. I have seen many capture images that are completely undecipherable and others that are way too long and complex.

The solution? An automated system to rename the file used for adding comments in conjunction with client side JavaScript encryption. Setup your webservers crontab to run a script every few minutes (or even every minute) to rename the file that is used for commenting to a hash, similar to a string that would be returned from the PHP md5 function. JavaScript can then be used to encrypt the new filename held within ACTION attribute in the FORM tag. So while Beelzebub can still monitor his POST data and attempt create a script to send data to the URL, the URL coded into the script will most likely be invalid by the time he tries to run it (depending on how often the name of the comment script is changed).

Only one solution will completely dissolve spam, and I tend to disagree with it - the approval of comments by website owners. You may choose to have your reasons for running this route, especially if your website deals with professional topics and does not want commenters to rambling about something completely off topic - understandable. However, for the mast amount of bloggers in the blogosphere, approving comment by comment detracts from a users experience on your blog. The web is quickly becoming more and more semantic and this requires tasks to happen within a split second. Comment approval by site admins will not be a feasible method for much longer.

Unless you manually approve each and every comment for your blog, no method will completely rid your website from spam, but it should at least slow it down to a crawl. Every blogger should moderate his website for spam at least once, preferably much more, a week, deleting comments that are blatant spam.

Spam Be Gone

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Two mornings ago I awoke to something a blogger never wants to awake to - a couple hundred spam comments on his blog. Never good. Especially when it was the first thing I had to deal with in the morning. Thinking it to be just a one time hit, I just deleted the comments with a simple mySQL query and went to take a shower. After getting out, and feeling quite a bit refreshed a might add, I checked my blog again. Ten more spam comments had been added. There goes the refreshed feeling.

From there I just turned off the comment system until I had a chance to deal with it. So now I have made some changes, rewrote some code, and hopefully now all will be well. If not, I will just have to keep working on an anti-spam system.

On tap for later this week, I am planning on writing up another blog post this week regarding the different types of spammers and methods to deal with them. So grab your favorite stein and fill it with some smooth, frothy knowledge.

Trimming Excess Memory in C++

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Allocation, allocation, allocation. So important, but yet easy to abuse. When reading a file of variable length or parsing input without knowing where it may end, allocating memory space on the heap is the best bet for a C++ programmer; unfortunately for novice coders, this requires the use of pointers. Some programmers may be tempted to use a very large value buffer value to try and ensure that there is enough space for the variable contents - DO NOT DO THIS. Using any statement such as int[99999], char[99999], and even string[99999], is prone to buffer overflows. Instead, allocate enough space on the heap, increment when needed if more space is needed, and trim any excess space off once the task is completed.

Consider a program that reads in a variable amount of employee names. The amount of employees can vary from company to company - some companies may only have 2 employees, others may have hundreds. The first step is to write a method to read in all the names and store them in the heap:

1.string *get_names(int &totalItems)
2. {
3.  int allocateSize = 0;
4.  string input = "";
5.  string *stringArray = NULL;
6.  string *tempArray = NULL;
7.  totalItems = 0;
8.
9.  while (1)
10.   {
11.    cout << "enter name: ";
12.    cin >> input;
13.
14.    if (input == "-1")
15.     {
16.      break;
17.     }
18.
19.    if (totalItems == allocateSize)
20.     {
21.      string *originalArray = stringArray;
22.      int enumArray = 0;
23.
24.      stringArray = new string[allocateSize * 2 + 1];
25.
26.      for (enumArray = 0; enumArray < allocateSize; enumArray++)
27.       {
28.        stringArray[enumArray] = originalArray[enumArray];
29.       }
30.
31.      delete[] originalArray;
32.      allocateSize = allocateSize * 2 + 1;
33.     }
34.
35.    stringArray[totalItems++] = input;
36.   }
37.
38.  tempArray = memtrim(stringArray, totalItems);
39.  delete[] stringArray;
40.
41.  return tempArray;
42. }

The function signature indicates the method will return a string pointer while requiring an integer to passed to it by reference - the integer will store the total number names in the array.

The method starts by starting a loop, requiring the user to enter in an employee name. If the employee name equals -1 the loop terminates. In reality, using a sentinel to indicate a termination of input would not be used, but fits easily enough into this example.

If the total number of items in the array equals the amount of space allocated for the array, the array must be resized. A pointer to the hold the contents of stringArray is created under the name oforiginalArray, while stringArray is resized exponentially. The rationalization for resizing exponentially is that allocating memory on the heap can be expensive - it is time consuming. Allocating on the heap as opposed to the stack is much slower. It is much faster to "over resize" and then trim the excess later in the program. Incrementing linearly and even quadratically is not (most of the time) good programming practice.

The for loop copies the contents from the originalArray to the newly resized stringArray. originalArray is then deleted to ensure a memory leak is not created and allocateSize is reset to match the new size of stringArray. input can then be placed in the stringArray array, while the number of totalItems is incremented.

This loop continues until the user enters a value of -1. Once this has happened it is time to remove the excess allocated memory, followed by deleting the oversized array and returning the pointer reference to the trimmed array. The template memtrim is now needed:

1.template <class Type> Type *memtrim(Type *array, int totalItems)
2. {
3.  Type *holdArray = NULL;
4.  int enumItems = 0;
5.
6.  holdArray = new Type[totalItems];
7.
8.  for (enumItems = 0; enumItems < totalItems; enumItems++)
9.   {
10.    holdArray[enumItems] = array[enumItems];
11.   }
12.
13.  return holdArray;
14. }

Using templates for this type of method is exactly what templates where created for. Having to rewrite this method for each variable type would be very redundant - using templates allows one method to be written and then reused for whatever variable type is needed.

memtrim returns a pointer to trimmed array, while requiring an array to be trimmed to be passed to it along with the number of total items in the array. The total number of items should never exceeded the allocation size of the array - it may be less than or equal to the allocation size, but never greater than.

holdArray is allocated with a size equal to the number of totalItems, and the contents of the oversized array are copied into the new array. Lastly, the method returns the pointer referencing the newly created trimmed array. As declared in the get_names method, the original oversized array must be deleted to ensure there is not a memory leak.

Putting the entire concept together, an example program would look like this:
1.#include <iostream>
2.#include <string>
3.
4.using namespace std;
5.
6.string *get_names(int &totalItems);
7.
8.template <class Type> Type *memtrim(Type *array, int totalItems)
9. {
10.  Type *holdArray = NULL;
11.  int enumItems = 0;
12.
13.  holdArray = new Type[totalItems];
14.
15.  for (enumItems = 0; enumItems < totalItems; enumItems++)
16.   {
17.    holdArray[enumItems] = array[enumItems];
18.   }
19.
20.  return holdArray;
21. }
22.
23.template <class Type> void dump_array(Type *array, int totalItems)
24. {
25.  int enumArray = 0;
26.
27.  for (enumArray = 0; enumArray < totalItems; enumArray++)
28.   {
29.    cout << array[enumArray] << endl;
30.   }
31. }
32.
33.string *get_names(int &totalItems)
34. {
35.  int allocateSize = 0;
36.  string input = "";
37.  string *stringArray = NULL;
38.  string *tempArray = NULL;
39.  totalItems = 0;
40.
41.  while (1)
42.   {
43.    cout << "enter name: ";
44.    cin >> input;
45.
46.    if (input == "-1")
47.     {
48.      break;
49.     }
50.
51.    if (totalItems == allocateSize)
52.     {
53.      string *originalArray = stringArray;
54.      int enumArray = 0;
55.
56.      stringArray = new string[allocateSize * 2 + 1];
57.
58.      for (enumArray = 0; enumArray < allocateSize; enumArray++)
59.       {
60.        stringArray[enumArray] = originalArray[enumArray];
61.       }
62.
63.      delete[] originalArray;
64.      allocateSize = allocateSize * 2 + 1;
65.     }
66.
67.    stringArray[totalItems++] = input;
68.   }
69.
70.  tempArray = memtrim(stringArray, totalItems);
71.  delete[] stringArray;
72.
73.  return tempArray;
74. }
75.
76.int main()
77. {
78.  string *stringArray = NULL;
79.  int totalItems = 0;
80.
81.  stringArray = get_names(totalItems);
82.  dump_array(stringArray, totalItems);
83.
84.  return 0;
85. }

Dynamically Allocating char[] in C++

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

This should be a fairly basic concept in C++, but beginners tend to struggle a lot with pointers, and to make things worse, when trying to debug them and find out where an error is, printing out the value of what a pointer is pointing at can quite cryptic. If during allocation there was a problem assigning a value then a bunch of garbage may be printed to the screen. A prime example is the following:

1.string line;
2.char *dynamicLine;
3.// assume line is given a value somewhere in the program
4.
5.dynamicLine = new char[line.length()];
6.
7.for (enumLine = 0; enumLine < line.length(); enumLine++)
8. {
9. dynamicLine[enumLine] = line[enumLine];
10. }
11.
12.// rest of program
13.delete[] dynamicLine;

Experienced C++ programmers should be able to find the error right away, but to beginners it may not be obvious, though the solution to the problem is incredibly obvious once you figure out what the issue is. To make matters even worse, the output from the above code may not even show there is a lurking error. Normally the problem lies when the code is looped through a handful of times, such as when reading file input.

The problem area is simply that there is not enough space in the dynamically created array for the null character which signifies the end of an array of chars. Without that, dynamicLine is not technically terminated and can lead to a problem when dealing with its contents. To fix this simply allow space for the null terminator and set the last index of the array to 0.

1.string line;
2.char *dynamicLine;
3.// assume line is given a value somewhere in the program
4.
5.dynamicLine = new char[line.length() + 1];
6.dynamicLine[line.length()] = 0;
7.// the following line is also equivalent:
8.// *(dynamicLine + length.length()) = 0;
9.
10.for (enumLine = 0; enumLine < line.length(); enumLine++)
11. {
12. dynamicLine[enumLine] = line[enumLine];
13. }
14.
15.// rest of program
16.delete[] dynamicLine;

So there you have it, a not so obvious problem (for those novice coders out there) with a very obvious and simple solution. Now get back to coding. You need practice.

DiggMatch Launch

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

I just launched my new project, DiggMatch. It has taken digg way too long to implement a recommended story algorithm so I took matters into my own hands and coded one myself. The results are quite satisfying.

To read the entire intro post, check out the DiggMatch blog page.

..And We Are Back

Adrian Rosebrock — Wed, 20 Aug 2008 15:32:28 -0400

Yes, the site was down for about a week. I am terribly sorry. Let me tell you a little story about what happened...

Originally, two of my friends, along with myself purchased a deluxe hosting plan from ixwebhosting.com. Over time it became extremely apparent that I was the only one that used the hosting service - the other two friends websites fell completely off the internet.

So in efforts to save money my friend cancelled the hosting account - which created a problem for me. Here I am trying to build a blog and all of a sudden my site is gone for a week? No, not good at all.

I scrambled to create my own account at ixwebhosting, had my friend transfer ownership of chaoscoding.com from him to me, and then let the new IP address percolate through the DNS system. When the site was finally set and ready, I had an awful realization - there was no fucking backup for my database. I know, dumbest move on my part ever. Luckily, the spectacular technical support at ixwebhosting hooked me up with a backup of my database and the day was saved.

There is the story. Yes, it sucks and it is (mostly) my fault. But, let me tell you what is on tap.

First off, a new design. What? Why? I have only had this one for a few months?! Honestly, I am a coder and a designer. This is what I do. Get the hell over it.

Secondly, I am launching another webapp (hopefully) next week. And guess what? It is related to digg. It is going to be intense...