idea drought

Colored glass balls

Colored Glass balls

I found this wallpaper while stumbling. Good use of colors, simple, yet effective. I love it, and it’s in my rotator at this moment

I had to add a bit of whitespace on each side to get it to the size I wanted (1920×1200), but it didn’t take anything away from it. Original can be found here

1 Comment »

Weighted raiting

Every website that gives its users access to rate items by a scale comes across a choice in how to best handle the calculation of ratings. There are some pretty advanced algorithms used to handle this. From algorithms based on the Bayesian prinsiple or using social networking(pdf) as a base, like digg.com

The easiest choice is to calculate the average rating and leave it at that. This means that an item which has been rated once as 10 will be ranked higher then an item which has been rated over a thousand times, and has an average of 9.8. There is of course an upside and a downside to this. For a website with a lot of activity, items voted up like this get an opportunity to be viewed and if it isn’t up to standard, it will be rated down. Though, this system has its downsides.

Lets say a website with user created content like youtube.com had this system running. This would mean every user could vote their own (or get a friend to vote – just to preempt you guys) item as a 10 and be on the top of the top-rated list. The result would be that the top-rated list would be filled by bad quality videos, and the top-rated list wouldn’t be very useful.

To combat this, we have weighted rating. Its main purpose is to give items with a low number of ratings a handicap. How weighted rating is implemented varies a lot, but a good place to start is by using the average number of votes for all items as a base for the weighting.
Here is an example of a simple algorithm that can be used.

0<= weight >= 3
weight = AVG(votes) / COUNT(ratings)
rating =( SUM(ratings) / COUNT(ratings) ) – weight

The first line says that weight cannot be smaller then 0 and not bigger then 3. If we do not set this kind of a limit we would get some unexpected results. Lets say AVG(votes) is 30, the item has one rating of nine on a scale from one to ten. This would give us:

weight = 30 / 1 = 30
rating = 9 / 1 – 30 = -29

Though, there is a problem with that algorithm. New items are treated the same as very old items. The way I see it, old items with few ratings should be weighted down a lot, while new items, with few ratings, should not be rated down as much, as this would give them an unfairly bad start. So what we want can do is add a time based weight decreaser.

daysOfEffect = 180
rateOfDecrease = daysOfEffect / 2

0<= timeWeightDecreaser => 2
timeWeightDecreaser = 2 – ( daysOnSite / rateOfDecrease )

0<=weight=>3
weight = (AVG(votes) / COUNT(ratings)) – timeWeightDecreaser

1<=rating=>10
rating = (SUM(ratings) / COUNT(ratings)) – weight

This is the timebased weight decreasing algorithm

daysOfEffect denotes how long an item should be considered new. A small period of time before the the time based weight decreaser kicks in can be a good idea. A visual on how this is done can be seen in the picture on the left. I used to grapher utility you get in OS X to create it. Never thought I’d get use for that

I also implemented a static function in php so you can see how this algorithm can be implemented in code.

<?php
  /**
  * Calculates weighted rating
  * Uses a time and vote amoun based algorithm
  *
  * @param $avg avarage number of votes for all items
  * @param $sum sum of ratings for item to be rated
  * @param $count number of ratings for item to be rated
  * @param $freePeriod number of days until the time based algorith will start having an effect
  * @param $daysOfEffect total number of days the time based algorithm will have an effect
  * @param $daysOnSite number of days the item has been on the site
  * @param $debug optional, default = false, set to true to see debug message
  * @return decimal weighted rating
  **/
  public static function CalculateWeightedRating($avg,
                                                 $sum,
                                                 $count,
                                                 $freePeriod,
                                                 $daysOfEffect,
                                                 $daysOnSite,
                                                 $debug = false )
  {
	  if( $count == 0) return 0;

	  $rating = $sum / $count;

	  if($count > $avg)
	  {
		  $weight = 0;
		  $timeWeightDecreaser = 0;
	  }
	  else
	  {
		  $weight = $avg / $count;

		  if( $weight > 3)
		  {
			    $weight = 3;
		  }

		  $rateOfDecrease = ($daysOfEffect - $freePeriod) / 2;
		  $timeWeightDecreaser = 2.3 - (($daysOnSite - $freePeriod) / $rateOfDecrease);

		  if( $timeWeightDecreaser > 2.3 ) $timeWeightDecreaser = 2.3;
		  if( $timeWeightDecreaser < 0 ) $timeWeightDecreaser = 0;

	  }

	  $weightedRating = $rating - ($weight - $timeWeightDecreaser);

	  if( $weightedRating < 1) $weightedRating = 1;
	  if( $weightedRating > 10) $weightedRating = 10;

    if( $debug == true)
    {
	    echo "<p>";
	    echo   "Average votes: " . $avg . "<br/>";
	    echo   "Sum rating: " . $sum . "<br/>";
	    echo   "Num votes: " . $count . "<br/>";
	    echo   "Days on site: " . $daysOnSite . "<br/>";
	    echo   "Rating: " . $rating . "<br/>";
	    echo   "Weight: " . $weight . "<br/>";
	    echo   "Time weight decrease: " . $timeWeightDecreaser . "<br/>";
	    echo   "Weighted Rating: " . $weightedRating;
	    echo "</p>";
    }

    return $weightedRating;
  }

1 Comment »

Do you really know three layered webdesign?

This is a topic that is well known and understood. Export the css layer and the javascript layer into files that can then be included in all your files, right? Not quite, that is an “object oriented” approach, which has nothing to do with three layered design. The “oo” approach is about unity and reuse of code. Many people have this misconception, and the result is that it is making an impact on the web in general.

It’s not only personal websites that fall into this trap. I stopped at McDonalds between Sydney and Canberra on a trip. To my great delight they had free wifi for their customers. So I pulled out my iPod and tried to connect. No dice. The EULA was Javascrip based, so I couldn’t click agree. And this is only one of many places that does this.

This brings us to the main question: What is the three layered webdesign theory?

three layered webdesign

There are many analogies used to explain this, and my favorite has always been the onion. Imagine a three layered onion. Look at it from the outside and you can see that it is an onion. Peal off one layer, and you can still see that it is an onion, a little bit smaller, but still an onion. Peal of one more and… you get the point.

Now after pealing off one or two layers, does the functionality of the onion decrease? No, it is still an onion and still does what it is supposed to do. In Webdesign; the outer layer is Javascript, middle is CSS and in the center is the content.

The theory states that you can not let the base functionality (submitting forms, navigating etc) of your site be dependent Javascript nor CSS. This has to be available in the content layer. You can’t use Javascript to load content, unless you make sure the content can also be loaded for a user who has Javascript disabled. Of course, you are allowed to change the content layer with the other layers, ie, changing the header text with an image using CSS. Or change how a form is submitted by changing the button using Javascript.

I am a firm believer of letting the majority of visitors have access to the content of my web sites. And I am sure all of your clients will agree to this. This means you will have to cater for people without 20/20 vision, or other disabilities. Using the three layered web design approach you are well on your way to achieve an AA, if not an AAA, WCAG rating.

Remember, about ten percent of the worlds male population (~300,000,000) is colorblind. Can you afford not to cater for this group?

No Comments »

Linux saves the day

My gfs computer crashed on the 1st of January (probably a sign of what is to come in the new year). It died during shutdown, and something went horrible wrong. From this moment the computer would just hang as soon as Windows started loading. First thing we tried was booting up with the Windows XP cd to see if we could fix it that way. No luck, computer hung as soon as it started loading the cd. If we disabled the hdd with windows on it, the cd would boot just fine though. The drive with Windows XP on it, also had Ubuntu on it, which booted just fine, as did the Ubuntu live cd. So it wasn’t a hardware problem.

Mounting the drive in Ubuntu didn’t work at all. Only option I saw was clearing out the windows partition and reinstall. Though, here comes the real problem…Backups. As often happens, they are out of date when they are needed.

This is were TestDisk (windows/linux/osx) comes into the picture. It’s a tool to test and rebuild broken HDDs. It can even copy deleted files and repear broken partitions. The computer has a couple of big drives in it, which happen to have about 200GiB free space. This meant we could mount one of these, and copy all the needed files to them.

We downloaded Ubuntu rescue remix 8.10 and fired it up. It’s a command line distro, so make sure you know your way around the command line and especially how to mount and unmount drives.

After mounting the drive where we could copy the files, we fired up TestDisk and let it analyze the drive in question. It found all the partitions, so we selected the windows partition and pressed ‘P’ to view files. Everything is there! Awesome. Hit ‘C’ to copy the selected file/folder and navigate to the mounted hard drive.

All files copied over and Windows is back and working.

I had to use linux some time earlier to clean out a pretty nasty virus that went around on Windows (couple of reformats ago). Funny how you need linux to save a windows crash

No Comments »

Handy Linux network stat commands

Netstat can be used to do a lot of things, I usually use it to keep track of how many connections there are to my server. These are my two most used commands

Show the number of connections on your http port

netstat -nta | grep :80 | wc -l

List the top10 ips using the highest number of connections to your server

netstat -atnp -A inet | awk -F " " '{print $5} ' | awk -F ":" '{print $1}' | sort | uniq -c | sort -nr | head -10

If there are some bad offenders in the list, you can ban their ip by using IP tables.

Ban:

sudo iptables -A INPUT -s <IPHERE> -j DROP

Unban:

sudo iptables -D INPUT -s <IPHERE> -j DROP

These are temporary bans, if you want to save them you will need to save the IP tables and restart the IP table service. This is for rhel, fedora and centOS.

sudo service iptables save
sudo service iptables restart

No Comments »

Wallpaper of the moment

Here is an amazing high-resolution wallpaper from Yang Media. It went straight onto my wallpaper rotator when I saw it.

From Norway - By Yang Media

1 Comment »

Variable as a function-call

In PHP there is one really cool thing you can do. You can use a variable as a function call. Microsoft created a sort of a similar thing in C#, but it’s not as easy to use.

It’s easier to show you with an example then trying to explain it, so that is what I will do

<?php
  function foo()
  {
    echo "Hello World!";
  }

  $bar = "foo";
  $bar(); // this calls foo()
?>

Neat eh?

This can be used to call different classes depending on the URI. New example for this:

<?php

  $uri = explode('/', $_SERVER['REQUEST_URI']);

  if( is_file($publicClassesPath . "/" . $this->uri[1] . ".php"))
  {
    include_once( $publicClassesPath. "/" . $this->uri[1] . ".php");

    $RunningClass = new $uri[1](); // initiate the class and bind an instance
                                   // to the variable $RunningClass
  }
?>

No Comments »

Cascading order in CSS

I’ve been working with CSS for years now, and I’ve been following two simple rules about overriding styling rules. I like to call them the position rule and the origin rule.

• The position rule
  If two or more rules with the same name exist, the latter wins and overrides the others.
• The origin rule
  In line style rules overrides style rules in the header, and styles in the header overrides external style sheets

Example

Just following these two rules works in most cases, but can give you some interesting bugs. A bit deeper understanding of a third rule will help with these bugs and how to avoid them.

The third rule you need to be aware of is the specificity rule. This dictates that the more specific a rule is, the more weight it has, and thus it’s more important. Even if a rule is after that one, the latter will be ignored and the more specific will be used. It is pretty straight forward, and the w3 cascading rules does a good job of explaining the idea behind it:

A selector’s specificity is calculated as follows:

• count the number of ID attributes in the selector (= a)
• count the number of other attributes and pseudo-classes in the selector (= b)
• count the number of element names in the selector (= c)
• ignore pseudo-elements.

Concatenating the three numbers a-b-c (in a number system with a large base) gives the specificity.

Example(s):

Some examples:

*             {}  /* a=0 b=0 c=0 -> specificity =   0 */
LI            {}  /* a=0 b=0 c=1 -> specificity =   1 */
UL LI         {}  /* a=0 b=0 c=2 -> specificity =   2 */
UL OL+LI      {}  /* a=0 b=0 c=3 -> specificity =   3 */
H1 + *[REL=up]{}  /* a=0 b=1 c=1 -> specificity =  11 */
UL OL LI.red  {}  /* a=0 b=1 c=3 -> specificity =  13 */
LI.red.level  {}  /* a=0 b=2 c=1 -> specificity =  21 */
#x34y         {}  /* a=1 b=0 c=0 -> specificity = 100 */

Order doesn’t matter, and it only counts for one group at the time. IE: “UL LI, LI #rawr {}” gives us a specificity of 2 for first group and 101 for the second group.

The only other rule that over rides everything else is the ‘!important’ rule (buggy in IE6). Though, you can read that in the w3 cascading rule 6.4.2.

No Comments »

Cross-site request forgery prevention

Cross-Site Request Forgeries is the new buzzword in web-security these days. What it is? Why it’s dangerous? How to prevent it?

Bill Zeller offers a good description of what this is

CSRF vulnerabilities occur when a website allows an authenticated user to perform a sensitive action but does not verify that the user herself is invoking that action. The key to understanding CSRF attacks is to recognize that websites typically don’t verify that a request came from an authorized user. Instead they verify only that the request came from the browser of an authorized user. Because browsers run code sent by multiple sites, there is a danger that one site will (unbeknownst to the user) send a request to a second site, and the second site will mistakenly think that the user authorized the request.

This means the potential hacker will insert a bit of code into a page, or trick a user to load one of their pages up. Then they will be able to post forms as the user.

GMail has been hit by several of these flaws, one being users having filters added to their gmail account (fixed now). With this, the hijacker could e.g. filter all emails from Godaddy.com to his own account and then take over the targets’ domain names. They are easy to do, and can be automated.

Google isn’t alone, many of the major websites have these kinds of vulnerabilities. Do you?

Luckily it is easy to stop these attacks

One way of doing it is to generate a random string and add that to a cookie and a hidden form element. Then make sure that they match before you process the form. This way the hacker has to guess what the random string is to be able to post the form. The chance of this happening is slim to none, so it is a pretty good way of doing it

Now, how do we do this in php?

First off, we have to generate a random string, this is easily done with the random function. Keep in mind, this string doesn’t have to be very random before it becomes difficult to impossible for the hacker to guess. I’ll keep it simple

$randomString = md5(rand(0,100000));

That should give us about 100 000 possibilities.

Next step is to send the cookie to the user

setcookie("XSRF_filter",
$randomString,
time() + 3600,
"/",
".yourdomain.com",
0);

And the hidden form element would look something like this

<form action="" method="POST">
<input type="hidden" name="XSRF_filter" value="<? echo $randomString; ?>">
<br>...other form elements
<br><input type="submit" name="submit" value="submit">
</form>

Pretty simple so far right? Good, now comes the validation checking

if ($_POST['XSRF_filter'] != $_COOKIE['XSRF_filter'])
{
echo “You’re trying to do an XSRF attack! Can’t allow you to post this form”;
}
else
{
//process form
}

That’s it!

Demo and complete source code can be found in the playground

Here is a tool for testing for XSRF vulnerabilities

No Comments »