Benchmarking LCS Systems with Random Systems

After completing a first running version of a LCS system^3.7 using the fitness function tttfit2() as well selection and crossover (but not yet mutation) this version has been tested against pure random systems.

For a first benchmark the following scenarios have been assumed (cf. the table below): Random (R) against Random (R)(cf. example of one experiment in diagram 3.39); R against Classifier System (C); C against R. The scenarios have been realized with experiments playing 3 times 88 tournaments with 11 games per tournament, thus having 3 x 968 games for each scenario. The table shows the percentage of wins of the beginner (left column) and the opponent (right column) as the mean values out of the three experiments with a standard deviation between 12 - 14% of the mean for the beginner and 11-14% for the opponent.

R	R
59%	27%
R	C
63%	27%
C	R
66%	23%

Figure 3.39: 88 Tournaments, 11 Games/T, Random Beginner (black) against Random Opponent (blue), MEANX=58.9, STDX=14.6, MEANO=27.8, STDO=13.8

The question is, what these numbers can tell us about the acting systems. An explanation is needed for the fact that the opponent is always only half as good as the beginner. Furthermore the opponent as classifier system (C) has the same values as a random system (R). The classifier system (C) as beginner shows 3-7% better results as a random system and at the same time shows a random system 4% worse results as opponent, but taking into account the deviations these minimal differences can be caused by the inherent variations and are therefore not really significant.