RandomDump

This is how it works..

Available events:
Keyboard input.
Mouse click (left/middle/right).
Mouse button release (left/middle/right)
Mouse wheel.
Mouse coordinate.
Maybe more...

So, my desktop environment (GuiGui) will capture all those events and record them in a file (xml or s-expression).
The file might look like this:

<code>
<gui-macro>
<date>
<day-of-week>Thursday</day-of-week>
<month>December</month><day>29</day>
<hh>08</hh><mm>56</mm><ss>25</ss>
<timezone>EST</timezone>
<year>2005</year>
</date>
<event-sequence>
<item class="LMB-on">
<coordinate><x>352</x><y>35</y></coordinate>
</item>
<item class="LMB-off">
<coordinate><x>124</x><y>436</y></coordinate>
</item>
<item class="RMB-on">
<coordinate><x>23</x><y>352</y></coordinate>
<item>
<item class="RMB-off">
<coordinate><x>23</x><y>352</y></coordinate>
<item>
<item class="LMB-on">
<coordinate><x>32</x><y>367</y></coordinate>
</item>
<item class="LMB-off">
<coordinate><x>32</x><y>367</y></coordinate>
</item>
<item class="LMB-on">
<coordinate><x>100</x><y>111</y></coordinate>
</item>
<item class="LMB-off">
<coordinate><x>100</x><y>111</y></coordinate>
</item>
<item class="Keyboard-in">
<string>mike\trowe\t1st Ave.\tNew York\tnnnnnnnnnnnn\t10001\r</string>
</item>
</event-sequence>
</gui-macro>
</code>

Above drags a window (supposedly) from 352,35 to 124,436.
Then it right clicks on 23,352 and clicks a button from the right click menu (supposedly).
Then it places caret on 100,111 where there can be a web form for registration.
And it fills out the form.

Maybe I can do better than that if X11 gives me the name of application being clicked, dragged..etc.
And each text field of an application (including web applications) should have some sort of id.
Web applications are usually marked up very well.
So, it might be possible to record where the left mouse click was.
For example, Firefox::http://www.website.com/registration.php::name can be the were I clicked before the keyboard input sequence.
For normal applications, I'm not sure if X11 gives name of the application being clicked.
And in the application, text fields might not have unique id...