nanogui: Thread: Can we make PIXELVAL a 16-bit entity?

Hello,

First off, the X11 stuff in 0.87pre1 works ok and is of great help
for testing!  As for speed, it seems much better, but we have not
tested extensively yet, e.g. Opera.

I've got a question: We will use a 16 bit display, and would like
to use GrArea/GrReadArea to draw nice widgets and stuff.  If we
included code to make PIXELVAL only 16 bits, would this require
extensive rewrites of mwin/nanox?  And more important: Would it
run counter to design philosophy?

I think it should work well, but I'd like to get a GA on this
before changing too much...

Should I/we try this?

Regards,
Morten Rolland, Screen Media

: First off, the X11 stuff in 0.87pre1 works ok and is of great help
: for testing!  As for speed, it seems much better, but we have not
: tested extensively yet, e.g. Opera.

Great.  Keep me posted on the speed stuff with Opera, when
your team finds the time.  It's very important to me that
both Nano-X and Microwindows can blaze...


: 
: I've got a question: We will use a 16 bit display, and would like
: to use GrArea/GrReadArea to draw nice widgets and stuff.  If we
: included code to make PIXELVAL only 16 bits, would this require
: extensive rewrites of mwin/nanox?  And more important: Would it
: run counter to design philosophy?

First - no this definitely doesn't run counter to my design philosophy;
in fact, if you check out device.h, you'll see that there's already an
#ifdef for the ELKS systems that will never use PIXELVALs larger
than a byte, and it'd be a shame to require 32bpp images when
only 8bpp are required on 16 bit systems. 

My original design idea was that the entire graphics computing
system could "inherit" characteristics of the lower level hardware
and use that for most effecient processing and memory use,
considering that's the point of this project, really.

So - I had previously considered that sizeof PIXELVAL would be
either only 8 or 32, but 16 is just fine.  What I'd like to see
would be minimal #ifdef's throughout the shared code, 
if possible; perhaps separate procedures for packing/unpacking PIXELVALs
into ReadArea buffers, with an upper level #define for which procedure
to call.  This keeps the main code extremely easy to read and understand,
which is another goal of mine for this project.

Now - there's one more consideration, and that is that when running
client/server, we need to tell the server which algorithm to use
as well, since it may not be running the same hardware. [I think
we can assume that it always will, but just in case].



: 
: I think it should work well, but I'd like to get a GA on this
: before changing too much...
: 
: Should I/we try this?
: 

Go ahead and do it, send me the mods.  Use 0.87pre1 as the
base.  Like I said above, keep it simple if possible.  You'll see
that this is already an item in the TODO list...

Greg

Hello!

Greg Haerr wrote:

> First - no this definitely doesn't run counter to my design philosophy;

Thats what I thought, just wanted to check, thanx.

> My original design idea was that the entire graphics computing
> system could "inherit" characteristics of the lower level hardware
> and use that for most effecient processing and memory use,
> considering that's the point of this project, really.

I guess you mean "inherit" WRT compiling an entire system with
same setup for both servers, libraries and user apps, right?
Adjusting the PIXELVAL or packing method should be done globally
to get the best results (so they match each other all the way).

> So - I had previously considered that sizeof PIXELVAL would be
> either only 8 or 32, but 16 is just fine.  What I'd like to see
> would be minimal #ifdef's throughout the shared code,
> if possible; perhaps separate procedures for packing/unpacking PIXELVALs
> into ReadArea buffers, with an upper level #define for which procedure
> to call.  This keeps the main code extremely easy to read and understand,
> which is another goal of mine for this project.

Yes I agree that separate functions are a clean solution. If you have
looked into GGI you will know that they do this, only to the
extreme; they use dynamic run time loading of shared libraries to match
up against the hardware/target with optimized functions, but with no
calling overhead.  We don't need this to the same extent, as we only
compile for a single target at a time (I guess...)

> Now - there's one more consideration, and that is that when running
> client/server, we need to tell the server which algorithm to use
> as well, since it may not be running the same hardware. [I think
> we can assume that it always will, but just in case].

OK, this is my take on this that I have started to code (for Nano-X):

The GrArea call in 'client.c' uses GrNumArea=40 IIRC and a struct with
the area-info and a bunch of pixles trailing it.

I figured one could solve the problem in a simple manner by introducing
a new protocol "opcode" GrNumArea_R5G6B5=47 that uses the same struct as
the original GrArea, but with 16 bit pixels trailing it.  This reduces
the memory usage and transfer time.  The pixels needs to be formatted
at the client to use the 16 bit encoding that this particular version
of the GrArea call supports, but this is the general idea like you
stated earlier.

If the Nano-X server happens to have 8, 24 or 32 bits/pixel, the 16 bit
values supplied with this call can still be converted and displayed.
I would like this to be optional though, to save space when you don't
need it.  Likewize, I'd like to be able to selectively decide which
GrArea_* functions I want in my library so that a 16 bit only setup can
have as small code footprint as possible.

As to the entery point in the Nano-X client library, it could be done
like:

 #define NANOX_PF  PF_TRUECOLOR565
 #include <mwin/nano-X.h>
          ...
 PIXELVAL *p;   /* sizeof(*p) == 2 */
          ...
 GrArea(id,gc,x,y,w,h,p)
 /* Gets expanded to GrArea_R5G6B5(id,...) by a #define in
  * nano-X.h depending on the value of NANOX_PF
  */

The NANOX_PF could default to 24 bit colour, to make life easier.
Also, when doing this, the RGB macro or some similar macro should
be defined according to the pixel format used, but I'm not quite
sure if RGB() is the right macro for this:  Judging from the
definition of RGB(r,g,b) it returns a COLORVAL value, which is used
for a couple of things; but is it intended to be used to build
PIXELVALs for GrArea as well?

If not, a macro 'RGB2PIXEL(r,g,b)' could be used to do this, along
with 'PIXEL2GREEN(PIXELVAL c)' etc. To manipulate the PIXELVAL
'images'.  Applications like MPEG players would like to avoid using
macros and instead do their own lookups etc. for speed, while some
applications like our own, would use precompiled binary files
in 565 format for buttons etc.

> Go ahead and do it, send me the mods.  Use 0.87pre1 as the
> base.  Like I said above, keep it simple if possible.

OK.  There are som issues with optimizing at the device independent
drawing code/device drawing code, but I'll leave that alone at the
moment.

- Morten

: I guess you mean "inherit" WRT compiling an entire system with
: same setup for both servers, libraries and user apps, right?
: Adjusting the PIXELVAL or packing method should be done globally
: to get the best results (so they match each other all the way).

Yep

: I figured one could solve the problem in a simple manner by introducing
: a new protocol "opcode" GrNumArea_R5G6B5=47 that uses the same struct as
: the original GrArea, but with 16 bit pixels trailing it. 

Well, if you agree that we should inherit PIXELVAL for the entire
system, then a new opcode isn't needed, since PIXELVAL will
be assumed to be 16 bits for all pieces of the software.  Only
in the case where the client wants to send different coded
PIXELVALs is this an issue.  IMHO, if the client is going to
send differently coded color values, then they should be sent
RGB, since we already have conversion code for those cases.

So perhaps, with a PIXELVAL set to unsigned short for you,
this entire problem goes away.  More on this below.

 This reduces
: the memory usage and transfer time.  The pixels needs to be formatted
: at the client to use the 16 bit encoding that this particular version
: of the GrArea call supports, 

Even if you decide that you need the separate opcodes, there's
no need for a GrNumArea_RGB565 (note the name matches 
the PF_XXX naming).  This is because the GdArea() server
function doesn't actually do anything with the data.  Also,
if you want the server to muck with the values passed (say in the
case that you're passing truecolor 565 but the hardware
is running palette mode), then you'd be better off to get the
system palette and convert them yourself once before ever
sending them, as it would be very much faster too.
: 
: If the Nano-X server happens to have 8, 24 or 32 bits/pixel, the 16 bit
: values supplied with this call can still be converted and displayed.
: I would like this to be optional though, to save space when you don't
: need it.  

I agree, but this still means that the engine has to get into the business
of PIXELVAL->PIXELVAL conversion, which I don't think
is a good idea. 

I think I just realized what the real issue is.  GrArea() only takes
PIXELVALs, not COLORVALs.  This is because, originally, the
only purpose of GrArea was to display screen contents that were
previously read by GrReadArea().   But you're using GrArea
to display images.  Now, normally, the Microwindows design
is to use RGB only for color display, since it's portable.  But,
for speed, I agree we need a mechanism to be able to pass
pre-computed PIXELVALs directly to the screen for higher
speed applications.  But the size of PIXELVALs is always
a constant for a particular system.  So, rather than sending
differently sized PIXELVALs across and reinterpreting
them (kindof like our RGB interpretations - slow), we
need a way for the client to build PIXELVALs, and 
then a single GrArea will still work.

So, the GrGetScreenInfo call will return the PF_ mode
in the pixtype member.  The client can use sizeof(PIXELVAL)
to compile in the proper conversion functions.  I suggest that
a small set of client-side functions be created, along with
some macros like you suggest below, that allow PIXELVALs
to be created on the client side, and then leave the GrArea
stuff as is, with the exception that of course an unsigned short
PIXELVAL be creatable at compiletime through some define.

: As to the entery point in the Nano-X client library, it could be done
: like:
: 
:  #define NANOX_PF  PF_TRUECOLOR565
:  #include <mwin/nano-X.h>
:           ...
:  PIXELVAL *p;   /* sizeof(*p) == 2 */
:           ...
:  GrArea(id,gc,x,y,w,h,p)
:  /* Gets expanded to GrArea_R5G6B5(id,...) by a #define in
:   * nano-X.h depending on the value of NANOX_PF
:   */
: 

Morten - this is quite cool and tricky, but we probably don't need
to use it.




: The NANOX_PF could default to 24 bit colour, to make life easier.
: Also, when doing this, the RGB macro or some similar macro should
: be defined according to the pixel format used,

Yes, I like that.  According to the sizeof(PIXELVAL) in
device.h, it could export some defines that are useful to the
client programmer.

 but I'm not quite
: sure if RGB() is the right macro for this:  Judging from the
: definition of RGB(r,g,b) it returns a COLORVAL value, which is used
: for a couple of things; but is it intended to be used to build
: PIXELVALs for GrArea as well?

Thats right, the RGB macro is not the one, it's only used to build
COLO:RVALs.


: 
: If not, a macro 'RGB2PIXEL(r,g,b)' could be used to do this, along
: with 'PIXEL2GREEN(PIXELVAL c)' etc. To manipulate the PIXELVAL
: 'images'.  Applications like MPEG players would like to avoid using
: macros and instead do their own lookups etc. for speed, while some
: applications like our own, would use precompiled binary files
: in 565 format for buttons etc.

Yes.  This is the set of client-side library code that could be included
in the client library, and unused if not linked in.  Or better yet,
be macros.


: OK.  There are som issues with optimizing at the device independent
: drawing code/device drawing code, but I'll leave that alone at the
: moment.

So, we don't really need to change any devdraw.c code, since it's just
going to copy this stuff out quickly, right?

Go ahead an modify the device.h stuff and add macros there, though.

I hope my comments haven't changed your design too much, but after
thinking about it, it seems that if we inherit PIXELVAL from the server
to the client, then there's a much simpler solution than both of us
were initially thinking...

Regards,

Greg

: If not, a macro 'RGB2PIXEL(r,g,b)' could be used to do this, along
: with 'PIXEL2GREEN(PIXELVAL c)' etc. To manipulate the PIXELVAL
: 'images'.  Applications like MPEG players would like to avoid using
: macros and instead do their own lookups etc. for speed, while some
: applications like our own, would use precompiled binary files
: in 565 format for buttons etc.

Morten,
    In thinking more about your needs and the last email, we could need the
following macros:

RGB2PIXEL(r,g,b)    returns PIXELVAL according to compiletime or runtime PF_XXX?
    It's possible that we might need this to operate at runtime, rather than
compiletime.
For the simpler compiletime case, we can only make it work when running in the
truecolor
modes:
    PF_TRUECOLOR24            (32bpp)
    PF_TRUECOLOR565          (16bpp)
    PF_TRUECOLOR332          (8bpp)
For PF_PALETTE, we're kinda in trouble, unless we're running a fixed palette.

So to solve this at compiletime (the fastest), I suggest the following macros:

RGB2PIXEL24(r,g,b)        convert RGB to 32bpp 24bit truecolor
RGB2PIXEL565(r,g.b)      convert RGB to 16bpp 5/6/5 truecolor
RGB2PIXEL332(r,g,b)      convert RGB to 8bpp 3/3/2 truecolor

For runtime conversion, a single RGBToPixel() routine could run on the client
side and statically get the PF_XXX pixtype thru GrGetScreenInfo once, and then
calculate a PIXELVAL from the passed parameters.  It could work in PF_PALETTE
mode by calling GrGetSystemPaletteEntries.  This is slower, but can work
regardless
of the server's PIXELVAL size.  (But probably isn't needed by you)

For reading PIXELVALs, the following macros could also be needed, again
depending on the type of PIXELVAL:


PIXEL24GREEN(pixelval)        returns the 8 bit green component of 24bpp
pixelval
PIXEL565GREEN(pixelval)      returns the 6 bit green component of 16bpp pixelval
PIXEL332GREEN(pixelval)      returns the 3 bit green component of 8bpp pixelval
and likewise for RED and BLUE.

For runtime conversion, the function
PixelToGreen(PIXELVAL pixelval) could be used and kept in the client library.
Again, it could get the system palette if running in palette, rather than
truecolor modes.

With the above design, a Nano-X applications programmer has two choices in which
to write his application:  (1) assume knowledge of the PIXELVAL implementation,
in
which case he uses the macros, and the application only runs on that
implementation,
or he can use the runtime functions, which are slower, and the application will
run on
any implementation.

Regards,

Greg

Greg Haerr wrote:
> 
> Yep
> 
> : I figured one could solve the problem in a simple manner by introducing
> : a new protocol "opcode" GrNumArea_R5G6B5=47 that uses the same struct as
> : the original GrArea, but with 16 bit pixels trailing it.
> 
> Well, if you agree that we should inherit PIXELVAL for the entire
> system, then a new opcode isn't needed, since PIXELVAL will
> be assumed to be 16 bits for all pieces of the software.

Yes, this is true. But I don't like the concept of *assuming* that
pixles are 16 bits - if they are not, we run into trouble.  The
reasoning behind my design was; make it possible to make it possible
(later) to write apps and have them run everywhere, if the Nano-X
server is compiled with the proper settings, without modifying
the clients.

I like the concept of writing an app using the color scheme best
suited for the application, not the other way around.  This is one
of the things I don't like with X11...

Obviously, pseudocolor will probably need special attention, but
it can be done.  There will always be a system palette of sorts, and
most often it will contain some sort of rainbow we can use a lookup
table to translate into (for ugly, but working results).

But, I'm not aiming as high as it may have seemed, not initially
anyway.  Se below.

> Only
> in the case where the client wants to send different coded
> PIXELVALs is this an issue.  IMHO, if the client is going to
> send differently coded color values, then they should be sent
> RGB, since we already have conversion code for those cases.

This will require more code in the client: if 16 bit; use that
for speed, else use RGB.  This is the stuff that non-portable
programs are made of.  NOTE: This is a statement in general
terms, my use will be fine with just about any solution.
But I'm concerned with the general usefullness if we introduce
too many hard-coded operational modes that are incompatible.

> Even if you decide that you need the separate opcodes, there's
> no need for a GrNumArea_RGB565 (note the name matches
> the PF_XXX naming).  This is because the GdArea() server
> function doesn't actually do anything with the data.

Well, not if the server uses the same definition for PIXELVAL,
but this is a fact I don't trust, and having a Nano-X server
not being able to detect an unsupported condition is bad IMHO.
To be able to detect unsupported use, extra opcodes are
needed, and a stub function to return, or report, an error for
the unsupported function.

This also brings up another issue; optimizing.  From a quick
read of the GdArea code, it seems like a good candidate for
optimizing.  Depending on what the underlying hardware or
graphics API provides WRT area painting, the GdArea function
may have to have special cases depending on the features of
the underlying device drawing routines.  Once you start to
optimize, merging layers of code has a very good payoff,
although it may not be as elegant.  This may require different
versions of the Area code one way or another, but this is
for later study.

> : If the Nano-X server happens to have 8, 24 or 32 bits/pixel, the 16 bit
> : values supplied with this call can still be converted and displayed.
> : I would like this to be optional though, to save space when you don't
> : need it.
> 
> I agree, but this still means that the engine has to get into the business
> of PIXELVAL->PIXELVAL conversion, which I don't think
> is a good idea.

Me neither, and I would probably not code any on the conversion
code as we will not need it, but it would be a good idea to be
able to include the conversion code at a later stage to support
old applications written to run fast on 8 or 16 bit displays etc.
Allocating more opcodes is a free for all; it will not automatically
result in a bigger Nano-X server (but possibly a useless one
for certain apps, which is unavoidable anyway).

> I think I just realized what the real issue is.  GrArea() only takes
> PIXELVALs, not COLORVALs.  This is because, originally, the
> only purpose of GrArea was to display screen contents that were
> previously read by GrReadArea().   But you're using GrArea
> to display images.  Now, normally, the Microwindows design
> is to use RGB only for color display, since it's portable.  But,
> for speed, I agree we need a mechanism to be able to pass
> pre-computed PIXELVALs directly to the screen for higher
> speed applications.  But the size of PIXELVALs is always
> a constant for a particular system.

I don't see why the PIXELVAL *has* to be the same in server
and application.  In my experimental code I have typedefs for
pixelval_r8g8b8_t and pixelval_r5g6b5_t, which are used for
the standard GrArea and GrArea_R5G6B5 functions respectively
(in the client API).  Depending on the NANOX_PF, the PIXELVAL
will be defined to the corresponding pixelval_*_t along with
the #define for GrArea.

My idea was to make sure the PIXELVALs are the same, and not
do any conversion (for our app) because there will not be any
code for it. But by tagging the protocol requests makes it
possible to

   1) Provide "unsupported" log messages rather than
      undefined behavior when mixing apps and server
      with differing pixelsizes.
   2) Include conversion code at a later stage if it is needed.
      The conversion code should always be configurable to
      save space where not needed.

> So, rather than sending
> differently sized PIXELVALs across and reinterpreting
> them (kindof like our RGB interpretations - slow), we
> need a way for the client to build PIXELVALs, and
> then a single GrArea will still work.

Hmm, I don't like this much.  In order to make things fast,
we have to use macros, which will not adapt to a new
environment (needs recompilation to match server), or
support multiple setups in the same application.

> : OK.  There are som issues with optimizing at the device independent
> : drawing code/device drawing code, but I'll leave that alone at the
> : moment.
> 
> So, we don't really need to change any devdraw.c code, since it's just
> going to copy this stuff out quickly, right?

If introducing new opcodes (which I think is a must), we need some
way of differentiating the various opcodes so we can reject those that
we don't support.

> Go ahead an modify the device.h stuff and add macros there, though.
> 
> I hope my comments haven't changed your design too much, but after
> thinking about it, it seems that if we inherit PIXELVAL from the server
> to the client, then there's a much simpler solution than both of us
> were initially thinking...

I guess my problem is that I don't want the client to inherit the
PIXELVAL in order for things to work, just to work fast.  But I
definitely don't want the translation code in *my* server, but that
shouldn't exclude the possibility for someone else to have it.

For this reason, I would really like GrArea_R5G6B5 to be defined to
have a solid fundament that can possibly support multiple pixel
formats (think shared library and two apps with different desires
for the pixel format).

This got way long... sorry about that.

Regards,
- Morten

: Yes, this is true. But I don't like the concept of *assuming* that
: pixles are 16 bits - if they are not, we run into trouble.  The
: reasoning behind my design was; make it possible to make it possible
: (later) to write apps and have them run everywhere, if the Nano-X
: server is compiled with the proper settings, without modifying
: the clients.

That was the whole idea behind using RGB on clients.  The clients
are compiled once, and run on any server, truecolor and palettized.
The apps are written in a standard color model and will guarantee
be runnable on any server.
What you are saying is that you want an additional, non-RGB
format, that also is client-portable, but is closer to the hardware,
so that it's faster.  Because of this, the client has to include
alot more code, since it needs to be able to run on palettized
systems also, not just fast 16bpp truecolor.  I think that
having a mechanism to allow a client to know about the
hardware color format and run quickly is a good idea,
but having two supposedly client-portable formats is
probably a bad idea, considering this is supposed to be a small
project.

However, I don't think this is the whole problem.  The
real issue is that GrArea() only takes PIXELVALs, and I 
don't like the idea that portable programs should have to know
about the hardware color format.  (yes, in your case, they might
want to, but don't have to).  Now, the Nano-X GrArea() is a
holdover from the original mini-X api, and wasn't modified
when I added full truecolor and pseudocolor support for the 
project.  It's original use was just to allow a client
to copy screen areas by calling GrReadArea, and then
GrArea().

I propose that the first solution be that GrAreaXXX() be allowed
to be passed COLORVALs, so that small images, and most
drawing be allowed to be standardized, and passed as RGB.
In this way, any image can be displayed in a standard
format from Nano-X on any server.

In addition, we implement 3 more GrAreaXXX's, one for 8, 16, 
and 32bpp PIXELVAL sizes.  These additional
opcodes can be used, _only_ if the SCREENINFO indicates
that the hardware supports it.  [I still don't see the reasoning
behind the need for multiple 16bpp GrArea's.  Since the
server doesn't modify the data, can you please be more
explicit as to the need?] 

[Also, we don't want to query
the opcodes, since, with the new client/server rewrite,
almost all opcodes return void, so that they can be buffered
and not have to wait for a return value]

: > Even if you decide that you need the separate opcodes, there's
: > no need for a GrNumArea_RGB565 (note the name matches
: > the PF_XXX naming).  This is because the GdArea() server
: > function doesn't actually do anything with the data.
: 
: Well, not if the server uses the same definition for PIXELVAL,
: but this is a fact I don't trust, and having a Nano-X server
: not being able to detect an unsupported condition is bad IMHO.
: To be able to detect unsupported use, extra opcodes are
: needed, and a stub function to return, or report, an error for
: the unsupported function.

Like I mentioned above, we definitely want to use SCREENINFO
for this, and not change the type of server procedure to return
something other than void.


: 
: This also brings up another issue; optimizing.  From a quick
: read of the GdArea code, it seems like a good candidate for
: optimizing.  Depending on what the underlying hardware or
: graphics API provides WRT area painting, the GdArea function
: may have to have special cases depending on the features of
: the underlying device drawing routines. 

Certainly.  The optimized routines are called GdBitBlt(), which
was written for very high speed.  GrArea was a holdover,
and almost tries to do the same thing, except it doesn't
support offscreen drawables. (Nano-X doesn't support offscreen
drawing yet, I was planning on adding all this later)


What did you think about my previous email?  The one that
described the proposed client macros?

Regards,

Greg