Go Back   EQEmulator Home > EQEmulator Forums > Development > Development::Development

Development::Development Forum for development topics and for those interested in EQEMu development. (Not a support forum)

Reply
 
Thread Tools Display Modes
  #1  
Old 06-10-2009, 02:58 AM
gaeorn
Developer
 
Join Date: Apr 2009
Location: USA
Posts: 478
Default Improving 64 bit Linux support

I know that a relative few of us actually use EQEmu under 64 bit linux. However, I was thinking it would be great if we could integrate changes for 64 bit into the codebase without breaking anything for everyone else.

I'm not sure the best method to accomplish this so I'm requesting ideas. I have a number of changes that I have made to correct either compile time or runtime behavior of the server under 64 bit linux. Perhaps some of the changes would need minor tweaking to provide maximum compatibility.

I know the developers have a lot to do already, so I want to make this as easy as possible for them. However, I'm sure it would be greatly appreciated by those who do use 64 bit linux. And 64 bit systems will only grow in prevalence, so perhaps dealing with this now would be a good idea.

So, please reply with suggestions/ideas/flames/whatever. Thanks.
Reply With Quote
  #2  
Old 06-10-2009, 03:10 AM
KLS
Administrator
 
Join Date: Sep 2006
Posts: 1,348
Default

Fine with me so long as it doesn't break things for everyone else.
Reply With Quote
  #3  
Old 06-10-2009, 11:23 AM
gaeorn
Developer
 
Join Date: Apr 2009
Location: USA
Posts: 478
Default

Do you, or anyone else, have a suggestion about how to do this to reduce the chances of breaking things for 32bit systems?

I personally do not have a 32bit system to test on. I do have access to a 32bit system at work that I can compile on, but can not use that system to actually run the server. I can then run the 32bit binaries on one of my 64bit systems, but that still leaves room for differences in system behavior. So my ability to test in a 32bit environment is very limited.

If this ends up being too much trouble for the few people that are using 64bit at this time, I am perfectly willing to wait until more people need this.
Reply With Quote
  #4  
Old 06-10-2009, 12:54 PM
pfyon's Avatar
pfyon
Discordant
 
Join Date: Mar 2009
Location: Ottawa
Posts: 495
Default

Aren't the problems occurring because of poor programming? I don't really know enough about why 64 bit systems cause problems to comment, but I'd thought the problems were occurring during casting of values?
Reply With Quote
  #5  
Old 06-10-2009, 01:54 PM
gaeorn
Developer
 
Join Date: Apr 2009
Location: USA
Posts: 478
Default

Casting values is only part of the problem. I've also run into different behavior in some of the libraries (not preserving argument values for example) and even ran into a g++ optimization bug that only appears to affect 64bit systems. And I'm sure there are more bugs to be found.
Reply With Quote
  #6  
Old 06-10-2009, 02:03 PM
Bulle
Hill Giant
 
Join Date: Jan 2008
Posts: 102
Default

It would be unfair to say it is poor programming not to have "natural" 64-bits support. 64-bits support when programming on a 32-bits platform has to be seriously planned in advance, and even then it requires corrections once you switch to the actual 64-bits platform. It is an outstanding effort to support both, C is like that, close to the "machine". If you want hardware indepence, use Java or something else. EQEmu is in C, we have to live with that.

Problems do not only occur when casting values, but for example when reading binary data from files (ints are not the same size). Cross-processor compatibility support (like in PowerPC vs Intel) could mitigate this problem, but EQEmu on Mac is not really something I have heard of. We have to live with that too !

That said, I will answer the question of gaeorn :
The surefire way to ensure you do not alter 32-bits code when adding changes for 64-bits support is to use #defines. Everytime you need to replace an existing portion of code with your new code you use something like this :
Code:
#ifdef     __64_bits__
<your new code>
#else  /* __64_bits__ */
<the old code>
#endif /* __64_bits__ */
The comments are there for readability (especially if you have tens of lines of code in there), and cost nothing.

Obviously you also have to #define the __64_bits__ macro somewhere when you are on a 64-bits plaform. It can be done on the compile-line with a -D__64_bits__ flag (read : your Makefile). There could be an existing flag for your compile platform but I cannot remember it off-hand. If it exists then just use it instead of my made-up macro.

I can remember trying to compile EQEmu for 64-bits a year ago (and succeeding), but running the executables failed miserably with a segmentation fault. Not a surprise really, as the code is not meant to support it. Be warned this can be very tough to fix.

One hint : on a 64-bits Linux platform you can always create a chroot 32-bits installation to run your programs with everything 32-bits. That is how I use EQEmu on my (native 64-bits) server. The details vary depending on your distribution, I am on Gentoo and they provide a nice "how-to" on how to create the 32-bits chroot environment. Just know that you will need as much HD space as you use on your main Linux root to support it. Yes, it works fine.

Good luck
Reply With Quote
  #7  
Old 06-10-2009, 02:30 PM
gaeorn
Developer
 
Join Date: Apr 2009
Location: USA
Posts: 478
Default

Quote:
Originally Posted by Bulle View Post
It would be unfair to say it is poor programming not to have "natural" 64-bits support. 64-bits support when programming on a 32-bits platform has to be seriously planned in advance, and even then it requires corrections once you switch to the actual 64-bits platform. It is an outstanding effort to support both, C is like that, close to the "machine". If you want hardware indepence, use Java or something else. EQEmu is in C, we have to live with that.
Especially when you consider that development started on this before 64bit microprocessors were available to most people.

Quote:
Problems do not only occur when casting values, but for example when reading binary data from files (ints are not the same size). Cross-processor compatibility support (like in PowerPC vs Intel) could mitigate this problem, but EQEmu on Mac is not really something I have heard of. We have to live with that too !

That said, I will answer the question of gaeorn :
The surefire way to ensure you do not alter 32-bits code when adding changes for 64-bits support is to use #defines. Everytime you need to replace an existing portion of code with your new code you use something like this :
Code:
#ifdef     __64_bits__
<your new code>
#else  /* __64_bits__ */
<the old code>
#endif /* __64_bits__ */
The comments are there for readability (especially if you have tens of lines of code in there), and cost nothing.

Obviously you also have to #define the __64_bits__ macro somewhere when you are on a 64-bits plaform. It can be done on the compile-line with a -D__64_bits__ flag (read : your Makefile). There could be an existing flag for your compile platform but I cannot remember it off-hand. If it exists then just use it instead of my made-up macro.
Yeah, I know about #ifdef. I just was hoping to keep that to a minimum. I'm sure many parts of the code could be written in a way to run happily on both 32bit and 64bit. Only a handful will likely require wrapping with #ifdef blocks for the two architectures.

But, if the developers would prefer me to do all of it as #ifdef blocks, I can. Perhaps KLS or another developer can chime in with their preference.

Quote:
I can remember trying to compile EQEmu for 64-bits a year ago (and succeeding), but running the executables failed miserably with a segmentation fault. Not a surprise really, as the code is not meant to support it. Be warned this can be very tough to fix.
I ran into the same problem. Found it and resolved it. I've done enough changes to my 64bit version that it is very playable in my testing. However, it doesn't have a user population to truly test out everything.

Quote:
One hint : on a 64-bits Linux platform you can always create a chroot 32-bits installation to run your programs with everything 32-bits. That is how I use EQEmu on my (native 64-bits) server. The details vary depending on your distribution, I am on Gentoo and they provide a nice "how-to" on how to create the 32-bits chroot environment. Just know that you will need as much HD space as you use on your main Linux root to support it. Yes, it works fine.

Good luck
I use Fedora and it will very cleanly run 32bit binaries without a chroot. Fedora has split out the library paths to /lib and /lib64 (same for others like /usr/lib). The problem is it doesn't make a clean build environment for 32bit. But I have built a 32bit version on a system I have at work. It ran fine, but was noticeably slower performance, much more so than I would have expected. So I prefer to stick with 64bit and fix it. Besides, its the "right" way to do it, lol.
Reply With Quote
  #8  
Old 06-10-2009, 02:42 PM
Shendare
Dragon
 
Join Date: Apr 2009
Location: California
Posts: 814
Default

My understanding is that most of the problems 64-bit compiling is going to run into are going to be type definition differences. For example, an "int" is 32 bits wide in a 32-bit environment, but 64 bits wide in a 64-bit environment. Pointers are also going to be 64 bits wide instead of 32.

This means that any struct that contains an int is going to be incorrectly sized and misaligned in an environment it wasn't designed for.

The cleanest and easiest way to avoid screwing up your structs and such between 32-bit and 64-bit environments would be to change any references to non-static data types ("short", "int", "long", etc.) to static references.

I believe gcpp uses "int16_t", "int32_t", "int64_t", etc. for its static integer types, defined in <stdint.h> or something. Not really sure.

Visual C++ (the environment I'm accustomed to) uses __int16, __int32, __int64 for its static types.

What I generally do in my own code is define a set of static typedefs for maximum portability:

typedef signed __int32 sint32;
typedef unsigned __int32 uint32;
typedef signed __int16 sint16;
typedef unsigned __int16 uint16;
typedef signed __int8 sint8;
typedef unsigned __int8 uint8;

etc.

Of course, coming from an assembly background, I also tend to use BYTE, WORD, DWORD, and QWORD for the unsigned integer types.

The gist of it is that these variables will retain their original size no matter what environment they're in, 32-bit, 64-bit, 128-bit, whatever.

It may be a simple matter of using #ifdef directives to set the appropriate typedefs for each C++ compiler environment (gcpp, Visual Studio, etc.)...

Code:
#ifdef (a gcpp-only define)
typedef int32_t  sint32;
typedef uint32_t uint32;
#endif
#ifdef (a vcpp-only define, __CLR_VER maybe?)
typedef signed __int32 sint32;
typedef unsigned __int32 uint32;
#endif
Then doing a search-and-replace to change all references to the non-static types into the static types.

Just my 2 coppers.

- Shendare
Reply With Quote
  #9  
Old 06-10-2009, 02:58 PM
KLS
Administrator
 
Join Date: Sep 2006
Posts: 1,348
Default

Personally, if you can avoid ifdef blocks and just make code that works on both platforms that's best. If it's unavoidable then it's unavoidable though. I definitely think the code looks best without 10 million ifdef blocks all over it /glare bot code.

I can't imagine there are too many changes that will break 32 bit compatibility. Maybe a few but we've got enough 32 bit developers to catch it pretty quickly.
Reply With Quote
  #10  
Old 06-10-2009, 03:02 PM
gaeorn
Developer
 
Join Date: Apr 2009
Location: USA
Posts: 478
Default

Thankfully the majority of the code already uses uint32, uint16 and uint8 (if not all of it). The type casting has thus far only been needed for printf formats using %lu when being fed a uint32.
Reply With Quote
  #11  
Old 06-10-2009, 03:19 PM
gaeorn
Developer
 
Join Date: Apr 2009
Location: USA
Posts: 478
Default

Quote:
Originally Posted by KLS View Post
Personally, if you can avoid ifdef blocks and just make code that works on both platforms that's best. If it's unavoidable then it's unavoidable though. I definitely think the code looks best without 10 million ifdef blocks all over it /glare bot code.
That's what I suspected. As long as it works on both, it makes it easier to maintain (not to mention readability). And maybe the new bot code WildcardX is working on won't require the ifdef blocks (crosses fingers).

Quote:
I can't imagine there are too many changes that will break 32 bit compatibility. Maybe a few but we've got enough 32 bit developers to catch it pretty quickly.
I felt this way as well, but since I'm relatively new to the project, I wanted to defer to those who have worked a lot longer on it.

I'll start submitting some diffs in the next day or so.
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

   

All times are GMT -4. The time now is 09:48 AM.


 

Everquest is a registered trademark of Daybreak Game Company LLC.
EQEmulator is not associated or affiliated in any way with Daybreak Game Company LLC.
Except where otherwise noted, this site is licensed under a Creative Commons License.
       
Powered by vBulletin®, Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Template by Bluepearl Design and vBulletin Templates - Ver3.3