Finding Opcodes
I have been working the past week to try to understand the processes of finding opcodes in an attempt to help fill out some missing ones for Titanium. And, to see if it would be possible to find the Anniversary opcodes without having packet collects from Live when Anniversary was being used.
The wiki pages we have on finding opcodes are far from complete and don't give quite as much information as I would like. Opcode Questions: 1. Are all opcodes 2-way? What I mean is; are the opcodes that the client sends to the server the same ones that the server sends back to the client? I need to know if there is something we actually MUST have live packet collects for, or if everything we need should be somewhere in the client itself. If everything is in the client, then maybe we can find them without using Live packet collects. 2. Do all opcodes change with every release of the client, or do some stay the same? Did SoE purposely change them so that it was hard for programs like MQ, Showeq, and EQEmu to keep up with live? Or is it the source that changes these automatically? 3. What would it take to get the packet collector updated so that it could at least collect opcodes as they came in? Does anyone have the source for any of the Packet Collectors? I imagine that if the packet collector has some logic or filters to pull opcodes from packet sniffs, then even if the packet collector itself would be too much work to get updated, maybe we could use another packet sniffer like Wireshark and just put a filter on it to only pull opcode packets. Using Wireshark, I can definitely see the opcodes, but without already knowing what you are looking for, it would be near impossible to get the ones that are still unknown. My Tests So far: Here is what I did to start trying to find them using Wireshark. Note that this is a learning process for me, so I used one that we are already aware of (Rewind - 0x4cfa): I loaded up Wireshark and ran EQ (connecting to my EQEmu Server, NOT Live) and then I set a couple of filters just to keep unnecessary traffic from showing up. Mainly, I set it to only show UDP packets and also only anything from 192.168.1.101 or to 192.168.1.100, which are my server and client IPs on my LAN. This blocked out most other web or whatever traffic so I was only seeing client to server packets. Then, I got EQ ready by creating a hotkey for /rewind. At this point, I started the sniffer collecting packets, and immediately switched back to EQ and spammed the crap out of my /rewind hotkey for about 30 secs. Then, I stopped the packet sniff so I could analyze. What I found was that the client was sending alot of packets to the server that had the opcode in them, but the opcode was backwards with the FA first and then the 4C, so it was showing up as "FA4C". But, this code wasn't just standing alone, it was mixed in with packets that were mostly 17 in size, but sometimes varied to be even bigger. For the most part, the FA4C showed up right before the last 4 characters in the data of the packet. So, if the packet was (not a real example), "FB 04 16 00 A9 90 00 20 B8 09 C2 85 3B FA 4C 8B 20", then the text in bold is where the opcode normally was (but not every time). More Notes: I can't seem to get the emu collector to run on my PC, even for Titanium. But, it seems to be a problem with it not finding my NIC card for some reason, and not the version I am trying to collect from. But, I think that if I could get it running, maybe it would show all of the missing opcodes we need just by sniffing the communication from emu server to emu client. So, the emu collector packet sniffers might not need to be updated very much to get to the point that they are needed. Though, it would probably be more work to get them updated to sniff Anniversary. Another thing I noticed is that some of the log files actually make note of opcodes, and I think even ones that are unknown. So, the source must have some way to convert these so they are easily readable. Maybe that part of the source could be used to help develop a working opcode packet collector. Idea to get Anniversary Edition working: My idea for Anniversary is that if we had a working sniffer that can see the opcodes and pull them out so they are easy to find, then we could use a step-by-step process to get anniversary working. By that, I mean that we could start an Anniversary client while already running a packet sniff and then collect the opcodes as they come in. It will get to a point where it fails, and we just find the opcode before that and figure out which opcode it is and add it to the .conf file and then start again. Basically just trying over and over until it gets logged in. Then, it is just figuring out the commands and what not, which should probably be much quicker since you shouldn't have to restart every time it fails. Another possibility would maybe be using IDA Pro to check the code of the client for opcodes listed. But, so far I have been completely unsuccessful in finding even a single known opcode in there. And, I have done quite a lot of searching and research. Though, my understanding of disassembly files is still very weak. If I could figure out how to get opcodes this way, I think it would make this a piece of cake. Some references I used when researching how to find opcodes: Method for hacking from the hackersquest forums that has some good disassembly information: http://hackersquest.org/boards/viewt...ghlight=opcode Page in the wiki for finding Opcodes, that I can't really make much sense from at all: http://www.eqemulator.net/wiki/wikka...vOpcodeFinding ASM Hunting: http://www.eqemulator.net/wiki/wikka...DevAsmRoutines Assembly Tools and Information Links: http://www.eqemulator.net/wiki/wikka...ka=DevAsmTools HexVis - A little tool for converting hex or other data to different types of output: http://www.eqemulator.net/wiki/wikka.php?wakka=HexVis To convert Decimal to Hex, you can just use windows Calculator and set it to Scientific mode. Then, put the number you want to convert and change the radio button to hex and it automatically converts it. Also, the eqstr_us.txt and eqls_us.txt both have some information in them that may relate to opcodes and could help in defining what the opcode itself relates to in the code. The hackersquest post above mentions the lines in these files. Though, I haven't been able to find their related hex versions in the ASM of the eqgame file. If anyone out there has info that might help this process, please respond here or PM me. I am sure we could all appreciate a working Anniversary edition, and it would be worth the work to get to that point. Once we find a good procedure for it, I don't think it will be too bad to get these. But, for now, I am still trying to refine a way to actually gather the opcodes without having to do too much guessing. |
After doing a bit more research on Anniversary Edition, it seems like we might be almost too late to use it to put a lot of work in getting an updated version of the emu working. It seems like copies are still easier to get than Platinum, but apparently many stores have already stopped selling the Anniversary Edition and it can even be a little hard to find any online. I am sure it will get even harder soon.
Here are some links to places that should normally be easy to get games from: http://www.newegg.com/Product/Produc...82E16832135022 http://www.ebgames.com/Catalog/Produ...spx?sku=647003 http://www.amazon.com/Everquest-I-Th...4879400&sr=8-1 As you can see, it appears that the games aren't being shipped much if at all anymore. It seems like they are just trying to sell off remaining stock and that will be it. I don't quite understand how SoE expects people to get their game if it isn't on shelves or even available to buy the CDs online. I don't suspect that everyone would want to just get a downloaded game file to play it. The only way I see to reliably get the game is online from SoE directly as a download only. But, we would need to decide on the version to find opcodes for so everyone could get it. https://store.station.sony.com/ Click on Everquest under the Digital Download section on the lower left. Then, you have to login and you should be able to click cancel on the subscription section. That will take you right to the digital downloads. I am wondering if those are just standard unpatched downloads, which I imagine they are. If so, then for 20$ we could have the option to go with the ""Starter Pack" which seems to be the exact same thing as Anniversary Edition. Or for 40$, we could go with "The Serpent Spine All-In-One" package that includes every expansion currently out. If I can figure out how to find opcodes well enough, I will either just get a copy of anniversary, or maybe one of these download packages and start working on finding the opcodes right away. Though, I would prefer an actual CD pack so we don't have to worry about SoE changing the opcodes by changing the versions of the download files. Maybe they will have another CD release soon :P |
From what I recall, vaguely, they downloads are, or at least used to be, unpatched versions. They don't update the downloads.
|
We use a protocol similar to the one used in eq2 I believe though it really isn't documented anywhere outside of the code. It's kind of difficult to put together via the code because of the layers of abstraction on abstraction. It would just be a matter of decoding the packets further to get them to load from something like WireShark.
Usually the opcodes change from exe version to version though some don't; sony does seem to like to change a lot of the ingame ones every patch; probably to hinder stuff like showeq. Usually if the client sends a packet and expects a direct response packet for it they will use the same opcode; example would be pickpocket or click object. It's a shame nothing ever came of openeq and we still have to deal with this. =/ |
Heh KLS... this sounds hauntingly familiar.
|
Sir, I thought I was leaving you alone with this!
Anyway basically the packets are encrypted or compressed, there's a flag on the session request for the stream that indicates which it will be; I believe it can be both as well. The server generates a key to send to the client to decode the data as well. You have larger packets that make up the protocol, things like session requests, whole packets and combined packets etc. Contained within can be the smaller packets, encoded with the key sent from the server etc. I think we send the key always as 11223344 but obviously were you to do this from live or something you would need to retrieve the actual one to decode the data into readable packets. in common/EQPacket.cpp there are several functions that deal with compression and encoding of the data as well. |
Thanks for the info KLS, it seems like it has been a while since anyone has considered taking on opcodes and I wouldn't mind giving it a try as long as I can find a viable option for it.
Here are some code sections from the source common directory that just skimming through caught my eye as being related to helping find opcodes: packetfile.h - Guessing this holds the string you were mentioning that the server sends the client, maybe? Code:
#ifndef PACKET_FILE_H Code:
#ifndef _EQPACKET_H Code:
#include "debug.h" Code:
#copy this file into your server's dir Code:
#!/usr/bin/perl Code:
#!/usr/bin/perl Code:
#ifndef LOG_CATEGORY It would also be nice if that was working and you could somehow filter out all of the known opcodes, so only ones you are looking for (unknowns) will show up to make finding them much easier and quicker. Then, once we were able to get enough opcodes to get a client logged all of the way in, the rest would probably be pretty easy. |
Hmm, I may have answered my own question about using the logs to find opcodes. To watch them as they come in, simply open a terminal window in Linux and type:
Code:
cd /home/eqemu/server/logs Unfortunately my other PC is windows only, and I don't know how to do a tail in windows so I can grep for OpCodes only. So, I may have to build my other PC as dual boot to Debian Linux. Unless I can figure out how to run another server from the same PC I run my main server on. So, I will have 1 that only I can use. Maybe if I make another directory and set the config to use different zone ports... I dunno, but I don't think it would talk right on port 9000 with both server sessions wanting to use the same port to communicate with the login server. |
Looks like I might be able to do this on my server even with players on it after-all. This tail shows the name of the char that sent the opcode, so if I am testing, I will just look for my own.
It has already found a few unknown opcodes from my players and these seem to be somewhat common. OP_Unknowns: 0x6a5f 0x45ff 0x7085 0x3b21 0x1241 I will do some testing tonight to see if I can force it to produce more opcodes and hopefully get some of the unknowns defined. I will let you all know of any progress I make. If this actually works, it is far easier than I could have ever expected lol. Maybe I was thinking too hard, but I imagine that if it was this easy, someone else would have been using this method already... |
I took a quick peek into the source, and it looks like this is where those OpCode errors you found are spit out:
zone/client_packet.cpp: Code:
375 case CLIENT_CONNECTED: { Code:
337 #if EQDEBUG >= 9 Code:
15 DFLAGS=-DEQDEBUG=5 -DCATCH_CRASH -DNO_PIDLOG -DSHAREMEM -DSPELL_EFFECT_SPAM -DFIELD_ITEMS -DCOMBINED -DAPP_OPCODE_SIZE=2 -Di386 |
I put some code into the server just to see how the code works further. To analyze live packets I believe you would need something to analyze the entire stream; or at least the session request since right now it appears the client receives a key from the server used to encode and decode.
Right now the emu always sends back: 287454020 which is (0x11223344), I actually added some logging to see what it was since the code has no commenting and can be kinda hard to follow at times, so I guess I got that part right the first time. Basically: Client creates protocol packet of OP_SessionRequest Code:
pragma pack(1) Code:
pragma pack(1) Client <- OP_SessionResponce <- Server the uint32 key is what we use to decode the packet if the flag for encoding (0x04) is set in format, and clearly the session would be the session id, format is the bitfield to store the encode and compression flags for the stream. I'm still trying to understand it myself though, wtb code commenting -.-. Oh yeah, also if we're trying to find a client -> server opcode that's pretty simple because you can just have it dump to log or terminal over the emu as stated above but a lot of the ops we're missing aren't client -> server and the ones we are is because the functionality hasn't been implemented and if it were finding the opcodes wouldn't be an issue obviously. |
Here is what I have found so far just from running the tail I mentioned above. I am sure some things I have tested must be sending opcodes in another format or something, because they don't seem to send the server anything even though I think they would be unknowns. One example would be when you right click an adventure merchant. Unless the opcode is just assigned to the wrong OP_ name in the .conf file for it already. Then, it wouldn't be getting logged by my tail, because it is currently only able to see opcodes that error for one reason or another.
Errors (unhandled): 0x381d - OP_WeaponUnequip2 - unequip weapon 2 or unequip weapon 1 0x63da - OP_WeaponEquip2 - equip weapon 2 0x6c5e - OP_WeaponEquip1 - unequip weapon 1 0x63da - OP_WeaponEquip2 - equip weapon 1 0x6c5e - OP_WeaponEquip1 - equip weapon 1 0x6f0c - OP_Bandolier - Added and named new bandolier name - same when bandolier is used to swap weapons 0x1ee9 - OP_BazaarSearch - /Bazaar Search or clicking "welcome" on the /bazaar window 0x5891 - OP_RaidInvite - Invited a player with /raid - got some packet info as well 0x6f82 - OP_LFPCommand - Shown on the player when they accepted group invite 0x7f9d - OP_Report - /report playername 0x5306 - OP_Feedback - /feedback and filled out a feedback and hit send. Got some packet as well I know that unhandled doesn't mean there are problems. But, in some cases, it might be useful to have the notes I made just in case someone decides to write some code to handle them. Unknowns: 0x524e - Begin /trader mode or Closed /trader window even if Begin Trader mode hadn't been started (on off toggle?) 0x6a5f - Comes in after not moving the mouse for exactly 15 minutes. And comes in again as soon as the mouse is moved again - Toggle Auto-AFK? 0x19d8 - /viewpetition command ran 0x5fc7 - Clicked "View Stats" on the Adventure Window - Same for clicking the Refresh button there 0x230a - Clicked "Leaderboard" on the View Stats Window for LDoN Adventures - Same for clicking the Refresh button there 0x48fe - Used the "Who" button in the friends window - Same for typing /who all friends or friend 0x224c - /zone command 0x35e8 - Right click a player in /becomenpc mode in bazaar (basically trader mode on) 0x3d05 - /veteranReward 0x5eba - Used Shift+T to open the Titles window And some of these probably aren't considered issues, because there are no systems in place for them. But, I am sure someone would love to get bazaar working, and Titles, and the petition manager window would be nice too. Also, I didn't even know EQ had an auto-afk feature, but that one opcode really seems to be auto AFK. It comes in exactly 15 minutes every time after the mouse hasn't moved on that window and comes in again as soon as it is moved again. That would be a cool feature to have IMO. Notes: Starting trader mode by opening the /trader window and setting items for sale in a Trader's Satchel didn't make my character show up on the /bazaar search window to be listed under "Traders". BUT, using /becomenpc on myself changes my name to add "Trader" in front of it the way that trader mode is supposed to on Live. Then, once I had /becomenpc on, I now showed up as a trader in the /bazaar search window. It still wouldn't list the items I had for sale from the search results though. I will continue looking for more opcodes and I am going to mess around with other versions of EQ as well to see how difficult it would be to get them working for the emu. If most of the opcodes to get another version of EQ working can be attained with the tail I have been running, then it really shouldn't be all too tough. I still need to check into those IDA Pro scripts as well to see what they do and if they are still useful in any way. I guess some progress is better than nothing. Might as well get as many opcodes in place as possible, so if someone wants to write code to get the systems working, they can do so without worrying about opcodes. |
the SWGemu people have a pretty decent writeup of the basic UDP transmission protocol sony uses for apparently all their MMOGs. Including a section on encryption, definitely easier to understand the code after reading it. I think it may differ in the footer a bit from the protocol sony uses in EQ, but it's similar enough to give a good idea of what's going on.
http://trac2.assembla.com/swgemu/wiki/Packets |
It's really really close actually, I wrote a small collector today that can identify separate streams and reveal their relevant information. Can't decode packets yet but hey that stuff's hard for an afternoon of work. Trying to devise a system for combined packets doesn't sound very fun at least since they can be combined as many times as the client wants apparently.
|
Unless I am mistaken, it looks like any change to the .conf files requires a server reset to take effect. Or, is there a way to do it via command or something without a complete reload? That would help quicken things up. Especially since I am testing on a live server with lots of players.
For anyone wondering if the Digital Downloads from SoE are a packaged set of files, I wanted to clarify that they are NOT. I went ahead and purchased the 20$ digital download "Starter Pack", but it didn't give me any actual link to download a package. The only way I was able to get the new files was to use the new beta sony patcher, which unfortunately patches to live. So, it looks like we are currently stuck with Anniversary Edition as being our only option for upgrading the emu unless we want to chase live, which is probably very unlikely. It is unfortunate that the Anniversary Edition is already almost as hard to get as Titanium. Very few stores still have copies on the shelf and many don't even sell it online either. Though, if someone wanted a copy, I don't think it would be too hard to get just yet. And, thanks KLS, for working on this. If you can come up with something, I am sure it would make a big difference and help the project to keep moving forward. I am still investigating the opcodes as well, but your knowledge and experience far exceeds my own lol. |
All times are GMT -4. The time now is 03:10 AM. |
Powered by vBulletin®, Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.