Converting Worlds from McRegion to PMAnvil (128 to 256 Height)

Since I am assuming there are thousands of servers out there that must eventually convert their McRegion (PocketMine default prior to 1.0.0) worlds to PMAnvil (PMMP default after 1.0.0) worlds, I wanted to start a discussion on the various methods to do this.

First a quick summary of WHY you would want to do this:
1) PMAnvil is much faster, primarily because it does not need to do on-the-fly conversion from XYZ full columns to a stack of XZY cubes.
2) PMAnvil supports 256 world height vs. 128 for McRegion.
3) PMAnvil is the PMMP default format, so will likely have long term support; and eventually McRegion will be deprecated.

Please post in this thread:
- Working methods of McRegion to PMAnvil conversion with as much detail as possible.
- Problems/solutions you have experienced with conversion.

 

Reference threads:

Solutions!
Awzaw has built a well tested and working JAVA MCRegion to PMAnvil/Anvil converter based on Mojang code

Older Reference Threads
dktapps says Intyre is working on a dedicated world conversion tool.
robske_110 (Tim) says we can convert by hacking PMMP source code to "redirect serialisation functions to dummy providers".
List of all current stable world formats. Just for reference: As entered in pocketmine.yml: mcregion, pmanvil, anvil

Older World Conversion Tools
pmimporter
MConverterUltimate

 

Has anyone installed dktapps C extension? Any performance improvements when using McRegion worlds?

EDIT: I guess not. Per dktapps : "No, the extension will make no difference to McRegion worlds. McRegion needs a conversion from XZY full columns to a stack of XZY cubes. Makes no logical sense, but just blame Mojang."

 

I've got Mojang's MCRegion->Anvil converter working, you can download it here: https://github.com/Awzaw/AnvilConverter

It takes about 5 minutes to convert a 1GB world, and if you record the console output you'll see which file is to blame when there's a problem. So far I've only had a couple of UTFDataFormatExceptions which aren't fatal, although I'm not yet sure what effect those errors might have. I have no idea if there's any hidden problems with this... so please note that I take no responsibility whatsoever for whatever havoc this may wreak on your data. That said, I've tested on several worlds now, and everything looks good; I tried it on a plot world on a production server and everyone loves the extra height, and less lag (see below). So good so far.

You'll need dktapps ChunkUtils PHP extension to see any performance benefits of converting mcregion to anvil, but it's not that hard to install. The next step is to get the converter to convert directly to PMAnvil, any help with that would be welcome.

 

Thank you Awzaw! This is a great start; as I believe the only difference between Anvil and PMAnvil is the XYZ to XZY conversion. So other than this performance hit; Anvil should be as much a "first class citizen" as PMAnvil.

Thank you for taking the time to tweak, hack, and package up this code.

I am currently converting a 25GB world using this method, and will report back on the results; including any errors given. If this is a good solution; then we just need the final conversion step in this code to have a complete MCRegion to PMAnvil converter. But as you said; performance even with Anvil should be far superior to MCRegion assuming ChunkUtils is installed (which is quite easy to do with the current php build scripts thanks to dktapps).

 

It seems to be entity nametags that are producing the errors, and I'm not sure that chunks that give that error won't be corrupted... But maybe a small price to pay? I'll look into it in more detail when I get the time, meanwhile reordering is not looking too hard, so it shouldn't be too long before this can convert to Anvil or PMAnvil.

 

The 25GB world converted without problems and several players are testing it now.

So far we aren't detecting anything "weird". Items seem to be in chests where they should be, not finding missing chunks, etc.

EDIT: See reports of door issues and replacement of blocks with extended IDs below.

Any thoughts on a good way to confirm the ChunkUtils extension is really working? I compiled php with the current script and -u; so I believe it is working.

 

If the chunkutils extension shows up when you list the PHP modules with php -m then I think you can assume it's working.

I've added conversion from MCRegion directly to PMAnvil, and it all seems to work nicely although conversion is even slower than converting to Anvil, of course. Again, I take no responsibility... and if anyone can help speed it up, or fix any bugs I overlooked, please do so: https://github.com/Awzaw/AnvilConverter

 

Conversion to PMAnvil is working great. I converted the 25GB world with no errors! We are testing for missing data now; and I will report back.

EDIT: Some strange errors have occurred; see posts below.

 

Cool

 

The first problem we have found is these split doors. It seems to be relatively rare; but it's happening:



If you open and close them they seem to sometimes fix themselves.

Even weirder; if you tap on the lower part of the door; this happens:

 

Did you try replacing the doors? And if so, what were the results?

Oh and BTW, how do you have a world that's 25GB?!

 

Yes. The new doors seem fine and have stayed in place.

Over a million players over a 2 year period.

 

Here is an odd "block replacement" we have found. Player quote:

Some blocks are being replaced by other blocks in a strange mathematical pattern. I've only observed this with clay, wool, and slabs so it probably only happens to blocks with extended IDs (ie 35:1 vs 35).



Here is an identical picture on the original mcregion world:



I have enclosed an example world that makes this easy to reproduce.

 

Another issue: The converter does not seem to be copying the spawn coordinates from the old level.dat to the new. This can be demonstrated by changing the spawn location (e.g. EssentialsPE /setspawn). Then run the conversion. The spawn will be set back to the original location.

 

I've no idea how spawn could be changed as the contents of level.dat aren't changed apart from the anvil flag, so maybe that's the plugins fault? Or are all the coordinates changed after conversion?!

From the problems you describe for wool, door positions etc, this is most probably a problem with the block metadata - did the same problem occur with regular Anvil conversion? If not, then there's probably a problem my reorderNibbleArray routine: https://github.com/Awzaw/AnvilConve...level/chunk/storage/OldChunkStorage.java#L270

 

Actually; I cleared everything and did a fresh conversion. IMMEDIATELY after conversion the spawn points are correct. But once you start PMMP and stop it, the spawn point changes to default. This is with no plugins loaded. You are correct...immediately after conversion the spawn data looks good.

My guess is PMMP doesn't like my spawn coordinates for some reason; so it resets them to default values. Sorry for the confusion; this issue has nothing to do with the conversion.

 

I just did a test using anvil instead, and both the door and block issues are NOT present. So you are correct; it seems to be a problem in the reorderNibbleArray routine.

 

I am trying to experiment with some changes to Awzaw's reorderNibbleArray routine. However, my Java fu is weak. If anyone could answer a simple compile question; it would be most appreciated.

I started by installing wget, Git and Java. I happen to be running MacOS, but the process should be the same for any *nix.

Step 1:

Code:

git clone --recursive https://github.com/Awzaw/AnvilConverter.git AnvilConverter.git

Step 2:

Code:

cd AnvilConverter.git
cd src
javac ./src/net/minecraft/world/level/storage/AnvilConverter.java

Step 3:

Code:

jar -cvfe ../AnvilConverter.jar AnvilConverter ./net/minecraft/world/level/storage/AnvilConverter.class

Step 3 is where I believe I went astray. I wasn't sure what the main class was, so I used "AnvilConverter". Here is the error I receive when I try to execute:

Code:

$ java -jar "./AnvilConverter.jar"
Error: Could not find or load main class AnvilConverter

What should I be using for the main class in the "jar -cvfe" line?

 

I have a question as well. Worlds that I created from PMMP in mcregion converted just fine to pmanvil. But, worlds that I had imported from PC or MCPE back in the day that are in mcregion do not convert. No errors or anything, just says 0 converted.

Spoiler

pocketmine@mcs01:~/server$ java -jar "/home/pocketmine/server/AnvilConverter/AnvilConverter.jar" /home/pocketmine/server/worlds StarfleetHQ pmanvil
Converting map to pmanvil
Format: pmanvil
Scanning folders...
Total conversion count is 0
Done!
To revert, replace level.dat with level.dat_mcr. Old mcr region files have not been modified.