Converting Worlds from McRegion to PMAnvil (128 to 256 Height)

Discussion in 'General discussion' started by MK500, Feb 24, 2017.

  1. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Since I am assuming there are thousands of servers out there that must eventually convert their McRegion (PocketMine default prior to 1.0.0) worlds to PMAnvil (PMMP default after 1.0.0) worlds, I wanted to start a discussion on the various methods to do this.

    First a quick summary of WHY you would want to do this:
    1) PMAnvil is much faster, primarily because it does not need to do on-the-fly conversion from XYZ full columns to a stack of XZY cubes.
    2) PMAnvil supports 256 world height vs. 128 for McRegion.
    3) PMAnvil is the PMMP default format, so will likely have long term support; and eventually McRegion will be deprecated.

    Please post in this thread:
    - Working methods of McRegion to PMAnvil conversion with as much detail as possible.
    - Problems/solutions you have experienced with conversion.
     
    Last edited: Mar 13, 2017
    robske_110 (Tim) likes this.
  2. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Reference threads:

    Solutions!
    Awzaw has built a well tested and working JAVA MCRegion to PMAnvil/Anvil converter based on Mojang code

    Older Reference Threads
    dktapps says Intyre is working on a dedicated world conversion tool.
    robske_110 (Tim) says we can convert by hacking PMMP source code to "redirect serialisation functions to dummy providers".
    List of all current stable world formats. Just for reference: As entered in pocketmine.yml: mcregion, pmanvil, anvil

    Older World Conversion Tools
    pmimporter
    MConverterUltimate
     
    Last edited: Apr 27, 2017
    jarne and robske_110 (Tim) like this.
  3. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Has anyone installed dktapps C extension? Any performance improvements when using McRegion worlds?

    EDIT: I guess not. Per dktapps : "No, the extension will make no difference to McRegion worlds. McRegion needs a conversion from XZY full columns to a stack of XZY cubes. Makes no logical sense, but just blame Mojang."
     
    Last edited: Apr 27, 2017
  4. Awzaw

    Awzaw Zombie Pigman Poggit Admin

    Messages:
    726
    GitHub:
    awzaw
    I've got Mojang's MCRegion->Anvil converter working, you can download it here: https://github.com/Awzaw/AnvilConverter

    It takes about 5 minutes to convert a 1GB world, and if you record the console output you'll see which file is to blame when there's a problem. So far I've only had a couple of UTFDataFormatExceptions which aren't fatal, although I'm not yet sure what effect those errors might have. I have no idea if there's any hidden problems with this... so please note that I take no responsibility whatsoever for whatever havoc this may wreak on your data. That said, I've tested on several worlds now, and everything looks good; I tried it on a plot world on a production server and everyone loves the extra height, and less lag (see below). So good so far.

    You'll need dktapps ChunkUtils PHP extension to see any performance benefits of converting mcregion to anvil, but it's not that hard to install. The next step is to get the converter to convert directly to PMAnvil, any help with that would be welcome.
     
    Last edited: Mar 26, 2017
  5. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Thank you Awzaw! This is a great start; as I believe the only difference between Anvil and PMAnvil is the XYZ to XZY conversion. So other than this performance hit; Anvil should be as much a "first class citizen" as PMAnvil.

    Thank you for taking the time to tweak, hack, and package up this code.

    I am currently converting a 25GB world using this method, and will report back on the results; including any errors given. If this is a good solution; then we just need the final conversion step in this code to have a complete MCRegion to PMAnvil converter. But as you said; performance even with Anvil should be far superior to MCRegion assuming ChunkUtils is installed (which is quite easy to do with the current php build scripts thanks to dktapps).
     
    jasonwynn10 and Awzaw like this.
  6. Awzaw

    Awzaw Zombie Pigman Poggit Admin

    Messages:
    726
    GitHub:
    awzaw
    It seems to be entity nametags that are producing the errors, and I'm not sure that chunks that give that error won't be corrupted... But maybe a small price to pay? I'll look into it in more detail when I get the time, meanwhile reordering is not looking too hard, so it shouldn't be too long before this can convert to Anvil or PMAnvil.
     
    MK500 likes this.
  7. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    The 25GB world converted without problems and several players are testing it now.

    So far we aren't detecting anything "weird". Items seem to be in chests where they should be, not finding missing chunks, etc.

    EDIT: See reports of door issues and replacement of blocks with extended IDs below.

    Any thoughts on a good way to confirm the ChunkUtils extension is really working? I compiled php with the current script and -u; so I believe it is working.
     
    Last edited: Apr 2, 2017
  8. Awzaw

    Awzaw Zombie Pigman Poggit Admin

    Messages:
    726
    GitHub:
    awzaw
    If the chunkutils extension shows up when you list the PHP modules with php -m then I think you can assume it's working.

    I've added conversion from MCRegion directly to PMAnvil, and it all seems to work nicely although conversion is even slower than converting to Anvil, of course. Again, I take no responsibility... and if anyone can help speed it up, or fix any bugs I overlooked, please do so: https://github.com/Awzaw/AnvilConverter
     
    Last edited: Mar 27, 2017
  9. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Conversion to PMAnvil is working great. I converted the 25GB world with no errors! We are testing for missing data now; and I will report back.

    EDIT: Some strange errors have occurred; see posts below.
     
    Last edited: Apr 2, 2017
    Awzaw likes this.
  10. robske_110 (Tim)

    robske_110 (Tim) Wither Skeleton Poggit Reviewer

    Messages:
    1,342
    GitHub:
    robske110
    Cool
     
  11. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    The first problem we have found is these split doors. It seems to be relatively rare; but it's happening:

    [​IMG]

    If you open and close them they seem to sometimes fix themselves.

    Even weirder; if you tap on the lower part of the door; this happens:

    [​IMG]
     
  12. EdwardHamHam

    EdwardHamHam Skeleton

    Messages:
    962
    GitHub:
    edwardhamham
    Did you try replacing the doors? And if so, what were the results?

    Oh and BTW, how do you have a world that's 25GB?!
     
    jasonwynn10 likes this.
  13. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Yes. The new doors seem fine and have stayed in place.

    Over a million players over a 2 year period.
     
    jasonwynn10 likes this.
  14. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Here is an odd "block replacement" we have found. Player quote:

    Some blocks are being replaced by other blocks in a strange mathematical pattern. I've only observed this with clay, wool, and slabs so it probably only happens to blocks with extended IDs (ie 35:1 vs 35).

    [​IMG]

    Here is an identical picture on the original mcregion world:

    [​IMG]

    I have enclosed an example world that makes this easy to reproduce.
     

    Attached Files:

    Last edited: Apr 4, 2017
  15. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Another issue: The converter does not seem to be copying the spawn coordinates from the old level.dat to the new. This can be demonstrated by changing the spawn location (e.g. EssentialsPE /setspawn). Then run the conversion. The spawn will be set back to the original location.
     
  16. Awzaw

    Awzaw Zombie Pigman Poggit Admin

    Messages:
    726
    GitHub:
    awzaw
    I've no idea how spawn could be changed as the contents of level.dat aren't changed apart from the anvil flag, so maybe that's the plugins fault? Or are all the coordinates changed after conversion?!

    From the problems you describe for wool, door positions etc, this is most probably a problem with the block metadata - did the same problem occur with regular Anvil conversion? If not, then there's probably a problem my reorderNibbleArray routine: https://github.com/Awzaw/AnvilConve...level/chunk/storage/OldChunkStorage.java#L270
     
  17. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    Actually; I cleared everything and did a fresh conversion. IMMEDIATELY after conversion the spawn points are correct. But once you start PMMP and stop it, the spawn point changes to default. This is with no plugins loaded. You are correct...immediately after conversion the spawn data looks good.

    My guess is PMMP doesn't like my spawn coordinates for some reason; so it resets them to default values. Sorry for the confusion; this issue has nothing to do with the conversion.
     
    Last edited: Apr 3, 2017
    jasonwynn10 likes this.
  18. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    I just did a test using anvil instead, and both the door and block issues are NOT present. So you are correct; it seems to be a problem in the reorderNibbleArray routine.
     
    jasonwynn10 likes this.
  19. MK500

    MK500 Slime

    Messages:
    77
    GitHub:
    markkrueg
    I am trying to experiment with some changes to Awzaw's reorderNibbleArray routine. However, my Java fu is weak. If anyone could answer a simple compile question; it would be most appreciated.

    I started by installing wget, Git and Java. I happen to be running MacOS, but the process should be the same for any *nix.

    Step 1:
    Code:
    git clone --recursive https://github.com/Awzaw/AnvilConverter.git AnvilConverter.git
    Step 2:
    Code:
    cd AnvilConverter.git
    cd src
    javac ./src/net/minecraft/world/level/storage/AnvilConverter.java
    Step 3:
    Code:
    jar -cvfe ../AnvilConverter.jar AnvilConverter ./net/minecraft/world/level/storage/AnvilConverter.class
    Step 3 is where I believe I went astray. I wasn't sure what the main class was, so I used "AnvilConverter". Here is the error I receive when I try to execute:

    Code:
    $ java -jar "./AnvilConverter.jar"
    Error: Could not find or load main class AnvilConverter
    What should I be using for the main class in the "jar -cvfe" line?
     
  20. josh

    josh Creeper

    Messages:
    2
    GitHub:
    joshnorth
    I have a question as well. Worlds that I created from PMMP in mcregion converted just fine to pmanvil. But, worlds that I had imported from PC or MCPE back in the day that are in mcregion do not convert. No errors or anything, just says 0 converted.
    [email protected]:~/server$ java -jar "/home/pocketmine/server/AnvilConverter/AnvilConverter.jar" /home/pocketmine/server/worlds StarfleetHQ pmanvil
    Converting map to pmanvil
    Format: pmanvil
    Scanning folders...
    Total conversion count is 0
    Done!
    To revert, replace level.dat with level.dat_mcr. Old mcr region files have not been modified.
     

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.