1 (edited by Kevin 2017-11-26 07:07:40)

Topic: Large arrays yield red lines in textures written

I've successfully created terrain and textures for a given lat/long bounding region.

I then increased the tile count, created a new terrain set (as recommended in the documentation) and then invoked the RWT 'Regenerate Textures' from the '\RealWorldTerrainItem.cs' script (e.g.:  \Real World Terrain Generator\Assets\Infinity Code\Real World Terrain\Scripts\Containers\RealWorldTerrainItem.cs).

That process completed without errors (both download and texture generation) -- though slowly:  the texture process only used one core of (8) available.

RWT created files for each texture tile at the correct size, but they are all simple white with red vertical stripes (some have a strip of partial texture, see below):

PunBB bbcode test

Is this a known problem?

What do you advise I try?  I assume I should regenerate the textures from the download cache with new settings, but I can't imagine what they would be.

Again, the 'Regenerate Textures' process completed without errors, system RAM usage was never above 50%, and I can detect no other signs of a problem.

Re: Large arrays yield red lines in textures written

Hello.

This is the first time I've ever seen such a result.
Please send me your settings, I will try to reproduce the problem and fix it.

Kind Regards,
Infinity Code Team

3 (edited by Kevin 2017-11-26 18:27:01)

Re: Large arrays yield red lines in textures written

Thanks, Alex, for your fast reply and offer to try to confirm my results on your side.

Here are my settings:

PunBB bbcode test

To review, I generated the terrains first (quite fast) then executed the texture settings shown.

I tried to regenerate textures again, with the result that all tiles are white with red vertical lines as shown above.

Note that the texture provider is Google -- naïvely, I would suppose that Google is returning these 'empty' textures as a response to too many requests.  That would explain why I have narrow strips for a few textures (the first computed?) and then nothing but white fields with red lines thereafter.

I see your note about transaction limits in the interface for RWT, but I don't see anything like a meter, a transaction log or a way to understand what constitutes a single transaction.  Are there notes I'm missing?

Again, this is a guess only.  I didn't see any errors.  If there is an error log I can explicitly check, let me know and I'll do that.

Thanks again,

-Kevin

Re: Large arrays yield red lines in textures written

Yes, with a 99% chance of a problem is Google.
Google has a limit on the number of downloaded tiles per day.
Solution: use any other tile provider.

Honestly, we are thinking about removing Google from the list of providers to prevent such problems.
There is only one reason why it is still here: in some places other tile providers have a quality much inferior to Google maps. These places are quite rare, but they are there.

A little trick if you need tiles like in Google:
Google Maps uses tiles from third-party vendors, such as DigitalGlobe.
Use DigitalGlobe directly in RWT, using «Provider - Custom».
URL:
https://a.tiles.mapbox.com/v4/digitalgl … cesstoken}
And specify your access token.

Kind Regards,
Infinity Code Team

5 (edited by Kevin 2017-11-26 19:49:41)

Re: Large arrays yield red lines in textures written

That's depressing to hear, since the imagery Google licenses is much better than the others in my case -- I tried all the providers available in RWT on small tests for my bounding region and the Google results were by far the best.

I see DigitalGlobe licenses per 'map view':
https://platform.digitalglobe.com/maps-api/pricing/

Do you know how that relates to API calls of the kind you suggested from RWT?

Re: Large arrays yield red lines in textures written

DigitalGlobe and Mapbox: 1 map view = 15 tiles.
I have not tested your area and I do not know what quality DigitalGlobe has.
But Mapbox has the same quality as Google Maps (tested), and gives 50000 map views for free.

https://a.tiles.mapbox.com/v4/mapbox.satellite/{zoom}/{x}/{y}.png?access_token={YOUR_ACCESS_TOKEN}
Kind Regards,
Infinity Code Team

Re: Large arrays yield red lines in textures written

Thanks again, Alex,

As I noted earlier, I see that the 'Generate Textures' step doesn't seem to support multithreading -- RWT is using only one physical core of the CPU (two logical cores with Intel's 'Hyperthreading').  It takes me > 1 day to compute a large area terrain set (12x12, 18x18).

Is there a way to invoke this process in parallel since it's piecewise linear computation?  Perhaps via the API?

A second question:  searching this forum it seems one 'tile' is equal to 256x256 pixels; their API documentation looks like 512x512:
https://www.mapbox.com/api-documentation/#maps

Is the definition for a tile laid out clearly somewhere?

I'll try the Mapbox API for textures, and assume they will best match Mapbox's terrain data.

Re: Large arrays yield red lines in textures written

At 1K or 2K texture resolution, Mapbox returns the full texture for my single terrain test.

At 4K or 8K, the texture RWT acquires is partial, as shown below at bottom:

See the RWT settings at right for the failure case.

Note that this partial texture is what I saw with Google, but where Google is acting over quota (no sat view appears now, even in the TWT helper map page), I'm able to request small textures successfully from Mapbox *after* the 4K/8K failure shown.  That is, it's not running out of the views, there's something else going on here:

PunBB bbcode test

Re: Large arrays yield red lines in textures written

Yes, we can try to optimize (parallelize) the generation of terrains and textures.
The problem is that Unity allows to work with most Unity Objects (TerrainData, Texture2D, etc.) only in the main thread.
In any case, we need some time to develop.

There are three types of tiles in Mapbox:
1. Vector tiles.
2. Raster tiles 512x512.
3. Raster tiles 256x256.
Satellite images have a size of 256x256.

Mapbox does not have a zoom 20 texture for this area.
To work around this, select "Textures / Max level - 19".

Kind Regards,
Infinity Code Team

Re: Large arrays yield red lines in textures written

Thanks for the real-world advice concerning the vagaries of Mapbox tiles, much appreciated.

A final question for now -- is there a way to selectively choose texture resolutions for a large grid of terrains?

To illustrate, here is a group of terrains, with a path overlaid where a bird flies:

PunBB bbcode overview

Since this is real-world data, we want to respect the positions shown, which get very close to the terrains in the local regions under the path.

Ideally, I'd like some way to identify just the terrains beneath the flight path and page those for higher resolution texture download, while downloading the balance of the textures at much lower resolution.

Is this possible in RWT?

Re: Large arrays yield red lines in textures written

RWT can generate textures for all terrains in the grid or for one selected terrain.
But it can not for N (> 1) selected terrains.
This means that you can generate textures of all terrains with low resolution, and regenerate textures with high resolution for each desired terrain one at a time.
Theoretically, it can be automated using RWT API, but in most cases it's much easier to do manually.

Kind Regards,
Infinity Code Team

12 (edited by Kevin 2017-12-01 21:11:09)

Re: Large arrays yield red lines in textures written

Thanks for the reply!

Yes, I thought about manually selecting each terrain close to the flight path, setting a new target resolution and re-downloading and re-computing, but of course for a very large number of terrains this will take days given the time to do the download, the single-threaded computation, and babysit the process.

This kind of repetitive, mindless work is exactly what computers should deliver us from.

I'd be happy for any pointers on deploying the API to that end.

Thanks again.

Re: Large arrays yield red lines in textures written

The example below.
Place the script in Editor folder, open "Window / Infinity Code / Real World Terrain / Tools / Automatic Regeneration", add 1+ locations and click Regenerate.

This is just an example of how to automate regeneration, and you can modify it to your own needs.

Script:

using System.Collections;
using System.Collections.Generic;
using InfinityCode.RealWorldTerrain;
using InfinityCode.RealWorldTerrain.Windows;
using UnityEditor;
using UnityEngine;

public class AutomaticRegenerationWindow:EditorWindow
{
    private static AutomaticRegenerationWindow wnd;
    private RealWorldTerrainContainer container;
    private bool isNewLocationCreated;
    private bool isStarted;
    private int itemIndex;
    private Vector3 lastCursorPosition;
    private List<Vector2> locations;
    private Vector2 newLocation;
    private Vector2 scrollPosition;
    private int textureResolution = 256;
    private List<RealWorldTerrainItem> terrainItems;

    private void OnCaptureCanceled()
    {
        Debug.Log("Canceled");
        isStarted = false;
        RemoveListeners();
    }

    private void OnCaptureCompleted()
    {
        // Delay to give RWT complete.
        EditorCoroutine.start(StartNextItemR());
    }

    IEnumerator StartNextItemR()
    {
        yield return new WaitForSeconds(1);
        StartNextItem();
    }

    private void OnDestroy()
    {
        wnd = null;
        SceneView.onSceneGUIDelegate -= OnSceneGUI;
        EditorApplication.update -= OnUpdate;
    }

    private void OnEnable()
    {
        isStarted = false;
        wnd = this;
        container = FindObjectOfType<RealWorldTerrainContainer>();
        SceneView.onSceneGUIDelegate += OnSceneGUI;
        EditorApplication.update += OnUpdate;
    }

    private void OnGUI()
    {
        if (!isStarted) OnIdleGUI();
        else OnRegenerateGUI();
    }

    private void OnIdleGUI()
    {
        if (locations == null) locations = new List<Vector2>();

        container = EditorGUILayout.ObjectField("Real World Terrain", container, typeof(RealWorldTerrainContainer), true) as RealWorldTerrainContainer;

        EditorGUILayout.LabelField("Locations (x - longitude, y - latitude)");

        int removeIndex = -1;

        scrollPosition = EditorGUILayout.BeginScrollView(scrollPosition);

        for (int i = 0; i < locations.Count; i++)
        {
            EditorGUILayout.BeginHorizontal();
            locations[i] = EditorGUILayout.Vector2Field((i + 1).ToString(), locations[i]);
            if (GUILayout.Button("X", GUILayout.ExpandWidth(false))) removeIndex = i;
            EditorGUILayout.EndHorizontal();
        }

        EditorGUILayout.EndScrollView();

        if (removeIndex != -1) locations.RemoveAt(removeIndex);

        bool isInfoDrawed = false;

        if (container != null && lastCursorPosition != Vector3.zero)
        {
            SceneView view = SceneView.lastActiveSceneView;
            if (view != null)
            {
                Vector3 cp = view.camera.transform.position;
                double longitude, latitude, altitude;

                container.GetCoordinatesByWorldPosition(cp, out longitude, out latitude, out altitude);

                container.GetCoordinatesByWorldPosition(lastCursorPosition, out longitude, out latitude, out altitude);

                EditorGUILayout.LabelField("Scene cursor latitude: " + latitude);
                EditorGUILayout.LabelField("Scene cursor longitude: " + longitude);
                EditorGUILayout.LabelField("Scene cursor altitude: " + altitude.ToString("F2") + " meters");
                EditorGUILayout.LabelField("Use CTRL+SHIFT to insert the coordinates.");

                isInfoDrawed = true;

                if (Event.current.control && Event.current.shift)
                {
                    if (!isNewLocationCreated)
                    {
                        newLocation = new Vector2((float) longitude, (float) latitude);
                        isNewLocationCreated = true;
                    }
                }
                else
                {
                    isNewLocationCreated = false;
                }
            }
        }

        if (!isInfoDrawed)
        {
            EditorGUILayout.LabelField("Scene cursor latitude: ");
            EditorGUILayout.LabelField("Scene cursor longitude: ");
            EditorGUILayout.LabelField("Scene cursor altitude: ");
            EditorGUILayout.LabelField("Use CTRL+SHIFT to insert the coordinates.");
        }

        textureResolution = EditorGUILayout.IntField("Texture resolution", textureResolution);

        if (GUILayout.Button("Regenerate"))
        {
            StartRegeneration();
        }
    }

    private void OnRegenerateGUI()
    {
        Rect r = EditorGUILayout.BeginVertical();
        float progress = (float)itemIndex / terrainItems.Count;
        EditorGUI.ProgressBar(r, progress, Mathf.FloorToInt(progress * 100) + "%");
        GUILayout.Space(16);
        EditorGUILayout.EndVertical();

        if (GUILayout.Button("Cancel")) RealWorldTerrainWindow.CancelCapture();
    }

    private void OnSceneGUI(SceneView view)
    {
        Vector2 mp = Event.current.mousePosition;
        mp.y = view.camera.pixelHeight - mp.y;

        RaycastHit hit;
        Ray ray = view.camera.ScreenPointToRay(mp);

        if (Physics.Raycast(ray.origin, ray.direction, out hit, 1000000000)) lastCursorPosition = hit.point;
        else lastCursorPosition = Vector3.zero;
    }

    private void OnUpdate()
    {
        if (newLocation != Vector2.zero)
        {
            locations.Add(newLocation);
            newLocation = Vector2.zero;
        }

        Repaint();
    }

    [MenuItem("Window/Infinity Code/Real World Terrain/Tools/Automatic Regeneration")]
    private static void OpenWindow()
    {
        if (wnd != null) wnd.Close();
        wnd = GetWindow<AutomaticRegenerationWindow>("Automatic Regeneration");
    }

    private void StartRegeneration()
    {
        if (locations == null || locations.Count == 0 || container == null) return;

        terrainItems = new List<RealWorldTerrainItem>();

        foreach (Vector2 location in locations)
        {
            Vector3 pos;
            container.GetWorldPosition(location, out pos);
            RealWorldTerrainItem item = container.GetItemByWorldPosition(pos);
            if (item != null && !terrainItems.Contains(item)) terrainItems.Add(item);
        }

        if (terrainItems.Count == 0)
        {
            Debug.LogError("No terrain items");
            return;
        }

        Debug.Log(terrainItems.Count);

        isStarted = true;

        RealWorldTerrainWindow.OnCaptureCanceled += OnCaptureCanceled;
        RealWorldTerrainWindow.OnCaptureCompleted += OnCaptureCompleted;

        itemIndex = -1;
        StartNextItem();
    }

    private void RemoveListeners()
    {
        RealWorldTerrainWindow.OnCaptureCanceled -= OnCaptureCanceled;
        RealWorldTerrainWindow.OnCaptureCompleted -= OnCaptureCompleted;
    }

    private void StartNextItem()
    {
        itemIndex++;
        if (itemIndex >= terrainItems.Count)
        {
            Debug.Log("Finish");
            isStarted = false;
            RemoveListeners();
            return;
        }

        RealWorldTerrainWindow.OpenWindow(RealWorldTerrainGenerateType.texture, terrainItems[itemIndex]);
        RealWorldTerrainWindow.prefs.textureSize = new RealWorldTerrainVector2i(textureResolution, textureResolution);
        RealWorldTerrainWindow.StartCapture();
    }

    public class EditorCoroutine
    {
        public static EditorCoroutine start(IEnumerator _routine)
        {
            EditorCoroutine coroutine = new EditorCoroutine(_routine);
            coroutine.start();
            return coroutine;
        }

        readonly IEnumerator routine;
        EditorCoroutine(IEnumerator _routine)
        {
            routine = _routine;
        }

        void start()
        {
            EditorApplication.update += update;
        }
        public void stop()
        {
            EditorApplication.update -= update;
        }

        void update()
        {
            if (!routine.MoveNext())
            {
                stop();
            }
        }
    }
}
Kind Regards,
Infinity Code Team

Re: Large arrays yield red lines in textures written

Thanks Alex for this, which is helpful!