Mapnik Generative AI workflow: Processing

Alexander Dunkel, Madalina Gugulica, Institute of Cartography, TU Dresden

No description has been provided for this image
•••
Out[6]:

Last updated: Oct-18-2023, Carto-Lab Docker Version 0.15.7

Mapnik rendering based on stable diffusion generative AI and social media data.

This notebook is a continuation from the previous notebook (01_mapnik_generativeai.html).

Prepare environment

•••
List of package versions used in this notebook
package python Fiona Shapely bokeh colorcet geopandas geoviews holoviews hvplot ipywidgets
version 3.9.15 1.8.20 1.7.1 2.4.3 3.0.1 0.13.2 1.9.5 1.14.8 0.8.4 8.0.7
package mapclassify matplotlib matplotlib-venn numpy pandas python-dotenv xarray
version 2.5.0 3.7.1 0.11.9 1.22.4 2.1.0 1.0.0 2023.8.0

Load base dependencies:

In [8]:
import os, sys
import re
import random
import shutil
import geopandas as gp
import pandas as pd
import geopandas
import matplotlib.pyplot as plt
import rasterio as rio
from pathlib import Path
from rasterio.plot import show

Install temporary package rembg

In [9]:
!../py/modules/base/pkginstall.sh "rembg"
rembg already installed.

Import every cell from the previous notebook, except those tagged with active-ipynb. This will make all variables and methods from the previous notebook available in the current runtime, so we can continue where we left.

In [13]:
module_path = str(Path.cwd().parents[0] / "py")
if module_path not in sys.path:
    sys.path.append(module_path)
from modules.base import raster
from _02_generativeai import *

Symlink font folder

In [14]:
!rm /fonts && ln -s {TMP}/fonts /fonts

Symlink remove white background model:

In [15]:
!ln -s {OUTPUT}/isnet-general-use.onnx /root/.u2net/isnet-general-use.onnx 2> /dev/null

Create new directories

In [179]:
dlist = [
    (OUTPUT / "images_gis"),
    (OUTPUT / "img2img"),
    (INPUT / "cluster_img"),
    (INPUT / "cluster_img_tags"),
]
for folder in dlist:
    folder.mkdir(exist_ok=True, parents=True)

Activate autoreload of changed python files:

In [16]:
%load_ext autoreload
%autoreload 2

Parameters

In [17]:
APIURL = "http://127.0.0.1:7861"
BASE_PROMPT_POS: str = \
    "white background,simple outline,masterpiece,best quality,high quality," \
    "<lora:Japanese_style_Minimalist_Line_Illustrations:0.2>"
BASE_PROMPT_NEG: str = \
    "(bad-artist:1),(worst quality, low quality:1.4),lowres,bad anatomy,bad hands," \
    "((text)),(watermark),error,missing fingers,extra digit,fewer digits,cropped,worst quality," \
    "low quality,normal quality,((username)),blurry,(extra limbs),bad-artist-anime," \
    "(three hands:1.6),(three legs:1.2),(more than two hands:1.4),(more than two legs,:1.2)," \
    "label,(isometric), (square)"

Set global SD-settings

In [18]:
payload = {
    "CLIP_stop_at_last_layers": 1,
    "sd_vae":"vae-ft-mse-840000-ema-pruned.safetensors",
    "sd_model_checkpoint":"hellofunnycity_V14.safetensors",
}
requests.post(url=f'{APIURL}/sdapi/v1/options', json=payload)
Out[18]:
<Response [200]>

Have a look at our per-job basis settings, loaded from the last notebook:

In [19]:
SD_CONFIG
Out[19]:
{'steps': 20, 'batch_size': 4, 'sampler_name': 'DPM++ 2M SDE Exponential'}

For this notebook, increase steps to 28

In [20]:
SD_CONFIG["steps"] = 28

Test image generation for tags and emoji

The next step is to process social media metadata (tags, emoji) in descending importance (cluster-size), generate images for clusters, and place images on the map, according to the center of gravity for the cluster shape from tagmaps package.

Test API for selected tags

In [21]:
PROMPT = "(Grosser Garten, Palais, Nature)"
In [22]:
output_name = "test_image_palais_default"
KWARGS = {
    "prompt": concat_prompt(PROMPT),
    "negative_prompt": BASE_PROMPT_NEG,
    "save_name": output_name,
    "sd_config": SD_CONFIG,
    "show": False
}
DKWARGS = {
    "resize":(350, 350),
    "figsize":(22, 60),
}
In [23]:
if not (OUTPUT / "images" / f'{output_name}.png').exists():
    generate(**KWARGS)
In [24]:
imgs = list((OUTPUT / "images").glob(f'{output_name}*'))
tools.image_grid(imgs, **DKWARGS)
No description has been provided for this image

We have to think about a way to better incorporate these square images in the map. Maybe if we add A thought bubble of to our prompt?

In [25]:
def generate_samples(
        prompt: str, save_name: str, kwargs=KWARGS, output=OUTPUT,
        dkwargs=DKWARGS, print_prompt: bool = None, rembg: bool = None):
    """Generate and show 4 sample images for prompt"""
    kwargs["prompt"] = concat_prompt(prompt)
    if print_prompt:
        print(kwargs["prompt"][:50])
    kwargs["save_name"] = save_name
    if not (output / "images" / f'{kwargs["save_name"]}.png').exists():
        if rembg:
            generate_rembg(**kwargs)
        else:
            generate(**kwargs)
    imgs = list((output / "images").glob(f'{kwargs["save_name"]}*'))
    tools.image_grid(imgs, **dkwargs)
In [26]:
generate_samples("(thought bubble of Grosser Garten, Palais, Nature)", save_name="test_image_palais_bubble")
No description has been provided for this image

or maybe icon?

In [27]:
generate_samples("(A map icon of Grosser Garten, Palais, Nature)", save_name="test_image_palais_icon")
No description has been provided for this image

Let's keep A map icon of as the pre-prompt.

Some more tests for other tags and terms

In [28]:
generate_samples("(A map icon of Botanischergarten), flower, grün, 🌵 🌿 🌱", save_name="test_image_botan_icon")
No description has been provided for this image
In [29]:
generate_samples("(A map icon of Gläsernemanufaktur), volkswagen, building", save_name="test_image_vw_icon")
No description has been provided for this image
In [30]:
generate_samples("(A map icon of zoo), zoodresden, animals", save_name="test_image_zoo_icon")
No description has been provided for this image
In [31]:
generate_samples("(A map icon of fussball stadion), dynamo, stadion", save_name="test_image_fussball_icon")
No description has been provided for this image
In [32]:
generate_samples("(people 🏃), activity", save_name="test_image_running_activity")
No description has been provided for this image

Enough tests. Now, we can move to collecting tag and emoji clusters and move on to batch generation.

Process clustered data

The overall workflow looks like this:

  1. Find all clusters above a weight of x
  2. Walk through clusters, get cluster centroid
  3. Select all other cluster-shapes that can be found at this location
  4. Concat prompt based on ascending importance
  5. Generate image, remove background, save
  6. Create Mapnik Stylesheet to place images as either symbols or raster images
  7. Render map
  8. (Adjust parameters and repeat, until map quality is acceptable)
In [33]:
data_src = Path(INPUT / "shapefiles_gg" / "allTagCluster.shp")
In [34]:
gdf = gp.read_file(INPUT / "shapefiles_gg" / "allTagCluster.shp", encoding='utf-8')
CRS_PROJ = gdf.crs
In [35]:
def sel_cluster(gdf: gp.GeoDataFrame, min_weight: int) -> gp.GeoSeries:
    """Return GeoSeries of Clusters above min_weight"""
    with fiona.open(data_src, encoding='UTF-8', mode="r") as shapefile:
        for feature in shapefile:
            properties = feature["properties"]
            if properties["HImpTag"] == 1 and properties["ImpTag"] == feature_name.lower():
                bounds = shape(feature["geometry"]).bounds
                if add_buffer:
                    bounds = add_buffer_bbox(bounds, buffer = add_buffer)
                return bounds
In [36]:
OUTPUT_MAPS = TMP / "bg"

Reproject raster

In [37]:
%%time
raster.reproject_raster(
    raster_in=f"{OUTPUT_MAPS}/grossergarten_carto_17.tif", 
    raster_out=f"{OUTPUT_MAPS}/grossergarten_carto_17_proj.tif",
    dst_crs=f'epsg:{CRS_PROJ.to_epsg()}')
CPU times: user 723 ms, sys: 60.8 ms, total: 784 ms
Wall time: 782 ms
In [38]:
basemap = rio.open(f"{OUTPUT_MAPS}/grossergarten_carto_17_proj.tif")

bbox_map = gdf.total_bounds.squeeze()
minx, miny = bbox_map[0], bbox_map[1]
maxx, maxy = bbox_map[2], bbox_map[3]
x_lim=(minx, maxx)
y_lim=(miny, maxy)

Plot all cluster shapes

In [39]:
fig, ax = plt.subplots(figsize=(10, 10))
rio.plot.show(basemap, ax=ax)
gdf.plot(ax=ax, facecolor='none', edgecolor='red', linewidth=0.1)
ax.set_xlim(*x_lim)
ax.set_ylim(*y_lim)
ax.set_axis_off()
No description has been provided for this image

Plot only cluster shapes above a certain weight

In [40]:
cluster_sel = gdf[gdf["Weights"]>300]
In [41]:
def plot_clustermap(cluster_sel: gp.GeoDataFrame, basemap: rio.DatasetReader, label: bool = None):
    """Plot a map with clusters, basemap, and cluster labels"""
    if label is None:
        label = True
    fig, ax = plt.subplots(figsize=(7, 10))
    rio.plot.show(basemap, ax=ax)
    cmap=plt.get_cmap('Paired')
    cluster_sel.plot(ax=ax, facecolor='none', cmap=cmap, linewidth=1)
    if label:
        tools.annotate_locations_fit(
            gdf=cluster_sel, ax=ax,
            text_col="ImpTag", arrowstyle='-', arrow_col='black', fontsize=10,
            font_path="/fonts/seguisym.ttf")
    ax.set_xlim(*x_lim)
    ax.set_ylim(*y_lim)
    ax.set_axis_off()
    with warnings.catch_warnings():
        # Ignore emoji "Variation-Selector" not found in font
        warnings.filterwarnings("ignore", category=UserWarning)
        plt.show()
In [42]:
plot_clustermap(cluster_sel=cluster_sel, basemap=basemap)
No description has been provided for this image

There are several clusters visible. On the upper left, we can see the Dynamo Dresden stadium. Several tag and emoji cluster shapes are can be found at in this area. There is also a big shape covering the Großer Garten. Two smaller shapes can be found hovering the Dresden Zoo and the Gläserne Manufaktur.

Process Emoji

We start with processing emoji. This seems like the easier part, since emoji are already highly abstracted concepts that can convey many meanings in a simplified form.

Some emoji, however, are very generic and used for arbitrary context. We use a broad positive filter list with 693 emoji (out of about 2000 available) to focus on specific activity and environment emoji.

In [45]:
emoji_filter_list = pd.read_csv(
    INPUT / 'SelectionList_EmojiLandscapePlanning.txt', header=None, names=["emoji"], encoding="utf-8", on_bad_lines='skip')
emoji_filter_list = emoji_filter_list.set_index("emoji").index
In [49]:
print(emoji_filter_list[:20])
Index(['🌊', '🌅', '🍻', '🎡', '📸', '🎢', '🎶', '💪', '📷', '🐶', '🍁', '🍂', '🌸', '💦',
       '👭', '🍀', '🏖', '👫', '🎈', '🍃'],
      dtype='object', name='emoji')
In [50]:
cluster_sel = gdf[(gdf["Weights"]>100) & (gdf["emoji"]==1) & (gdf["ImpTag"].isin(emoji_filter_list))].copy()
In [51]:
plot_clustermap(cluster_sel=cluster_sel, basemap=basemap)
No description has been provided for this image

We can see four spatial groups of emoji clusters, the football stadium (upper left), the Zoo (below), the botanical garden (upper group) and the Junge Garde (lower right), an outdoor music venue.

Concat emoji based on cluster group/spatial intersection

In [52]:
intersects = cluster_sel.sjoin(cluster_sel[["geometry"]], how="left", predicate="intersects").reset_index()
cluster_groups = intersects.dissolve("index_right", aggfunc="min")

Join back the group-id's

In [53]:
cluster_sel["group"] = cluster_groups["index"]
In [54]:
cluster_lists = cluster_sel.groupby("group")["ImpTag"].apply(list)
In [55]:
cluster_lists
Out[55]:
group
1     [⚽, 💪, 🍻, 💪🏻, 🏈, 🏃, 💪🏼, 🏆, 📸]
26                        [🌵, 🌿, 🌱]
62                     [🐒, 🦁, 🐘, 🐨]
71                              [🎶]
Name: ImpTag, dtype: object

Generate images for cluster-groups

In [56]:
emoji_cluster_1 = list(cluster_sel[cluster_sel["group"]==1]["ImpTag"])
emoji_cluster_1
Out[56]:
['⚽', '💪', '🍻', '💪🏻', '🏈', '🏃', '💪🏼', '🏆', '📸']

Generate sample images for clusters

In [57]:
for ix, cluster_list in enumerate(cluster_lists):
    print(cluster_list)
    generate_samples(
        f"A map icon of happy ({cluster_list[0]}), {''.join(cluster_list[1:])}", save_name=f"emoji_{ix:03d}", rembg=True)
['⚽', '💪', '🍻', '💪🏻', '🏈', '🏃', '💪🏼', '🏆', '📸']
['🌵', '🌿', '🌱']
['🐒', '🦁', '🐘', '🐨']
['🎶']