Link

https://tinyurl.com/KJGSL

Morinaga Milk Caramels

Morinaga in Korea, 1922

Figure 1: Morinaga advertisement from Korea & Japan. Left: Tonga Ilbo, November 6, 1922. Right: Ōsaka mainichi shinbun, November 2, 1922, 4.

For example, a reporter at the Tonga ilbo later recalled that in the 1920s and 1930s he often saw the staff at the advertising department “cutting out advertising copy [written] in Japanese and replacing it with Korean,” on design layouts sent from Japan

Morinaga in Korea, 1930

Figure 2: advertisement for Morinaga maeil sinbo, july 7, 1930, 4

• Change in rhetoric
  • For twenty million people in the country
  • The seventy million compatriots
• Highlight on product rather than modern life
• Logo in Han’gŭl highlighted
• Step further in localising

Morinaga in Korea, 1931

Figure 3: advertisement for morinaga Tonga Ilbo, February 14, 1931, 7

• Visuals more prominent
  • Saektong Chŏgori Hanbok
  • Kite tradition with New Year (Sŏllal)
  • Despite suppression of the tradition
• Imperial perceptions
• Acknowledment of Identity

Example of Morinaga taken from Chun,¹ who has done much work on the study of advertising and branding in Colonial Korea.

1. “Becoming Imperial Brands”.

Searching for visuals?

Methodology

Data

import { createHeatmap } from "./heatmap.js"
data = await FileAttachment("unified_newspaper_data.csv").csv()
createHeatmap(data, d3)

(a)

(b)

(c)

Figure 4: Data coverage 1920-1950

Corpus consists of 212,000 pages from the Tonga Ilbo, Chosŏn Ilbo, and Maeil Sinbo.

Method

Figure 5: Samples from YOLO detected illustrations

1. Label 1000 newspaper pages
2. Train YOLO Model to recognise advertisement illustrations¹
3. ‘Cut Out’ detected illustrations
4. Use CLIP model to generate ‘embeddings’²
5. Work with embeddings to examine and search through visuals

1. Redmon et al., “You Only Look Once”.

2. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision”.

Embeddings Visualised

Clustering

Use Clustering Algorithm to form groups of visually similar illustrations

Figure 6: Embeddings Clustered using K-means

Clusters

(a) Cluster 16

(b) Cluster 22

(c) Cluster 41

(d) Cluster 51

(e) Cluster 28

(f) Cluster 24

Figure 7: Samples of images in a given cluster

Temporal Trends

(a) Samples

(b) Trend

Figure 8: Cluster 0 Visualization

Temporal Trends

(a) Samples

(b) Trend

Figure 9: Cluster 0 Visualization

Temporal Trends

(a) Samples

(b) Trend

Figure 10: Cluster 28 Visualization

Word Driven Search

Search for keywords or concepts such as woman

(a) Cluster 16

(b) Cluster 22

Figure 11: Samples of images in a given cluster

Demo

https://tinyurl.com/KJGSAADS

https://ads.aronvandepol.com/

Problem

• Metadata
• What are we detecting?

Parsing with SAM

(a) 1. Original Image

(b) 2. Binarization

(c) 3. Pixel morphing

Figure 12: Pre-Processing

Segmentation (multimodal)

With SAM (Segment Anything)¹

(a) Segmentation Map

(b) Bounding Boxes

Figure 13

1. Kirillov et al., “Segment Anything”.

Boxes can be cutout…

(a) box 1

(b) box 2

(c) box 3

Figure 14: Box Cutouts

LLM (Large Language Model) transciption

Figure 15: box 1

{
  "image_id": "box_0.png",
  "raw_text": "無代進呈\n附屬品及 明年掛歷\n오늘부터\n오늘부터?\n今番歲暮를利用하여서\n半島諸君의厚誼를表하 고저아\n岳原價로드립니다\n京城寬勳洞\n朴德裕洋靴店\n二四八三番\n地方代金引換\n原價大賣\n歲暮割引\n新舊曆\n親切應酬\n歲暮問",

  "metadata": {
    "company": {
      "name_hanja": "朴德裕洋靴店",
      "name_mr": "Pak Tŏk-yu Yanghwa Chŏm",
      "category_hanja": "洋靴店",
      "category_mr": "Yanghwa chŏm",
      "nationality_prediction": "korean"
    },
    "address": {
      "full_hanja": "京城寬勳洞 二四八三番",
      "city_hanja": "京城",
      "city_mr": "Kyŏngsŏng",
      "district_hanja": "寬勳洞",
      "district_mr": "Kwanhun-dong",
      "number": "2483"
    },
    "linguistic": {
      "scripts_detected": ["hanja", "hangul"],
      "primary_language": "korean"
    }
  }
}

LLM (Large Language Model) transciption

Figure 16: box 6

{
  "image_id": "box_1.png",
  "raw_text": "學生帽\n各種\n小兒色帽種\n秋期大割引\n都賣散賣\n地方代金引換\n各種付屬品都賣\n地番二十六洞平公城京\n海昌商會",
  "metadata": {
    "company": {
      "name_hanja": "海昌商會",
      "name_mr": "Hae Chang Sanghoe",
      "category_hanja": "商會",
      "category_mr": "Sanghoe",
      "nationality_prediction": "korean"
    },
    "address": {
      "full_hanja": "京城公平洞二十六番地",
      "city_hanja": "京城",
      "city_mr": "Kyŏngsŏng",
      "district_hanja": "公 平 洞",
      "district_mr": "Kongpyŏng-dong",
      "number": "26"
    },
    "linguistic": {
      "scripts_detected": ["hanja", "hangul"],
      "primary_language": "korean"
    }
  }
}

Morinaga’s ads?

• When we Cutout boxes, Detect Ads, Extract Metadata

Tracking as of yet unseen changes becomes easy

Explaining these is a second

Thank you

Referenced Works

Chun, Yongkeun. “Becoming Imperial Brands: Japanese Advertising in Colonial Korea, 1920–1932.” Journal of Design History, December 2023, epad050. https://doi.org/10.1093/jdh/epad050.

Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, et al. “Segment Anything.” arXiv, April 2023. https://doi.org/10.48550/arXiv.2304.02643.

Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, et al. “Learning Transferable Visual Models From Natural Language Supervision.” arXiv, February 2021. https://doi.org/10.48550/arXiv.2103.00020.

Redmon, Joseph, Santosh Divvala, Ross Girshick, and Ali Farhadi. “You Only Look Once: Unified, Real-Time Object Detection.” In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779–88. Las Vegas, NV, USA: IEEE, 2016. https://doi.org/10.1109/CVPR.2016.91.