Project

General

Profile

Actions
Copy link

Support #239

open

Refining SCPC generation from RuleMaker

Added by Joseph Potvin 12 months ago. Updated about 1 month ago.

Status:
Feedback
Priority:
High
Assignee:
Joseph Potvin
Category:
-
Start date:
05/10/2024
Due date:
% Done:

0%

Estimated time:

Description

I tested my Rock Types CSV and am getting up to 78 columns being counted. Is that correct?

Color|1,6,11,16,21,26,31,36,41,46,51,56,61,66,71,76
Texture|2,7,12,17,22,27,32,37,42,47,52,57,62,67,72,77
Constituents|3,8,13,18,23,28,33,38,43,48,53,58,63,68,73,78
Igneous|1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18
Sedimentary|19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36
Metamorphic|37,38,39,40,41,42,43,44,45,46,47,48
Granite|1,2,3
Basalt|4,5,6
Obsidian|7,8,9
Pumice|10,11,12
Rhyolite|13,14,15
Gabbro|16,17,18
Limestone|19,20,21
Sandstone|22,23,24
Conglomerate|25,26,27
Shale|28,29,30
Dolomite|31,32,33
Coal|34,35,36
Gneiss|37,38,39
Schist|40,41,42
Marble|43,44,45
Quartzite|46,47,48
pink, gray, white, black|1
crystaline, coarse granular|2
quartz, feldspar, mica|3
gray, black|4
fine granular|5
olivine, pyroxene, plagioclase feldspar|6
black|7
smooth glassy|8
silica (SiO2), trace elements|9
gray|10
foamy, sharp edges|11
silica (SiO2), water vapor|12
gray, pink|13
fine granular, vesiclar|14
quartz, feldspar, mica|15
gray, black|16
coarse granular|17
pyroxene, olivine|18
grey|19
fine granular, smooth|20
calcium carbonate (CaCO3), organic matter|21
beige, brown, red ochre, yellow ochre|22
coarse granular|23
quartz, feldspar|24
various|25
coarse granular|26
various|27
gray, black|28
fine granular|29
kaolinite, illite, smectite, quartz, organic matter|30
white to gray or brown|31
fine granular|32
calcium magnesium carbonate (CaMg(CO3)2), other minerals|33
black, brown|34
chucky, dusty,brittle|35
carbon, hydrogen, oxygen, sulfur, nitrogen.|36
various|37
banded, foliated, fibrous|38
quartz, feldspar, mica|39
various|40
foliated, shiny|41
quartz, feldspar, mica|42
white, gray, pink, blue|43
hard, smooth, crystalline|44
recrystallized calcite (CaCO3)|45
various|46
hard, crystalline|47
quartz, other minerals|48

Files

computing-history-table.csv computing-history-table.csv 620 Bytes Joseph Potvin, 05/10/2024 08:18 PM
RockTypes_SampleLookupTable.csv RockTypes_SampleLookupTable.csv 1.5 KB Joseph Potvin, 05/10/2024 10:20 PM
nibsunspsc-gsinunspsc_FIXED-SEQUENTIAL+ADAPTED-FOR-MISSING-1ST-COLUMN-HEADER-2.csv nibsunspsc-gsinunspsc_FIXED-SEQUENTIAL+ADAPTED-FOR-MISSING-1ST-COLUMN-HEADER-2.csv 1.42 MB Joseph Potvin, 05/11/2024 01:31 AM
unspsc_hs_SCRC.txt unspsc_hs_SCRC.txt 2.84 MB Joseph Potvin, 05/11/2024 02:00 AM
clipboard-202503221551-xmqz2.png View clipboard-202503221551-xmqz2.png 23.8 KB Huda Hussain, 03/22/2025 07:51 PM
Actions
Copy link
 #1

Updated by Joseph Potvin 12 months ago

  • Subject changed from Refining SCRC generation from RuleMaker to Refining SCPC generation from RuleMaker

Reminder to use : instead of | in the SCPC.

Actions
Copy link
 #2

Updated by Joseph Potvin 12 months ago

I attach another table for your testing.

Actions
Copy link
 #3

Updated by Huda Hussain 12 months ago

Can you email me the rock types csv

Actions
Copy link
 #4

Updated by Joseph Potvin 12 months ago

  1. Attaching the Rock Types .CSV here also.

  2. Using the computing history table, something is wrong with the section for years since "Laptops/Tablets; Smartphones|11" is missing the "11" in "2000s-2010s".

1960s-1970s|1,7,13,19,25
1980s-1990s|2,8,14,20,26
2000s-2010s|3,9,15,21,27
2020s-2030s|4,10,16,22,28
Communication Devices|1,2,3,4,5,6,7,8
General-Purpose Computers|9,10,11,12,13,14,15,16,17,18,19,20
Wired|1,2,3,4
Wireless|5,6,7,8
Platform|9,10,11,12
Mobility|13,14,15,16
Storage|17,18,19,20
Landline Phones|1
Fax Machines|2
ISDN/DSL; Fibre-Optic|3
Multiplexed Fibre-Optic; Brain-Computer|4
Radio; Television|5
Wireless Phones|6
Smartphones|7
IoT; 5G/6G; Satellite Internet|8
Mainframe Computers|9
Desktop Computers|10
Laptops/Tablets; Smartphones|11
IoT; mRNA|12
Stationary|13
Movable|14
Mobile|15
Ubiquitous; Ambient|16
Magnetic Tapes|17
Floppy Disks; CDs|18
Flash Drives; Central Cloud|19
ReRAM/3D-NAND Drives; IPFS Cloud; DNA|20

Actions
Copy link
 #5

Updated by Huda Hussain 12 months ago

I just pushed a change, let me know if the issue persists

Actions
Copy link
 #6

Updated by Joseph Potvin 12 months ago

Looks good!

1960s-1970s:1,5,9,13,17
1980s-1990s:2,6,10,14,18
2000s-2010s:3,7,11,15,19
2020s-2030s:4,8,12,16,20
Communication Devices:1,2,3,4,5,6,7,8
General-Purpose Computers:9,10,11,12,13,14,15,16,17,18,19,20
Wired:1,2,3,4
Wireless:5,6,7,8
Platform:9,10,11,12
Mobility:13,14,15,16
Storage:17,18,19,20
Landline Phones:1
Fax Machines:2
ISDN/DSL; Fibre-Optic:3
Multiplexed Fibre-Optic; Brain-Computer:4
Radio; Television:5
Wireless Phones:6
Smartphones:7
IoT; 5G/6G; Satellite Internet:8
Mainframe Computers:9
Desktop Computers:10
Laptops/Tablets; Smartphones:11
IoT; mRNA:12
Stationary:13
Movable:14
Mobile:15
Ubiquitous; Ambient:16
Magnetic Tapes:17
Floppy Disks; CDs:18
Flash Drives; Central Cloud:19
ReRAM/3D-NAND Drives; IPFS Cloud; DNA:20

Actions
Copy link
 #7

Updated by Joseph Potvin 12 months ago

What do you think if "Save Copy of Rule to Local Machine" either now asks the user if they want SCPR or SCPC, to generate the entire lookup .JSON include metadata etc, with one or the other? Or have two separate buttons:

Save Copy of Rule (SCPR format) to Local Machine

Save Copy of Rule (SCPC format) to Local Machine

Actions
Copy link
 #8

Updated by Joseph Potvin 12 months ago

I tested with the 65,000+ row table of UNSPSC codes mapped to HS codes, and it seems to work elegantly! Still checking.

Both the CSV and the SCPC are attached here if you want to check details.

It performed the transformation very quickly, and the result is VERY human-readable.

The UNSPSC codes show up like this:

10102100:65,66,67,68,69,70,71,72
10110000:73,74,75,76,77,78,79,80
10111300:81,82,83,84,85,86,87,88
10120000:89,90,91,92,93,94,95,96
10121500:97,98,99,100,101,102,103,104
10121600:105,106,107,108,109,110,111,112

And those categories for SCPR columns are found like this:

Birds and fowl hatching eggs:65,66,67,68,69,70,71,72 
Domestic pet products:73,74,75,76,77,78,79,80 
Domestic pet treatments and accessories and equipment:81,82,83,84,85,86,87,88 
Animal feed:89,90,91,92,93,94,95,96 
Livestock feed:97,98,99,100,101,102,103,104 
Bird and fowl food:105,106,107,108,109,110,111,112

Which is to say, machine lookups should be faster and simpler too!

Actions
Copy link
 #9

Updated by Joseph Potvin 12 months ago

I guess that for conformance with JSON, the output should appear this way:

{
  "10102100": [65, 66, 67, 68, 69, 70, 71, 72],
  "10110000": [73, 74, 75, 76, 77, 78, 79, 80],
  "10111300": [81, 82, 83, 84, 85, 86, 87, 88],
  "10120000": [89, 90, 91, 92, 93, 94, 95, 96],
  "10121500": [97, 98, 99, 100, 101, 102, 103, 104],
  "10121600": [105, 106, 107, 108, 109, 110, 111, 112]
}

{
  "Birds and fowl hatching eggs": [65, 66, 67, 68, 69, 70, 71, 72],
  "Domestic pet products": [73, 74, 75, 76, 77, 78, 79, 80],
  "Domestic pet treatments and accessories and equipment": [81, 82, 83, 84, 85, 86, 87, 88],
  "Animal feed": [89, 90, 91, 92, 93, 94, 95, 96],
  "Livestock feed": [97, 98, 99, 100, 101, 102, 103, 104],
  "Bird and fowl food": [105, 106, 107, 108, 109, 110, 111, 112]
}
Actions
Copy link
 #10

Updated by Joseph Potvin 11 months ago

RE: When sorting into a category, does the order of the categories need to have the ability to be manipulated? Or is that not necessary.

Not necessary "in RuleMaker". We will assume that anyone bringing a lookup table into RuleMaker looks after the sorting in their own environment (e.g. a spreadsheet).

We assume that the table is already structured the way the user wants it.

(This is different from out assumption with a "rule", because RuleMaker provides users a new way to 'see' their rule with all those colored icons.)

Actions
Copy link
 #11

Updated by Huda Hussain 9 months ago

  • Status changed from New to Feedback
  • Assignee changed from Huda Hussain to Joseph Potvin

I believe this one has been solved.

Actions
Copy link
 #12

Updated by Joseph Potvin 7 months ago

  • Assignee changed from Joseph Potvin to Huda Hussain

Hi, Currently the SCPR is not rendering so I can't fully check the resulting SCPC. But I am curious about the following: In the sample table of rock types there are three rows with |1|2|3|4|...|60|61|62|63|64 (in bold text below). This doesn't seem correct; however I'll wait until I see the SCPR before offering a suggestion.

INDEX|DATA|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64
W1|"Column"|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64
W1.1|"Secondary Rock Group"|1|5|9|13|17|21|25|29|33|37|41|45|49|53|57|61
W1.2|"Color"|2|6|10|14|18|22|26|30|34|38|42|46|50|54|58|62
W1.3|"Texture"|3|7|11|15|19|23|27|31|35|39|43|47|51|55|59|63
W1.4|"Constituents"|4|8|12|16|20|24|28|32|36|40|44|48|52|56|60|64
W2|"Primary Rock Group"|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64
W2.1|"Igneous"|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24
W2.2|"Sedimentary"|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48
W2.3|"Metamorphic"|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64
T_W1.1_W2.1|"Granite"|1
T_W1.2_W2.1|"pink, gray, white, black"|2
T_W1.3_W2.1|"crystaline, coarse granular"|3
T_W1.4_W2.1|"quartz, feldspar, mica"|4
T_W1.1_W2.1|"Basalt"|5
T_W1.2_W2.1|"gray, black"|6
T_W1.3_W2.1|"fine granular"|7
T_W1.4_W2.1|"olivine, pyroxene, plagioclase feldspar"|8
T_W1.1_W2.1|"Obsidian"|9
T_W1.2_W2.1|"black"|10
T_W1.3_W2.1|"smooth glassy"|11
T_W1.4_W2.1|"silica (SiO2), trace elements"|12
T_W1.1_W2.1|"Pumice"|13
T_W1.2_W2.1|"gray"|14
T_W1.3_W2.1|"foamy, sharp edges"|15
T_W1.4_W2.1|"silica (SiO2), water vapor"|16
T_W1.1_W2.1|"Rhyolite"|17
T_W1.2_W2.1|"gray, pink"|18
T_W1.3_W2.1|"fine granular, vesiclar"|19
T_W1.4_W2.1|"quartz, feldspar, mica"|20
T_W1.1_W2.1|"Gabbro"|21
T_W1.2_W2.1|"gray, black"|22
T_W1.3_W2.1|"coarse granular"|23
T_W1.4_W2.1|"pyroxene, olivine"|24
T_W1.1_W2.2|"Limestone"|25
T_W1.2_W2.2|"grey"|26
T_W1.3_W2.2|"fine granular, smooth"|27
T_W1.4_W2.2|"calcium carbonate (CaCO3), organic matter"|28
T_W1.1_W2.2|"Sandstone"|29
T_W1.2_W2.2|"beige, brown, red ochre, yellow ochre"|30
T_W1.3_W2.2|"coarse granular"|31
T_W1.4_W2.2|"quartz, feldspar"|32
T_W1.1_W2.2|"Conglomerate"|33
T_W1.2_W2.2|"various"|34
T_W1.3_W2.2|"coarse granular"|35
T_W1.4_W2.2|"various"|36
T_W1.1_W2.2|"Shale"|37
T_W1.2_W2.2|"gray, black"|38
T_W1.3_W2.2|"fine granular"|39
T_W1.4_W2.2|"kaolinite, illite, smectite, quartz, organic matter"|40
T_W1.1_W2.2|"Dolomite"|41
T_W1.2_W2.2|"white to gray or brown"|42
T_W1.3_W2.2|"fine granular"|43
T_W1.4_W2.2|"calcium magnesium carbonate (CaMg(CO3)2), other minerals"|44
T_W1.1_W2.2|"Coal"|45
T_W1.2_W2.2|"black, brown"|46
T_W1.3_W2.2|"chucky, dusty,brittle"|47
T_W1.4_W2.2|"carbon, hydrogen, oxygen, sulfur, nitrogen."|48
T_W1.1_W2.3|"Gneiss"|49
T_W1.2_W2.3|"various"|50
T_W1.3_W2.3|"banded, foliated, fibrous"|51
T_W1.4_W2.3|"quartz, feldspar, mica"|52
T_W1.1_W2.3|"Schist"|53
T_W1.2_W2.3|"various"|54
T_W1.3_W2.3|"foliated, shiny"|55
T_W1.4_W2.3|"quartz, feldspar, mica"|56
T_W1.1_W2.3|"Marble"|57
T_W1.2_W2.3|"white, gray, pink, blue"|58
T_W1.3_W2.3|"hard, smooth, crystalline"|59
T_W1.4_W2.3|"recrystallized calcite (CaCO3)"|60
T_W1.1_W2.3|"Quartzite"|61
T_W1.2_W2.3|"various"|62
T_W1.3_W2.3|"hard, crystalline"|63
T_W1.4_W2.3|"quartz, other minerals"|64

Actions
Copy link
 #13

Updated by Huda Hussain about 1 month ago

So the reason I formatted like this, was because

INDEX|DATA|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64
W1|"Column"|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64
W1.1|"Secondary Rock Group"|1|5|9|13|17|21|25|29|33|37|41|45|49|53|57|61
W1.2|"Color"|2|6|10|14|18|22|26|30|34|38|42|46|50|54|58|62
W1.3|"Texture"|3|7|11|15|19|23|27|31|35|39|43|47|51|55|59|63
W1.4|"Constituents"|4|8|12|16|20|24|28|32|36|40|44|48|52|56|60|64
W2|"Primary Rock Group"|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64
W2.1|"Igneous"|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24
W2.2|"Sedimentary"|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48
W2.3|"Metamorphic"|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64
T_W1.1_W2.1|"Granite"|1
T_W1.2_W2.1|"pink, gray, white, black"|2
T_W1.3_W2.1|"crystaline, coarse granular"|3
T_W1.4_W2.1|"quartz, feldspar, mica"|4

Since “Granite” is classified under “Igneous”, and “Igneous” is part of the “Primary Rock Group”, then all the columns (1–24) associated with "Igneous" also belong to the parent category "Primary Rock Group".
That’s why the row for "Primary Rock Group" has all columns marked, as it inherits the scope of its child categories.
Apologies, this may require more discussion, since columns start at the cell level, not a categorical one.

Actions
Copy link
 #14

Updated by Huda Hussain about 1 month ago

Decided to not make a change on this.

Actions
Copy link

Also available in: Atom PDF