Merge develop into master (#678)

* Split window.ToDurationOffset into DurationOffset and DurationOffsetStrings

* Check and register annotations collectors if enabled.

* add csv fallback (#641) (#646)

* add csv fallback

* log class match

* add counts by source

* add test for pricing source counter and make the names of sources public

* Fix the issue with empty pod name on annotation metric.

* Only Emit label_ and annotation_ metrics if they have values!

* Simplify label and annotation metric to labels.

* Add annotations costmodel

* Additional annotations additions

* Ajay tripathy pvc error fix (#644)

* add csv fallback

* log class match

* filter empty volumenames out

* Ajay tripathy fix e2custom (#623)

* pass offset to ccdr

* remove conflict

* add e2custom support

* revert cntext background change

* fix improperly named constant for govcloud lookup (#651)

* Use compatibility region implementation. (#653)

* Add filter by annotations to AggregateCostModelHandler

* WIP AWS idle investigation

* Fix bug with multiple filters on label and annotations

* Fix idle allocation bug for windows < 1h; improve DurationOffset string conversion

* Commit missing test file

* aggregate on annotations

* Merge master into develop. (#658)

* add csv fallback (#641)

* add csv fallback

* log class match

* add counts by source

* add test for pricing source counter and make the names of sources public

* Bump to version 1.71.1

* Bump version to 1.72.0 (#659)

* Aggregation by label

Updated AssetSet and AssetSetRange to aggregate by a new property aggStrings []string.
The props []AssetProperty value is still maintained on a given AssetSet, but rather than use this for aggregation,
we now use aggStrings, which can include values other than enumerated props. Specifically, strings prefixed with "label:"
are interpreted as labels and can be grouped on.

Strings in aggStrings which match the enumerated AssetProperty strings are stored in AssetSet.props, as before.

Also updated the relevant asset_test tests to call the Aggregation funcs with []string rather than []AssetProperty

* WIP add labels to nodes

* Now copying all labels into node objects

* Updates:

- Factored out `AssetSet.properties []AssetProperty`, which has been replaced by `AssetSet.aggregateBy []string`.

- Function `key()` now uses reserved word `__unallocated__` in keys for assets that do not have the given prop defined. Previous behavior was to omit that part of the key.

- Function `key()` now emits errors for given aggregation keys that are not either an `AssetProperty` or a string prefixed with `"label:"`.

- Updated tests to expect `__unallocated__` in relevant parts of keys.

* Fixes from Nikos comments

* Implement kubecost.AllocationSetRange.InsertRange and test; refactor custom approx implementations into util.IsApproximately for testing

* Returning single Error from key() func and disallowing grouping on empty label

* Capacity Optimizations (#664)

* Update to latest bingen read optimizations by allocating the required map space ahead of time. Apply to Properties as well.

* A few more smaller optimizations on Clone()

* simple pvfix (#663)

* Add annotation to allocation key

* go fmt

* Adds the concept of an AssetCredit to pkg/kubecost/asset

* undo go fmt

* Checking for errors from key() everywhere and made __undefined__ a constant

* process annotations in etl

* Simplified Cloud.Credit down to a simple float64 rather than objects containing credit metadata

* Allocation ETL: on-demand idle cost with unit testing

* Make AllocationSet.insert safer

* Ajay tripathy fix pvcalls (#667)

* simple pvfix

* Update to latest bingen read optimizations by allocating the required map space ahead of time. Apply to Properties as well.

* A few more smaller optimizations on Clone()

* fix extra PVlookups

Co-authored-by: Matt Bolt <mbolt35@gmail.com>

* Re-implemented 'propsEqual' check in accumulate(). Can't be exactly propsEqual() because they aren't props anymore...but wrote equivalent functions for []string

* Allocation ETL: on-demand external cost

* Allocation ETL: on-demand external cost; implement Properties.AggregationStrings

* Allocation ETL: on-demand external cost: fix Properties

* camel case json property

* refactor map merging function

* WIP logging for Allocation ETL: on-demand external cost

* Added a unit test for Aggregating by label.

* Added a comment explaining the nature of Credit

* Changed the label-key format to match allocation. Format for a label's key entry is now '/key=value/' rather than '/value/'

* undefined labels don't list key= before the __undefined__ value

* Now treating labels with value '' as unset labels

* Fix test broken in merge

* Ajay tripathy remove 2d cache (#668)

* simple pvfix

* Update to latest bingen read optimizations by allocating the required map space ahead of time. Apply to Properties as well.

* A few more smaller optimizations on Clone()

* fix extra PVlookups

* remove 2d cache

Co-authored-by: Matt Bolt <mbolt35@gmail.com>

* Active minutes query includes provider id

* Refactor ClusterNodes and use provider ID

See comments for more detailed explanation.

* Detailed comment describing buildNodeMap

* Unit test for mergeTypeMaps

* Initial simple test for buildNodeMap

* More buildNodeMap tests

* Renamed test file

* Removed check made obsolete by refactor

* Grammar and spelling fix in a comment

* Moved labels map into helper function

Also updated tests and followed the new query name
for preemptible, resNodeLabels -> resIsSpot.

* Allocation ETL: on-demand external cost: fix naming convention

* Fix bugs with external cost AggregateBy; fix test funcs; remove logs

* Update TODOs; add log

* Unit tests for E2 manual cost adjustment (#676)

* e2 fixes

* Added tests for CPU cost adjustment for e2

Co-authored-by: Ajay Tripathy <4tripathy@gmail.com>

* Bump version (#677)

Co-authored-by: Niko Kovacevic <nikovacevic@gmail.com>
Co-authored-by: Matt Bolt <mbolt35@gmail.com>
Co-authored-by: Sean Holcomb <seanholcomb@gmail.com>
Co-authored-by: Neal Ormsbee <neal.ormsbee@gmail.com>
Co-authored-by: Sean Holcomb <sean@kubecost.com>
Co-authored-by: Michael Dresser <michael@kubecost.com>

go.mod

@@ -8,6 +8,7 @@ require (
 	github.com/Azure/azure-sdk-for-go v24.1.0+incompatible
 	github.com/Azure/go-autorest v11.3.2+incompatible
 	github.com/aws/aws-sdk-go v1.28.9
+	github.com/davecgh/go-spew v1.1.1
 	github.com/dimchansky/utfbom v1.1.0 // indirect
 	github.com/getsentry/sentry-go v0.6.1
 	github.com/google/martian v2.1.0+incompatible

pkg/costmodel/aggregation.go

@@ -375,6 +375,20 @@ func AggregateCostData(costData map[string]*CostData, field string, subfields []
 				if !found {
 					aggregateDatum(cp, aggregations, costDatum, field, subfields, rate, UnallocatedSubfield, discount, customDiscount, idleCoefficient, false)
 				}
+			} else if field == "annotation" {
+				found := false
+				if costDatum.Annotations != nil {
+					for _, sf := range subfields {
+						if subfieldName, ok := costDatum.Annotations[sf]; ok {
+							aggregateDatum(cp, aggregations, costDatum, field, subfields, rate, subfieldName, discount, customDiscount, idleCoefficient, false)
+							found = true
+							break
+						}
+					}
+				}
+				if !found {
+					aggregateDatum(cp, aggregations, costDatum, field, subfields, rate, UnallocatedSubfield, discount, customDiscount, idleCoefficient, false)
+				}
 			} else if field == "pod" {
 				aggregateDatum(cp, aggregations, costDatum, field, subfields, rate, costDatum.Namespace+"/"+costDatum.PodName, discount, customDiscount, idleCoefficient, false)
 			} else if field == "container" {
@@ -585,6 +599,7 @@ func aggregateDatum(cp cloud.Provider, aggregations map[string]*Aggregation, cos
 				props.SetControllerKind(kind)
 			}
 			props.SetLabels(costDatum.Labels)
+			props.SetAnnotations(costDatum.Annotations)
 			props.SetNamespace(costDatum.Namespace)
 			props.SetPod(costDatum.PodName)
 			props.SetServices(costDatum.Services)
@@ -1127,6 +1142,14 @@ func (a *Accesses) ComputeAggregateCostModel(promClient prometheusClient.Client,
 					}
 				}
 			}
+		} else if field == "annotation" {
+			if costDatum.Annotations != nil {
+				for _, sf := range subfields {
+					if subfieldName, ok := costDatum.Annotations[sf]; ok {
+						return fmt.Sprintf("%s=%s", sf, subfieldName)
+					}
+				}
+			}
 		} else if field == "pod" {
 			return costDatum.Namespace + "/" + costDatum.PodName
 		} else if field == "container" {
@@ -1248,15 +1271,15 @@ func (a *Accesses) ComputeAggregateCostModel(promClient prometheusClient.Client,
 					ae := aggregateEnvironment(cd)
 					for _, v := range vs {
 						if v == "__unallocated__" { // Special case. __unallocated__ means return all pods without the attached label
-							if _, ok := cd.Labels[label]; !ok {
+							if _, ok := cd.Labels[l]; !ok {
 								return true, ae
 							}
 						}
-						if cd.Labels[label] == v {
+						if cd.Labels[l] == v {
 							return true, ae
 						} else if strings.HasSuffix(v, "*") { // trigger wildcard prefix filtering
 							vTrim := strings.TrimSuffix(v, "*")
-							if strings.HasPrefix(cd.Labels[label], vTrim) {
+							if strings.HasPrefix(cd.Labels[l], vTrim) {
 								return true, ae
 							}
 						}
@@ -1268,6 +1291,55 @@ func (a *Accesses) ComputeAggregateCostModel(promClient prometheusClient.Client,
 		}
 	}
 
+	if filters["annotations"] != "" {
+		// annotations are expected to be comma-separated and to take the form key=value
+		// e.g. app=cost-analyzer,app.kubernetes.io/instance=kubecost
+		// each different annotation will be applied as an AND
+		// multiple values for a single annotation will be evaluated as an OR
+		annotationValues := map[string][]string{}
+		as := strings.Split(filters["annotations"], ",")
+		for _, annot := range as {
+			aTrim := strings.TrimSpace(annot)
+			annotation := strings.Split(aTrim, "=")
+			if len(annotation) == 2 {
+				an := prom.SanitizeLabelName(strings.TrimSpace(annotation[0]))
+				av := strings.TrimSpace(annotation[1])
+				annotationValues[an] = append(annotationValues[an], av)
+			} else {
+				// annotation is not of the form name=value, so log it and move on
+				log.Warningf("ComputeAggregateCostModel: skipping illegal annotation filter: %s", annot)
+			}
+		}
+
+		// Generate FilterFunc for each set of annotation filters by invoking a function instead of accessing
+		// values by closure to prevent reference-type looping bug.
+		// (see https://github.com/golang/go/wiki/CommonMistakes#using-reference-to-loop-iterator-variable)
+		for annotation, values := range annotationValues {
+			ff := (func(l string, vs []string) FilterFunc {
+				return func(cd *CostData) (bool, string) {
+					ae := aggregateEnvironment(cd)
+					for _, v := range vs {
+						if v == "__unallocated__" { // Special case. __unallocated__ means return all pods without the attached label
+							if _, ok := cd.Annotations[l]; !ok {
+								return true, ae
+							}
+						}
+						if cd.Annotations[l] == v {
+							return true, ae
+						} else if strings.HasSuffix(v, "*") { // trigger wildcard prefix filtering
+							vTrim := strings.TrimSuffix(v, "*")
+							if strings.HasPrefix(cd.Annotations[l], vTrim) {
+								return true, ae
+							}
+						}
+					}
+					return false, ae
+				}
+			})(annotation, values)
+			filterFuncs = append(filterFuncs, ff)
+		}
+	}
+
 	// clear cache prior to checking the cache so that a clearCache=true
 	// request always returns a freshly computed value
 	if clearCache {
@@ -1619,7 +1691,28 @@ func GenerateAggKey(window kubecost.Window, field string, subfields []string, op
 	sort.Strings(lFilters)
 	lFilterStr := strings.Join(lFilters, ",")
 
-	filterStr := fmt.Sprintf("%s:%s:%s:%s:%s", nsFilterStr, nodeFilterStr, cFilterStr, lFilterStr, podPrefixFiltersStr)
+	// parse, trim, and sort annotation filters
+	aFilters := []string{}
+	if afs, ok := opts.Filters["annotations"]; ok && afs != "" {
+		for _, af := range strings.Split(afs, ",") {
+			// trim whitespace from the annotation name and the annotation value
+			// of each annotation name/value pair, then reconstruct
+			// e.g. "tier = frontend, app = kubecost" == "app=kubecost,tier=frontend"
+			afa := strings.Split(af, "=")
+			if len(afa) == 2 {
+				afn := strings.TrimSpace(afa[0])
+				afv := strings.TrimSpace(afa[1])
+				aFilters = append(aFilters, fmt.Sprintf("%s=%s", afn, afv))
+			} else {
+				// annotation is not of the form name=value, so log it and move on
+				klog.V(2).Infof("[Warning] GenerateAggKey: skipping illegal annotation filter: %s", af)
+			}
+		}
+	}
+	sort.Strings(aFilters)
+	aFilterStr := strings.Join(aFilters, ",")
+
+	filterStr := fmt.Sprintf("%s:%s:%s:%s:%s:%s", nsFilterStr, nodeFilterStr, cFilterStr, lFilterStr, aFilterStr, podPrefixFiltersStr)
 
 	sort.Strings(subfields)
 	fieldStr := fmt.Sprintf("%s:%s", field, strings.Join(subfields, ","))
@@ -1726,26 +1819,26 @@ func (a *Accesses) warmAggregateCostModelCache() {
 		}
 	}(sem)
 
-	// 2 day
-	go func(sem *util.Semaphore) {
-		defer errors.HandlePanic()
+	if !env.IsETLEnabled() {
+		// 2 day
+		go func(sem *util.Semaphore) {
+			defer errors.HandlePanic()
 
-		duration := "2d"
-		offset := "1m"
-		durHrs := "48h"
-		dur := 2 * 24 * time.Hour
+			duration := "2d"
+			offset := "1m"
+			durHrs := "48h"
+			dur := 2 * 24 * time.Hour
 
-		for {
-			sem.Acquire()
-			warmFunc(duration, durHrs, offset, false)
-			sem.Return()
+			for {
+				sem.Acquire()
+				warmFunc(duration, durHrs, offset, false)
+				sem.Return()
 
-			log.Infof("aggregation: warm cache: %s", duration)
-			time.Sleep(a.GetCacheRefresh(dur))
-		}
-	}(sem)
+				log.Infof("aggregation: warm cache: %s", duration)
+				time.Sleep(a.GetCacheRefresh(dur))
+			}
+		}(sem)
 
-	if !env.IsETLEnabled() {
 		// 7 day
 		go func(sem *util.Semaphore) {
 			defer errors.HandlePanic()
@@ -1858,10 +1951,8 @@ func (a *Accesses) AggregateCostModelHandler(w http.ResponseWriter, r *http.Requ
 	namespace := r.URL.Query().Get("namespace")
 	cluster := r.URL.Query().Get("cluster")
 	labels := r.URL.Query().Get("labels")
+	annotations := r.URL.Query().Get("annotations")
 	podprefix := r.URL.Query().Get("podprefix")
-	labelArray := strings.Split(labels, "=")
-	labelArray[0] = strings.ReplaceAll(labelArray[0], "-", "_")
-	labels = strings.Join(labelArray, "=")
 	field := r.URL.Query().Get("aggregation")
 	sharedNamespaces := r.URL.Query().Get("sharedNamespaces")
 	sharedLabelNames := r.URL.Query().Get("sharedLabelNames")
@@ -1920,7 +2011,7 @@ func (a *Accesses) AggregateCostModelHandler(w http.ResponseWriter, r *http.Requ
 	}
 
 	// aggregation subfield is required when aggregation field is "label"
-	if field == "label" && len(subfields) == 0 {
+	if (field == "label" || field == "annotation") && len(subfields) == 0 {
 		WriteError(w, BadRequest("Missing aggregation field parameter"))
 		return
 	}
@@ -1936,10 +2027,11 @@ func (a *Accesses) AggregateCostModelHandler(w http.ResponseWriter, r *http.Requ
 	// labels are expected to be comma-separated and to take the form key=value
 	// e.g. app=cost-analyzer,app.kubernetes.io/instance=kubecost
 	opts.Filters = map[string]string{
-		"namespace": namespace,
-		"cluster":   cluster,
-		"labels":    labels,
-		"podprefix": podprefix,
+		"namespace":   namespace,
+		"cluster":     cluster,
+		"labels":      labels,
+		"annotations": annotations,
+		"podprefix":   podprefix,
 	}
 
 	// parse shared resources
@@ -1953,7 +2045,7 @@ func (a *Accesses) AggregateCostModelHandler(w http.ResponseWriter, r *http.Requ
 		sln = strings.Split(sharedLabelNames, ",")
 		slv = strings.Split(sharedLabelValues, ",")
 		if len(sln) != len(slv) || slv[0] == "" {
-			WriteError(w, BadRequest("Supply exacly one shared label value per shared label name"))
+			WriteError(w, BadRequest("Supply exactly one shared label value per shared label name"))
 			return
 		}
 	}

pkg/costmodel/cluster.go

@@ -391,15 +391,30 @@ type Node struct {
 	Start        time.Time
 	End          time.Time
 	Minutes      float64
+	Labels       map[string]string
 }
 
+// GKE lies about the number of cores e2 nodes have. This table
+// contains a mapping from node type -> actual CPU cores
+// for those cases.
 var partialCPUMap = map[string]float64{
 	"e2-micro":  0.25,
 	"e2-small":  0.5,
 	"e2-medium": 1.0,
 }
 
-func ClusterNodes(cp cloud.Provider, client prometheus.Client, duration, offset time.Duration) (map[string]*Node, error) {
+type NodeIdentifier struct {
+	Cluster    string
+	Name       string
+	ProviderID string
+}
+
+type nodeIdentifierNoProviderID struct {
+	Cluster string
+	Name    string
+}
+
+func ClusterNodes(cp cloud.Provider, client prometheus.Client, duration, offset time.Duration) (map[NodeIdentifier]*Node, error) {
 	durationStr := fmt.Sprintf("%dm", int64(duration.Minutes()))
 	offsetStr := fmt.Sprintf(" offset %dm", int64(offset.Minutes()))
 	if offset < time.Minute {
@@ -425,11 +440,12 @@ func ClusterNodes(cp cloud.Provider, client prometheus.Client, duration, offset
 	queryNodeRAMCost := fmt.Sprintf(`sum_over_time((avg(kube_node_status_capacity_memory_bytes) by (cluster_id, node) * on(cluster_id, node) group_right avg(node_ram_hourly_cost) by (cluster_id, node, instance_type, provider_id))[%s:%dm]%s) / 1024 / 1024 / 1024 * %f`, durationStr, minsPerResolution, offsetStr, hourlyToCumulative)
 	queryNodeRAMBytes := fmt.Sprintf(`avg_over_time(avg(kube_node_status_capacity_memory_bytes) by (cluster_id, node)[%s:%dm]%s)`, durationStr, minsPerResolution, offsetStr)
 	queryNodeGPUCost := fmt.Sprintf(`sum_over_time((avg(node_gpu_hourly_cost * %d.0 / 60.0) by (cluster_id, node, provider_id))[%s:%dm]%s)`, minsPerResolution, durationStr, minsPerResolution, offsetStr)
-	queryNodeLabels := fmt.Sprintf(`avg_over_time(kubecost_node_is_spot[%s:%dm]%s)`, durationStr, minsPerResolution, offsetStr)
 	queryNodeCPUModeTotal := fmt.Sprintf(`sum(rate(node_cpu_seconds_total[%s:%dm]%s)) by (kubernetes_node, cluster_id, mode)`, durationStr, minsPerResolution, offsetStr)
 	queryNodeRAMSystemPct := fmt.Sprintf(`sum(sum_over_time(container_memory_working_set_bytes{container_name!="POD",container_name!="",namespace="kube-system"}[%s:%dm]%s)) by (instance, cluster_id) / avg(label_replace(sum(sum_over_time(kube_node_status_capacity_memory_bytes[%s:%dm]%s)) by (node, cluster_id), "instance", "$1", "node", "(.*)")) by (instance, cluster_id)`, durationStr, minsPerResolution, offsetStr, durationStr, minsPerResolution, offsetStr)
 	queryNodeRAMUserPct := fmt.Sprintf(`sum(sum_over_time(container_memory_working_set_bytes{container_name!="POD",container_name!="",namespace!="kube-system"}[%s:%dm]%s)) by (instance, cluster_id) / avg(label_replace(sum(sum_over_time(kube_node_status_capacity_memory_bytes[%s:%dm]%s)) by (node, cluster_id), "instance", "$1", "node", "(.*)")) by (instance, cluster_id)`, durationStr, minsPerResolution, offsetStr, durationStr, minsPerResolution, offsetStr)
-	queryActiveMins := fmt.Sprintf(`avg(node_total_hourly_cost) by (node,cluster_id)[%s:%dm]%s`, durationStr, minsPerResolution, offsetStr)
+	queryActiveMins := fmt.Sprintf(`avg(node_total_hourly_cost) by (node, cluster_id, provider_id)[%s:%dm]%s`, durationStr, minsPerResolution, offsetStr)
+	queryIsSpot := fmt.Sprintf(`avg_over_time(kubecost_node_is_spot[%s:%dm]%s)`, durationStr, minsPerResolution, offsetStr)
+	queryLabels := fmt.Sprintf(`count_over_time(kube_node_labels[%s:%dm]%s)`, durationStr, minsPerResolution, offsetStr)
 
 	// Return errors if these fail
 	resChNodeCPUCost := requiredCtx.Query(queryNodeCPUCost)
@@ -437,24 +453,26 @@ func ClusterNodes(cp cloud.Provider, client prometheus.Client, duration, offset
 	resChNodeRAMCost := requiredCtx.Query(queryNodeRAMCost)
 	resChNodeRAMBytes := requiredCtx.Query(queryNodeRAMBytes)
 	resChNodeGPUCost := requiredCtx.Query(queryNodeGPUCost)
-	resChNodeLabels := requiredCtx.Query(queryNodeLabels)
 	resChActiveMins := requiredCtx.Query(queryActiveMins)
+	resChIsSpot := requiredCtx.Query(queryIsSpot)
 
 	// Do not return errors if these fail, but log warnings
 	resChNodeCPUModeTotal := optionalCtx.Query(queryNodeCPUModeTotal)
 	resChNodeRAMSystemPct := optionalCtx.Query(queryNodeRAMSystemPct)
 	resChNodeRAMUserPct := optionalCtx.Query(queryNodeRAMUserPct)
+	resChLabels := optionalCtx.Query(queryLabels)
 
 	resNodeCPUCost, _ := resChNodeCPUCost.Await()
 	resNodeCPUCores, _ := resChNodeCPUCores.Await()
 	resNodeGPUCost, _ := resChNodeGPUCost.Await()
 	resNodeRAMCost, _ := resChNodeRAMCost.Await()
 	resNodeRAMBytes, _ := resChNodeRAMBytes.Await()
-	resNodeLabels, _ := resChNodeLabels.Await()
+	resIsSpot, _ := resChIsSpot.Await()
 	resNodeCPUModeTotal, _ := resChNodeCPUModeTotal.Await()
 	resNodeRAMSystemPct, _ := resChNodeRAMSystemPct.Await()
 	resNodeRAMUserPct, _ := resChNodeRAMUserPct.Await()
 	resActiveMins, _ := resChActiveMins.Await()
+	resLabels, _ := resChLabels.Await()
 
 	if optionalCtx.HasErrors() {
 		for _, err := range optionalCtx.Errors() {
@@ -469,343 +487,35 @@ func ClusterNodes(cp cloud.Provider, client prometheus.Client, duration, offset
 		return nil, requiredCtx.ErrorCollection()
 	}
 
-	nodeMap := map[string]*Node{}
-
-	for _, result := range resNodeCPUCost {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := result.GetString("node")
-		if err != nil {
-			log.Warningf("ClusterNodes: CPU cost data missing node")
-			continue
-		}
-
-		nodeType, _ := result.GetString("instance_type")
-		providerID, _ := result.GetString("provider_id")
-
-		cpuCost := result.Values[0].Value
-
-		key := fmt.Sprintf("%s/%s", cluster, name)
-		if _, ok := nodeMap[key]; !ok {
-			nodeMap[key] = &Node{
-				Cluster:      cluster,
-				Name:         name,
-				NodeType:     nodeType,
-				ProviderID:   cp.ParseID(providerID),
-				CPUBreakdown: &ClusterCostsBreakdown{},
-				RAMBreakdown: &ClusterCostsBreakdown{},
-			}
-		}
-		nodeMap[key].CPUCost += cpuCost
-		nodeMap[key].NodeType = nodeType
-		if nodeMap[key].ProviderID == "" {
-			nodeMap[key].ProviderID = cp.ParseID(providerID)
-		}
-	}
-
-	for _, result := range resNodeCPUCores {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := result.GetString("node")
-		if err != nil {
-			log.Warningf("ClusterNodes: CPU cores data missing node")
-			continue
-		}
-
-		cpuCores := result.Values[0].Value
-
-		key := fmt.Sprintf("%s/%s", cluster, name)
-		if _, ok := nodeMap[key]; !ok {
-			nodeMap[key] = &Node{
-				Cluster:      cluster,
-				Name:         name,
-				CPUBreakdown: &ClusterCostsBreakdown{},
-				RAMBreakdown: &ClusterCostsBreakdown{},
-			}
-		}
-		node := nodeMap[key]
-		if v, ok := partialCPUMap[node.NodeType]; ok {
-			node.CPUCores = v
-			if cpuCores > 0 {
-				adjustmentFactor := v / cpuCores
-				node.CPUCost = node.CPUCost * adjustmentFactor
-			}
-		} else {
-			nodeMap[key].CPUCores = cpuCores
-		}
-	}
-
-	for _, result := range resNodeRAMCost {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := result.GetString("node")
-		if err != nil {
-			log.Warningf("ClusterNodes: RAM cost data missing node")
-			continue
-		}
-
-		nodeType, _ := result.GetString("instance_type")
-		providerID, _ := result.GetString("provider_id")
-
-		ramCost := result.Values[0].Value
-
-		key := fmt.Sprintf("%s/%s", cluster, name)
-		if _, ok := nodeMap[key]; !ok {
-			nodeMap[key] = &Node{
-				Cluster:      cluster,
-				Name:         name,
-				NodeType:     nodeType,
-				ProviderID:   cp.ParseID(providerID),
-				CPUBreakdown: &ClusterCostsBreakdown{},
-				RAMBreakdown: &ClusterCostsBreakdown{},
-			}
-		}
-		nodeMap[key].RAMCost += ramCost
-		nodeMap[key].NodeType = nodeType
-		if nodeMap[key].ProviderID == "" {
-			nodeMap[key].ProviderID = cp.ParseID(providerID)
-		}
-	}
-
-	for _, result := range resNodeRAMBytes {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := result.GetString("node")
-		if err != nil {
-			log.Warningf("ClusterNodes: RAM bytes data missing node")
-			continue
-		}
-
-		ramBytes := result.Values[0].Value
-
-		key := fmt.Sprintf("%s/%s", cluster, name)
-		if _, ok := nodeMap[key]; !ok {
-			nodeMap[key] = &Node{
-				Cluster:      cluster,
-				Name:         name,
-				CPUBreakdown: &ClusterCostsBreakdown{},
-				RAMBreakdown: &ClusterCostsBreakdown{},
-			}
-		}
-		nodeMap[key].RAMBytes = ramBytes
-	}
-
-	for _, result := range resNodeGPUCost {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := result.GetString("node")
-		if err != nil {
-			log.Warningf("ClusterNodes: GPU cost data missing node")
-			continue
-		}
-
-		nodeType, _ := result.GetString("instance_type")
-		providerID, _ := result.GetString("provider_id")
-
-		gpuCost := result.Values[0].Value
-
-		key := fmt.Sprintf("%s/%s", cluster, name)
-		if _, ok := nodeMap[key]; !ok {
-			nodeMap[key] = &Node{
-				Cluster:      cluster,
-				Name:         name,
-				NodeType:     nodeType,
-				ProviderID:   cp.ParseID(providerID),
-				CPUBreakdown: &ClusterCostsBreakdown{},
-				RAMBreakdown: &ClusterCostsBreakdown{},
-			}
-		}
-		nodeMap[key].GPUCost += gpuCost
-		if nodeMap[key].ProviderID == "" {
-			nodeMap[key].ProviderID = cp.ParseID(providerID)
-		}
-	}
-
-	// Mapping of cluster/node=cpu for computing resource efficiency
-	clusterNodeCPUTotal := map[string]float64{}
-	// Mapping of cluster/node:mode=cpu for computing resource efficiency
-	clusterNodeModeCPUTotal := map[string]map[string]float64{}
+	cpuCostMap, clusterAndNameToType1 := buildCPUCostMap(resNodeCPUCost, cp.ParseID)
+	ramCostMap, clusterAndNameToType2 := buildRAMCostMap(resNodeRAMCost, cp.ParseID)
+	gpuCostMap, clusterAndNameToType3 := buildGPUCostMap(resNodeGPUCost, cp.ParseID)
 
-	// Build intermediate structures for CPU usage by (cluster, node) and by
-	// (cluster, node, mode) for computing resouce efficiency
-	for _, result := range resNodeCPUModeTotal {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
+	clusterAndNameToTypeIntermediate := mergeTypeMaps(clusterAndNameToType1, clusterAndNameToType2)
+	clusterAndNameToType := mergeTypeMaps(clusterAndNameToTypeIntermediate, clusterAndNameToType3)
 
-		node, err := result.GetString("kubernetes_node")
-		if err != nil {
-			log.DedupedWarningf(5, "ClusterNodes: CPU mode data missing node")
-			continue
-		}
+	cpuCoresMap := buildCPUCoresMap(resNodeCPUCores, clusterAndNameToType)
 
-		mode, err := result.GetString("mode")
-		if err != nil {
-			log.Warningf("ClusterNodes: unable to read CPU mode: %s", err)
-			mode = "other"
-		}
-
-		key := fmt.Sprintf("%s/%s", cluster, node)
+	ramBytesMap := buildRAMBytesMap(resNodeRAMBytes)
 
-		total := result.Values[0].Value
+	ramUserPctMap := buildRAMUserPctMap(resNodeRAMUserPct)
+	ramSystemPctMap := buildRAMSystemPctMap(resNodeRAMSystemPct)
 
-		// Increment total
-		clusterNodeCPUTotal[key] += total
-
-		// Increment mode
-		if _, ok := clusterNodeModeCPUTotal[key]; !ok {
-			clusterNodeModeCPUTotal[key] = map[string]float64{}
-		}
-		clusterNodeModeCPUTotal[key][mode] += total
-	}
-
-	// Compute resource efficiency from intermediate structures
-	for key, total := range clusterNodeCPUTotal {
-		if modeTotals, ok := clusterNodeModeCPUTotal[key]; ok {
-			for mode, subtotal := range modeTotals {
-				// Compute percentage for the current cluster, node, mode
-				pct := 0.0
-				if total > 0 {
-					pct = subtotal / total
-				}
-
-				if _, ok := nodeMap[key]; !ok {
-					log.Warningf("ClusterNodes: CPU mode data for unidentified node")
-					continue
-				}
-
-				switch mode {
-				case "idle":
-					nodeMap[key].CPUBreakdown.Idle += pct
-				case "system":
-					nodeMap[key].CPUBreakdown.System += pct
-				case "user":
-					nodeMap[key].CPUBreakdown.User += pct
-				default:
-					nodeMap[key].CPUBreakdown.Other += pct
-				}
-			}
-		}
-	}
-
-	for _, result := range resNodeRAMSystemPct {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := result.GetString("instance")
-		if err != nil {
-			log.Warningf("ClusterNodes: RAM system percent missing node")
-			continue
-		}
-
-		pct := result.Values[0].Value
-
-		key := fmt.Sprintf("%s/%s", cluster, name)
-		if _, ok := nodeMap[key]; !ok {
-			log.Warningf("ClusterNodes: RAM system percent for unidentified node")
-			continue
-		}
-
-		nodeMap[key].RAMBreakdown.System += pct
-	}
-
-	for _, result := range resNodeRAMUserPct {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := result.GetString("instance")
-		if err != nil {
-			log.Warningf("ClusterNodes: RAM system percent missing node")
-			continue
-		}
+	cpuBreakdownMap := buildCPUBreakdownMap(resNodeCPUModeTotal)
+	activeDataMap := buildActiveDataMap(resActiveMins, resolution, cp.ParseID)
+	preemptibleMap := buildPreemptibleMap(resIsSpot, cp.ParseID)
+	labelsMap := buildLabelsMap(resLabels)
 
-		pct := result.Values[0].Value
-
-		key := fmt.Sprintf("%s/%s", cluster, name)
-		if _, ok := nodeMap[key]; !ok {
-			log.Warningf("ClusterNodes: RAM system percent for unidentified node")
-			continue
-		}
-
-		nodeMap[key].RAMBreakdown.User += pct
-	}
-
-	for _, result := range resActiveMins {
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := result.GetString("node")
-		if err != nil {
-			log.Warningf("ClusterNodes: active mins missing node")
-			continue
-		}
-
-		key := fmt.Sprintf("%s/%s", cluster, name)
-		if _, ok := nodeMap[key]; !ok {
-			log.Warningf("ClusterNodes: active mins for unidentified node")
-			continue
-		}
-
-		if len(result.Values) == 0 {
-			continue
-		}
-
-		s := time.Unix(int64(result.Values[0].Timestamp), 0)
-		e := time.Unix(int64(result.Values[len(result.Values)-1].Timestamp), 0).Add(resolution)
-		mins := e.Sub(s).Minutes()
-
-		// TODO niko/assets if mins >= threshold, interpolate for missing data?
-
-		nodeMap[key].End = e
-		nodeMap[key].Start = s
-		nodeMap[key].Minutes = mins
-	}
-
-	// Determine preemptibility with node labels
-	for _, result := range resNodeLabels {
-		nodeName, err := result.GetString("node")
-		if err != nil {
-			continue
-		}
-
-		// GCP preemptible label
-		pre := result.Values[0].Value
-
-		cluster, err := result.GetString("cluster_id")
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-		key := fmt.Sprintf("%s/%s", cluster, nodeName)
-		if node, ok := nodeMap[key]; pre > 0.0 && ok {
-			node.Preemptible = true
-		}
-
-		// TODO AWS preemptible
-
-		// TODO Azure preemptible
-	}
+	nodeMap := buildNodeMap(
+		cpuCostMap, ramCostMap, gpuCostMap,
+		cpuCoresMap, ramBytesMap, ramUserPctMap,
+		ramSystemPctMap,
+		cpuBreakdownMap,
+		activeDataMap,
+		preemptibleMap,
+		labelsMap,
+		clusterAndNameToType,
+	)
 
 	c, err := cp.GetConfig()
 	if err != nil {

pkg/costmodel/cluster_helpers.go

@@ -0,0 +1,658 @@
+package costmodel
+
+import (
+	"time"
+
+	"github.com/kubecost/cost-model/pkg/env"
+	"github.com/kubecost/cost-model/pkg/log"
+	"github.com/kubecost/cost-model/pkg/prom"
+)
+
+// mergeTypeMaps takes two maps of (cluster name, node name) -> node type
+// and combines them into a single map, preferring the k/v pairs in
+// the first map.
+func mergeTypeMaps(clusterAndNameToType1, clusterAndNameToType2 map[nodeIdentifierNoProviderID]string) map[nodeIdentifierNoProviderID]string {
+	merged := map[nodeIdentifierNoProviderID]string{}
+	for k, v := range clusterAndNameToType2 {
+		merged[k] = v
+	}
+
+	// This ordering ensures the mappings in the first arg are preferred.
+	for k, v := range clusterAndNameToType1 {
+		merged[k] = v
+	}
+
+	return merged
+}
+
+func buildCPUCostMap(
+	resNodeCPUCost []*prom.QueryResult,
+	providerIDParser func(string) string,
+) (
+	map[NodeIdentifier]float64,
+	map[nodeIdentifierNoProviderID]string,
+) {
+
+	cpuCostMap := make(map[NodeIdentifier]float64)
+	clusterAndNameToType := make(map[nodeIdentifierNoProviderID]string)
+
+	for _, result := range resNodeCPUCost {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("node")
+		if err != nil {
+			log.Warningf("ClusterNodes: CPU cost data missing node")
+			continue
+		}
+
+		nodeType, _ := result.GetString("instance_type")
+		providerID, _ := result.GetString("provider_id")
+
+		cpuCost := result.Values[0].Value
+
+		key := NodeIdentifier{
+			Cluster:    cluster,
+			Name:       name,
+			ProviderID: providerIDParser(providerID),
+		}
+		keyNon := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    name,
+		}
+
+		clusterAndNameToType[keyNon] = nodeType
+
+		cpuCostMap[key] = cpuCost
+	}
+
+	return cpuCostMap, clusterAndNameToType
+}
+
+func buildRAMCostMap(
+	resNodeRAMCost []*prom.QueryResult,
+	providerIDParser func(string) string,
+) (
+	map[NodeIdentifier]float64,
+	map[nodeIdentifierNoProviderID]string,
+) {
+
+	ramCostMap := make(map[NodeIdentifier]float64)
+	clusterAndNameToType := make(map[nodeIdentifierNoProviderID]string)
+
+	for _, result := range resNodeRAMCost {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("node")
+		if err != nil {
+			log.Warningf("ClusterNodes: RAM cost data missing node")
+			continue
+		}
+
+		nodeType, _ := result.GetString("instance_type")
+		providerID, _ := result.GetString("provider_id")
+
+		ramCost := result.Values[0].Value
+
+		key := NodeIdentifier{
+			Cluster:    cluster,
+			Name:       name,
+			ProviderID: providerIDParser(providerID),
+		}
+		keyNon := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    name,
+		}
+
+		clusterAndNameToType[keyNon] = nodeType
+		ramCostMap[key] = ramCost
+	}
+
+	return ramCostMap, clusterAndNameToType
+}
+
+func buildGPUCostMap(
+	resNodeGPUCost []*prom.QueryResult,
+	providerIDParser func(string) string,
+) (
+	map[NodeIdentifier]float64,
+	map[nodeIdentifierNoProviderID]string,
+) {
+
+	gpuCostMap := make(map[NodeIdentifier]float64)
+	clusterAndNameToType := make(map[nodeIdentifierNoProviderID]string)
+
+	for _, result := range resNodeGPUCost {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("node")
+		if err != nil {
+			log.Warningf("ClusterNodes: GPU cost data missing node")
+			continue
+		}
+
+		nodeType, _ := result.GetString("instance_type")
+		providerID, _ := result.GetString("provider_id")
+
+		gpuCost := result.Values[0].Value
+
+		key := NodeIdentifier{
+			Cluster:    cluster,
+			Name:       name,
+			ProviderID: providerIDParser(providerID),
+		}
+		keyNon := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    name,
+		}
+
+		clusterAndNameToType[keyNon] = nodeType
+
+		gpuCostMap[key] = gpuCost
+	}
+
+	return gpuCostMap, clusterAndNameToType
+}
+
+func buildCPUCoresMap(
+	resNodeCPUCores []*prom.QueryResult,
+	clusterAndNameToType map[nodeIdentifierNoProviderID]string,
+) map[nodeIdentifierNoProviderID]float64 {
+
+	m := make(map[nodeIdentifierNoProviderID]float64)
+
+	for _, result := range resNodeCPUCores {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("node")
+		if err != nil {
+			log.Warningf("ClusterNodes: CPU cores data missing node")
+			continue
+		}
+
+		cpuCores := result.Values[0].Value
+
+		key := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    name,
+		}
+		m[key] = cpuCores
+	}
+
+	return m
+}
+
+func buildRAMBytesMap(resNodeRAMBytes []*prom.QueryResult) map[nodeIdentifierNoProviderID]float64 {
+
+	m := make(map[nodeIdentifierNoProviderID]float64)
+
+	for _, result := range resNodeRAMBytes {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("node")
+		if err != nil {
+			log.Warningf("ClusterNodes: RAM bytes data missing node")
+			continue
+		}
+
+		ramBytes := result.Values[0].Value
+
+		key := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    name,
+		}
+		m[key] = ramBytes
+	}
+
+	return m
+}
+
+// Mapping of cluster/node=cpu for computing resource efficiency
+func buildCPUBreakdownMap(resNodeCPUModeTotal []*prom.QueryResult) map[nodeIdentifierNoProviderID]*ClusterCostsBreakdown {
+
+	cpuBreakdownMap := make(map[nodeIdentifierNoProviderID]*ClusterCostsBreakdown)
+
+	// Mapping of cluster/node=cpu for computing resource efficiency
+	clusterNodeCPUTotal := map[nodeIdentifierNoProviderID]float64{}
+	// Mapping of cluster/node:mode=cpu for computing resource efficiency
+	clusterNodeModeCPUTotal := map[nodeIdentifierNoProviderID]map[string]float64{}
+
+	// Build intermediate structures for CPU usage by (cluster, node) and by
+	// (cluster, node, mode) for computing resouce efficiency
+	for _, result := range resNodeCPUModeTotal {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		node, err := result.GetString("kubernetes_node")
+		if err != nil {
+			log.DedupedWarningf(5, "ClusterNodes: CPU mode data missing node")
+			continue
+		}
+
+		mode, err := result.GetString("mode")
+		if err != nil {
+			log.Warningf("ClusterNodes: unable to read CPU mode: %s", err)
+			mode = "other"
+		}
+
+		key := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    node,
+		}
+
+		total := result.Values[0].Value
+
+		// Increment total
+		clusterNodeCPUTotal[key] += total
+
+		// Increment mode
+		if _, ok := clusterNodeModeCPUTotal[key]; !ok {
+			clusterNodeModeCPUTotal[key] = map[string]float64{}
+		}
+		clusterNodeModeCPUTotal[key][mode] += total
+	}
+
+	// Compute resource efficiency from intermediate structures
+	for key, total := range clusterNodeCPUTotal {
+		if modeTotals, ok := clusterNodeModeCPUTotal[key]; ok {
+			for mode, subtotal := range modeTotals {
+				// Compute percentage for the current cluster, node, mode
+				pct := 0.0
+				if total > 0 {
+					pct = subtotal / total
+				}
+
+				if _, ok := cpuBreakdownMap[key]; !ok {
+					cpuBreakdownMap[key] = &ClusterCostsBreakdown{}
+				}
+
+				switch mode {
+				case "idle":
+					cpuBreakdownMap[key].Idle += pct
+				case "system":
+					cpuBreakdownMap[key].System += pct
+				case "user":
+					cpuBreakdownMap[key].User += pct
+				default:
+					cpuBreakdownMap[key].Other += pct
+				}
+			}
+		}
+	}
+
+	return cpuBreakdownMap
+}
+
+func buildRAMUserPctMap(resNodeRAMUserPct []*prom.QueryResult) map[nodeIdentifierNoProviderID]float64 {
+
+	m := make(map[nodeIdentifierNoProviderID]float64)
+
+	for _, result := range resNodeRAMUserPct {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("instance")
+		if err != nil {
+			log.Warningf("ClusterNodes: RAM user percent missing node")
+			continue
+		}
+
+		pct := result.Values[0].Value
+
+		key := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    name,
+		}
+
+		m[key] = pct
+	}
+
+	return m
+}
+
+func buildRAMSystemPctMap(resNodeRAMSystemPct []*prom.QueryResult) map[nodeIdentifierNoProviderID]float64 {
+
+	m := make(map[nodeIdentifierNoProviderID]float64)
+
+	for _, result := range resNodeRAMSystemPct {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("instance")
+		if err != nil {
+			log.Warningf("ClusterNodes: RAM system percent missing node")
+			continue
+		}
+
+		pct := result.Values[0].Value
+
+		key := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    name,
+		}
+
+		m[key] = pct
+	}
+
+	return m
+}
+
+type activeData struct {
+	start   time.Time
+	end     time.Time
+	minutes float64
+}
+
+func buildActiveDataMap(resActiveMins []*prom.QueryResult, resolution time.Duration, providerIDParser func(string) string) map[NodeIdentifier]activeData {
+
+	m := make(map[NodeIdentifier]activeData)
+
+	for _, result := range resActiveMins {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("node")
+		if err != nil {
+			log.Warningf("ClusterNodes: active mins missing node")
+			continue
+		}
+
+		providerID, _ := result.GetString("provider_id")
+
+		key := NodeIdentifier{
+			Cluster:    cluster,
+			Name:       name,
+			ProviderID: providerIDParser(providerID),
+		}
+
+		if len(result.Values) == 0 {
+			continue
+		}
+
+		s := time.Unix(int64(result.Values[0].Timestamp), 0)
+		e := time.Unix(int64(result.Values[len(result.Values)-1].Timestamp), 0).Add(resolution)
+		mins := e.Sub(s).Minutes()
+
+		// TODO niko/assets if mins >= threshold, interpolate for missing data?
+		m[key] = activeData{
+			start:   s,
+			end:     e,
+			minutes: mins,
+		}
+	}
+
+	return m
+}
+
+// Determine preemptibility with node labels
+// node id -> is preemptible?
+func buildPreemptibleMap(
+	resIsSpot []*prom.QueryResult,
+	providerIDParser func(string) string,
+) map[NodeIdentifier]bool {
+
+	m := make(map[NodeIdentifier]bool)
+
+	for _, result := range resIsSpot {
+		nodeName, err := result.GetString("node")
+		if err != nil {
+			continue
+		}
+
+		// GCP preemptible label
+		pre := result.Values[0].Value
+
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		providerID, _ := result.GetString("provider_id")
+
+		key := NodeIdentifier{
+			Cluster:    cluster,
+			Name:       nodeName,
+			ProviderID: providerIDParser(providerID),
+		}
+
+		// TODO(michaelmdresser): check this condition at merge time?
+		// if node, ok := nodeMap[key]; pre > 0.0 && ok {
+		// 	node.Preemptible = true
+		// }
+		m[key] = pre > 0.0
+
+		// TODO AWS preemptible
+
+		// TODO Azure preemptible
+	}
+
+	return m
+}
+
+func buildLabelsMap(
+	resLabels []*prom.QueryResult,
+) map[nodeIdentifierNoProviderID]map[string]string {
+
+	m := make(map[nodeIdentifierNoProviderID]map[string]string)
+
+	// Copy labels into node
+	for _, result := range resLabels {
+		cluster, err := result.GetString("cluster_id")
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+		node, err := result.GetString("kubernetes_node")
+		if err != nil {
+			log.DedupedWarningf(5, "ClusterNodes: label data missing node")
+			continue
+		}
+		key := nodeIdentifierNoProviderID{
+			Cluster: cluster,
+			Name:    node,
+		}
+
+		m[key] = make(map[string]string)
+
+		for name, value := range result.Metric {
+			if val, ok := value.(string); ok {
+				m[key][name] = val
+			}
+		}
+	}
+	return m
+}
+
+// checkForKeyAndInitIfMissing inits a key in the provided nodemap if
+// it does not exist. Intended to be called ONLY by buildNodeMap
+func checkForKeyAndInitIfMissing(
+	nodeMap map[NodeIdentifier]*Node,
+	key NodeIdentifier,
+	clusterAndNameToType map[nodeIdentifierNoProviderID]string,
+) {
+	if _, ok := nodeMap[key]; !ok {
+		// default nodeType in case we don't have the mapping
+		var nodeType string
+		if t, ok := clusterAndNameToType[nodeIdentifierNoProviderID{
+			Cluster: key.Cluster,
+			Name:    key.Name,
+		}]; ok {
+			nodeType = t
+		} else {
+			log.Warningf("ClusterNodes: Type does not exist for node identifier %s", key)
+		}
+
+		nodeMap[key] = &Node{
+			Cluster:      key.Cluster,
+			Name:         key.Name,
+			NodeType:     nodeType,
+			ProviderID:   key.ProviderID,
+			CPUBreakdown: &ClusterCostsBreakdown{},
+			RAMBreakdown: &ClusterCostsBreakdown{},
+		}
+	}
+}
+
+// buildNodeMap creates the main set of node data for ClusterNodes from
+// the data maps built from Prometheus queries. Some of the Prometheus
+// data has access to the provider_id field and some does not. To get
+// around this problem, we use the data that includes provider_id
+// to build up the definitive set of nodes and then use the data
+// with less-specific identifiers (i.e. without provider_id) to fill
+// in the remaining fields.
+//
+// For example, let's say we have nodes identified like so:
+// cluster name/node name/provider_id. For the sake of the example,
+// we will also limit data to CPU cost, CPU cores, and preemptibility.
+//
+// We have CPU cost data that looks like this:
+// cluster1/node1/prov_node1_A: $10
+// cluster1/node1/prov_node1_B: $8
+// cluster1/node2/prov_node2: $15
+//
+// We have Preemptible data that looks like this:
+// cluster1/node1/prov_node1_A: true
+// cluster1/node1/prov_node1_B: false
+// cluster1/node2/prov_node2_B: false
+//
+// We have CPU cores data that looks like this:
+// cluster1/node1: 4
+// cluster1/node2: 6
+//
+// This function first combines the data that is fully identified,
+// creating the following:
+// cluster1/node1/prov_node1_A: CPUCost($10), Preemptible(true)
+// cluster1/node1/prov_node1_B: CPUCost($8), Preemptible(false)
+// cluster1/node2/prov_node2: CPUCost($15), Preemptible(false)
+//
+// It then uses the less-specific data to extend the specific data,
+// making the following:
+// cluster1/node1/prov_node1_A: CPUCost($10), Preemptible(true), Cores(4)
+// cluster1/node1/prov_node1_B: CPUCost($8), Preemptible(false), Cores(4)
+// cluster1/node2/prov_node2: CPUCost($15), Preemptible(false), Cores(6)
+//
+// In the situation where provider_id doesn't exist for any metrics,
+// that is the same as all provider_ids being empty strings. If
+// provider_id doesn't exist at all, then we (without having to do
+// extra work) easily fall back on identifying nodes only by cluster name
+// and node name because the provider_id part of the key will always
+// be the empty string.
+//
+// It is worth nothing that, in this approach, if a node is not present
+// in the more specific data but is present in the less-specific data,
+// that data is never processed into the final node map. For example,
+// let's say the CPU cores map has the following entry:
+// cluster1/node8: 6
+// But none of the maps with provider_id (CPU cost, RAM cost, etc.)
+// have an identifier for cluster1/node8 (regardless of provider_id).
+// In this situation, the final node map will not have a cluster1/node8
+// entry. This could be fixed by iterating over all of the less specific
+// identifiers and, inside that iteration, all of the identifiers in
+// the node map, but this would introduce a roughly quadratic time
+// complexity.
+func buildNodeMap(
+	cpuCostMap, ramCostMap, gpuCostMap map[NodeIdentifier]float64,
+	cpuCoresMap, ramBytesMap, ramUserPctMap,
+	ramSystemPctMap map[nodeIdentifierNoProviderID]float64,
+	cpuBreakdownMap map[nodeIdentifierNoProviderID]*ClusterCostsBreakdown,
+	activeDataMap map[NodeIdentifier]activeData,
+	preemptibleMap map[NodeIdentifier]bool,
+	labelsMap map[nodeIdentifierNoProviderID]map[string]string,
+	clusterAndNameToType map[nodeIdentifierNoProviderID]string,
+) map[NodeIdentifier]*Node {
+
+	nodeMap := make(map[NodeIdentifier]*Node)
+
+	// Initialize the map with the most-specific data:
+
+	for id, cost := range cpuCostMap {
+		checkForKeyAndInitIfMissing(nodeMap, id, clusterAndNameToType)
+		nodeMap[id].CPUCost = cost
+	}
+
+	for id, cost := range ramCostMap {
+		checkForKeyAndInitIfMissing(nodeMap, id, clusterAndNameToType)
+		nodeMap[id].RAMCost = cost
+	}
+
+	for id, cost := range gpuCostMap {
+		checkForKeyAndInitIfMissing(nodeMap, id, clusterAndNameToType)
+		nodeMap[id].GPUCost = cost
+	}
+
+	for id, preemptible := range preemptibleMap {
+		checkForKeyAndInitIfMissing(nodeMap, id, clusterAndNameToType)
+		nodeMap[id].Preemptible = preemptible
+	}
+
+	for id, activeData := range activeDataMap {
+		checkForKeyAndInitIfMissing(nodeMap, id, clusterAndNameToType)
+		nodeMap[id].Start = activeData.start
+		nodeMap[id].End = activeData.end
+		nodeMap[id].Minutes = activeData.minutes
+	}
+
+	// We now merge in data that doesn't have a provider id by looping over
+	// all keys already added and inserting data according to their
+	// cluster name/node name combos.
+	for id, nodePtr := range nodeMap {
+		clusterAndNameID := nodeIdentifierNoProviderID{
+			Cluster: id.Cluster,
+			Name:    id.Name,
+		}
+
+		if cores, ok := cpuCoresMap[clusterAndNameID]; ok {
+			nodePtr.CPUCores = cores
+			if v, ok := partialCPUMap[nodePtr.NodeType]; ok {
+				if cores > 0 {
+					nodePtr.CPUCores = v
+					adjustmentFactor := v / cores
+					nodePtr.CPUCost = nodePtr.CPUCost * adjustmentFactor
+				}
+			}
+		}
+
+		if ramBytes, ok := ramBytesMap[clusterAndNameID]; ok {
+			nodePtr.RAMBytes = ramBytes
+		}
+
+		if ramUserPct, ok := ramUserPctMap[clusterAndNameID]; ok {
+			nodePtr.RAMBreakdown.User = ramUserPct
+		}
+
+		if ramSystemPct, ok := ramSystemPctMap[clusterAndNameID]; ok {
+			nodePtr.RAMBreakdown.System = ramSystemPct
+		}
+
+		if cpuBreakdown, ok := cpuBreakdownMap[clusterAndNameID]; ok {
+			nodePtr.CPUBreakdown = cpuBreakdown
+		}
+
+		if labels, ok := labelsMap[clusterAndNameID]; ok {
+			nodePtr.Labels = labels
+		}
+	}
+
+	return nodeMap
+}

pkg/costmodel/cluster_helpers_test.go

@@ -0,0 +1,673 @@
+package costmodel
+
+import (
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/davecgh/go-spew/spew"
+)
+
+func TestMergeTypeMaps(t *testing.T) {
+	cases := []struct {
+		name     string
+		map1     map[nodeIdentifierNoProviderID]string
+		map2     map[nodeIdentifierNoProviderID]string
+		expected map[nodeIdentifierNoProviderID]string
+	}{
+		{
+			name:     "both empty",
+			map1:     map[nodeIdentifierNoProviderID]string{},
+			map2:     map[nodeIdentifierNoProviderID]string{},
+			expected: map[nodeIdentifierNoProviderID]string{},
+		},
+		{
+			name: "map2 empty",
+			map1: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+			map2: map[nodeIdentifierNoProviderID]string{},
+			expected: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+		},
+		{
+			name: "map2 empty",
+			map1: map[nodeIdentifierNoProviderID]string{},
+			map2: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+			expected: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+		},
+		{
+			name: "no overlap",
+			map1: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+			map2: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: "type2",
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node4",
+				}: "type4",
+			},
+			expected: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: "type2",
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node4",
+				}: "type4",
+			},
+		},
+		{
+			name: "with overlap",
+			map1: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+			map2: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: "type2",
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type4",
+			},
+			expected: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: "type2",
+			},
+		},
+	}
+
+	for _, testCase := range cases {
+		result := mergeTypeMaps(testCase.map1, testCase.map2)
+
+		if !reflect.DeepEqual(result, testCase.expected) {
+			t.Errorf("mergeTypeMaps case %s failed. Got %+v but expected %+v", testCase.name, result, testCase.expected)
+		}
+	}
+}
+
+func TestBuildNodeMap(t *testing.T) {
+	cases := []struct {
+		name                 string
+		cpuCostMap           map[NodeIdentifier]float64
+		ramCostMap           map[NodeIdentifier]float64
+		gpuCostMap           map[NodeIdentifier]float64
+		cpuCoresMap          map[nodeIdentifierNoProviderID]float64
+		ramBytesMap          map[nodeIdentifierNoProviderID]float64
+		ramUserPctMap        map[nodeIdentifierNoProviderID]float64
+		ramSystemPctMap      map[nodeIdentifierNoProviderID]float64
+		cpuBreakdownMap      map[nodeIdentifierNoProviderID]*ClusterCostsBreakdown
+		activeDataMap        map[NodeIdentifier]activeData
+		preemptibleMap       map[NodeIdentifier]bool
+		labelsMap            map[nodeIdentifierNoProviderID]map[string]string
+		clusterAndNameToType map[nodeIdentifierNoProviderID]string
+		expected             map[NodeIdentifier]*Node
+	}{
+		{
+			name:     "empty",
+			expected: map[NodeIdentifier]*Node{},
+		},
+		{
+			name: "just cpu cost",
+			cpuCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1",
+				}: 0.048,
+			},
+			clusterAndNameToType: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+			expected: map[NodeIdentifier]*Node{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1",
+				}: &Node{
+					Cluster:      "cluster1",
+					Name:         "node1",
+					ProviderID:   "prov_node1",
+					NodeType:     "type1",
+					CPUCost:      0.048,
+					CPUBreakdown: &ClusterCostsBreakdown{},
+					RAMBreakdown: &ClusterCostsBreakdown{},
+				},
+			},
+		},
+		{
+			name: "just cpu cost with empty provider ID",
+			cpuCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: 0.048,
+			},
+			clusterAndNameToType: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+			expected: map[NodeIdentifier]*Node{
+				NodeIdentifier{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: &Node{
+					Cluster:      "cluster1",
+					Name:         "node1",
+					NodeType:     "type1",
+					CPUCost:      0.048,
+					CPUBreakdown: &ClusterCostsBreakdown{},
+					RAMBreakdown: &ClusterCostsBreakdown{},
+				},
+			},
+		},
+		{
+			name: "cpu cost with overlapping node names",
+			cpuCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+				}: 0.048,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+				}: 0.087,
+			},
+			clusterAndNameToType: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+			},
+			expected: map[NodeIdentifier]*Node{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+				}: &Node{
+					Cluster:      "cluster1",
+					Name:         "node1",
+					ProviderID:   "prov_node1_A",
+					NodeType:     "type1",
+					CPUCost:      0.048,
+					CPUBreakdown: &ClusterCostsBreakdown{},
+					RAMBreakdown: &ClusterCostsBreakdown{},
+				},
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+				}: &Node{
+					Cluster:      "cluster1",
+					Name:         "node1",
+					ProviderID:   "prov_node1_B",
+					NodeType:     "type1",
+					CPUCost:      0.087,
+					CPUBreakdown: &ClusterCostsBreakdown{},
+					RAMBreakdown: &ClusterCostsBreakdown{},
+				},
+			},
+		},
+		{
+			name: "all fields + overlapping node names",
+			cpuCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+				}: 0.048,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+				}: 0.087,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node2",
+					ProviderID: "prov_node2_A",
+				}: 0.033,
+			},
+			ramCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+				}: 0.09,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+				}: 0.3,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node2",
+					ProviderID: "prov_node2_A",
+				}: 0.024,
+			},
+			gpuCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+				}: 0.8,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+				}: 1.4,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node2",
+					ProviderID: "prov_node2_A",
+				}: 3.1,
+			},
+			cpuCoresMap: map[nodeIdentifierNoProviderID]float64{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: 2.0,
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: 5.0,
+			},
+			ramBytesMap: map[nodeIdentifierNoProviderID]float64{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: 2048.0,
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: 6303.0,
+			},
+			ramUserPctMap: map[nodeIdentifierNoProviderID]float64{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: 30.0,
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: 42.6,
+			},
+			ramSystemPctMap: map[nodeIdentifierNoProviderID]float64{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: 15.0,
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: 20.1,
+			},
+			cpuBreakdownMap: map[nodeIdentifierNoProviderID]*ClusterCostsBreakdown{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: &ClusterCostsBreakdown{
+					System: 20.2,
+					User:   68.0,
+				},
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: &ClusterCostsBreakdown{
+					System: 28.9,
+					User:   34.0,
+				},
+			},
+			activeDataMap: map[NodeIdentifier]activeData{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+				}: activeData{
+					start:   time.Date(2020, 6, 16, 3, 45, 28, 0, time.UTC),
+					end:     time.Date(2020, 6, 16, 9, 20, 39, 0, time.UTC),
+					minutes: 5*60 + 35 + (11.0 / 60.0),
+				},
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+				}: activeData{
+					start:   time.Date(2020, 6, 16, 3, 45, 28, 0, time.UTC),
+					end:     time.Date(2020, 6, 16, 9, 21, 39, 0, time.UTC),
+					minutes: 5*60 + 36 + (11.0 / 60.0),
+				},
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node2",
+					ProviderID: "prov_node2_A",
+				}: activeData{
+					start:   time.Date(2020, 6, 16, 3, 45, 28, 0, time.UTC),
+					end:     time.Date(2020, 6, 16, 9, 10, 39, 0, time.UTC),
+					minutes: 5*60 + 25 + (11.0 / 60.0),
+				},
+			},
+			preemptibleMap: map[NodeIdentifier]bool{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+				}: true,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+				}: false,
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node2",
+					ProviderID: "prov_node2_A",
+				}: false,
+			},
+			labelsMap: map[nodeIdentifierNoProviderID]map[string]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: map[string]string{
+					"labelname1_A": "labelvalue1_A",
+					"labelname1_B": "labelvalue1_B",
+				},
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: map[string]string{
+					"labelname2_A": "labelvalue2_A",
+					"labelname2_B": "labelvalue2_B",
+				},
+			},
+			clusterAndNameToType: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "type1",
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node2",
+				}: "type2",
+			},
+			expected: map[NodeIdentifier]*Node{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+				}: &Node{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_A",
+					NodeType:   "type1",
+					CPUCost:    0.048,
+					RAMCost:    0.09,
+					GPUCost:    0.8,
+					CPUCores:   2.0,
+					RAMBytes:   2048.0,
+					RAMBreakdown: &ClusterCostsBreakdown{
+						User:   30.0,
+						System: 15.0,
+					},
+					CPUBreakdown: &ClusterCostsBreakdown{
+						System: 20.2,
+						User:   68.0,
+					},
+					Start:       time.Date(2020, 6, 16, 3, 45, 28, 0, time.UTC),
+					End:         time.Date(2020, 6, 16, 9, 20, 39, 0, time.UTC),
+					Minutes:     5*60 + 35 + (11.0 / 60.0),
+					Preemptible: true,
+					Labels: map[string]string{
+						"labelname1_A": "labelvalue1_A",
+						"labelname1_B": "labelvalue1_B",
+					},
+				},
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+				}: &Node{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1_B",
+					NodeType:   "type1",
+					CPUCost:    0.087,
+					RAMCost:    0.3,
+					GPUCost:    1.4,
+					CPUCores:   2.0,
+					RAMBytes:   2048.0,
+					RAMBreakdown: &ClusterCostsBreakdown{
+						User:   30.0,
+						System: 15.0,
+					},
+					CPUBreakdown: &ClusterCostsBreakdown{
+						System: 20.2,
+						User:   68.0,
+					},
+					Start:       time.Date(2020, 6, 16, 3, 45, 28, 0, time.UTC),
+					End:         time.Date(2020, 6, 16, 9, 21, 39, 0, time.UTC),
+					Minutes:     5*60 + 36 + (11.0 / 60.0),
+					Preemptible: false,
+					Labels: map[string]string{
+						"labelname1_A": "labelvalue1_A",
+						"labelname1_B": "labelvalue1_B",
+					},
+				},
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node2",
+					ProviderID: "prov_node2_A",
+				}: &Node{
+					Cluster:    "cluster1",
+					Name:       "node2",
+					ProviderID: "prov_node2_A",
+					NodeType:   "type2",
+					CPUCost:    0.033,
+					RAMCost:    0.024,
+					GPUCost:    3.1,
+					CPUCores:   5.0,
+					RAMBytes:   6303.0,
+					RAMBreakdown: &ClusterCostsBreakdown{
+						User:   42.6,
+						System: 20.1,
+					},
+					CPUBreakdown: &ClusterCostsBreakdown{
+						System: 28.9,
+						User:   34.0,
+					},
+					Start:       time.Date(2020, 6, 16, 3, 45, 28, 0, time.UTC),
+					End:         time.Date(2020, 6, 16, 9, 10, 39, 0, time.UTC),
+					Minutes:     5*60 + 25 + (11.0 / 60.0),
+					Preemptible: false,
+					Labels: map[string]string{
+						"labelname2_A": "labelvalue2_A",
+						"labelname2_B": "labelvalue2_B",
+					},
+				},
+			},
+		},
+		{
+			name: "e2-micro cpu cost adjustment",
+			cpuCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1",
+				}: 0.048,
+			},
+			cpuCoresMap: map[nodeIdentifierNoProviderID]float64{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: 6.0, // GKE lies about number of cores
+			},
+			clusterAndNameToType: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "e2-micro", // for this node type
+			},
+			expected: map[NodeIdentifier]*Node{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1",
+				}: &Node{
+					Cluster:      "cluster1",
+					Name:         "node1",
+					ProviderID:   "prov_node1",
+					NodeType:     "e2-micro",
+					CPUCost:      0.048 * (partialCPUMap["e2-micro"] / 6.0), // adjustmentFactor is (v / GKE cores)
+					CPUCores:     partialCPUMap["e2-micro"],
+					CPUBreakdown: &ClusterCostsBreakdown{},
+					RAMBreakdown: &ClusterCostsBreakdown{},
+				},
+			},
+		},
+		{
+			name: "e2-small cpu cost adjustment",
+			cpuCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1",
+				}: 0.048,
+			},
+			cpuCoresMap: map[nodeIdentifierNoProviderID]float64{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: 6.0, // GKE lies about number of cores
+			},
+			clusterAndNameToType: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "e2-small", // for this node type
+			},
+			expected: map[NodeIdentifier]*Node{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1",
+				}: &Node{
+					Cluster:      "cluster1",
+					Name:         "node1",
+					ProviderID:   "prov_node1",
+					NodeType:     "e2-small",
+					CPUCost:      0.048 * (partialCPUMap["e2-small"] / 6.0), // adjustmentFactor is (v / GKE cores)
+					CPUCores:     partialCPUMap["e2-small"],
+					CPUBreakdown: &ClusterCostsBreakdown{},
+					RAMBreakdown: &ClusterCostsBreakdown{},
+				},
+			},
+		},
+		{
+			name: "e2-medium cpu cost adjustment",
+			cpuCostMap: map[NodeIdentifier]float64{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1",
+				}: 0.048,
+			},
+			cpuCoresMap: map[nodeIdentifierNoProviderID]float64{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: 6.0, // GKE lies about number of cores
+			},
+			clusterAndNameToType: map[nodeIdentifierNoProviderID]string{
+				nodeIdentifierNoProviderID{
+					Cluster: "cluster1",
+					Name:    "node1",
+				}: "e2-medium", // for this node type
+			},
+			expected: map[NodeIdentifier]*Node{
+				NodeIdentifier{
+					Cluster:    "cluster1",
+					Name:       "node1",
+					ProviderID: "prov_node1",
+				}: &Node{
+					Cluster:      "cluster1",
+					Name:         "node1",
+					ProviderID:   "prov_node1",
+					NodeType:     "e2-medium",
+					CPUCost:      0.048 * (partialCPUMap["e2-medium"] / 6.0), // adjustmentFactor is (v / GKE cores)
+					CPUCores:     partialCPUMap["e2-medium"],
+					CPUBreakdown: &ClusterCostsBreakdown{},
+					RAMBreakdown: &ClusterCostsBreakdown{},
+				},
+			},
+		},
+	}
+
+	for _, testCase := range cases {
+
+		result := buildNodeMap(
+			testCase.cpuCostMap, testCase.ramCostMap, testCase.gpuCostMap,
+			testCase.cpuCoresMap, testCase.ramBytesMap, testCase.ramUserPctMap,
+			testCase.ramSystemPctMap,
+			testCase.cpuBreakdownMap,
+			testCase.activeDataMap,
+			testCase.preemptibleMap,
+			testCase.labelsMap,
+			testCase.clusterAndNameToType,
+		)
+
+		if !reflect.DeepEqual(result, testCase.expected) {
+			t.Errorf("buildNodeMap case %s failed. Got %+v but expected %+v", testCase.name, result, testCase.expected)
+
+			// Use spew because we have to follow pointers to figure out
+			// what isn't matching up
+			t.Logf("Got: %s", spew.Sdump(result))
+			t.Logf("Expected: %s", spew.Sdump(testCase.expected))
+		}
+	}
+}

pkg/costmodel/costmodel.go

@@ -85,6 +85,7 @@ type CostData struct {
 	GPUReq          []*util.Vector               `json:"gpureq,omitempty"`
 	PVCData         []*PersistentVolumeClaimData `json:"pvcData,omitempty"`
 	NetworkData     []*util.Vector               `json:"network,omitempty"`
+	Annotations     map[string]string            `json:"annotations,omitempty"`
 	Labels          map[string]string            `json:"labels,omitempty"`
 	NamespaceLabels map[string]string            `json:"namespaceLabels,omitempty"`
 	ClusterID       string                       `json:"clusterId"`
@@ -177,10 +178,10 @@ const (
 		)
 	) by (namespace,container_name,pod_name,node,cluster_id)
 	* on (pod_name, namespace, cluster_id) group_left(container) label_replace(avg(avg_over_time(kube_pod_status_phase{phase="Running"}[%s] %s)) by (pod,namespace,cluster_id), "pod_name","$1","pod","(.+)")`
-	queryPVRequestsStr = `avg(avg(kube_persistentvolumeclaim_info{volumename != ""}) by (persistentvolumeclaim, storageclass, namespace, volumename, cluster_id)
+	queryPVRequestsStr = `avg(avg(kube_persistentvolumeclaim_info{volumename != ""}) by (persistentvolumeclaim, storageclass, namespace, volumename, cluster_id, kubernetes_node)
 	*
-	on (persistentvolumeclaim, namespace, cluster_id) group_right(storageclass, volumename)
-	sum(kube_persistentvolumeclaim_resource_requests_storage_bytes{volumename != ""}) by (persistentvolumeclaim, namespace, cluster_id, kubernetes_name)) by (persistentvolumeclaim, storageclass, namespace, volumename, cluster_id)`
+	on (persistentvolumeclaim, namespace, cluster_id, kubernetes_node) group_right(storageclass, volumename)
+	sum(kube_persistentvolumeclaim_resource_requests_storage_bytes{}) by (persistentvolumeclaim, namespace, cluster_id, kubernetes_node, kubernetes_name)) by (persistentvolumeclaim, storageclass, namespace, cluster_id, volumename, kubernetes_node)`
 	// queryRAMAllocationByteHours yields the total byte-hour RAM allocation over the given
 	// window, aggregated by container.
 	//  [line 3]  sum_over_time(each byte) = [byte*scrape] by metric
@@ -211,6 +212,8 @@ const (
 	queryPVHourlyCostFmt      = `avg_over_time(pv_hourly_cost[%s])`
 	queryNSLabels             = `avg_over_time(kube_namespace_labels[%s])`
 	queryPodLabels            = `avg_over_time(kube_pod_labels[%s])`
+	queryNSAnnotations        = `avg_over_time(kube_namespace_annotations[%s])`
+	queryPodAnnotations       = `avg_over_time(kube_pod_annotations[%s])`
 	queryDeploymentLabels     = `avg_over_time(deployment_match_labels[%s])`
 	queryStatefulsetLabels    = `avg_over_time(statefulSet_match_labels[%s])`
 	queryPodDaemonsets        = `sum(kube_pod_owner{owner_kind="DaemonSet"}) by (namespace,pod,owner_name,cluster_id)`
@@ -268,6 +271,11 @@ func (cm *CostModel) ComputeCostData(cli prometheusClient.Client, cp costAnalyze
 		return nil, err
 	}
 
+	namespaceAnnotationsMapping, err := getNamespaceAnnotations(cm.Cache, clusterID)
+	if err != nil {
+		return nil, err
+	}
+
 	// Process Prometheus query results. Handle errors using ctx.Errors.
 	resRAMRequests, _ := resChRAMRequests.Await()
 	resRAMUsage, _ := resChRAMUsage.Await()
@@ -409,6 +417,18 @@ func (cm *CostModel) ComputeCostData(cli prometheusClient.Client, cp costAnalyze
 				}
 			}
 
+			nsAnnotations := namespaceAnnotationsMapping[ns+","+clusterID]
+			podAnnotations := pod.GetObjectMeta().GetAnnotations()
+			if podAnnotations == nil {
+				podAnnotations = make(map[string]string)
+			}
+
+			for k, v := range nsAnnotations {
+				if _, ok := podAnnotations[k]; !ok {
+					podAnnotations[k] = v
+				}
+			}
+
 			nodeName := pod.Spec.NodeName
 			var nodeData *costAnalyzerCloud.Node
 			if _, ok := nodes[nodeName]; ok {
@@ -518,6 +538,7 @@ func (cm *CostModel) ComputeCostData(cli prometheusClient.Client, cp costAnalyze
 					GPUReq:          GPUReqV,
 					PVCData:         pvReq,
 					NetworkData:     netReq,
+					Annotations:     podAnnotations,
 					Labels:          podLabels,
 					NamespaceLabels: nsLabels,
 					ClusterID:       clusterID,
@@ -579,6 +600,11 @@ func (cm *CostModel) ComputeCostData(cli prometheusClient.Client, cp costAnalyze
 				klog.V(3).Infof("Missing data for namespace %s", c.Namespace)
 			}
 
+			namespaceAnnotations, ok := namespaceAnnotationsMapping[c.Namespace+","+c.ClusterID]
+			if !ok {
+				klog.V(3).Infof("Missing data for namespace %s", c.Namespace)
+			}
+
 			costs := &CostData{
 				Name:            c.ContainerName,
 				PodName:         c.PodName,
@@ -590,6 +616,7 @@ func (cm *CostModel) ComputeCostData(cli prometheusClient.Client, cp costAnalyze
 				CPUReq:          CPUReqV,
 				CPUUsed:         CPUUsedV,
 				GPUReq:          GPUReqV,
+				Annotations:     namespaceAnnotations,
 				NamespaceLabels: namespacelabels,
 				ClusterID:       c.ClusterID,
 				ClusterName:     cm.ClusterMap.NameFor(c.ClusterID),
@@ -607,7 +634,7 @@ func (cm *CostModel) ComputeCostData(cli prometheusClient.Client, cp costAnalyze
 	}
 	// Use unmounted pvs to create a mapping of "Unmounted-<Namespace>" containers
 	// to pass along the cost data
-	unmounted := findUnmountedPVCostData(cm.ClusterMap, unmountedPVs, namespaceLabelsMapping)
+	unmounted := findUnmountedPVCostData(cm.ClusterMap, unmountedPVs, namespaceLabelsMapping, namespaceAnnotationsMapping)
 	for k, costs := range unmounted {
 		klog.V(4).Infof("Unmounted PVs in Namespace/ClusterID: %s/%s", costs.Namespace, costs.ClusterID)
 
@@ -630,7 +657,7 @@ func (cm *CostModel) ComputeCostData(cli prometheusClient.Client, cp costAnalyze
 	return containerNameCost, err
 }
 
-func findUnmountedPVCostData(clusterMap clusters.ClusterMap, unmountedPVs map[string][]*PersistentVolumeClaimData, namespaceLabelsMapping map[string]map[string]string) map[string]*CostData {
+func findUnmountedPVCostData(clusterMap clusters.ClusterMap, unmountedPVs map[string][]*PersistentVolumeClaimData, namespaceLabelsMapping map[string]map[string]string, namespaceAnnotationsMapping map[string]map[string]string) map[string]*CostData {
 	costs := make(map[string]*CostData)
 	if len(unmountedPVs) == 0 {
 		return costs
@@ -650,6 +677,11 @@ func findUnmountedPVCostData(clusterMap clusters.ClusterMap, unmountedPVs map[st
 			klog.V(3).Infof("Missing data for namespace %s", ns)
 		}
 
+		namespaceAnnotations, ok := namespaceAnnotationsMapping[ns+","+clusterID]
+		if !ok {
+			klog.V(3).Infof("Missing data for namespace %s", ns)
+		}
+
 		// Should be a unique "Unmounted" cost data type
 		name := "unmounted-pvs"
 
@@ -661,6 +693,7 @@ func findUnmountedPVCostData(clusterMap clusters.ClusterMap, unmountedPVs map[st
 				Name:            name,
 				PodName:         name,
 				NodeName:        "",
+				Annotations:     namespaceAnnotations,
 				Namespace:       ns,
 				NamespaceLabels: namespacelabels,
 				Labels:          namespacelabels,
@@ -1568,6 +1601,8 @@ func (cm *CostModel) costDataRange(cli prometheusClient.Client, cp costAnalyzerC
 	resChNetInternetRequests := ctx.QueryRange(queryNetInternetRequests, start, end, resolution)
 	resChNSLabels := ctx.QueryRange(fmt.Sprintf(queryNSLabels, resStr), start, end, resolution)
 	resChPodLabels := ctx.QueryRange(fmt.Sprintf(queryPodLabels, resStr), start, end, resolution)
+	resChNSAnnotations := ctx.QueryRange(fmt.Sprintf(queryNSAnnotations, resStr), start, end, resolution)
+	resChPodAnnotations := ctx.QueryRange(fmt.Sprintf(queryPodAnnotations, resStr), start, end, resolution)
 	resChServiceLabels := ctx.QueryRange(fmt.Sprintf(queryServiceLabels, resStr), start, end, resolution)
 	resChDeploymentLabels := ctx.QueryRange(fmt.Sprintf(queryDeploymentLabels, resStr), start, end, resolution)
 	resChStatefulsetLabels := ctx.QueryRange(fmt.Sprintf(queryStatefulsetLabels, resStr), start, end, resolution)
@@ -1598,6 +1633,11 @@ func (cm *CostModel) costDataRange(cli prometheusClient.Client, cp costAnalyzerC
 		return nil, fmt.Errorf("error querying the kubernetes API: %s", err)
 	}
 
+	namespaceAnnotationsMapping, err := getNamespaceAnnotations(cm.Cache, clusterID)
+	if err != nil {
+		return nil, fmt.Errorf("error querying the kubernetes API: %s", err)
+	}
+
 	// Process query results. Handle errors afterwards using ctx.Errors.
 	resRAMRequests, _ := resChRAMRequests.Await()
 	resRAMUsage, _ := resChRAMUsage.Await()
@@ -1614,6 +1654,8 @@ func (cm *CostModel) costDataRange(cli prometheusClient.Client, cp costAnalyzerC
 	resNetInternetRequests, _ := resChNetInternetRequests.Await()
 	resNSLabels, _ := resChNSLabels.Await()
 	resPodLabels, _ := resChPodLabels.Await()
+	resNSAnnotations, _ := resChNSAnnotations.Await()
+	resPodAnnotations, _ := resChPodAnnotations.Await()
 	resServiceLabels, _ := resChServiceLabels.Await()
 	resDeploymentLabels, _ := resChDeploymentLabels.Await()
 	resStatefulsetLabels, _ := resChStatefulsetLabels.Await()
@@ -1679,7 +1721,7 @@ func (cm *CostModel) costDataRange(cli prometheusClient.Client, cp costAnalyzerC
 		klog.V(1).Infof("Unable to get Namespace Labels for Metrics: %s", err.Error())
 	}
 	if nsLabels != nil {
-		appendNamespaceLabels(namespaceLabelsMapping, nsLabels)
+		mergeStringMap(namespaceLabelsMapping, nsLabels)
 	}
 
 	podLabels, err := GetPodLabelsMetrics(resPodLabels, clusterID)
@@ -1687,6 +1729,19 @@ func (cm *CostModel) costDataRange(cli prometheusClient.Client, cp costAnalyzerC
 		klog.V(1).Infof("Unable to get Pod Labels for Metrics: %s", err.Error())
 	}
 
+	nsAnnotations, err := GetNamespaceAnnotationsMetrics(resNSAnnotations, clusterID)
+	if err != nil {
+		klog.V(1).Infof("Unable to get Namespace Annotations for Metrics: %s", err.Error())
+	}
+	if nsAnnotations != nil {
+		mergeStringMap(namespaceAnnotationsMapping, nsAnnotations)
+	}
+
+	podAnnotations, err := GetPodAnnotationsMetrics(resPodAnnotations, clusterID)
+	if err != nil {
+		klog.V(1).Infof("Unable to get Pod Annotations for Metrics: %s", err.Error())
+	}
+
 	serviceLabels, err := GetServiceSelectorLabelsMetrics(resServiceLabels, clusterID)
 	if err != nil {
 		klog.V(1).Infof("Unable to get Service Selector Labels for Metrics: %s", err.Error())
@@ -1874,6 +1929,22 @@ func (cm *CostModel) costDataRange(cli prometheusClient.Client, cp costAnalyzerC
 			}
 		}
 
+		namespaceAnnotations, ok := namespaceAnnotationsMapping[nsKey]
+		if !ok {
+			klog.V(4).Infof("Missing data for namespace %s", c.Namespace)
+		}
+
+		pAnnotations := podAnnotations[podKey]
+		if pAnnotations == nil {
+			pAnnotations = make(map[string]string)
+		}
+
+		for k, v := range namespaceAnnotations {
+			if _, ok := pAnnotations[k]; !ok {
+				pAnnotations[k] = v
+			}
+		}
+
 		var podDeployments []string
 		if _, ok := podDeploymentsMapping[nsKey]; ok {
 			if ds, ok := podDeploymentsMapping[nsKey][c.PodName]; ok {
@@ -1965,6 +2036,7 @@ func (cm *CostModel) costDataRange(cli prometheusClient.Client, cp costAnalyzerC
 			RAMAllocation:   RAMAllocsV,
 			CPUAllocation:   CPUAllocsV,
 			GPUReq:          GPUReqV,
+			Annotations:     pAnnotations,
 			Labels:          pLabels,
 			NamespaceLabels: namespaceLabels,
 			PVCData:         podPVs,
@@ -1979,7 +2051,7 @@ func (cm *CostModel) costDataRange(cli prometheusClient.Client, cp costAnalyzerC
 		}
 	}
 
-	unmounted := findUnmountedPVCostData(cm.ClusterMap, unmountedPVs, namespaceLabelsMapping)
+	unmounted := findUnmountedPVCostData(cm.ClusterMap, unmountedPVs, namespaceLabelsMapping, namespaceAnnotationsMapping)
 	for k, costs := range unmounted {
 		klog.V(4).Infof("Unmounted PVs in Namespace/ClusterID: %s/%s", costs.Namespace, costs.ClusterID)
 
@@ -2056,11 +2128,11 @@ func addMetricPVData(pvAllocationMap map[string][]*PersistentVolumeClaimData, pv
 	}
 }
 
-// Append labels into nsLabels iff the ns key doesn't already exist
-func appendNamespaceLabels(nsLabels map[string]map[string]string, labels map[string]map[string]string) {
-	for k, v := range labels {
-		if _, ok := nsLabels[k]; !ok {
-			nsLabels[k] = v
+// Add values that don't already exist in origMap from mergeMap into origMap
+func mergeStringMap(origMap map[string]map[string]string, mergeMap map[string]map[string]string) {
+	for k, v := range mergeMap {
+		if _, ok := origMap[k]; !ok {
+			origMap[k] = v
 		}
 	}
 }
@@ -2084,6 +2156,19 @@ func getNamespaceLabels(cache clustercache.ClusterCache, clusterID string) (map[
 	return nsToLabels, nil
 }
 
+func getNamespaceAnnotations(cache clustercache.ClusterCache, clusterID string) (map[string]map[string]string, error) {
+	nsToAnnotations := make(map[string]map[string]string)
+	nss := cache.GetAllNamespaces()
+	for _, ns := range nss {
+		annotations := make(map[string]string)
+		for k, v := range ns.Annotations {
+			annotations[prom.SanitizeLabelName(k)] = v
+		}
+		nsToAnnotations[ns.Name+","+clusterID] = annotations
+	}
+	return nsToAnnotations, nil
+}
+
 func getDaemonsetsOfPod(pod v1.Pod) []string {
 	for _, ownerReference := range pod.ObjectMeta.OwnerReferences {
 		if ownerReference.Kind == "DaemonSet" {

pkg/costmodel/metrics.go

@@ -950,44 +950,45 @@ func (cmme *CostModelMetricsEmitter) Start() bool {
 				} else {
 					containerSeen[labelKey] = false
 				}
+			}
 
-				storageClasses := cmme.KubeClusterCache.GetAllStorageClasses()
-				storageClassMap := make(map[string]map[string]string)
-				for _, storageClass := range storageClasses {
-					params := storageClass.Parameters
-					storageClassMap[storageClass.ObjectMeta.Name] = params
-					if storageClass.GetAnnotations()["storageclass.kubernetes.io/is-default-class"] == "true" || storageClass.GetAnnotations()["storageclass.beta.kubernetes.io/is-default-class"] == "true" {
-						storageClassMap["default"] = params
-						storageClassMap[""] = params
-					}
+			storageClasses := cmme.KubeClusterCache.GetAllStorageClasses()
+			storageClassMap := make(map[string]map[string]string)
+			for _, storageClass := range storageClasses {
+				params := storageClass.Parameters
+				storageClassMap[storageClass.ObjectMeta.Name] = params
+				if storageClass.GetAnnotations()["storageclass.kubernetes.io/is-default-class"] == "true" || storageClass.GetAnnotations()["storageclass.beta.kubernetes.io/is-default-class"] == "true" {
+					storageClassMap["default"] = params
+					storageClassMap[""] = params
 				}
+			}
 
-				pvs := cmme.KubeClusterCache.GetAllPersistentVolumes()
-				for _, pv := range pvs {
-					parameters, ok := storageClassMap[pv.Spec.StorageClassName]
-					if !ok {
-						klog.V(4).Infof("Unable to find parameters for storage class \"%s\". Does pv \"%s\" have a storageClassName?", pv.Spec.StorageClassName, pv.Name)
-					}
-					var region string
-					if r, ok := pv.Labels[v1.LabelZoneRegion]; ok {
-						region = r
-					} else {
-						region = defaultRegion
-					}
-					cacPv := &cloud.PV{
-						Class:      pv.Spec.StorageClassName,
-						Region:     region,
-						Parameters: parameters,
-					}
-
-					// TODO: GetPVCost should be a method in CostModel?
-					GetPVCost(cacPv, pv, cmme.CloudProvider, region)
-					c, _ := strconv.ParseFloat(cacPv.Cost, 64)
-					cmme.PersistentVolumePriceRecorder.WithLabelValues(pv.Name, pv.Name, cacPv.ProviderID).Set(c)
-					labelKey := getKeyFromLabelStrings(pv.Name, pv.Name)
-					pvSeen[labelKey] = true
+			pvs := cmme.KubeClusterCache.GetAllPersistentVolumes()
+			for _, pv := range pvs {
+				parameters, ok := storageClassMap[pv.Spec.StorageClassName]
+				if !ok {
+					klog.V(4).Infof("Unable to find parameters for storage class \"%s\". Does pv \"%s\" have a storageClassName?", pv.Spec.StorageClassName, pv.Name)
+				}
+				var region string
+				if r, ok := pv.Labels[v1.LabelZoneRegion]; ok {
+					region = r
+				} else {
+					region = defaultRegion
 				}
+				cacPv := &cloud.PV{
+					Class:      pv.Spec.StorageClassName,
+					Region:     region,
+					Parameters: parameters,
+				}
+
+				// TODO: GetPVCost should be a method in CostModel?
+				GetPVCost(cacPv, pv, cmme.CloudProvider, region)
+				c, _ := strconv.ParseFloat(cacPv.Cost, 64)
+				cmme.PersistentVolumePriceRecorder.WithLabelValues(pv.Name, pv.Name, cacPv.ProviderID).Set(c)
+				labelKey := getKeyFromLabelStrings(pv.Name, pv.Name)
+				pvSeen[labelKey] = true
 			}
+
 			for labelString, seen := range nodeSeen {
 				if !seen {
 					klog.V(4).Infof("Removing %s from nodes", labelString)

pkg/costmodel/promparsers.go

@@ -210,6 +210,66 @@ func GetPodLabelsMetrics(qrs []*prom.QueryResult, defaultClusterID string) (map[
 	return toReturn, nil
 }
 
+func GetNamespaceAnnotationsMetrics(qrs []*prom.QueryResult, defaultClusterID string) (map[string]map[string]string, error) {
+	toReturn := make(map[string]map[string]string)
+
+	for _, val := range qrs {
+		// We want Namespace and ClusterID for key generation purposes
+		ns, err := val.GetString("namespace")
+		if err != nil {
+			return toReturn, err
+		}
+
+		clusterID, err := val.GetString("cluster_id")
+		if clusterID == "" {
+			clusterID = defaultClusterID
+		}
+
+		nsKey := ns + "," + clusterID
+		if nsAnnotations, ok := toReturn[nsKey]; ok {
+			for k, v := range val.GetAnnotations() {
+				nsAnnotations[k] = v // override with more recently assigned if we changed labels within the window.
+			}
+		} else {
+			toReturn[nsKey] = val.GetAnnotations()
+		}
+	}
+	return toReturn, nil
+}
+
+func GetPodAnnotationsMetrics(qrs []*prom.QueryResult, defaultClusterID string) (map[string]map[string]string, error) {
+	toReturn := make(map[string]map[string]string)
+
+	for _, val := range qrs {
+		// We want Pod, Namespace and ClusterID for key generation purposes
+		pod, err := val.GetString("pod")
+		if err != nil {
+			return toReturn, err
+		}
+
+		ns, err := val.GetString("namespace")
+		if err != nil {
+			return toReturn, err
+		}
+
+		clusterID, err := val.GetString("cluster_id")
+		if clusterID == "" {
+			clusterID = defaultClusterID
+		}
+
+		nsKey := ns + "," + pod + "," + clusterID
+		if labels, ok := toReturn[nsKey]; ok {
+			for k, v := range val.GetAnnotations() {
+				labels[k] = v
+			}
+		} else {
+			toReturn[nsKey] = val.GetAnnotations()
+		}
+	}
+
+	return toReturn, nil
+}
+
 func GetStatefulsetMatchLabelsMetrics(qrs []*prom.QueryResult, defaultClusterID string) (map[string]map[string]string, error) {
 	toReturn := make(map[string]map[string]string)
 

pkg/env/costmodelenv.go

@@ -64,7 +64,7 @@ const (
 // GetAWSAccessKeyID returns the environment variable value for AWSAccessKeyIDEnvVar which represents
 // the AWS access key for authentication
 func GetAppVersion() string {
-	return Get(AppVersionEnvVar, "1.72.0")
+	return Get(AppVersionEnvVar, "1.73.0")
 }
 
 // IsEmitNamespaceAnnotationsMetric returns true if cost-model is configured to emit the kube_namespace_annotations metric

pkg/kubecost/allocation.go

@@ -11,6 +11,14 @@ import (
 	"github.com/kubecost/cost-model/pkg/log"
 )
 
+// TODO Clean-up use of IsEmpty; nil checks should be separated for safety.
+
+// TODO Consider making Allocation an interface, which is fulfilled by structs
+// like KubernetesAllocation, IdleAllocation, and ExternalAllocation.
+
+// ExternalSuffix indicates an external allocation
+const ExternalSuffix = "__external__"
+
 // IdleSuffix indicates an idle allocation property
 const IdleSuffix = "__idle__"
 
@@ -53,9 +61,9 @@ type Allocation struct {
 	RAMCost         float64    `json:"ramCost"`
 	RAMEfficiency   float64    `json:"ramEfficiency"`
 	SharedCost      float64    `json:"sharedCost"`
+	ExternalCost    float64    `json:"externalCost"`
 	TotalCost       float64    `json:"totalCost"`
 	TotalEfficiency float64    `json:"totalEfficiency"`
-	// Profiler        *log.Profiler `json:"-"`
 }
 
 // AllocationMatchFunc is a function that can be used to match Allocations by
@@ -75,7 +83,6 @@ func (a *Allocation) Add(that *Allocation) (*Allocation, error) {
 	}
 
 	agg := a.Clone()
-	// agg.Profiler = a.Profiler
 	agg.add(that, false, false)
 
 	return agg, nil
@@ -106,11 +113,14 @@ func (a *Allocation) Clone() *Allocation {
 		RAMCost:         a.RAMCost,
 		RAMEfficiency:   a.RAMEfficiency,
 		SharedCost:      a.SharedCost,
+		ExternalCost:    a.ExternalCost,
 		TotalCost:       a.TotalCost,
 		TotalEfficiency: a.TotalEfficiency,
 	}
 }
 
+// Equal returns true if the values held in the given Allocation precisely
+// match those of the receiving Allocation. nil does not match nil.
 func (a *Allocation) Equal(that *Allocation) bool {
 	if a == nil || that == nil {
 		return false
@@ -167,6 +177,9 @@ func (a *Allocation) Equal(that *Allocation) bool {
 	if a.SharedCost != that.SharedCost {
 		return false
 	}
+	if a.ExternalCost != that.ExternalCost {
+		return false
+	}
 	if a.TotalCost != that.TotalCost {
 		return false
 	}
@@ -191,6 +204,11 @@ func (a *Allocation) IsAggregated() bool {
 	return a == nil || a.Properties == nil
 }
 
+// IsExternal is true if the given Allocation represents external costs.
+func (a *Allocation) IsExternal() bool {
+	return strings.Contains(a.Name, ExternalSuffix)
+}
+
 // IsIdle is true if the given Allocation represents idle costs.
 func (a *Allocation) IsIdle() bool {
 	return strings.Contains(a.Name, IdleSuffix)
@@ -201,45 +219,6 @@ func (a *Allocation) IsUnallocated() bool {
 	return strings.Contains(a.Name, UnallocatedSuffix)
 }
 
-// MatchesFilter returns true if the Allocation passes the given AllocationFilter
-func (a *Allocation) MatchesFilter(f AllocationMatchFunc) bool {
-	return f(a)
-}
-
-// MatchesAll takes a variadic list of Properties, returning true iff the
-// Allocation matches each set of Properties.
-func (a *Allocation) MatchesAll(ps ...Properties) bool {
-	// nil Allocation don't match any Properties
-	if a == nil {
-		return false
-	}
-
-	for _, p := range ps {
-		if !a.Properties.Matches(p) {
-			return false
-		}
-	}
-
-	return true
-}
-
-// MatchesOne takes a variadic list of Properties, returning true iff the
-// Allocation matches at least one of the set of Properties.
-func (a *Allocation) MatchesOne(ps ...Properties) bool {
-	// nil Allocation don't match any Properties
-	if a == nil {
-		return false
-	}
-
-	for _, p := range ps {
-		if a.Properties.Matches(p) {
-			return true
-		}
-	}
-
-	return false
-}
-
 // Share works like Add, but converts the entire cost of the given Allocation
 // to SharedCost, rather than adding to the individual resource costs.
 func (a *Allocation) Share(that *Allocation) (*Allocation, error) {
@@ -267,13 +246,7 @@ func (a *Allocation) String() string {
 
 func (a *Allocation) add(that *Allocation, isShared, isAccumulating bool) {
 	if a == nil {
-		a = that
-
-		// reset properties
-		thatCluster, _ := that.Properties.GetCluster()
-		thatNode, _ := that.Properties.GetNode()
-		a.Properties = Properties{ClusterProp: thatCluster, NodeProp: thatNode}
-
+		log.Warningf("Allocation.AggregateBy: trying to add a nil receiver")
 		return
 	}
 
@@ -349,6 +322,7 @@ func (a *Allocation) add(that *Allocation, isShared, isAccumulating bool) {
 		}
 
 		a.SharedCost += that.SharedCost
+		a.ExternalCost += that.ExternalCost
 		a.CPUCost += that.CPUCost
 		a.GPUCost += that.GPUCost
 		a.NetworkCost += that.NetworkCost
@@ -363,20 +337,22 @@ func (a *Allocation) add(that *Allocation, isShared, isAccumulating bool) {
 // a window. An AllocationSet is mutable, so treat it like a threadsafe map.
 type AllocationSet struct {
 	sync.RWMutex
-	// Profiler    *log.Profiler
-	allocations map[string]*Allocation
-	idleKeys    map[string]bool
-	Window      Window
-	Warnings    []string
-	Errors      []string
+	allocations  map[string]*Allocation
+	externalKeys map[string]bool
+	idleKeys     map[string]bool
+	Window       Window
+	Warnings     []string
+	Errors       []string
 }
 
 // NewAllocationSet instantiates a new AllocationSet and, optionally, inserts
 // the given list of Allocations
 func NewAllocationSet(start, end time.Time, allocs ...*Allocation) *AllocationSet {
 	as := &AllocationSet{
-		allocations: map[string]*Allocation{},
-		Window:      NewWindow(&start, &end),
+		allocations:  map[string]*Allocation{},
+		externalKeys: map[string]bool{},
+		idleKeys:     map[string]bool{},
+		Window:       NewWindow(&start, &end),
 	}
 
 	for _, a := range allocs {
@@ -407,20 +383,25 @@ type AllocationAggregationOptions struct {
 // given Property; e.g. Containers can be divided by Namespace, but not vice-a-versa.
 func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationAggregationOptions) error {
 	// The order of operations for aggregating allocations is as follows:
-	// 1. move shared and/or idle allocations to separate sets if options
-	//    indicate that they should be shared
-	// 2. idle coefficients
-	// 2.a) if idle allocation is to be shared, compute idle coefficients
-	//      (do not compute shared coefficients here, see step 5)
-	// 2.b) if idle allocation is NOT shared, but filters are present, compute
-	//      idle filtration coefficients for the purpose of only returning the
-	//      portion of idle allocation that would have been shared with the
-	//      unfiltered results set. (See unit tests 5.a,b,c)
-	// 3. ignore allocation if it fails any of the FilterFuncs
-	// 4. generate aggregation key and insert allocation into the output set
-	// 5. if there are shared allocations, compute sharing coefficients on
+	// 1. Partition external, idle, and shared allocations into separate sets
+	// 2. Compute idle coefficients (if necessary)
+	//    a) if idle allocation is to be shared, compute idle coefficients
+	//       (do not compute shared coefficients here, see step 5)
+	//    b) if idle allocation is NOT shared, but filters are present, compute
+	//       idle filtration coefficients for the purpose of only returning the
+	//       portion of idle allocation that would have been shared with the
+	//       unfiltered results set. (See unit tests 5.a,b,c)
+	// 3. Ignore allocation if it fails any of the FilterFuncs
+	// 4. Distribute idle allocations among remaining non-idle, non-external
+	//    allocations
+	// 5. Generate aggregation key and insert allocation into the output set
+	// 6. Scale un-aggregated idle coefficients by filtration coefficient
+	// 7. If there are shared allocations, compute sharing coefficients on
 	//    the aggregated set, then share allocation accordingly
-	// 6. if the merge idle option is enabled, merge any remaining idle
+	// 8. If there are external allocations that can be aggregated into
+	//    the output (i.e. they can be used to generate a valid key for
+	//    the given properties) then aggregate; otherwise... ignore them?
+	// 9. If the merge idle option is enabled, merge any remaining idle
 	//    allocations into a single idle allocation
 
 	// TODO niko/etl revisit (ShareIdle: ShareEven) case, which is probably wrong
@@ -436,24 +417,27 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 
 	// aggSet will collect the aggregated allocations
 	aggSet := &AllocationSet{
-		// Profiler: as.Profiler,
+		Window: as.Window.Clone(),
+	}
+
+	// externalSet will collect external allocations
+	externalSet := &AllocationSet{
 		Window: as.Window.Clone(),
 	}
 
 	// idleSet will be shared among aggSet after initial aggregation
 	// is complete
 	idleSet := &AllocationSet{
-		// Profiler: as.Profiler,
 		Window: as.Window.Clone(),
 	}
 
 	// shareSet will be shared among aggSet after initial aggregation
 	// is complete
 	shareSet := &AllocationSet{
-		// Profiler: as.Profiler
 		Window: as.Window.Clone(),
 	}
 
+	// Convert SharedHourlyCosts to Allocations in the shareSet
 	for name, cost := range options.SharedHourlyCosts {
 		if cost > 0.0 {
 			hours := as.Resolution().Hours()
@@ -479,21 +463,29 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 	as.Lock()
 	defer as.Unlock()
 
-	// Loop and find all of the idle and shared allocations initially. Add
-	// them to their respective sets, removing them from the set of
-	// allocations to aggregate.
+	// (1) Loop and find all of the external, idle, and shared allocations. Add
+	// them to their respective sets, removing them from the set of allocations
+	// to aggregate.
 	for _, alloc := range as.allocations {
+		// External allocations get aggregated post-hoc (see step 6) and do
+		// not necessarily contain complete sets of properties, so they are
+		// moved to a separate AllocationSet.
+		if alloc.IsExternal() {
+			delete(as.externalKeys, alloc.Name)
+			delete(as.allocations, alloc.Name)
+			externalSet.Insert(alloc)
+			continue
+		}
+
 		cluster, err := alloc.Properties.GetCluster()
 		if err != nil {
 			log.Warningf("AllocationSet.AggregateBy: missing cluster for allocation: %s", alloc.Name)
 			return err
 		}
 
-		// Idle allocation doesn't get aggregated, so it can be passed through,
-		// whether or not it is shared. If it is shared, it is put in idleSet
-		// because shareSet may be split by different rules (even/weighted).
+		// Idle allocations should be separated into idleSet if they are to be
+		// shared later on. If they are not to be shared, then aggregate them.
 		if alloc.IsIdle() {
-			// Can't recursively call Delete() due to lock acquisition
 			delete(as.idleKeys, alloc.Name)
 			delete(as.allocations, alloc.Name)
 
@@ -502,14 +494,15 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 			} else {
 				aggSet.Insert(alloc)
 			}
+
+			continue
 		}
 
-		// If any of the share funcs succeed, share the allocation. Do this
-		// prior to filtering so that shared namespaces, etc do not get
-		// filtered out before we have a chance to share them.
+		// Shared allocations must be identified and separated prior to
+		// aggregation and filtering. That is, if any of the ShareFuncs
+		// return true, then move the allocation to shareSet.
 		for _, sf := range options.ShareFuncs {
 			if sf(alloc) {
-				// Can't recursively call Delete() due to lock acquisition
 				delete(as.idleKeys, alloc.Name)
 				delete(as.allocations, alloc.Name)
 
@@ -520,6 +513,8 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 		}
 	}
 
+	// It's possible that no more un-shared, non-idle, non-external allocations
+	// remain at this point. This always results in an emptySet.
 	if len(as.allocations) == 0 {
 		log.Warningf("ETL: AggregateBy: no allocations to aggregate")
 		emptySet := &AllocationSet{
@@ -529,24 +524,30 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 		return nil
 	}
 
-	// In order to correctly apply idle and shared resource coefficients appropriately,
-	// we need to determine the coefficients for the full set of data. The ensures that
-	// the ratios are maintained through filtering.
+	// (2) In order to correctly apply idle and shared resource coefficients
+	// appropriately, we need to determine the coefficients for the full set
+	// of data. The ensures that the ratios are maintained through filtering.
+
 	// idleCoefficients are organized by [cluster][allocation][resource]=coeff
 	var idleCoefficients map[string]map[string]map[string]float64
+
 	// shareCoefficients are organized by [allocation][resource]=coeff (no cluster)
 	var shareCoefficients map[string]float64
+
 	var err error
 
+	// (2a) If there are idle costs and we intend to share them, compute the
+	// coefficients for sharing the cost among the non-idle, non-aggregated
+	// allocations.
 	if idleSet.Length() > 0 && options.ShareIdle != ShareNone {
 		idleCoefficients, err = computeIdleCoeffs(properties, options, as)
 		if err != nil {
 			log.Warningf("AllocationSet.AggregateBy: compute idle coeff: %s", err)
-			return err
+			return fmt.Errorf("error computing idle coefficients: %s", err)
 		}
 	}
 
-	// If we're not sharing idle and we're filtering, we need to track the
+	// (2b) If we're not sharing idle and we're filtering, we need to track the
 	// amount of each idle allocation to "delete" in order to maintain parity
 	// with the idle-allocated results. That is, we want to return only the
 	// idle cost that would have been shared with the unfiltered portion of
@@ -556,10 +557,11 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 		idleFiltrationCoefficients, err = computeIdleCoeffs(properties, options, as)
 		if err != nil {
 			log.Warningf("AllocationSet.AggregateBy: compute idle coeff: %s", err)
-			return err
+			return fmt.Errorf("error computing idle filtration coefficients: %s", err)
 		}
 	}
 
+	// (3-5) Filter, distribute idle cost, and aggregate (in that order)
 	for _, alloc := range as.allocations {
 		cluster, err := alloc.Properties.GetCluster()
 		if err != nil {
@@ -569,7 +571,7 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 
 		skip := false
 
-		// If any of the filter funcs fail, immediately skip the allocation.
+		// (3) If any of the filter funcs fail, immediately skip the allocation.
 		for _, ff := range options.FilterFuncs {
 			if !ff(alloc) {
 				skip = true
@@ -590,9 +592,11 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 			continue
 		}
 
-		// Split idle allocations and distribute among aggregated allocations
-		// NOTE: if idle allocation is off (i.e. ShareIdle == ShareNone) then all
-		// idle allocations will be in the aggSet at this point.
+		// (4) Split idle allocations and distribute among remaining
+		// un-aggregated allocations.
+		// NOTE: if idle allocation is off (i.e. ShareIdle == ShareNone) then
+		// all idle allocations will be in the aggSet at this point, so idleSet
+		// will be empty and we won't enter this block.
 		if idleSet.Length() > 0 {
 			// Distribute idle allocations by coefficient per-cluster, per-allocation
 			for _, idleAlloc := range idleSet.allocations {
@@ -630,6 +634,7 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 			}
 		}
 
+		// (5) generate key to use for aggregation-by-key and allocation name
 		key, err := alloc.generateKey(properties)
 		if err != nil {
 			return err
@@ -640,9 +645,15 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 			alloc.Name = UnallocatedSuffix
 		}
 
+		// Inserting the allocation with the generated key for a name will
+		// perform the actual basic aggregation step.
 		aggSet.Insert(alloc)
 	}
 
+	// clusterIdleFiltrationCoeffs is used to track per-resource idle
+	// coefficients on a cluster-by-cluster basis. It is, essentailly, an
+	// aggregation of idleFiltrationCoefficients after they have been
+	// filtered above (in step 3)
 	var clusterIdleFiltrationCoeffs map[string]map[string]float64
 	if idleFiltrationCoefficients != nil {
 		clusterIdleFiltrationCoeffs = map[string]map[string]float64{}
@@ -664,9 +675,10 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 		}
 	}
 
-	// If we have filters, and so have computed coefficients for scaling idle
-	// allocation costs by cluster, then use those coefficients to scale down
-	// each idle coefficient in the aggSet.
+	// (6) If we have both un-shared idle allocations and idle filtration
+	// coefficients (i.e. we have computed coefficients for scaling idle
+	// allocation costs by cluster) then use those coefficients to scale down
+	// each idle allocation.
 	if len(aggSet.idleKeys) > 0 && clusterIdleFiltrationCoeffs != nil {
 		for idleKey := range aggSet.idleKeys {
 			idleAlloc := aggSet.Get(idleKey)
@@ -687,7 +699,7 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 		}
 	}
 
-	// Split shared allocations and distribute among aggregated allocations
+	// (7) Split shared allocations and distribute among aggregated allocations
 	if shareSet.Length() > 0 {
 		shareCoefficients, err = computeShareCoeffs(properties, options, aggSet)
 		if err != nil {
@@ -716,7 +728,21 @@ func (as *AllocationSet) AggregateBy(properties Properties, options *AllocationA
 		}
 	}
 
-	// Combine all idle allocations into a single "__idle__" allocation
+	// (8) Aggregate external allocations into aggregated allocations. This may
+	// not be possible for every external allocation, but attempt to find an
+	// exact key match, given each external allocation's proerties, and
+	// aggregate if an exact match is found.
+	for _, alloc := range externalSet.allocations {
+		key, err := alloc.generateKey(properties)
+		if err != nil {
+			continue
+		}
+
+		alloc.Name = key
+		aggSet.Insert(alloc)
+	}
+
+	// (9) Combine all idle allocations into a single "__idle__" allocation
 	if !options.SplitIdle {
 		for _, idleAlloc := range aggSet.IdleAllocations() {
 			aggSet.Delete(idleAlloc.Name)
@@ -959,6 +985,42 @@ func (alloc *Allocation) generateKey(properties Properties) (string, error) {
 		}
 	}
 
+	if properties.HasAnnotations() {
+		annotations, err := alloc.Properties.GetAnnotations() // annotations that the individual allocation possesses
+		if err != nil {
+			// Indicate that allocation has no annotations
+			names = append(names, UnallocatedSuffix)
+		} else {
+			annotationNames := []string{}
+
+			aggAnnotations, err := properties.GetAnnotations() // potential annotations to aggregate on supplied by the API caller
+			if err != nil {
+				// We've already checked HasAnnotation, so this should never occur
+				return "", err
+			}
+			// calvin - support multi-annotation aggregation
+			for annotationName := range aggAnnotations {
+				if val, ok := annotations[annotationName]; ok {
+					annotationNames = append(annotationNames, fmt.Sprintf("%s=%s", annotationName, val))
+				} else if indexOf(UnallocatedSuffix, annotationNames) == -1 { // if UnallocatedSuffix not already in names
+					annotationNames = append(annotationNames, UnallocatedSuffix)
+				}
+			}
+			// resolve arbitrary ordering. e.g., app=app0/env=env0 is the same agg as env=env0/app=app0
+			if len(annotationNames) > 1 {
+				sort.Strings(annotationNames)
+			}
+			unallocatedSuffixIndex := indexOf(UnallocatedSuffix, annotationNames)
+			// suffix should be at index 0 if it exists b/c of underscores
+			if unallocatedSuffixIndex != -1 {
+				annotationNames = append(annotationNames[:unallocatedSuffixIndex], annotationNames[unallocatedSuffixIndex+1:]...)
+				annotationNames = append(annotationNames, UnallocatedSuffix) // append to end
+			}
+
+			names = append(names, annotationNames...)
+		}
+	}
+
 	if properties.HasLabel() {
 		labels, err := alloc.Properties.GetLabels() // labels that the individual allocation possesses
 		if err != nil {
@@ -998,6 +1060,7 @@ func (alloc *Allocation) generateKey(properties Properties) (string, error) {
 	return strings.Join(names, "/"), nil
 }
 
+// TODO clean up
 // Helper function to check for slice membership. Not sure if repeated elsewhere in our codebase.
 func indexOf(v string, arr []string) int {
 	for i, s := range arr {
@@ -1024,12 +1087,133 @@ func (as *AllocationSet) Clone() *AllocationSet {
 		allocs[k] = v.Clone()
 	}
 
+	externalKeys := map[string]bool{}
+	for k, v := range as.externalKeys {
+		externalKeys[k] = v
+	}
+
+	idleKeys := map[string]bool{}
+	for k, v := range as.idleKeys {
+		idleKeys[k] = v
+	}
+
 	return &AllocationSet{
-		allocations: allocs,
-		Window:      as.Window.Clone(),
+		allocations:  allocs,
+		externalKeys: externalKeys,
+		idleKeys:     idleKeys,
+		Window:       as.Window.Clone(),
 	}
 }
 
+// ComputeIdleAllocations computes the idle allocations for the AllocationSet,
+// given a set of Assets. Ideally, assetSet should contain only Nodes, but if
+// it contains other Assets, they will be ignored; only CPU, GPU and RAM are
+// considered for idle allocation. One idle allocation per-cluster will be
+// computed and returned, keyed by cluster_id.
+func (as *AllocationSet) ComputeIdleAllocations(assetSet *AssetSet) (map[string]*Allocation, error) {
+	if as == nil {
+		return nil, fmt.Errorf("cannot compute idle allocation for nil AllocationSet")
+	}
+
+	// TODO: external allocation: remove after testing and benchmarking
+	profStart := time.Now()
+	defer log.Profile(profStart, fmt.Sprintf("ComputeIdleAllocations: %s", as.Window))
+
+	if assetSet == nil {
+		return nil, fmt.Errorf("cannot compute idle allocation with nil AssetSet")
+	}
+
+	if !as.Window.Equal(assetSet.Window) {
+		return nil, fmt.Errorf("cannot compute idle allocation for sets with mismatched windows: %s != %s", as.Window, assetSet.Window)
+	}
+
+	window := as.Window
+
+	// Build a map of cumulative cluster asset costs, per resource; i.e.
+	// cluster-to-{cpu|gpu|ram}-to-cost.
+	assetClusterResourceCosts := map[string]map[string]float64{}
+	assetSet.Each(func(key string, a Asset) {
+		if node, ok := a.(*Node); ok {
+			if _, ok := assetClusterResourceCosts[node.Properties().Cluster]; !ok {
+				assetClusterResourceCosts[node.Properties().Cluster] = map[string]float64{}
+			}
+			assetClusterResourceCosts[node.Properties().Cluster]["cpu"] += node.CPUCost * (1.0 - node.Discount)
+			assetClusterResourceCosts[node.Properties().Cluster]["gpu"] += node.GPUCost * (1.0 - node.Discount)
+			assetClusterResourceCosts[node.Properties().Cluster]["ram"] += node.RAMCost * (1.0 - node.Discount)
+		}
+	})
+
+	// Determine start, end on a per-cluster basis
+	clusterStarts := map[string]time.Time{}
+	clusterEnds := map[string]time.Time{}
+
+	// Subtract allocated costs from asset costs, leaving only the remaining
+	// idle costs.
+	as.Each(func(name string, a *Allocation) {
+		cluster, err := a.Properties.GetCluster()
+		if err != nil {
+			// Failed to find allocation's cluster
+			return
+		}
+
+		if _, ok := assetClusterResourceCosts[cluster]; !ok {
+			// Failed to find assets for allocation's cluster
+			return
+		}
+
+		// Set cluster (start, end) if they are either not currently set,
+		// or if the detected (start, end) of the current allocation falls
+		// before or after, respectively, the current values.
+		if s, ok := clusterStarts[cluster]; !ok || a.Start.Before(s) {
+			clusterStarts[cluster] = a.Start
+		}
+		if e, ok := clusterEnds[cluster]; !ok || a.End.Before(e) {
+			clusterEnds[cluster] = a.End
+		}
+
+		assetClusterResourceCosts[cluster]["cpu"] -= a.CPUCost
+		assetClusterResourceCosts[cluster]["gpu"] -= a.GPUCost
+		assetClusterResourceCosts[cluster]["ram"] -= a.RAMCost
+	})
+
+	// Turn remaining un-allocated asset costs into idle allocations
+	idleAllocs := map[string]*Allocation{}
+	for cluster, resources := range assetClusterResourceCosts {
+		// Default start and end to the (start, end) of the given window, but
+		// use the actual, detected (start, end) pair if they are available.
+		start := *window.Start()
+		if s, ok := clusterStarts[cluster]; ok && window.Contains(s) {
+			start = s
+		}
+		end := *window.End()
+		if e, ok := clusterEnds[cluster]; ok && window.Contains(e) {
+			end = e
+		}
+
+		idleAlloc := &Allocation{
+			Name:       fmt.Sprintf("%s/%s", cluster, IdleSuffix),
+			Properties: Properties{ClusterProp: cluster},
+			Start:      start,
+			End:        end,
+			Minutes:    end.Sub(start).Minutes(), // TODO deprecate w/ niko/allocation-minutes
+			CPUCost:    resources["cpu"],
+			GPUCost:    resources["gpu"],
+			RAMCost:    resources["ram"],
+		}
+		idleAlloc.TotalCost = idleAlloc.CPUCost + idleAlloc.GPUCost + idleAlloc.RAMCost
+
+		// Do not continue if multiple idle allocations are computed for a
+		// single cluster.
+		if _, ok := idleAllocs[cluster]; ok {
+			return nil, fmt.Errorf("duplicate idle allocations for cluster %s", cluster)
+		}
+
+		idleAllocs[cluster] = idleAlloc
+	}
+
+	return idleAllocs, nil
+}
+
 // Delete removes the allocation with the given name from the set
 func (as *AllocationSet) Delete(name string) {
 	if as == nil {
@@ -1038,6 +1222,7 @@ func (as *AllocationSet) Delete(name string) {
 
 	as.Lock()
 	defer as.Unlock()
+	delete(as.externalKeys, name)
 	delete(as.idleKeys, name)
 	delete(as.allocations, name)
 }
@@ -1078,6 +1263,44 @@ func (as *AllocationSet) Get(key string) *Allocation {
 	return nil
 }
 
+// ExternalAllocations returns a map of the external allocations in the set.
+// Returns clones of the actual Allocations, so mutability is not a problem.
+func (as *AllocationSet) ExternalAllocations() map[string]*Allocation {
+	externals := map[string]*Allocation{}
+
+	if as.IsEmpty() {
+		return externals
+	}
+
+	as.RLock()
+	defer as.RUnlock()
+
+	for key := range as.externalKeys {
+		if alloc, ok := as.allocations[key]; ok {
+			externals[key] = alloc.Clone()
+		}
+	}
+
+	return externals
+}
+
+// ExternalCost returns the total aggregated external costs of the set
+func (as *AllocationSet) ExternalCost() float64 {
+	if as.IsEmpty() {
+		return 0.0
+	}
+
+	as.RLock()
+	defer as.RUnlock()
+
+	externalCost := 0.0
+	for _, alloc := range as.allocations {
+		externalCost += alloc.ExternalCost
+	}
+
+	return externalCost
+}
+
 // IdleAllocations returns a map of the idle allocations in the AllocationSet.
 // Returns clones of the actual Allocations, so mutability is not a problem.
 func (as *AllocationSet) IdleAllocations() map[string]*Allocation {
@@ -1107,16 +1330,25 @@ func (as *AllocationSet) Insert(that *Allocation) error {
 }
 
 func (as *AllocationSet) insert(that *Allocation, accumulate bool) error {
-	if as.IsEmpty() {
-		as.Lock()
-		as.allocations = map[string]*Allocation{}
-		as.idleKeys = map[string]bool{}
-		as.Unlock()
+	if as == nil {
+		return fmt.Errorf("cannot insert into nil AllocationSet")
 	}
 
 	as.Lock()
 	defer as.Unlock()
 
+	if as.allocations == nil {
+		as.allocations = map[string]*Allocation{}
+	}
+
+	if as.externalKeys == nil {
+		as.externalKeys = map[string]bool{}
+	}
+
+	if as.idleKeys == nil {
+		as.idleKeys = map[string]bool{}
+	}
+
 	// Add the given Allocation to the existing entry, if there is one;
 	// otherwise just set directly into allocations
 	if _, ok := as.allocations[that.Name]; !ok {
@@ -1125,6 +1357,11 @@ func (as *AllocationSet) insert(that *Allocation, accumulate bool) error {
 		as.allocations[that.Name].add(that, false, accumulate)
 	}
 
+	// If the given Allocation is an external one, record that
+	if that.IsExternal() {
+		as.externalKeys[that.Name] = true
+	}
+
 	// If the given Allocation is an idle one, record that
 	if that.IsIdle() {
 		as.idleKeys[that.Name] = true
@@ -1177,10 +1414,13 @@ func (as *AllocationSet) Resolution() time.Duration {
 	return as.Window.Duration()
 }
 
+// Set uses the given Allocation to overwrite the existing entry in the
+// AllocationSet under the Allocation's name.
 func (as *AllocationSet) Set(alloc *Allocation) error {
 	if as.IsEmpty() {
 		as.Lock()
 		as.allocations = map[string]*Allocation{}
+		as.externalKeys = map[string]bool{}
 		as.idleKeys = map[string]bool{}
 		as.Unlock()
 	}
@@ -1190,6 +1430,11 @@ func (as *AllocationSet) Set(alloc *Allocation) error {
 
 	as.allocations[alloc.Name] = alloc
 
+	// If the given Allocation is an external one, record that
+	if alloc.IsExternal() {
+		as.externalKeys[alloc.Name] = true
+	}
+
 	// If the given Allocation is an idle one, record that
 	if alloc.IsIdle() {
 		as.idleKeys[alloc.Name] = true
@@ -1236,6 +1481,7 @@ func (as *AllocationSet) TotalCost() float64 {
 	return tc
 }
 
+// UTCOffset returns the AllocationSet's configured UTCOffset.
 func (as *AllocationSet) UTCOffset() time.Duration {
 	_, zone := as.Start().Zone()
 	return time.Duration(zone) * time.Second
@@ -1303,11 +1549,17 @@ func (as *AllocationSet) accumulate(that *AllocationSet) (*AllocationSet, error)
 	return acc, nil
 }
 
+// AllocationSetRange is a thread-safe slice of AllocationSets. It is meant to
+// be used such that the AllocationSets held are consecutive and coherent with
+// respect to using the same aggregation properties, UTC offset, and
+// resolution. However these rules are not necessarily enforced, so use wisely.
 type AllocationSetRange struct {
 	sync.RWMutex
 	allocations []*AllocationSet
 }
 
+// NewAllocationSetRange instantiates a new range composed of the given
+// AllocationSets in the order provided.
 func NewAllocationSetRange(allocs ...*AllocationSet) *AllocationSetRange {
 	return &AllocationSetRange{
 		allocations: allocs,
@@ -1336,6 +1588,8 @@ func (asr *AllocationSetRange) Accumulate() (*AllocationSet, error) {
 // TODO niko/etl accumulate into lower-resolution chunks of the given resolution
 // func (asr *AllocationSetRange) AccumulateBy(resolution time.Duration) *AllocationSetRange
 
+// AggregateBy aggregates each AllocationSet in the range by the given
+// properties and options.
 func (asr *AllocationSetRange) AggregateBy(properties Properties, options *AllocationAggregationOptions) error {
 	aggRange := &AllocationSetRange{allocations: []*AllocationSet{}}
 
@@ -1355,6 +1609,8 @@ func (asr *AllocationSetRange) AggregateBy(properties Properties, options *Alloc
 	return nil
 }
 
+// Append appends the given AllocationSet to the end of the range. It does not
+// validate whether or not that violates window continuity.
 func (asr *AllocationSetRange) Append(that *AllocationSet) {
 	asr.Lock()
 	defer asr.Unlock()
@@ -1372,6 +1628,7 @@ func (asr *AllocationSetRange) Each(f func(int, *AllocationSet)) {
 	}
 }
 
+// Get retrieves the AllocationSet at the given index of the range.
 func (asr *AllocationSetRange) Get(i int) (*AllocationSet, error) {
 	if i < 0 || i >= len(asr.allocations) {
 		return nil, fmt.Errorf("AllocationSetRange: index out of range: %d", i)
@@ -1382,6 +1639,64 @@ func (asr *AllocationSetRange) Get(i int) (*AllocationSet, error) {
 	return asr.allocations[i], nil
 }
 
+// InsertRange merges the given AllocationSetRange into the receiving one by
+// lining up sets with matching windows, then inserting each allocation from
+// the given ASR into the respective set in the receiving ASR. If the given
+// ASR contains an AllocationSet from a window that does not exist in the
+// receiving ASR, then an error is returned. However, the given ASR does not
+// need to cover the full range of the receiver.
+func (asr *AllocationSetRange) InsertRange(that *AllocationSetRange) error {
+	if asr == nil {
+		return fmt.Errorf("cannot insert range into nil AllocationSetRange")
+	}
+
+	// keys maps window to index in asr
+	keys := map[string]int{}
+	asr.Each(func(i int, as *AllocationSet) {
+		if as == nil {
+			return
+		}
+		keys[as.Window.String()] = i
+	})
+
+	// Nothing to merge, so simply return
+	if len(keys) == 0 {
+		return nil
+	}
+
+	var err error
+	that.Each(func(j int, thatAS *AllocationSet) {
+		if thatAS == nil || err != nil {
+			return
+		}
+
+		// Find matching AllocationSet in asr
+		i, ok := keys[thatAS.Window.String()]
+		if !ok {
+			err = fmt.Errorf("cannot merge AllocationSet into window that does not exist: %s", thatAS.Window.String())
+			return
+		}
+		as, err := asr.Get(i)
+		if err != nil {
+			err = fmt.Errorf("AllocationSetRange index does not exist: %d", i)
+			return
+		}
+
+		// Insert each Allocation from the given set
+		thatAS.Each(func(k string, alloc *Allocation) {
+			err = as.Insert(alloc)
+			if err != nil {
+				err = fmt.Errorf("error inserting allocation: %s", err)
+				return
+			}
+		})
+	})
+
+	// err might be nil
+	return err
+}
+
+// Length returns the length of the range, which is zero if nil
 func (asr *AllocationSetRange) Length() int {
 	if asr == nil || asr.allocations == nil {
 		return 0
@@ -1392,12 +1707,15 @@ func (asr *AllocationSetRange) Length() int {
 	return len(asr.allocations)
 }
 
+// MarshalJSON JSON-encodes the range
 func (asr *AllocationSetRange) MarshalJSON() ([]byte, error) {
 	asr.RLock()
 	asr.RUnlock()
 	return json.Marshal(asr.allocations)
 }
 
+// Slice copies the underlying slice of AllocationSets, maintaining order,
+// and returns the copied slice.
 func (asr *AllocationSetRange) Slice() []*AllocationSet {
 	if asr == nil || asr.allocations == nil {
 		return nil
@@ -1420,6 +1738,9 @@ func (asr *AllocationSetRange) String() string {
 	return fmt.Sprintf("AllocationSetRange{length: %d}", asr.Length())
 }
 
+// UTCOffset returns the detected UTCOffset of the AllocationSets within the
+// range. Defaults to 0 if the range is nil or empty. Does not warn if there
+// are sets with conflicting UTCOffsets (just returns the first).
 func (asr *AllocationSetRange) UTCOffset() time.Duration {
 	if asr.Length() == 0 {
 		return 0

pkg/kubecost/allocation_test.go

@@ -5,6 +5,8 @@ import (
 	"math"
 	"testing"
 	"time"
+
+	util "github.com/kubecost/cost-model/pkg/util"
 )
 
 const day = 24 * time.Hour
@@ -96,106 +98,6 @@ func TestAllocation_Add(t *testing.T) {
 // TODO niko/etl
 // func TestAllocation_IsIdle(t *testing.T) {}
 
-func TestAllocation_MatchesAll(t *testing.T) {
-	var alloc *Allocation
-
-	// nil Allocations never match
-	if alloc.MatchesAll() {
-		t.Fatalf("Allocation.MatchesAll: expected no match on nil allocation")
-	}
-
-	today := time.Now().UTC().Truncate(day)
-	alloc = NewUnitAllocation("", today, day, nil)
-
-	// Matches when no Properties are given
-	if !alloc.MatchesAll() {
-		t.Fatalf("Allocation.MatchesAll: expected match on no conditions")
-	}
-
-	// Matches when all Properties match
-	if !alloc.MatchesAll(Properties{
-		NamespaceProp: "namespace1",
-	}, Properties{
-		ClusterProp:        "cluster1",
-		ControllerKindProp: "deployment",
-	}, Properties{
-		NodeProp: "node1",
-	}) {
-		t.Fatalf("Allocation.MatchesAll: expected match when all Properties are met")
-	}
-
-	// Doesn't match when one Property doesn't match
-	if alloc.MatchesAll(Properties{
-		NamespaceProp: "namespace1",
-		ServiceProp:   []string{"missing"},
-	}, Properties{
-		ClusterProp:        "cluster1",
-		ControllerKindProp: "deployment",
-	}) {
-		t.Fatalf("Allocation.MatchesAll: expected no match when one Properties is not met")
-	}
-
-	// Doesn't match when no Properties are met
-	if alloc.MatchesAll(Properties{
-		NamespaceProp: "namespace1",
-		ServiceProp:   []string{"missing"},
-	}, Properties{
-		ClusterProp:        "cluster2",
-		ControllerKindProp: "deployment",
-	}) {
-		t.Fatalf("Allocation.MatchesAll: expected no match when no Properties are met")
-	}
-}
-
-func TestAllocation_MatchesOne(t *testing.T) {
-	var alloc *Allocation
-
-	// nil Allocations never match
-	if alloc.MatchesOne() {
-		t.Fatalf("Allocation.MatchesOne: expected no match on nil allocation")
-	}
-
-	today := time.Now().UTC().Truncate(day)
-	alloc = NewUnitAllocation("", today, day, nil)
-
-	// Doesn't match when no Properties are given
-	if alloc.MatchesOne() {
-		t.Fatalf("Allocation.MatchesOne: expected no match on no conditions")
-	}
-
-	// Matches when all Properties match
-	if !alloc.MatchesOne(Properties{
-		NamespaceProp: "namespace1",
-	}, Properties{
-		ClusterProp:        "cluster1",
-		ControllerKindProp: "deployment",
-	}) {
-		t.Fatalf("Allocation.MatchesOne: expected match when all Properties are met")
-	}
-
-	// Matches when one Property doesn't match
-	if !alloc.MatchesOne(Properties{
-		NamespaceProp: "namespace1",
-		ServiceProp:   []string{"missing"},
-	}, Properties{
-		ClusterProp:        "cluster1",
-		ControllerKindProp: "deployment",
-	}) {
-		t.Fatalf("Allocation.MatchesOne: expected match when one Properties is met")
-	}
-
-	// Doesn't match when no Properties are met
-	if alloc.MatchesOne(Properties{
-		NamespaceProp: "namespace1",
-		ServiceProp:   []string{"missing"},
-	}, Properties{
-		ClusterProp:        "cluster2",
-		ControllerKindProp: "deployment",
-	}) {
-		t.Fatalf("Allocation.MatchesOne: expected no match when no Properties are met")
-	}
-}
-
 func TestAllocation_String(t *testing.T) {
 	// TODO niko/etl
 }
@@ -208,7 +110,7 @@ func generateAllocationSet(start time.Time) *AllocationSet {
 	// Idle allocations
 	a1i := NewUnitAllocation(fmt.Sprintf("cluster1/%s", IdleSuffix), start, day, &Properties{
 		ClusterProp: "cluster1",
-		NodeProp: "node1",
+		NodeProp:    "node1",
 	})
 	a1i.CPUCost = 5.0
 	a1i.RAMCost = 15.0
@@ -347,6 +249,11 @@ func generateAllocationSet(start time.Time) *AllocationSet {
 	a22mno4.Properties.SetLabels(map[string]string{"app": "app2"})
 	a22mno5.Properties.SetLabels(map[string]string{"app": "app2"})
 
+	//Annotations
+	a23stu7.Properties.SetAnnotations(map[string]string{"team": "team1"})
+	a23vwx8.Properties.SetAnnotations(map[string]string{"team": "team2"})
+	a23vwx9.Properties.SetAnnotations(map[string]string{"team": "team1"})
+
 	// Services
 
 	a12jkl6.Properties.SetServices([]string{"service1"})
@@ -445,10 +352,10 @@ func TestAllocationSet_AggregateBy(t *testing.T) {
 	//         container6: {service1}              5.00   1.00   1.00   1.00   1.00   1.00
 	//     namespace3:
 	//       pod-stu: (deployment3)
-	//         container7:                         5.00   1.00   1.00   1.00   1.00   1.00
+	//         container7: an[team=team1]          5.00   1.00   1.00   1.00   1.00   1.00
 	//       pod-vwx: (statefulset1)
-	//         container8:                         5.00   1.00   1.00   1.00   1.00   1.00
-	//         container9:                         5.00   1.00   1.00   1.00   1.00   1.00
+	//         container8: an[team=team2]          5.00   1.00   1.00   1.00   1.00   1.00
+	//         container9: an[team=team1]          5.00   1.00   1.00   1.00   1.00   1.00
 	// +----------------------------------------+------+------+------+------+------+------+
 	//   cluster2 subtotal                        40.00  11.00  11.00   6.00   6.00   6.00
 	// +----------------------------------------+------+------+------+------+------+------+
@@ -669,6 +576,18 @@ func TestAllocationSet_AggregateBy(t *testing.T) {
 	})
 	assertAllocationWindow(t, as, "1i", startYesterday, endYesterday, 1440.0)
 
+	// 1j AggregationProperties=(Annotation:team)
+	as = generateAllocationSet(start)
+	err = as.AggregateBy(Properties{AnnotationProp: map[string]string{"team": ""}}, nil)
+	assertAllocationSetTotals(t, as, "1j", err, 2+numIdle+numUnallocated, activeTotalCost+idleTotalCost)
+	assertAllocationTotals(t, as, "1j", map[string]float64{
+		"team=team1":      10.00,
+		"team=team2":      5.00,
+		IdleSuffix:        30.00,
+		UnallocatedSuffix: 55.00,
+	})
+	assertAllocationWindow(t, as, "1i", startYesterday, endYesterday, 1440.0)
+
 	// 2  Multi-aggregation
 
 	// 2a AggregationProperties=(Cluster, Namespace)
@@ -701,6 +620,24 @@ func TestAllocationSet_AggregateBy(t *testing.T) {
 		"cluster2/" + UnallocatedSuffix:          20.00,
 	})
 
+	// 2f AggregationProperties=(annotation:team, pod)
+	as = generateAllocationSet(start)
+	err = as.AggregateBy(Properties{AnnotationProp: map[string]string{"team": ""}, PodProp: ""}, nil)
+	assertAllocationSetTotals(t, as, "2e", err, 11, activeTotalCost+idleTotalCost)
+	assertAllocationTotals(t, as, "2e", map[string]float64{
+		"pod-jkl/" + UnallocatedSuffix: 5.00,
+		"pod-stu/team=team1":           5.00,
+		"pod-abc/" + UnallocatedSuffix: 5.00,
+		"pod-pqr/" + UnallocatedSuffix: 5.00,
+		"pod-def/" + UnallocatedSuffix: 5.00,
+		"pod-vwx/team=team1":           5.00,
+		"pod-vwx/team=team2":           5.00,
+		"pod1/" + UnallocatedSuffix:    15.00,
+		"pod-mno/" + UnallocatedSuffix: 10.00,
+		"pod-ghi/" + UnallocatedSuffix: 10.00,
+		IdleSuffix:                     30.00,
+	})
+
 	// // TODO niko/etl
 
 	// // 3  Share idle
@@ -941,6 +878,106 @@ func TestAllocationSet_AggregateBy(t *testing.T) {
 // TODO niko/etl
 //func TestAllocationSet_Clone(t *testing.T) {}
 
+func TestAllocationSet_ComputeIdleAllocations(t *testing.T) {
+	var as *AllocationSet
+	var err error
+	var idles map[string]*Allocation
+
+	end := time.Now().UTC().Truncate(day)
+	start := end.Add(-day)
+
+	// Generate AllocationSet and strip out any existing idle allocations
+	as = generateAllocationSet(start)
+	for key := range as.idleKeys {
+		as.Delete(key)
+	}
+
+	// Create an AssetSet representing cluster costs for two clusters (cluster1
+	// and cluster2). Include Nodes and Disks for both, even though only
+	// Nodes will be counted. Whereas in practice, Assets should be aggregated
+	// by type, here we will provide multiple Nodes for one of the clusters to
+	// make sure the function still holds.
+
+	// NOTE: we're re-using generateAllocationSet so this has to line up with
+	// the allocated node costs from that function. See table above.
+
+	// | Hierarchy                               | Cost |  CPU |  RAM |  GPU |
+	// +-----------------------------------------+------+------+------+------+
+	//   cluster1:
+	//     nodes                                  100.00  50.00  40.00  10.00
+	// +-----------------------------------------+------+------+------+------+
+	//   cluster1 subtotal                        100.00  50.00  40.00  10.00
+	// +-----------------------------------------+------+------+------+------+
+	//   cluster1 allocated                        48.00   6.00  16.00   6.00
+	// +-----------------------------------------+------+------+------+------+
+	//   cluster1 idle                             72.00  44.00  24.00   4.00
+	// +-----------------------------------------+------+------+------+------+
+	//   cluster2:
+	//     node1                                   35.00  20.00  15.00   0.00
+	//     node2                                   35.00  20.00  15.00   0.00
+	//     node3                                   30.00  10.00  10.00  10.00
+	//     (disks should not matter for idle)
+	// +-----------------------------------------+------+------+------+------+
+	//   cluster2 subtotal                        100.00  50.00  40.00  10.00
+	// +-----------------------------------------+------+------+------+------+
+	//   cluster2 allocated                        28.00   6.00   6.00   6.00
+	// +-----------------------------------------+------+------+------+------+
+	//   cluster2 idle                             82.00  44.00  34.00   4.00
+	// +-----------------------------------------+------+------+------+------+
+
+	cluster1Nodes := NewNode("", "cluster1", "", start, end, NewWindow(&start, &end))
+	cluster1Nodes.CPUCost = 50.0
+	cluster1Nodes.RAMCost = 40.0
+	cluster1Nodes.GPUCost = 10.0
+
+	cluster2Node1 := NewNode("node1", "cluster2", "node1", start, end, NewWindow(&start, &end))
+	cluster2Node1.CPUCost = 20.0
+	cluster2Node1.RAMCost = 15.0
+	cluster2Node1.GPUCost = 0.0
+
+	cluster2Node2 := NewNode("node2", "cluster2", "node2", start, end, NewWindow(&start, &end))
+	cluster2Node2.CPUCost = 20.0
+	cluster2Node2.RAMCost = 15.0
+	cluster2Node2.GPUCost = 0.0
+
+	cluster2Node3 := NewNode("node3", "cluster2", "node3", start, end, NewWindow(&start, &end))
+	cluster2Node3.CPUCost = 10.0
+	cluster2Node3.RAMCost = 10.0
+	cluster2Node3.GPUCost = 10.0
+
+	cluster2Disk1 := NewDisk("disk1", "cluster2", "disk1", start, end, NewWindow(&start, &end))
+	cluster2Disk1.Cost = 5.0
+
+	assetSet := NewAssetSet(start, end, cluster1Nodes, cluster2Node1, cluster2Node2, cluster2Node3, cluster2Disk1)
+
+	idles, err = as.ComputeIdleAllocations(assetSet)
+	if err != nil {
+		t.Fatalf("unexpected error: %s", err)
+	}
+
+	if len(idles) != 2 {
+		t.Fatalf("idles: expected length %d; got length %d", 2, len(idles))
+	}
+
+	if idle, ok := idles["cluster1"]; !ok {
+		t.Fatalf("expected idle cost for %s", "cluster1")
+	} else {
+		if !util.IsApproximately(idle.TotalCost, 72.0) {
+			t.Fatalf("%s idle: expected total cost %f; got total cost %f", "cluster1", 72.0, idle.TotalCost)
+		}
+	}
+
+	if idle, ok := idles["cluster2"]; !ok {
+		t.Fatalf("expected idle cost for %s", "cluster2")
+	} else {
+		if !util.IsApproximately(idle.TotalCost, 82.0) {
+			t.Fatalf("%s idle: expected total cost %f; got total cost %f", "cluster2", 82.0, idle.TotalCost)
+		}
+	}
+
+	// TODO assert value of each resource cost precisely
+}
+
 // TODO niko/etl
 //func TestAllocationSet_Delete(t *testing.T) {}
 
@@ -1140,6 +1177,169 @@ func TestAllocationSetRange_Accumulate(t *testing.T) {
 // TODO niko/etl
 // func TestAllocationSetRange_Append(t *testing.T) {}
 
+// TODO niko/etl
+// func TestAllocationSetRange_Each(t *testing.T) {}
+
+// TODO niko/etl
+// func TestAllocationSetRange_Get(t *testing.T) {}
+
+func TestAllocationSetRange_InsertRange(t *testing.T) {
+	// Set up
+	ago2d := time.Now().UTC().Truncate(day).Add(-2 * day)
+	yesterday := time.Now().UTC().Truncate(day).Add(-day)
+	today := time.Now().UTC().Truncate(day)
+	tomorrow := time.Now().UTC().Truncate(day).Add(day)
+
+	unit := NewUnitAllocation("", today, day, nil)
+
+	ago2dAS := NewAllocationSet(ago2d, yesterday)
+	ago2dAS.Set(NewUnitAllocation("a", ago2d, day, nil))
+	ago2dAS.Set(NewUnitAllocation("b", ago2d, day, nil))
+	ago2dAS.Set(NewUnitAllocation("c", ago2d, day, nil))
+
+	yesterdayAS := NewAllocationSet(yesterday, today)
+	yesterdayAS.Set(NewUnitAllocation("a", yesterday, day, nil))
+	yesterdayAS.Set(NewUnitAllocation("b", yesterday, day, nil))
+	yesterdayAS.Set(NewUnitAllocation("c", yesterday, day, nil))
+
+	todayAS := NewAllocationSet(today, tomorrow)
+	todayAS.Set(NewUnitAllocation("a", today, day, nil))
+	todayAS.Set(NewUnitAllocation("b", today, day, nil))
+	todayAS.Set(NewUnitAllocation("c", today, day, nil))
+
+	var nilASR *AllocationSetRange
+	thisASR := NewAllocationSetRange(yesterdayAS.Clone(), todayAS.Clone())
+	thatASR := NewAllocationSetRange(yesterdayAS.Clone())
+	longASR := NewAllocationSetRange(ago2dAS.Clone(), yesterdayAS.Clone(), todayAS.Clone())
+	var err error
+
+	// Expect an error calling InsertRange on nil
+	err = nilASR.InsertRange(thatASR)
+	if err == nil {
+		t.Fatalf("expected error, got nil")
+	}
+
+	// Expect nothing to happen calling InsertRange(nil) on non-nil ASR
+	err = thisASR.InsertRange(nil)
+	if err != nil {
+		t.Fatalf("unexpected error: %s", err)
+	}
+	thisASR.Each(func(i int, as *AllocationSet) {
+		as.Each(func(k string, a *Allocation) {
+			if !util.IsApproximately(a.CPUCoreHours, unit.CPUCoreHours) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCoreHours, a.CPUCoreHours)
+			}
+			if !util.IsApproximately(a.CPUCost, unit.CPUCost) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCost, a.CPUCost)
+			}
+			if !util.IsApproximately(a.RAMByteHours, unit.RAMByteHours) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.RAMByteHours, a.RAMByteHours)
+			}
+			if !util.IsApproximately(a.RAMCost, unit.RAMCost) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.RAMCost, a.RAMCost)
+			}
+			if !util.IsApproximately(a.GPUHours, unit.GPUHours) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.GPUHours, a.GPUHours)
+			}
+			if !util.IsApproximately(a.GPUCost, unit.GPUCost) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.GPUCost, a.GPUCost)
+			}
+			if !util.IsApproximately(a.PVByteHours, unit.PVByteHours) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.PVByteHours, a.PVByteHours)
+			}
+			if !util.IsApproximately(a.PVCost, unit.PVCost) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.PVCost, a.PVCost)
+			}
+			if !util.IsApproximately(a.NetworkCost, unit.NetworkCost) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.NetworkCost, a.NetworkCost)
+			}
+			if !util.IsApproximately(a.TotalCost, unit.TotalCost) {
+				t.Fatalf("allocation %s: expected %f; got %f", k, unit.TotalCost, a.TotalCost)
+			}
+		})
+	})
+
+	// Expect an error calling InsertRange with a range exceeding the receiver
+	err = thisASR.InsertRange(longASR)
+	if err == nil {
+		t.Fatalf("expected error calling InsertRange with a range exceeding the receiver")
+	}
+
+	// Expect each Allocation in "today" to stay the same, but "yesterday" to
+	// precisely double when inserting a range that only has a duplicate of
+	// "yesterday", but no entry for "today"
+	err = thisASR.InsertRange(thatASR)
+	if err != nil {
+		t.Fatalf("unexpected error: %s", err)
+	}
+	yAS, err := thisASR.Get(0)
+	yAS.Each(func(k string, a *Allocation) {
+		if !util.IsApproximately(a.CPUCoreHours, 2*unit.CPUCoreHours) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCoreHours, a.CPUCoreHours)
+		}
+		if !util.IsApproximately(a.CPUCost, 2*unit.CPUCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCost, a.CPUCost)
+		}
+		if !util.IsApproximately(a.RAMByteHours, 2*unit.RAMByteHours) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.RAMByteHours, a.RAMByteHours)
+		}
+		if !util.IsApproximately(a.RAMCost, 2*unit.RAMCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.RAMCost, a.RAMCost)
+		}
+		if !util.IsApproximately(a.GPUHours, 2*unit.GPUHours) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.GPUHours, a.GPUHours)
+		}
+		if !util.IsApproximately(a.GPUCost, 2*unit.GPUCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.GPUCost, a.GPUCost)
+		}
+		if !util.IsApproximately(a.PVByteHours, 2*unit.PVByteHours) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.PVByteHours, a.PVByteHours)
+		}
+		if !util.IsApproximately(a.PVCost, 2*unit.PVCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.PVCost, a.PVCost)
+		}
+		if !util.IsApproximately(a.NetworkCost, 2*unit.NetworkCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.NetworkCost, a.NetworkCost)
+		}
+		if !util.IsApproximately(a.TotalCost, 2*unit.TotalCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.TotalCost, a.TotalCost)
+		}
+	})
+	tAS, err := thisASR.Get(1)
+	tAS.Each(func(k string, a *Allocation) {
+		if !util.IsApproximately(a.CPUCoreHours, unit.CPUCoreHours) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCoreHours, a.CPUCoreHours)
+		}
+		if !util.IsApproximately(a.CPUCost, unit.CPUCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCost, a.CPUCost)
+		}
+		if !util.IsApproximately(a.RAMByteHours, unit.RAMByteHours) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.RAMByteHours, a.RAMByteHours)
+		}
+		if !util.IsApproximately(a.RAMCost, unit.RAMCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.RAMCost, a.RAMCost)
+		}
+		if !util.IsApproximately(a.GPUHours, unit.GPUHours) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.GPUHours, a.GPUHours)
+		}
+		if !util.IsApproximately(a.GPUCost, unit.GPUCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.GPUCost, a.GPUCost)
+		}
+		if !util.IsApproximately(a.PVByteHours, unit.PVByteHours) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.PVByteHours, a.PVByteHours)
+		}
+		if !util.IsApproximately(a.PVCost, unit.PVCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.PVCost, a.PVCost)
+		}
+		if !util.IsApproximately(a.NetworkCost, unit.NetworkCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.NetworkCost, a.NetworkCost)
+		}
+		if !util.IsApproximately(a.TotalCost, unit.TotalCost) {
+			t.Fatalf("allocation %s: expected %f; got %f", k, unit.TotalCost, a.TotalCost)
+		}
+	})
+}
+
 // TODO niko/etl
 // func TestAllocationSetRange_Length(t *testing.T) {}
 

pkg/kubecost/asset.go

@@ -15,6 +15,10 @@ import (
 
 const timeFmt = "2006-01-02T15:04:05-0700"
 
+// UndefinedKey is used in composing Asset group keys if the group does not have that property defined.
+// E.g. if aggregating on Cluster, Assets in the AssetSet where Asset has no cluster will be grouped under key "__undefined__"
+const UndefinedKey = "__undefined__"
+
 // Asset defines an entity within a cluster that has a defined cost over a
 // given period of time.
 type Asset interface {
@@ -56,52 +60,207 @@ type Asset interface {
 	fmt.Stringer
 }
 
+// AssetToExternalAllocation converts the given asset to an Allocation, given
+// the properties to use to aggregate, and the mapping from Allocation property
+// to Asset label. For example, consider this asset:
+//
+//   Cloud {
+// 	   TotalCost: 10.00,
+// 	   Labels{
+//       "kubernetes_namespace":"monitoring",
+// 	     "env":"prod"
+// 	   }
+//   }
+//
+// Given the following parameters, we expect to return:
+//
+//   1) single-prop full match
+//   aggregateBy = ["namespace"]
+//   allocationPropertyLabels = {"namespace":"kubernetes_namespace"}
+//   => Allocation{Name: "monitoring", ExternalCost: 10.00, TotalCost: 10.00}, nil
+//
+//   2) multi-prop full match
+//   aggregateBy = ["namespace", "label:env"]
+//   allocationPropertyLabels = {"namespace":"kubernetes_namespace"}
+//   => Allocation{Name: "monitoring/env=prod", ExternalCost: 10.00, TotalCost: 10.00}, nil
+//
+//   3) multi-prop partial match
+//   aggregateBy = ["namespace", "label:foo"]
+//   allocationPropertyLabels = {"namespace":"kubernetes_namespace"}
+//   => Allocation{Name: "monitoring/__unallocated__", ExternalCost: 10.00, TotalCost: 10.00}, nil
+//
+//   4) no match
+//   aggregateBy = ["cluster"]
+//   allocationPropertyLabels = {"namespace":"kubernetes_namespace"}
+//   => nil, err
+//
+// (See asset_test.go for assertions of these examples and more.)
+func AssetToExternalAllocation(asset Asset, aggregateBy []string, allocationPropertyLabels map[string]string) (*Allocation, error) {
+	if asset == nil {
+		return nil, fmt.Errorf("asset is nil")
+	}
+
+	// names will collect the slash-separated names accrued by iterating over
+	// aggregateBy and checking the relevant labels.
+	names := []string{}
+
+	// match records whether or not a match was found in the Asset labels,
+	// such that is can genuinely be turned into an external Allocation.
+	match := false
+
+	// props records the relevant Properties to set on the resultant Allocation
+	props := Properties{}
+
+	for _, aggBy := range aggregateBy {
+		// labelName should be derived from the mapping of properties to
+		// label names, unless the aggBy is explicitly a label, in which
+		// case we should pull the label name from the aggBy string.
+		labelName := allocationPropertyLabels[aggBy]
+		if strings.HasPrefix(aggBy, "label:") {
+			labelName = strings.TrimPrefix(aggBy, "label:")
+		}
+
+		if labelName == "" {
+			// No matching label has been defined in the cost-analyzer label config
+			// relating to the given aggregateBy property.
+			names = append(names, UnallocatedSuffix)
+			continue
+		}
+
+		if value := asset.Labels()[labelName]; value != "" {
+			// Valid label value was found for one of the aggregation properties,
+			// so add it to the name.
+			if strings.HasPrefix(aggBy, "label:") {
+				// Use naming convention labelName=labelValue for labels
+				// e.g. aggBy="label:env", value="prod" => "env=prod"
+				names = append(names, fmt.Sprintf("%s=%s", strings.TrimPrefix(aggBy, "label:"), value))
+				match = true
+
+				// Set the corresponding label in props
+				labels, err := props.GetLabels()
+				if err != nil {
+					labels = map[string]string{}
+				}
+				labels[labelName] = value
+				props.SetLabels(labels)
+			} else {
+				names = append(names, value)
+				match = true
+
+				// Set the corresponding property on props
+				switch aggBy {
+				case ClusterProp.String():
+					props.SetCluster(value)
+				case NodeProp.String():
+					props.SetNode(value)
+				case NamespaceProp.String():
+					props.SetNamespace(value)
+				case ControllerKindProp.String():
+					props.SetControllerKind(value)
+				case ControllerProp.String():
+					props.SetController(value)
+				case PodProp.String():
+					props.SetPod(value)
+				case ContainerProp.String():
+					props.SetContainer(value)
+				case ServiceProp.String():
+					// TODO: external allocation: how to do this? multi-service?
+					props.SetServices([]string{value})
+				}
+			}
+		} else {
+			// No value label value was found on the Asset; consider it
+			// unallocated. Note that this case is only truly relevant if at
+			// least one other property matches (e.g. case 3 in the examples)
+			// because if there are no matches, then an error is returned.
+			names = append(names, UnallocatedSuffix)
+		}
+	}
+
+	if !match {
+		return nil, fmt.Errorf("asset does not qualify as an external allocation")
+	}
+
+	names = append(names, ExternalSuffix)
+
+	// TODO: external allocation: efficiency?
+	// TODO: external allocation: resource totals?
+	return &Allocation{
+		Name:         strings.Join(names, "/"),
+		Properties:   props,
+		ExternalCost: asset.TotalCost(),
+		TotalCost:    asset.TotalCost(),
+	}, nil
+}
+
 // key is used to determine uniqueness of an Asset, for instance during Insert
-// to determine if two Assets should be combined. Passing nil props indicates
-// that all available props should be used. Passing empty props indicates that
-// no props should be used (e.g. to aggregate all assets). Passing one or more
-// props will key by only those props.
-func key(a Asset, props []AssetProperty) string {
+// to determine if two Assets should be combined. Passing `nil` `aggregateBy` indicates
+// that all available `AssetProperty` keys should be used. Passing empty `aggregateBy` indicates that
+// no key should be used (e.g. to aggregate all assets). Passing one or more `aggregateBy`
+// values will key by only those values.
+// Valid values of `aggregateBy` elements are strings which are an `AssetProperty`, and strings prefixed
+// with `"label:"`.
+func key(a Asset, aggregateBy []string) (string, error) {
 	keys := []string{}
 
-	if props == nil {
-		props = []AssetProperty{
-			AssetProviderProp,
-			AssetAccountProp,
-			AssetProjectProp,
-			AssetCategoryProp,
-			AssetClusterProp,
-			AssetTypeProp,
-			AssetServiceProp,
-			AssetProviderIDProp,
-			AssetNameProp,
+	if aggregateBy == nil {
+		aggregateBy = []string{
+			string(AssetProviderProp),
+			string(AssetAccountProp),
+			string(AssetProjectProp),
+			string(AssetCategoryProp),
+			string(AssetClusterProp),
+			string(AssetTypeProp),
+			string(AssetServiceProp),
+			string(AssetProviderIDProp),
+			string(AssetNameProp),
 		}
 	}
 
-	for _, prop := range props {
+	for _, s := range aggregateBy {
+		key := ""
 		switch true {
-		case prop == AssetProviderProp && a.Properties().Provider != "":
-			keys = append(keys, a.Properties().Provider)
-		case prop == AssetAccountProp && a.Properties().Account != "":
-			keys = append(keys, a.Properties().Account)
-		case prop == AssetProjectProp && a.Properties().Project != "":
-			keys = append(keys, a.Properties().Project)
-		case prop == AssetClusterProp && a.Properties().Cluster != "":
-			keys = append(keys, a.Properties().Cluster)
-		case prop == AssetCategoryProp && a.Properties().Category != "":
-			keys = append(keys, a.Properties().Category)
-		case prop == AssetTypeProp && a.Type().String() != "":
-			keys = append(keys, a.Type().String())
-		case prop == AssetServiceProp && a.Properties().Service != "":
-			keys = append(keys, a.Properties().Service)
-		case prop == AssetProviderIDProp && a.Properties().ProviderID != "":
-			keys = append(keys, a.Properties().ProviderID)
-		case prop == AssetNameProp && a.Properties().Name != "":
-			keys = append(keys, a.Properties().Name)
+		case s == string(AssetProviderProp):
+			key = a.Properties().Provider
+		case s == string(AssetAccountProp):
+			key = a.Properties().Account
+		case s == string(AssetProjectProp):
+			key = a.Properties().Project
+		case s == string(AssetClusterProp):
+			key = a.Properties().Cluster
+		case s == string(AssetCategoryProp):
+			key = a.Properties().Category
+		case s == string(AssetTypeProp):
+			key = a.Type().String()
+		case s == string(AssetServiceProp):
+			key = a.Properties().Service
+		case s == string(AssetProviderIDProp):
+			key = a.Properties().ProviderID
+		case s == string(AssetNameProp):
+			key = a.Properties().Name
+		case strings.HasPrefix(s, "label:"):
+			if labelKey := strings.TrimPrefix(s, "label:"); labelKey != "" {
+				labelVal := a.Labels()[labelKey]
+				if labelVal == "" {
+					key = "__undefined__"
+				} else {
+					key = fmt.Sprintf("%s=%s", labelKey, labelVal)
+				}
+			} else {
+				// Don't allow aggregating on label ""
+				return "", fmt.Errorf("Attempted to aggregate on invalid key: %s", s)
+			}
+		default:
+			return "", fmt.Errorf("Attempted to aggregate on invalid key: %s", s)
 		}
-	}
 
-	return strings.Join(keys, "/")
+		if key != "" {
+			keys = append(keys, key)
+		} else {
+			keys = append(keys, UndefinedKey)
+		}
+	}
+	return strings.Join(keys, "/"), nil
 }
 
 func toString(a Asset) string {
@@ -114,7 +273,7 @@ type AssetLabels map[string]string
 
 // Clone returns a cloned map of labels
 func (al AssetLabels) Clone() AssetLabels {
-	clone := AssetLabels{}
+	clone := make(AssetLabels, len(al))
 
 	for label, value := range al {
 		clone[label] = value
@@ -424,6 +583,7 @@ type Cloud struct {
 	window     Window
 	adjustment float64
 	Cost       float64
+	Credit     float64 // Credit is a negative value representing dollars credited back to a given line-item
 }
 
 // NewCloud returns a new Cloud Asset
@@ -479,7 +639,7 @@ func (ca *Cloud) SetAdjustment(adj float64) {
 
 // TotalCost returns the Asset's total cost
 func (ca *Cloud) TotalCost() float64 {
-	return ca.Cost + ca.adjustment
+	return ca.Cost + ca.adjustment + ca.Credit
 }
 
 // Start returns the Asset's precise start time within the window
@@ -550,7 +710,7 @@ func (ca *Cloud) Add(a Asset) Asset {
 	any.SetProperties(props)
 	any.SetLabels(labels)
 	any.adjustment = ca.Adjustment() + a.Adjustment()
-	any.Cost = (ca.TotalCost() - ca.Adjustment()) + (a.TotalCost() - a.Adjustment())
+	any.Cost = (ca.TotalCost() - ca.Adjustment() - ca.Credit) + (a.TotalCost() - a.Adjustment() - ca.Credit)
 
 	return any
 }
@@ -581,6 +741,7 @@ func (ca *Cloud) add(that *Cloud) {
 	ca.SetLabels(labels)
 	ca.adjustment += that.adjustment
 	ca.Cost += that.Cost
+	ca.Credit += that.Credit
 }
 
 // Clone returns a cloned instance of the Asset
@@ -593,6 +754,7 @@ func (ca *Cloud) Clone() Asset {
 		window:     ca.window.Clone(),
 		adjustment: ca.adjustment,
 		Cost:       ca.Cost,
+		Credit:     ca.Credit,
 	}
 }
 
@@ -627,6 +789,9 @@ func (ca *Cloud) Equal(a Asset) bool {
 	if ca.Cost != that.Cost {
 		return false
 	}
+	if ca.Credit != that.Credit {
+		return false
+	}
 
 	return true
 }
@@ -642,6 +807,7 @@ func (ca *Cloud) MarshalJSON() ([]byte, error) {
 	jsonEncodeString(buffer, "end", ca.End().Format(timeFmt), ",")
 	jsonEncodeFloat64(buffer, "minutes", ca.Minutes(), ",")
 	jsonEncodeFloat64(buffer, "adjustment", ca.Adjustment(), ",")
+	jsonEncodeFloat64(buffer, "credit", ca.Credit, ",")
 	jsonEncodeFloat64(buffer, "totalCost", ca.TotalCost(), "")
 	buffer.WriteString("}")
 	return buffer.Bytes(), nil
@@ -2304,11 +2470,11 @@ func (sa *SharedAsset) String() string {
 // a window. An AssetSet is mutable, so treat it like a threadsafe map.
 type AssetSet struct {
 	sync.RWMutex
-	assets   map[string]Asset
-	props    []AssetProperty
-	Window   Window
-	Warnings []string
-	Errors   []string
+	aggregateBy []string
+	assets      map[string]Asset
+	Window      Window
+	Warnings    []string
+	Errors      []string
 }
 
 // NewAssetSet instantiates a new AssetSet and, optionally, inserts
@@ -2329,7 +2495,7 @@ func NewAssetSet(start, end time.Time, assets ...Asset) *AssetSet {
 // AggregateBy aggregates the Assets in the AssetSet by the given list of
 // AssetProperties, such that each asset is binned by a key determined by its
 // relevant property values.
-func (as *AssetSet) AggregateBy(props []AssetProperty, opts *AssetAggregationOptions) error {
+func (as *AssetSet) AggregateBy(aggregateBy []string, opts *AssetAggregationOptions) error {
 	if opts == nil {
 		opts = &AssetAggregationOptions{}
 	}
@@ -2342,7 +2508,7 @@ func (as *AssetSet) AggregateBy(props []AssetProperty, opts *AssetAggregationOpt
 	defer as.Unlock()
 
 	aggSet := NewAssetSet(as.Start(), as.End())
-	aggSet.props = props
+	aggSet.aggregateBy = aggregateBy
 
 	// Compute hours of the given AssetSet, and if it ends in the future,
 	// adjust the hours accordingly
@@ -2357,7 +2523,10 @@ func (as *AssetSet) AggregateBy(props []AssetProperty, opts *AssetAggregationOpt
 		sa := NewSharedAsset(name, as.Window.Clone())
 		sa.Cost = hourlyCost * hours
 
-		aggSet.Insert(sa)
+		err := aggSet.Insert(sa)
+		if err != nil {
+			return err
+		}
 	}
 
 	// Delete the Assets that don't pass each filter
@@ -2369,15 +2538,18 @@ func (as *AssetSet) AggregateBy(props []AssetProperty, opts *AssetAggregationOpt
 		}
 	}
 
-	// Insert each asset into the new set, which will be keyed by the props
+	// Insert each asset into the new set, which will be keyed by the `aggregateBy`
 	// on aggSet, resulting in aggregation.
 	for _, asset := range as.assets {
-		aggSet.Insert(asset)
+		err := aggSet.Insert(asset)
+		if err != nil {
+			return err
+		}
 	}
 
 	// Assign the aggregated values back to the original set
 	as.assets = aggSet.assets
-	as.props = props
+	as.aggregateBy = aggregateBy
 
 	return nil
 }
@@ -2392,26 +2564,26 @@ func (as *AssetSet) Clone() *AssetSet {
 	as.RLock()
 	defer as.RUnlock()
 
+	var aggregateBy []string
+	if as.aggregateBy != nil {
+		aggregateBy := []string{}
+		for _, s := range as.aggregateBy {
+			aggregateBy = append(aggregateBy, s)
+		}
+	}
+
 	assets := map[string]Asset{}
 	for k, v := range as.assets {
 		assets[k] = v.Clone()
 	}
 
-	var props []AssetProperty
-	if as.props != nil {
-		props = []AssetProperty{}
-		for _, p := range as.props {
-			props = append(props, p)
-		}
-	}
-
 	s := as.Start()
 	e := as.End()
 
 	return &AssetSet{
-		Window: NewWindow(&s, &e),
-		assets: assets,
-		props:  props,
+		Window:      NewWindow(&s, &e),
+		aggregateBy: aggregateBy,
+		assets:      assets,
 	}
 }
 
@@ -2434,23 +2606,28 @@ func (as *AssetSet) End() time.Time {
 // FindMatch attempts to find a match in the AssetSet for the given Asset on
 // the provided properties and labels. If a match is not found, FindMatch
 // returns nil and a Not Found error.
-func (as *AssetSet) FindMatch(query Asset, props []AssetProperty) (Asset, error) {
+func (as *AssetSet) FindMatch(query Asset, aggregateBy []string) (Asset, error) {
 	as.RLock()
 	defer as.RUnlock()
 
-	matchKey := key(query, props)
+	matchKey, err := key(query, aggregateBy)
+	if err != nil {
+		return nil, err
+	}
 	for _, asset := range as.assets {
-		if key(asset, props) == matchKey {
+		if k, err := key(asset, aggregateBy); err != nil {
+			return nil, err
+		} else if k == matchKey {
 			return asset, nil
 		}
 	}
 
-	return nil, fmt.Errorf("Asset not found to match %s on %v", query, props)
+	return nil, fmt.Errorf("Asset not found to match %s on %v", query, aggregateBy)
 }
 
 // ReconciliationMatch attempts to find an exact match in the AssetSet on
 // (Category, ProviderID). If a match is found, it returns the Asset with the
-// intent to adjuts it. If no match exists, it attempts to find one on only
+// intent to adjust it. If no match exists, it attempts to find one on only
 // (ProviderID). If that match is found, it returns the Asset with the intent
 // to insert the associated Cloud cost.
 func (as *AssetSet) ReconciliationMatch(query Asset) (Asset, bool, error) {
@@ -2458,20 +2635,36 @@ func (as *AssetSet) ReconciliationMatch(query Asset) (Asset, bool, error) {
 	defer as.RUnlock()
 
 	// Full match means matching on (Category, ProviderID)
-	fullMatchProps := []AssetProperty{AssetCategoryProp, AssetProviderIDProp}
-	fullMatchKey := key(query, fullMatchProps)
+	fullMatchProps := []string{string(AssetCategoryProp), string(AssetProviderIDProp)}
+	fullMatchKey, err := key(query, fullMatchProps)
+
+	// This should never happen because we are using enumerated properties,
+	// but the check is here in case that changes
+	if err != nil {
+		return nil, false, err
+	}
 
 	// Partial match means matching only on (ProviderID)
-	providerIDMatchProps := []AssetProperty{AssetProviderIDProp}
-	providerIDMatchKey := key(query, providerIDMatchProps)
+	providerIDMatchProps := []string{string(AssetProviderIDProp)}
+	providerIDMatchKey, err := key(query, providerIDMatchProps)
+
+	// This should never happen because we are using enumerated properties,
+	// but the check is here in case that changes
+	if err != nil {
+		return nil, false, err
+	}
 
 	var providerIDMatch Asset
 	for _, asset := range as.assets {
-		if key(asset, fullMatchProps) == fullMatchKey {
+		if k, err := key(asset, fullMatchProps); err != nil {
+			return nil, false, err
+		} else if k == fullMatchKey {
 			log.DedupedInfof(10, "Asset ETL: Reconciliation[rcnw]: ReconcileRange Match: %s", fullMatchKey)
 			return asset, true, nil
 		}
-		if key(asset, providerIDMatchProps) == providerIDMatchKey {
+		if k, err := key(asset, providerIDMatchProps); err != nil {
+			return nil, false, err
+		} else if k == providerIDMatchKey {
 			// Found a partial match. Save it until after all other options
 			// have been checked for full matches.
 			providerIDMatch = asset
@@ -2512,7 +2705,10 @@ func (as *AssetSet) Insert(asset Asset) error {
 	defer as.Unlock()
 
 	// Determine key into which to Insert the Asset.
-	k := key(asset, as.props)
+	k, err := key(asset, as.aggregateBy)
+	if err != nil {
+		return err
+	}
 
 	// Add the given Asset to the existing entry, if there is one;
 	// otherwise just set directly into assets
@@ -2567,7 +2763,7 @@ func (as *AssetSet) MarshalJSON() ([]byte, error) {
 	return json.Marshal(as.assets)
 }
 
-func (as *AssetSet) Set(asset Asset, props []AssetProperty) {
+func (as *AssetSet) Set(asset Asset, aggregateBy []string) error {
 	if as.IsEmpty() {
 		as.Lock()
 		as.assets = map[string]Asset{}
@@ -2579,7 +2775,12 @@ func (as *AssetSet) Set(asset Asset, props []AssetProperty) {
 
 	// Expand the window to match the AssetSet, then set it
 	asset.ExpandWindow(as.Window)
-	as.assets[key(asset, props)] = asset
+	k, err := key(asset, aggregateBy)
+	if err != nil {
+		return err
+	}
+	as.assets[k] = asset
+	return nil
 }
 
 func (as *AssetSet) Start() time.Time {
@@ -2614,11 +2815,11 @@ func (as *AssetSet) accumulate(that *AssetSet) (*AssetSet, error) {
 	}
 
 	// In the case of an AssetSetRange with empty entries, we may end up with
-	// an incoming as without props, even though we are trying to aggregate
-	// by props. This handles that case, assigning the correct props.
-	if !propsEqual(as.props, that.props) {
-		if len(as.props) == 0 {
-			as.props = that.props
+	// an incoming `as` without an `aggregateBy`, even though we are tring to
+	// aggregate here. This handles that case by assigning the correct `aggregateBy`.
+	if !sameContents(as.aggregateBy, that.aggregateBy) {
+		if len(as.aggregateBy) == 0 {
+			as.aggregateBy = that.aggregateBy
 		}
 	}
 
@@ -2637,7 +2838,7 @@ func (as *AssetSet) accumulate(that *AssetSet) (*AssetSet, error) {
 	}
 
 	acc := NewAssetSet(start, end)
-	acc.props = as.props
+	acc.aggregateBy = as.aggregateBy
 
 	as.RLock()
 	defer as.RUnlock()
@@ -2697,14 +2898,14 @@ type AssetAggregationOptions struct {
 	FilterFuncs       []AssetMatchFunc
 }
 
-func (asr *AssetSetRange) AggregateBy(props []AssetProperty, opts *AssetAggregationOptions) error {
+func (asr *AssetSetRange) AggregateBy(aggregateBy []string, opts *AssetAggregationOptions) error {
 	aggRange := &AssetSetRange{assets: []*AssetSet{}}
 
 	asr.Lock()
 	defer asr.Unlock()
 
 	for _, as := range asr.assets {
-		err := as.AggregateBy(props, opts)
+		err := as.AggregateBy(aggregateBy, opts)
 		if err != nil {
 			return err
 		}
@@ -2811,3 +3012,25 @@ func jsonEncode(buffer *bytes.Buffer, name string, obj interface{}, comma string
 	}
 	buffer.WriteString(comma)
 }
+
+// Returns true if string slices a and b contain all of the same strings, in any order.
+func sameContents(a, b []string) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	for i := range a {
+		if !contains(b, a[i]) {
+			return false
+		}
+	}
+	return true
+}
+
+func contains(slice []string, item string) bool {
+	for _, element := range slice {
+		if element == item {
+			return true
+		}
+	}
+	return false
+}

pkg/kubecost/asset_test.go

@@ -6,6 +6,8 @@ import (
 	"math"
 	"testing"
 	"time"
+
+	util "github.com/kubecost/cost-model/pkg/util"
 )
 
 var start1 = time.Date(2020, time.January, 1, 0, 0, 0, 0, time.UTC)
@@ -19,11 +21,151 @@ var windows = []Window{
 	NewWindow(&start3, &start4),
 }
 
-const delta = 0.00001
 const gb = 1024 * 1024 * 1024
 
-func approx(a, b, delta float64) bool {
-	return math.Abs(a-b) < delta
+// generateAssetSet generates the following topology:
+//
+// | Asset                        | Cost |  Adj |
+// +------------------------------+------+------+
+//   cluster1:
+//     node1:                        6.00   1.00
+//     node2:                        4.00   1.50
+//     node3:                        7.00  -0.50
+//     disk1:                        2.50   0.00
+//     disk2:                        1.50   0.00
+//     clusterManagement1:           3.00   0.00
+// +------------------------------+------+------+
+//   cluster1 subtotal              24.00   2.00
+// +------------------------------+------+------+
+//   cluster2:
+//     node4:                       12.00  -1.00
+//     disk3:                        2.50   0.00
+//     disk4:                        1.50   0.00
+//     clusterManagement2:           0.00   0.00
+// +------------------------------+------+------+
+//   cluster2 subtotal              16.00  -1.00
+// +------------------------------+------+------+
+//   cluster3:
+//     node5:                       17.00   2.00
+// +------------------------------+------+------+
+//   cluster3 subtotal              17.00   2.00
+// +------------------------------+------+------+
+//   total                          57.00   3.00
+// +------------------------------+------+------+
+func generateAssetSet(start time.Time) *AssetSet {
+	end := start.Add(day)
+	window := NewWindow(&start, &end)
+
+	hours := window.Duration().Hours()
+
+	node1 := NewNode("node1", "cluster1", "gcp-node1", *window.Clone().start, *window.Clone().end, window.Clone())
+	node1.CPUCost = 4.0
+	node1.RAMCost = 4.0
+	node1.GPUCost = 2.0
+	node1.Discount = 0.5
+	node1.CPUCoreHours = 2.0 * hours
+	node1.RAMByteHours = 4.0 * gb * hours
+	node1.SetAdjustment(1.0)
+	node1.SetLabels(map[string]string{"test": "test"})
+
+	node2 := NewNode("node2", "cluster1", "gcp-node2", *window.Clone().start, *window.Clone().end, window.Clone())
+	node2.CPUCost = 4.0
+	node2.RAMCost = 4.0
+	node2.GPUCost = 0.0
+	node2.Discount = 0.5
+	node2.CPUCoreHours = 2.0 * hours
+	node2.RAMByteHours = 4.0 * gb * hours
+	node2.SetAdjustment(1.5)
+
+	node3 := NewNode("node3", "cluster1", "gcp-node3", *window.Clone().start, *window.Clone().end, window.Clone())
+	node3.CPUCost = 4.0
+	node3.RAMCost = 4.0
+	node3.GPUCost = 3.0
+	node3.Discount = 0.5
+	node3.CPUCoreHours = 2.0 * hours
+	node3.RAMByteHours = 4.0 * gb * hours
+	node3.SetAdjustment(-0.5)
+
+	node4 := NewNode("node4", "cluster2", "gcp-node4", *window.Clone().start, *window.Clone().end, window.Clone())
+	node4.CPUCost = 10.0
+	node4.RAMCost = 6.0
+	node4.GPUCost = 0.0
+	node4.Discount = 0.25
+	node4.CPUCoreHours = 4.0 * hours
+	node4.RAMByteHours = 12.0 * gb * hours
+	node4.SetAdjustment(-1.0)
+
+	node5 := NewNode("node5", "cluster3", "aws-node5", *window.Clone().start, *window.Clone().end, window.Clone())
+	node5.CPUCost = 10.0
+	node5.RAMCost = 7.0
+	node5.GPUCost = 0.0
+	node5.Discount = 0.0
+	node5.CPUCoreHours = 8.0 * hours
+	node5.RAMByteHours = 24.0 * gb * hours
+	node5.SetAdjustment(2.0)
+
+	disk1 := NewDisk("disk1", "cluster1", "gcp-disk1", *window.Clone().start, *window.Clone().end, window.Clone())
+	disk1.Cost = 2.5
+	disk1.ByteHours = 100 * gb * hours
+
+	disk2 := NewDisk("disk2", "cluster1", "gcp-disk2", *window.Clone().start, *window.Clone().end, window.Clone())
+	disk2.Cost = 1.5
+	disk2.ByteHours = 60 * gb * hours
+
+	disk3 := NewDisk("disk3", "cluster2", "gcp-disk3", *window.Clone().start, *window.Clone().end, window.Clone())
+	disk3.Cost = 2.5
+	disk3.ByteHours = 100 * gb * hours
+
+	disk4 := NewDisk("disk4", "cluster2", "gcp-disk4", *window.Clone().start, *window.Clone().end, window.Clone())
+	disk4.Cost = 1.5
+	disk4.ByteHours = 100 * gb * hours
+
+	cm1 := NewClusterManagement("gcp", "cluster1", window.Clone())
+	cm1.Cost = 3.0
+
+	cm2 := NewClusterManagement("gcp", "cluster2", window.Clone())
+	cm2.Cost = 0.0
+
+	return NewAssetSet(
+		start, end,
+		// cluster 1
+		node1, node2, node3, disk1, disk2, cm1,
+		// cluster 2
+		node4, disk3, disk4, cm2,
+		// cluster 3
+		node5,
+	)
+}
+
+func assertAssetSet(t *testing.T, as *AssetSet, msg string, window Window, exps map[string]float64, err error) {
+	if err != nil {
+		t.Fatalf("AssetSet.AggregateBy[%s]: unexpected error: %s", msg, err)
+	}
+	if as.Length() != len(exps) {
+		t.Fatalf("AssetSet.AggregateBy[%s]: expected set of length %d, actual %d", msg, len(exps), as.Length())
+	}
+	if !as.Window.Equal(window) {
+		t.Fatalf("AssetSet.AggregateBy[%s]: expected window %s, actual %s", msg, window, as.Window)
+	}
+	as.Each(func(key string, a Asset) {
+		if exp, ok := exps[key]; ok {
+			if math.Round(a.TotalCost()*100) != math.Round(exp*100) {
+				t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected total cost %.2f, actual %.2f", msg, key, exp, a.TotalCost())
+			}
+			if !a.Window().Equal(window) {
+				t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected window %s, actual %s", msg, key, window, as.Window)
+			}
+		} else {
+			t.Fatalf("AssetSet.AggregateBy[%s]: unexpected asset: %s", msg, key)
+		}
+	})
+}
+
+func printAssetSet(msg string, as *AssetSet) {
+	fmt.Printf("--- %s ---\n", msg)
+	as.Each(func(key string, a Asset) {
+		fmt.Printf(" > %s: %s\n", key, a)
+	})
 }
 
 func TestAny_Add(t *testing.T) {
@@ -186,7 +328,7 @@ func TestDisk_Add(t *testing.T) {
 	if diskT.Bytes() != 160.0*gb {
 		t.Fatalf("Disk.Add: expected %f; got %f", 160.0*gb, diskT.Bytes())
 	}
-	if !approx(diskT.Local, 0.333333, delta) {
+	if !util.IsApproximately(diskT.Local, 0.333333) {
 		t.Fatalf("Disk.Add: expected %f; got %f", 0.333333, diskT.Local)
 	}
 
@@ -381,7 +523,7 @@ func TestNode_Add(t *testing.T) {
 	nodeT := node1.Add(node2).(*Node)
 
 	// Check that the sums and properties are correct
-	if !approx(nodeT.TotalCost(), 15.0, delta) {
+	if !util.IsApproximately(nodeT.TotalCost(), 15.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 15.0, nodeT.TotalCost())
 	}
 	if nodeT.Adjustment() != 2.6 {
@@ -407,13 +549,13 @@ func TestNode_Add(t *testing.T) {
 	}
 
 	// Check that the original assets are unchanged
-	if !approx(node1.TotalCost(), 10.0, delta) {
+	if !util.IsApproximately(node1.TotalCost(), 10.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 10.0, node1.TotalCost())
 	}
 	if node1.Adjustment() != 1.6 {
 		t.Fatalf("Node.Add: expected %f; got %f", 1.0, node1.Adjustment())
 	}
-	if !approx(node2.TotalCost(), 5.0, delta) {
+	if !util.IsApproximately(node2.TotalCost(), 5.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 5.0, node2.TotalCost())
 	}
 	if node2.Adjustment() != 1.0 {
@@ -471,7 +613,7 @@ func TestNode_Add(t *testing.T) {
 	nodeAT := nodeA1.Add(nodeA2).(*Node)
 
 	// Check that the sums and properties are correct
-	if !approx(nodeAT.TotalCost(), 15.0, delta) {
+	if !util.IsApproximately(nodeAT.TotalCost(), 15.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 15.0, nodeAT.TotalCost())
 	}
 	if nodeAT.Adjustment() != 2.6 {
@@ -497,13 +639,13 @@ func TestNode_Add(t *testing.T) {
 	}
 
 	// Check that the original assets are unchanged
-	if !approx(nodeA1.TotalCost(), 10.0, delta) {
+	if !util.IsApproximately(nodeA1.TotalCost(), 10.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 10.0, nodeA1.TotalCost())
 	}
 	if nodeA1.Adjustment() != 1.6 {
 		t.Fatalf("Node.Add: expected %f; got %f", 1.0, nodeA1.Adjustment())
 	}
-	if !approx(nodeA2.TotalCost(), 5.0, delta) {
+	if !util.IsApproximately(nodeA2.TotalCost(), 5.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 5.0, nodeA2.TotalCost())
 	}
 	if nodeA2.Adjustment() != 1.0 {
@@ -618,6 +760,7 @@ func TestAssetSet_AggregateBy(t *testing.T) {
 	// 1b []AssetProperty=[Type]
 	// 1c []AssetProperty=[Nil]
 	// 1d []AssetProperty=nil
+	// 1e aggregateBy []string=["label:test"]
 
 	// 2  Multi-aggregation
 	// 2a []AssetProperty=[Cluster,Type]
@@ -636,7 +779,7 @@ func TestAssetSet_AggregateBy(t *testing.T) {
 
 	// 1a []AssetProperty=[Cluster]
 	as = generateAssetSet(startYesterday)
-	err = as.AggregateBy([]AssetProperty{AssetClusterProp}, nil)
+	err = as.AggregateBy([]string{string(AssetClusterProp)}, nil)
 	if err != nil {
 		t.Fatalf("AssetSet.AggregateBy: unexpected error: %s", err)
 	}
@@ -648,7 +791,7 @@ func TestAssetSet_AggregateBy(t *testing.T) {
 
 	// 1b []AssetProperty=[Type]
 	as = generateAssetSet(startYesterday)
-	err = as.AggregateBy([]AssetProperty{AssetTypeProp}, nil)
+	err = as.AggregateBy([]string{string(AssetTypeProp)}, nil)
 	if err != nil {
 		t.Fatalf("AssetSet.AggregateBy: unexpected error: %s", err)
 	}
@@ -660,7 +803,7 @@ func TestAssetSet_AggregateBy(t *testing.T) {
 
 	// 1c []AssetProperty=[Nil]
 	as = generateAssetSet(startYesterday)
-	err = as.AggregateBy([]AssetProperty{}, nil)
+	err = as.AggregateBy([]string{}, nil)
 	if err != nil {
 		t.Fatalf("AssetSet.AggregateBy: unexpected error: %s", err)
 	}
@@ -675,24 +818,36 @@ func TestAssetSet_AggregateBy(t *testing.T) {
 		t.Fatalf("AssetSet.AggregateBy: unexpected error: %s", err)
 	}
 	assertAssetSet(t, as, "1d", window, map[string]float64{
-		"Compute/cluster1/Node/Kubernetes/gcp-node1/node1":     7.00,
-		"Compute/cluster1/Node/Kubernetes/gcp-node2/node2":     5.50,
-		"Compute/cluster1/Node/Kubernetes/gcp-node3/node3":     6.50,
-		"Storage/cluster1/Disk/Kubernetes/gcp-disk1/disk1":     2.50,
-		"Storage/cluster1/Disk/Kubernetes/gcp-disk2/disk2":     1.50,
-		"GCP/Management/cluster1/ClusterManagement/Kubernetes": 3.00,
-		"Compute/cluster2/Node/Kubernetes/gcp-node4/node4":     11.00,
-		"Storage/cluster2/Disk/Kubernetes/gcp-disk3/disk3":     2.50,
-		"Storage/cluster2/Disk/Kubernetes/gcp-disk4/disk4":     1.50,
-		"GCP/Management/cluster2/ClusterManagement/Kubernetes": 0.00,
-		"Compute/cluster3/Node/Kubernetes/aws-node5/node5":     19.00,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster1/Node/Kubernetes/gcp-node1/node1":                   7.00,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster1/Node/Kubernetes/gcp-node2/node2":                   5.50,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster1/Node/Kubernetes/gcp-node3/node3":                   6.50,
+		"__undefined__/__undefined__/__undefined__/Storage/cluster1/Disk/Kubernetes/gcp-disk1/disk1":                   2.50,
+		"__undefined__/__undefined__/__undefined__/Storage/cluster1/Disk/Kubernetes/gcp-disk2/disk2":                   1.50,
+		"GCP/__undefined__/__undefined__/Management/cluster1/ClusterManagement/Kubernetes/__undefined__/__undefined__": 3.00,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster2/Node/Kubernetes/gcp-node4/node4":                   11.00,
+		"__undefined__/__undefined__/__undefined__/Storage/cluster2/Disk/Kubernetes/gcp-disk3/disk3":                   2.50,
+		"__undefined__/__undefined__/__undefined__/Storage/cluster2/Disk/Kubernetes/gcp-disk4/disk4":                   1.50,
+		"GCP/__undefined__/__undefined__/Management/cluster2/ClusterManagement/Kubernetes/__undefined__/__undefined__": 0.00,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster3/Node/Kubernetes/aws-node5/node5":                   19.00,
+	}, nil)
+
+	// 1e aggregateBy []string=["label:test"]
+	as = generateAssetSet(startYesterday)
+	err = as.AggregateBy([]string{"label:test"}, nil)
+	if err != nil {
+		t.Fatalf("AssetSet.AggregateBy: unexpected error: %s", err)
+	}
+	fmt.Println(as.assets)
+	assertAssetSet(t, as, "1e", window, map[string]float64{
+		"__undefined__": 53.00,
+		"test=test":     7.00,
 	}, nil)
 
 	// 2  Multi-aggregation
 
 	// 2a []AssetProperty=[Cluster,Type]
 	as = generateAssetSet(startYesterday)
-	err = as.AggregateBy([]AssetProperty{AssetClusterProp, AssetTypeProp}, nil)
+	err = as.AggregateBy([]string{string(AssetClusterProp), string(AssetTypeProp)}, nil)
 	if err != nil {
 		t.Fatalf("AssetSet.AggregateBy: unexpected error: %s", err)
 	}
@@ -710,7 +865,7 @@ func TestAssetSet_AggregateBy(t *testing.T) {
 
 	// 3a Shared hourly cost > 0.0
 	as = generateAssetSet(startYesterday)
-	err = as.AggregateBy([]AssetProperty{AssetTypeProp}, &AssetAggregationOptions{
+	err = as.AggregateBy([]string{string(AssetTypeProp)}, &AssetAggregationOptions{
 		SharedHourlyCosts: map[string]float64{"shared1": 0.5},
 	})
 	if err != nil {
@@ -737,7 +892,7 @@ func TestAssetSet_FindMatch(t *testing.T) {
 	// Assert success of a simple match of Type and ProviderID
 	as = generateAssetSet(startYesterday)
 	query = NewNode("", "", "gcp-node3", s, e, w)
-	match, err = as.FindMatch(query, []AssetProperty{AssetTypeProp, AssetProviderIDProp})
+	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)})
 	if err != nil {
 		t.Fatalf("AssetSet.FindMatch: unexpected error: %s", err)
 	}
@@ -745,7 +900,7 @@ func TestAssetSet_FindMatch(t *testing.T) {
 	// Assert error of a simple non-match of Type and ProviderID
 	as = generateAssetSet(startYesterday)
 	query = NewNode("", "", "aws-node3", s, e, w)
-	match, err = as.FindMatch(query, []AssetProperty{AssetTypeProp, AssetProviderIDProp})
+	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)})
 	if err == nil {
 		t.Fatalf("AssetSet.FindMatch: expected error (no match); found %s", match)
 	}
@@ -753,7 +908,7 @@ func TestAssetSet_FindMatch(t *testing.T) {
 	// Assert error of matching ProviderID, but not Type
 	as = generateAssetSet(startYesterday)
 	query = NewCloud(ComputeCategory, "gcp-node3", s, e, w)
-	match, err = as.FindMatch(query, []AssetProperty{AssetTypeProp, AssetProviderIDProp})
+	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)})
 	if err == nil {
 		t.Fatalf("AssetSet.FindMatch: expected error (no match); found %s", match)
 	}
@@ -784,17 +939,17 @@ func TestAssetSetRange_Accumulate(t *testing.T) {
 		t.Fatalf("AssetSetRange.AggregateBy: unexpected error: %s", err)
 	}
 	assertAssetSet(t, as, "1a", window, map[string]float64{
-		"Compute/cluster1/Node/Kubernetes/gcp-node1/node1":     21.00,
-		"Compute/cluster1/Node/Kubernetes/gcp-node2/node2":     16.50,
-		"Compute/cluster1/Node/Kubernetes/gcp-node3/node3":     19.50,
-		"Storage/cluster1/Disk/Kubernetes/gcp-disk1/disk1":     7.50,
-		"Storage/cluster1/Disk/Kubernetes/gcp-disk2/disk2":     4.50,
-		"GCP/Management/cluster1/ClusterManagement/Kubernetes": 9.00,
-		"Compute/cluster2/Node/Kubernetes/gcp-node4/node4":     33.00,
-		"Storage/cluster2/Disk/Kubernetes/gcp-disk3/disk3":     7.50,
-		"Storage/cluster2/Disk/Kubernetes/gcp-disk4/disk4":     4.50,
-		"GCP/Management/cluster2/ClusterManagement/Kubernetes": 0.00,
-		"Compute/cluster3/Node/Kubernetes/aws-node5/node5":     57.00,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster1/Node/Kubernetes/gcp-node1/node1":                   21.00,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster1/Node/Kubernetes/gcp-node2/node2":                   16.50,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster1/Node/Kubernetes/gcp-node3/node3":                   19.50,
+		"__undefined__/__undefined__/__undefined__/Storage/cluster1/Disk/Kubernetes/gcp-disk1/disk1":                   7.50,
+		"__undefined__/__undefined__/__undefined__/Storage/cluster1/Disk/Kubernetes/gcp-disk2/disk2":                   4.50,
+		"GCP/__undefined__/__undefined__/Management/cluster1/ClusterManagement/Kubernetes/__undefined__/__undefined__": 9.00,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster2/Node/Kubernetes/gcp-node4/node4":                   33.00,
+		"__undefined__/__undefined__/__undefined__/Storage/cluster2/Disk/Kubernetes/gcp-disk3/disk3":                   7.50,
+		"__undefined__/__undefined__/__undefined__/Storage/cluster2/Disk/Kubernetes/gcp-disk4/disk4":                   4.50,
+		"GCP/__undefined__/__undefined__/Management/cluster2/ClusterManagement/Kubernetes/__undefined__/__undefined__": 0.00,
+		"__undefined__/__undefined__/__undefined__/Compute/cluster3/Node/Kubernetes/aws-node5/node5":                   57.00,
 	}, nil)
 
 	asr = NewAssetSetRange(
@@ -802,7 +957,7 @@ func TestAssetSetRange_Accumulate(t *testing.T) {
 		generateAssetSet(startD1),
 		generateAssetSet(startD2),
 	)
-	err = asr.AggregateBy([]AssetProperty{}, nil)
+	err = asr.AggregateBy([]string{}, nil)
 	as, err = asr.Accumulate()
 	if err != nil {
 		t.Fatalf("AssetSetRange.AggregateBy: unexpected error: %s", err)
@@ -816,7 +971,7 @@ func TestAssetSetRange_Accumulate(t *testing.T) {
 		generateAssetSet(startD1),
 		generateAssetSet(startD2),
 	)
-	err = asr.AggregateBy([]AssetProperty{AssetTypeProp}, nil)
+	err = asr.AggregateBy([]string{string(AssetTypeProp)}, nil)
 	if err != nil {
 		t.Fatalf("AssetSetRange.AggregateBy: unexpected error: %s", err)
 	}
@@ -835,7 +990,7 @@ func TestAssetSetRange_Accumulate(t *testing.T) {
 		generateAssetSet(startD1),
 		generateAssetSet(startD2),
 	)
-	err = asr.AggregateBy([]AssetProperty{AssetClusterProp}, nil)
+	err = asr.AggregateBy([]string{string(AssetClusterProp)}, nil)
 	if err != nil {
 		t.Fatalf("AssetSetRange.AggregateBy: unexpected error: %s", err)
 	}
@@ -856,7 +1011,7 @@ func TestAssetSetRange_Accumulate(t *testing.T) {
 		generateAssetSet(startD1),
 		generateAssetSet(startD2),
 	)
-	err = asr.AggregateBy([]AssetProperty{AssetTypeProp}, nil)
+	err = asr.AggregateBy([]string{string(AssetTypeProp)}, nil)
 	as, err = asr.Accumulate()
 	if err != nil {
 		t.Fatalf("AssetSetRange.AggregateBy: unexpected error: %s", err)
@@ -868,146 +1023,245 @@ func TestAssetSetRange_Accumulate(t *testing.T) {
 	}, nil)
 }
 
-func assertAssetSet(t *testing.T, as *AssetSet, msg string, window Window, exps map[string]float64, err error) {
+func TestAssetToExternalAllocation(t *testing.T) {
+	var asset Asset
+	var alloc *Allocation
+	var err error
+
+	// default allocationPropertyLabels, which should be compatible with result
+	// of LabelConfig.AllocationPropertyLabels()
+	apls := map[string]string{"namespace": "kubernetes_namespace"}
+
+	alloc, err = AssetToExternalAllocation(asset, []string{"namespace"}, apls)
+	if err == nil {
+		t.Fatalf("expected error due to nil asset")
+	}
+
+	// Consider this Asset:
+	//   Cloud {
+	// 	   TotalCost: 10.00,
+	// 	   Labels{
+	//       "kubernetes_namespace":"monitoring",
+	// 	     "env":"prod"
+	// 	   }
+	//   }
+	cloud := NewCloud(ComputeCategory, "abc123", start1, start2, windows[0])
+	cloud.SetLabels(map[string]string{
+		"kubernetes_namespace": "monitoring",
+		"env":                  "prod",
+	})
+	cloud.Cost = 10.00
+	asset = cloud
+
+	// Providing nil params with a non-nil Asset should not panic, but it
+	// should return an error in both cases (no matching is possible).
+	alloc, err = AssetToExternalAllocation(asset, []string{"namespace"}, nil)
+	if err == nil {
+		t.Fatalf("expected error due to nil allocationPropertyLabels")
+	}
+	alloc, err = AssetToExternalAllocation(asset, nil, apls)
+	if err == nil {
+		t.Fatalf("expected error due to nil aggregateBy")
+	}
+
+	// Given the following parameters, we expect to return:
+	//
+	//   1) single-prop full match
+	//   aggregateBy = ["namespace"]
+	//   allocationPropertyLabels = {"namespace":"kubernetes_namespace"}
+	//   => Allocation{Name: "monitoring", ExternalCost: 10.00, TotalCost: 10.00}, nil
+	//
+	//   2) multi-prop full match
+	//   aggregateBy = ["namespace", "label:env"]
+	//   allocationPropertyLabels = {"namespace":"kubernetes_namespace"}
+	//   => Allocation{Name: "monitoring/env=prod", ExternalCost: 10.00, TotalCost: 10.00}, nil
+	//
+	//   3) multi-prop partial match
+	//   aggregateBy = ["namespace", "label:foo"]
+	//   allocationPropertyLabels = {"namespace":"kubernetes_namespace"}
+	//   => Allocation{Name: "monitoring/__unallocated__", ExternalCost: 10.00, TotalCost: 10.00}, nil
+	//
+	//   4) no match
+	//   aggregateBy = ["cluster"]
+	//   allocationPropertyLabels = {"namespace":"kubernetes_namespace"}
+	//   => nil, err
+
+	// 1) single-prop full match
+	alloc, err = AssetToExternalAllocation(asset, []string{"namespace"}, apls)
 	if err != nil {
-		t.Fatalf("AssetSet.AggregateBy[%s]: unexpected error: %s", msg, err)
+		t.Fatalf("unexpected error: %s", err)
 	}
-	if as.Length() != len(exps) {
-		t.Fatalf("AssetSet.AggregateBy[%s]: expected set of length %d, actual %d", msg, len(exps), as.Length())
+	if alloc.Name != "monitoring/__external__" {
+		t.Fatalf("expected external allocation with name '%s'; got '%s'", "monitoring/__external__", alloc.Name)
 	}
-	if !as.Window.Equal(window) {
-		t.Fatalf("AssetSet.AggregateBy[%s]: expected window %s, actual %s", msg, window, as.Window)
+	if ns, err := alloc.Properties.GetNamespace(); err != nil || ns != "monitoring" {
+		t.Fatalf("expected external allocation with Properties.Namespace '%s'; got '%s' (%s)", "monitoring", ns, err)
+	}
+	if alloc.ExternalCost != 10.00 {
+		t.Fatalf("expected external allocation with ExternalCost %f; got %f", 10.00, alloc.ExternalCost)
+	}
+	if alloc.TotalCost != 10.00 {
+		t.Fatalf("expected external allocation with TotalCost %f; got %f", 10.00, alloc.TotalCost)
 	}
-	as.Each(func(key string, a Asset) {
-		if exp, ok := exps[key]; ok {
-			if math.Round(a.TotalCost()*100) != math.Round(exp*100) {
-				t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected total cost %.2f, actual %.2f", msg, key, exp, a.TotalCost())
-			}
-			if !a.Window().Equal(window) {
-				t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected window %s, actual %s", msg, key, window, as.Window)
-			}
-		} else {
-			t.Fatalf("AssetSet.AggregateBy[%s]: unexpected asset: %s", msg, key)
-		}
-	})
-}
-
-// generateAssetSet generates the following topology:
-//
-// | Asset                        | Cost |  Adj |
-// +------------------------------+------+------+
-//   cluster1:
-//     node1:                        6.00   1.00
-//     node2:                        4.00   1.50
-//     node3:                        7.00  -0.50
-//     disk1:                        2.50   0.00
-//     disk2:                        1.50   0.00
-//     clusterManagement1:           3.00   0.00
-// +------------------------------+------+------+
-//   cluster1 subtotal              24.00   2.00
-// +------------------------------+------+------+
-//   cluster2:
-//     node4:                       12.00  -1.00
-//     disk3:                        2.50   0.00
-//     disk4:                        1.50   0.00
-//     clusterManagement2:           0.00   0.00
-// +------------------------------+------+------+
-//   cluster2 subtotal              16.00  -1.00
-// +------------------------------+------+------+
-//   cluster3:
-//     node5:                       17.00   2.00
-// +------------------------------+------+------+
-//   cluster3 subtotal              17.00   2.00
-// +------------------------------+------+------+
-//   total                          57.00   3.00
-// +------------------------------+------+------+
-func generateAssetSet(start time.Time) *AssetSet {
-	end := start.Add(day)
-	window := NewWindow(&start, &end)
-
-	hours := window.Duration().Hours()
-
-	node1 := NewNode("node1", "cluster1", "gcp-node1", *window.Clone().start, *window.Clone().end, window.Clone())
-	node1.CPUCost = 4.0
-	node1.RAMCost = 4.0
-	node1.GPUCost = 2.0
-	node1.Discount = 0.5
-	node1.CPUCoreHours = 2.0 * hours
-	node1.RAMByteHours = 4.0 * gb * hours
-	node1.SetAdjustment(1.0)
-
-	node2 := NewNode("node2", "cluster1", "gcp-node2", *window.Clone().start, *window.Clone().end, window.Clone())
-	node2.CPUCost = 4.0
-	node2.RAMCost = 4.0
-	node2.GPUCost = 0.0
-	node2.Discount = 0.5
-	node2.CPUCoreHours = 2.0 * hours
-	node2.RAMByteHours = 4.0 * gb * hours
-	node2.SetAdjustment(1.5)
-
-	node3 := NewNode("node3", "cluster1", "gcp-node3", *window.Clone().start, *window.Clone().end, window.Clone())
-	node3.CPUCost = 4.0
-	node3.RAMCost = 4.0
-	node3.GPUCost = 3.0
-	node3.Discount = 0.5
-	node3.CPUCoreHours = 2.0 * hours
-	node3.RAMByteHours = 4.0 * gb * hours
-	node3.SetAdjustment(-0.5)
-
-	node4 := NewNode("node4", "cluster2", "gcp-node4", *window.Clone().start, *window.Clone().end, window.Clone())
-	node4.CPUCost = 10.0
-	node4.RAMCost = 6.0
-	node4.GPUCost = 0.0
-	node4.Discount = 0.25
-	node4.CPUCoreHours = 4.0 * hours
-	node4.RAMByteHours = 12.0 * gb * hours
-	node4.SetAdjustment(-1.0)
-
-	node5 := NewNode("node5", "cluster3", "aws-node5", *window.Clone().start, *window.Clone().end, window.Clone())
-	node5.CPUCost = 10.0
-	node5.RAMCost = 7.0
-	node5.GPUCost = 0.0
-	node5.Discount = 0.0
-	node5.CPUCoreHours = 8.0 * hours
-	node5.RAMByteHours = 24.0 * gb * hours
-	node5.SetAdjustment(2.0)
-
-	disk1 := NewDisk("disk1", "cluster1", "gcp-disk1", *window.Clone().start, *window.Clone().end, window.Clone())
-	disk1.Cost = 2.5
-	disk1.ByteHours = 100 * gb * hours
-
-	disk2 := NewDisk("disk2", "cluster1", "gcp-disk2", *window.Clone().start, *window.Clone().end, window.Clone())
-	disk2.Cost = 1.5
-	disk2.ByteHours = 60 * gb * hours
-
-	disk3 := NewDisk("disk3", "cluster2", "gcp-disk3", *window.Clone().start, *window.Clone().end, window.Clone())
-	disk3.Cost = 2.5
-	disk3.ByteHours = 100 * gb * hours
-
-	disk4 := NewDisk("disk4", "cluster2", "gcp-disk4", *window.Clone().start, *window.Clone().end, window.Clone())
-	disk4.Cost = 1.5
-	disk4.ByteHours = 100 * gb * hours
 
-	cm1 := NewClusterManagement("gcp", "cluster1", window.Clone())
-	cm1.Cost = 3.0
+	// 2) multi-prop full match
+	alloc, err = AssetToExternalAllocation(asset, []string{"namespace", "label:env"}, apls)
+	if err != nil {
+		t.Fatalf("unexpected error: %s", err)
+	}
+	if alloc.Name != "monitoring/env=prod/__external__" {
+		t.Fatalf("expected external allocation with name '%s'; got '%s'", "monitoring/env=prod/__external__", alloc.Name)
+	}
+	if ns, err := alloc.Properties.GetNamespace(); err != nil || ns != "monitoring" {
+		t.Fatalf("expected external allocation with Properties.Namespace '%s'; got '%s' (%s)", "monitoring", ns, err)
+	}
+	if ls, err := alloc.Properties.GetLabels(); err != nil || ls["env"] != "prod" {
+		t.Fatalf("expected external allocation with Properties.Labels[\"env\"] '%s'; got '%s' (%s)", "prod", ls["env"], err)
+	}
+	if alloc.ExternalCost != 10.00 {
+		t.Fatalf("expected external allocation with ExternalCost %f; got %f", 10.00, alloc.ExternalCost)
+	}
+	if alloc.TotalCost != 10.00 {
+		t.Fatalf("expected external allocation with TotalCost %f; got %f", 10.00, alloc.TotalCost)
+	}
 
-	cm2 := NewClusterManagement("gcp", "cluster2", window.Clone())
-	cm2.Cost = 0.0
+	// 3) multi-prop partial match
+	alloc, err = AssetToExternalAllocation(asset, []string{"namespace", "label:foo"}, apls)
+	if err != nil {
+		t.Fatalf("unexpected error: %s", err)
+	}
+	if alloc.Name != "monitoring/__unallocated__/__external__" {
+		t.Fatalf("expected external allocation with name '%s'; got '%s'", "monitoring/__unallocated__/__external__", alloc.Name)
+	}
+	if ns, err := alloc.Properties.GetNamespace(); err != nil || ns != "monitoring" {
+		t.Fatalf("expected external allocation with Properties.Namespace '%s'; got '%s' (%s)", "monitoring", ns, err)
+	}
+	if alloc.ExternalCost != 10.00 {
+		t.Fatalf("expected external allocation with ExternalCost %f; got %f", 10.00, alloc.ExternalCost)
+	}
+	if alloc.TotalCost != 10.00 {
+		t.Fatalf("expected external allocation with TotalCost %f; got %f", 10.00, alloc.TotalCost)
+	}
 
-	return NewAssetSet(
-		start, end,
-		// cluster 1
-		node1, node2, node3, disk1, disk2, cm1,
-		// cluster 2
-		node4, disk3, disk4, cm2,
-		// cluster 3
-		node5,
-	)
+	// 3) no match
+	alloc, err = AssetToExternalAllocation(asset, []string{"cluster"}, apls)
+	if err == nil {
+		t.Fatalf("expected 'no match' error")
+	}
 }
 
-func printAssetSet(msg string, as *AssetSet) {
-	fmt.Printf("--- %s ---\n", msg)
-	as.Each(func(key string, a Asset) {
-		fmt.Printf(" > %s: %s\n", key, a)
-	})
-}
+// TODO merge conflict had this:
+
+// as.Each(func(key string, a Asset) {
+// 	if exp, ok := exps[key]; ok {
+// 		if math.Round(a.TotalCost()*100) != math.Round(exp*100) {
+// 			t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected total cost %.2f, actual %.2f", msg, key, exp, a.TotalCost())
+// 		}
+// 		if !a.Window().Equal(window) {
+// 			t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected window %s, actual %s", msg, key, window, as.Window)
+// 		}
+// 	} else {
+// 		t.Fatalf("AssetSet.AggregateBy[%s]: unexpected asset: %s", msg, key)
+// 	}
+// })
+// }
+
+// // generateAssetSet generates the following topology:
+// //
+// // | Asset                        | Cost |  Adj |
+// // +------------------------------+------+------+
+// //   cluster1:
+// //     node1:                        6.00   1.00
+// //     node2:                        4.00   1.50
+// //     node3:                        7.00  -0.50
+// //     disk1:                        2.50   0.00
+// //     disk2:                        1.50   0.00
+// //     clusterManagement1:           3.00   0.00
+// // +------------------------------+------+------+
+// //   cluster1 subtotal              24.00   2.00
+// // +------------------------------+------+------+
+// //   cluster2:
+// //     node4:                       12.00  -1.00
+// //     disk3:                        2.50   0.00
+// //     disk4:                        1.50   0.00
+// //     clusterManagement2:           0.00   0.00
+// // +------------------------------+------+------+
+// //   cluster2 subtotal              16.00  -1.00
+// // +------------------------------+------+------+
+// //   cluster3:
+// //     node5:                       17.00   2.00
+// // +------------------------------+------+------+
+// //   cluster3 subtotal              17.00   2.00
+// // +------------------------------+------+------+
+// //   total                          57.00   3.00
+// // +------------------------------+------+------+
+// func generateAssetSet(start time.Time) *AssetSet {
+// end := start.Add(day)
+// window := NewWindow(&start, &end)
+
+// hours := window.Duration().Hours()
+
+// node1 := NewNode("node1", "cluster1", "gcp-node1", *window.Clone().start, *window.Clone().end, window.Clone())
+// node1.CPUCost = 4.0
+// node1.RAMCost = 4.0
+// node1.GPUCost = 2.0
+// node1.Discount = 0.5
+// node1.CPUCoreHours = 2.0 * hours
+// node1.RAMByteHours = 4.0 * gb * hours
+// node1.SetAdjustment(1.0)
+// node1.SetLabels(map[string]string{"test": "test"})
+
+// node2 := NewNode("node2", "cluster1", "gcp-node2", *window.Clone().start, *window.Clone().end, window.Clone())
+// node2.CPUCost = 4.0
+// node2.RAMCost = 4.0
+// node2.GPUCost = 0.0
+// node2.Discount = 0.5
+// node2.CPUCoreHours = 2.0 * hours
+// node2.RAMByteHours = 4.0 * gb * hours
+// node2.SetAdjustment(1.5)
+
+// node3 := NewNode("node3", "cluster1", "gcp-node3", *window.Clone().start, *window.Clone().end, window.Clone())
+// node3.CPUCost = 4.0
+// node3.RAMCost = 4.0
+// node3.GPUCost = 3.0
+// node3.Discount = 0.5
+// node3.CPUCoreHours = 2.0 * hours
+// node3.RAMByteHours = 4.0 * gb * hours
+// node3.SetAdjustment(-0.5)
+
+// node4 := NewNode("node4", "cluster2", "gcp-node4", *window.Clone().start, *window.Clone().end, window.Clone())
+// node4.CPUCost = 10.0
+// node4.RAMCost = 6.0
+// node4.GPUCost = 0.0
+// node4.Discount = 0.25
+// node4.CPUCoreHours = 4.0 * hours
+// node4.RAMByteHours = 12.0 * gb * hours
+// node4.SetAdjustment(-1.0)
+
+// node5 := NewNode("node5", "cluster3", "aws-node5", *window.Clone().start, *window.Clone().end, window.Clone())
+// node5.CPUCost = 10.0
+// node5.RAMCost = 7.0
+// node5.GPUCost = 0.0
+// node5.Discount = 0.0
+// node5.CPUCoreHours = 8.0 * hours
+// node5.RAMByteHours = 24.0 * gb * hours
+// node5.SetAdjustment(2.0)
+
+// disk1 := NewDisk("disk1", "cluster1", "gcp-disk1", *window.Clone().start, *window.Clone().end, window.Clone())
+// disk1.Cost = 2.5
+// disk1.ByteHours = 100 * gb * hours
+
+// disk2 := NewDisk("disk2", "cluster1", "gcp-disk2", *window.Clone().start, *window.Clone().end, window.Clone())
+// disk2.Cost = 1.5
+// disk2.ByteHours = 60 * gb * hours
+
+// disk3 := NewDisk("disk3", "cluster2", "gcp-disk3", *window.Clone().start, *window.Clone().end, window.Clone())
+// disk3.Cost = 2.5
+// disk3.ByteHours = 100 * gb * hours
+
+// disk4 := NewDisk("disk4", "cluster2", "gcp-disk4", *window.Clone().start, *window.Clone().end, window.Clone())
+// disk4.Cost = 1.5
+// disk4.ByteHours = 100 * gb * hours
+
+// cm1 := NewClusterManagement("gcp", "cluster1", window.Clone())
+// cm1.Cost = 3.0

pkg/kubecost/assetprops.go

@@ -55,7 +55,7 @@ func ParseAssetProperty(text string) (AssetProperty, error) {
 		return AssetProjectProp, nil
 	case "provider":
 		return AssetProviderProp, nil
-	case "providerID":
+	case "providerid":
 		return AssetProviderIDProp, nil
 	case "service":
 		return AssetServiceProp, nil

pkg/kubecost/bingen.go

@@ -21,4 +21,4 @@ package kubecost
 // @bingen:generate:AllocationSet
 // @bingen:generate:AllocationSetRange
 
-//go:generate bingen -package=kubecost -version=4 -buffer=github.com/kubecost/cost-model/pkg/util
+//go:generate bingen -package=kubecost -version=5 -buffer=github.com/kubecost/cost-model/pkg/util

pkg/kubecost/config.go

@@ -0,0 +1,198 @@
+package kubecost
+
+import (
+	"fmt"
+	"strings"
+)
+
+// LabelConfig is a port of type AnalyzerConfig. We need to be more thoughtful
+// about design at some point, but this is a stop-gap measure, which is required
+// because AnalyzerConfig is defined in package main, so it can't be imported.
+type LabelConfig struct {
+	DepartmentLabel          string `json:"department_label"`
+	EnvironmentLabel         string `json:"environment_label"`
+	OwnerLabel               string `json:"owner_label"`
+	ProductLabel             string `json:"product_label"`
+	TeamLabel                string `json:"team_label"`
+	ClusterExternalLabel     string `json:"cluster_external_label"`
+	NamespaceExternalLabel   string `json:"namespace_external_label"`
+	ControllerExternalLabel  string `json:"controller_external_label"`
+	DaemonsetExternalLabel   string `json:"daemonset_external_label"`
+	DeploymentExternalLabel  string `json:"deployment_external_label"`
+	StatefulsetExternalLabel string `json:"statefulset_external_label"`
+	ServiceExternalLabel     string `json:"service_external_label"`
+	PodExternalLabel         string `json:"pod_external_label"`
+	DepartmentExternalLabel  string `json:"department_external_label"`
+	EnvironmentExternalLabel string `json:"environment_external_label"`
+	OwnerExternalLabel       string `json:"owner_external_label"`
+	ProductExternalLabel     string `json:"product_external_label"`
+	TeamExternalLabel        string `json:"team_external_label"`
+}
+
+// Map returns the config as a basic string map, with default values if not set
+func (lc *LabelConfig) Map() map[string]string {
+	// Start with default values
+	m := map[string]string{
+		"department_label":           "department",
+		"environment_label":          "env",
+		"owner_label":                "owner",
+		"product_label":              "app",
+		"team_label":                 "team",
+		"cluster_external_label":     "kubernetes_cluster",
+		"namespace_external_label":   "kubernetes_namespace",
+		"controller_external_label":  "kubernetes_controller",
+		"daemonset_external_label":   "kubernetes_daemonset",
+		"deployment_external_label":  "kubernetes_deployment",
+		"statefulset_external_label": "kubernetes_statefulset",
+		"service_external_label":     "kubernetes_service",
+		"pod_external_label":         "kubernetes_pod",
+		"department_external_label":  "kubernetes_label_department",
+		"environment_external_label": "kubernetes_label_env",
+		"owner_external_label":       "kubernetes_label_owner",
+		"product_external_label":     "kubernetes_label_app",
+		"team_external_label":        "kubernetes_label_team",
+	}
+
+	if lc == nil {
+		return m
+	}
+
+	if lc.DepartmentLabel != "" {
+		m["department_label"] = lc.DepartmentLabel
+	}
+
+	if lc.EnvironmentLabel != "" {
+		m["environment_label"] = lc.EnvironmentLabel
+	}
+
+	if lc.OwnerLabel != "" {
+		m["owner_label"] = lc.OwnerLabel
+	}
+
+	if lc.ProductLabel != "" {
+		m["product_label"] = lc.ProductLabel
+	}
+
+	if lc.TeamLabel != "" {
+		m["team_label"] = lc.TeamLabel
+	}
+
+	if lc.ClusterExternalLabel != "" {
+		m["cluster_external_label"] = lc.ClusterExternalLabel
+	}
+
+	if lc.NamespaceExternalLabel != "" {
+		m["namespace_external_label"] = lc.NamespaceExternalLabel
+	}
+
+	if lc.ControllerExternalLabel != "" {
+		m["controller_external_label"] = lc.ControllerExternalLabel
+	}
+
+	if lc.DaemonsetExternalLabel != "" {
+		m["daemonset_external_label"] = lc.DaemonsetExternalLabel
+	}
+
+	if lc.DeploymentExternalLabel != "" {
+		m["deployment_external_label"] = lc.DeploymentExternalLabel
+	}
+
+	if lc.StatefulsetExternalLabel != "" {
+		m["statefulset_external_label"] = lc.StatefulsetExternalLabel
+	}
+
+	if lc.ServiceExternalLabel != "" {
+		m["service_external_label"] = lc.ServiceExternalLabel
+	}
+
+	if lc.PodExternalLabel != "" {
+		m["pod_external_label"] = lc.PodExternalLabel
+	}
+
+	if lc.DepartmentExternalLabel != "" {
+		m["department_external_label"] = lc.DepartmentExternalLabel
+	} else if lc.DepartmentLabel != "" {
+		m["department_external_label"] = "kubernetes_label_" + lc.DepartmentLabel
+	}
+
+	if lc.EnvironmentExternalLabel != "" {
+		m["environment_external_label"] = lc.EnvironmentExternalLabel
+	} else if lc.EnvironmentLabel != "" {
+		m["environment_external_label"] = "kubernetes_label_" + lc.EnvironmentLabel
+	}
+
+	if lc.OwnerExternalLabel != "" {
+		m["owner_external_label"] = lc.OwnerExternalLabel
+	} else if lc.OwnerLabel != "" {
+		m["owner_external_label"] = "kubernetes_label_" + lc.OwnerLabel
+	}
+
+	if lc.ProductExternalLabel != "" {
+		m["product_external_label"] = lc.ProductExternalLabel
+	} else if lc.ProductLabel != "" {
+		m["product_external_label"] = "kubernetes_label_" + lc.ProductLabel
+	}
+
+	if lc.TeamExternalLabel != "" {
+		m["team_external_label"] = lc.TeamExternalLabel
+	} else if lc.TeamLabel != "" {
+		m["team_external_label"] = "kubernetes_label_" + lc.TeamLabel
+	}
+
+	return m
+}
+
+// ExternalQueryLabels returns the config's external labels as a mapping of the
+// query column to the label it should set;
+// e.g. if the config stores "statefulset_external_label": "kubernetes_sset",
+//      then this would return "kubernetes_sset": "statefulset"
+func (lc *LabelConfig) ExternalQueryLabels() map[string]string {
+	queryLabels := map[string]string{}
+
+	for label, query := range lc.Map() {
+		if strings.HasSuffix(label, "external_label") && query != "" {
+			queryLabels[query] = label
+		}
+	}
+
+	return queryLabels
+}
+
+// AllocationPropertyLabels returns the config's external resource labels
+// as a mapping from k8s resource-to-label name.
+// e.g. if the config stores "statefulset_external_label": "kubernetes_sset",
+//      then this would return "statefulset": "kubernetes_sset"
+// e.g. if the config stores "owner_label": "product_owner",
+//      then this would return "label:product_owner": "product_owner"
+func (lc *LabelConfig) AllocationPropertyLabels() map[string]string {
+	labels := map[string]string{}
+
+	for labelKind, labelName := range lc.Map() {
+		if labelName != "" {
+			switch labelKind {
+			case "namespace_external_label":
+				labels["namespace"] = labelName
+			case "cluster_external_label":
+				labels["cluster"] = labelName
+			case "controller_external_label":
+				labels["controller"] = labelName
+			case "product_external_label":
+				labels["product"] = labelName
+			case "service_external_label":
+				labels["service"] = labelName
+			case "deployment_external_label":
+				labels["deployment"] = labelName
+			case "statefulset_external_label":
+				labels["statefulset"] = labelName
+			case "daemonset_external_label":
+				labels["daemonset"] = labelName
+			case "pod_external_label":
+				labels["pod"] = labelName
+			default:
+				labels[fmt.Sprintf("label:%s", labelName)] = labelName
+			}
+		}
+	}
+
+	return labels
+}

pkg/kubecost/config_test.go

@@ -0,0 +1,94 @@
+package kubecost
+
+import "testing"
+
+func TestLabelConfig_Map(t *testing.T) {
+	var m map[string]string
+	var lc *LabelConfig
+
+	m = lc.Map()
+	if len(m) != 18 {
+		t.Fatalf("Map: expected length %d; got length %d", 18, len(m))
+	}
+	if val, ok := m["deployment_external_label"]; !ok || val != "kubernetes_deployment" {
+		t.Fatalf("Map: expected %s; got %s", "kubernetes_deployment", val)
+	}
+	if val, ok := m["namespace_external_label"]; !ok || val != "kubernetes_namespace" {
+		t.Fatalf("Map: expected %s; got %s", "kubernetes_namespace", val)
+	}
+
+	lc = &LabelConfig{
+		DaemonsetExternalLabel: "kubernetes_ds",
+	}
+	m = lc.Map()
+	if len(m) != 18 {
+		t.Fatalf("Map: expected length %d; got length %d", 18, len(m))
+	}
+	if val, ok := m["daemonset_external_label"]; !ok || val != "kubernetes_ds" {
+		t.Fatalf("Map: expected %s; got %s", "kubernetes_ds", val)
+	}
+	if val, ok := m["namespace_external_label"]; !ok || val != "kubernetes_namespace" {
+		t.Fatalf("Map: expected %s; got %s", "kubernetes_namespace", val)
+	}
+}
+
+func TestLabelConfig_ExternalQueryLabels(t *testing.T) {
+	var qls map[string]string
+	var lc *LabelConfig
+
+	qls = lc.ExternalQueryLabels()
+	if len(qls) != 13 {
+		t.Fatalf("ExternalQueryLabels: expected length %d; got length %d", 13, len(qls))
+	}
+	if val, ok := qls["kubernetes_deployment"]; !ok || val != "deployment_external_label" {
+		t.Fatalf("ExternalQueryLabels: expected %s; got %s", "deployment_external_label", val)
+	}
+	if val, ok := qls["kubernetes_namespace"]; !ok || val != "namespace_external_label" {
+		t.Fatalf("ExternalQueryLabels: expected %s; got %s", "namespace_external_label", val)
+	}
+
+	lc = &LabelConfig{
+		DaemonsetExternalLabel: "kubernetes_ds",
+	}
+	qls = lc.ExternalQueryLabels()
+	if len(qls) != 13 {
+		t.Fatalf("ExternalQueryLabels: expected length %d; got length %d", 13, len(qls))
+	}
+	if val, ok := qls["kubernetes_ds"]; !ok || val != "daemonset_external_label" {
+		t.Fatalf("ExternalQueryLabels: expected %s; got %s", "daemonset_external_label", val)
+	}
+	if val, ok := qls["kubernetes_namespace"]; !ok || val != "namespace_external_label" {
+		t.Fatalf("ExternalQueryLabels: expected %s; got %s", "namespace_external_label", val)
+	}
+}
+
+func TestTestLabelConfig_AllocationPropertyLabels(t *testing.T) {
+	var labels map[string]string
+	var lc *LabelConfig
+
+	labels = lc.AllocationPropertyLabels()
+	if len(labels) != 18 {
+		t.Fatalf("AllocationPropertyLabels: expected length %d; got length %d", 18, len(labels))
+	}
+	if val, ok := labels["namespace"]; !ok || val != "kubernetes_namespace" {
+		t.Fatalf("AllocationPropertyLabels: expected %s; got %s", "kubernetes_namespace", val)
+	}
+	if val, ok := labels["label:env"]; !ok || val != "env" {
+		t.Fatalf("AllocationPropertyLabels: expected %s; got %s", "env", val)
+	}
+
+	lc = &LabelConfig{
+		NamespaceExternalLabel: "kubens",
+		EnvironmentLabel:       "kubeenv",
+	}
+	labels = lc.AllocationPropertyLabels()
+	if len(labels) != 18 {
+		t.Fatalf("AllocationPropertyLabels: expected length %d; got length %d", 18, len(labels))
+	}
+	if val, ok := labels["namespace"]; !ok || val != "kubens" {
+		t.Fatalf("AllocationPropertyLabels: expected %s; got %s", "kubens", val)
+	}
+	if val, ok := labels["label:kubeenv"]; !ok || val != "kubeenv" {
+		t.Fatalf("AllocationPropertyLabels: expected %s; got %s", "kubeenv", val)
+	}
+}

pkg/kubecost/kubecost_codecs.go

@@ -14,11 +14,10 @@ package kubecost
 import (
 	"encoding"
 	"fmt"
+	util "github.com/kubecost/cost-model/pkg/util"
 	"reflect"
 	"strings"
 	"time"
-
-	util "github.com/kubecost/cost-model/pkg/util"
 )
 
 const (
@@ -26,7 +25,7 @@ const (
 	GeneratorPackageName string = "kubecost"
 
 	// CodecVersion is the version passed into the generator
-	CodecVersion uint8 = 4
+	CodecVersion uint8 = 6
 )
 
 //--------------------------------------------------------------------------
@@ -930,6 +929,19 @@ func (target *AssetSet) MarshalBinary() (data []byte, err error) {
 	buff := util.NewBuffer()
 	buff.WriteUInt8(CodecVersion) // version
 
+	if target.aggregateBy == nil {
+		buff.WriteUInt8(uint8(0)) // write nil byte
+	} else {
+		buff.WriteUInt8(uint8(1)) // write non-nil byte
+
+		// --- [begin][write][slice]([]string) ---
+		buff.WriteInt(len(target.aggregateBy)) // array length
+		for i := 0; i < len(target.aggregateBy); i++ {
+			buff.WriteString(target.aggregateBy[i]) // write string
+		}
+		// --- [end][write][slice]([]string) ---
+
+	}
 	if target.assets == nil {
 		buff.WriteUInt8(uint8(0)) // write nil byte
 	} else {
@@ -963,22 +975,6 @@ func (target *AssetSet) MarshalBinary() (data []byte, err error) {
 		}
 		// --- [end][write][map](map[string]Asset) ---
 
-	}
-	if target.props == nil {
-		buff.WriteUInt8(uint8(0)) // write nil byte
-	} else {
-		buff.WriteUInt8(uint8(1)) // write non-nil byte
-
-		// --- [begin][write][slice]([]AssetProperty) ---
-		buff.WriteInt(len(target.props)) // array length
-		for i := 0; i < len(target.props); i++ {
-			// --- [begin][write][alias](AssetProperty) ---
-			buff.WriteString(string(target.props[i])) // write string
-			// --- [end][write][alias](AssetProperty) ---
-
-		}
-		// --- [end][write][slice]([]AssetProperty) ---
-
 	}
 	// --- [begin][write][struct](Window) ---
 	d, errB := target.Window.MarshalBinary()
@@ -1042,93 +1038,89 @@ func (target *AssetSet) UnmarshalBinary(data []byte) (err error) {
 		return fmt.Errorf("Invalid Version Unmarshaling AssetSet. Expected %d, got %d", CodecVersion, version)
 	}
 
+	if buff.ReadUInt8() == uint8(0) {
+		target.aggregateBy = nil
+	} else {
+		// --- [begin][read][slice]([]string) ---
+		b := buff.ReadInt() // array len
+		a := make([]string, b)
+		for i := 0; i < b; i++ {
+			var c string
+			d := buff.ReadString() // read string
+			c = d
+
+			a[i] = c
+		}
+		target.aggregateBy = a
+		// --- [end][read][slice]([]string) ---
+
+	}
 	if buff.ReadUInt8() == uint8(0) {
 		target.assets = nil
 	} else {
 		// --- [begin][read][map](map[string]Asset) ---
-		a := make(map[string]Asset)
-		b := buff.ReadInt() // map len
-		for i := 0; i < b; i++ {
+		e := make(map[string]Asset)
+		f := buff.ReadInt() // map len
+		for j := 0; j < f; j++ {
 			var k string
-			c := buff.ReadString() // read string
-			k = c
+			g := buff.ReadString() // read string
+			k = g
 
 			var v Asset
 			if buff.ReadUInt8() == uint8(0) {
 				v = nil
 			} else {
 				// --- [begin][read][interface](Asset) ---
-				d := buff.ReadString()
-				_, e, _ := resolveType(d)
-				if _, ok := typeMap[e]; !ok {
-					return fmt.Errorf("Unknown Type: %s", e)
+				h := buff.ReadString()
+				_, l, _ := resolveType(h)
+				if _, ok := typeMap[l]; !ok {
+					return fmt.Errorf("Unknown Type: %s", l)
 				}
-				f, okA := reflect.New(typeMap[e]).Interface().(interface{ UnmarshalBinary([]byte) error })
+				m, okA := reflect.New(typeMap[l]).Interface().(interface{ UnmarshalBinary([]byte) error })
 				if !okA {
-					return fmt.Errorf("Type: %s does not implement UnmarshalBinary([]byte) error", e)
+					return fmt.Errorf("Type: %s does not implement UnmarshalBinary([]byte) error", l)
 				}
-				g := buff.ReadInt()    // byte array length
-				h := buff.ReadBytes(g) // byte array
-				errA := f.UnmarshalBinary(h)
+				n := buff.ReadInt()    // byte array length
+				o := buff.ReadBytes(n) // byte array
+				errA := m.UnmarshalBinary(o)
 				if errA != nil {
 					return errA
 				}
-				v = f.(Asset)
+				v = m.(Asset)
 				// --- [end][read][interface](Asset) ---
 
 			}
-			a[k] = v
+			e[k] = v
 		}
-		target.assets = a
+		target.assets = e
 		// --- [end][read][map](map[string]Asset) ---
 
-	}
-	if buff.ReadUInt8() == uint8(0) {
-		target.props = nil
-	} else {
-		// --- [begin][read][slice]([]AssetProperty) ---
-		m := buff.ReadInt() // array len
-		l := make([]AssetProperty, m)
-		for j := 0; j < m; j++ {
-			// --- [begin][read][alias](AssetProperty) ---
-			var o string
-			p := buff.ReadString() // read string
-			o = p
-
-			n := AssetProperty(o)
-			// --- [end][read][alias](AssetProperty) ---
-
-			l[j] = n
-		}
-		target.props = l
-		// --- [end][read][slice]([]AssetProperty) ---
-
 	}
 	// --- [begin][read][struct](Window) ---
-	q := &Window{}
-	r := buff.ReadInt()    // byte array length
-	s := buff.ReadBytes(r) // byte array
-	errB := q.UnmarshalBinary(s)
+	p := &Window{}
+	q := buff.ReadInt()    // byte array length
+	r := buff.ReadBytes(q) // byte array
+	errB := p.UnmarshalBinary(r)
 	if errB != nil {
 		return errB
 	}
-	target.Window = *q
+	target.Window = *p
 	// --- [end][read][struct](Window) ---
 
 	if buff.ReadUInt8() == uint8(0) {
 		target.Warnings = nil
 	} else {
 		// --- [begin][read][slice]([]string) ---
-		u := buff.ReadInt() // array len
-		t := make([]string, u)
-		for ii := 0; ii < u; ii++ {
-			var w string
-			x := buff.ReadString() // read string
-			w = x
+		t := buff.ReadInt() // array len
+		s := make([]string, t)
+		for ii := 0; ii < t; ii++ {
+			var u string
+			w := buff.ReadString() // read string
+			u = w
 
-			t[ii] = w
+			s[ii] = u
 		}
-		target.Warnings = t
+		target.Warnings = s
 		// --- [end][read][slice]([]string) ---
 
 	}
@@ -1136,16 +1128,16 @@ func (target *AssetSet) UnmarshalBinary(data []byte) (err error) {
 		target.Errors = nil
 	} else {
 		// --- [begin][read][slice]([]string) ---
-		z := buff.ReadInt() // array len
-		y := make([]string, z)
-		for jj := 0; jj < z; jj++ {
-			var aa string
-			bb := buff.ReadString() // read string
-			aa = bb
+		y := buff.ReadInt() // array len
+		x := make([]string, y)
+		for jj := 0; jj < y; jj++ {
+			var z string
+			aa := buff.ReadString() // read string
+			z = aa
 
-			y[jj] = aa
+			x[jj] = z
 		}
-		target.Errors = y
+		target.Errors = x
 		// --- [end][read][slice]([]string) ---
 
 	}

pkg/kubecost/properties.go

@@ -65,7 +65,7 @@ func (p *Properties) Clone() Properties {
 		return nil
 	}
 
-	clone := Properties{}
+	clone := make(Properties, len(*p))
 	for k, v := range *p {
 		clone[k] = v
 	}
@@ -222,119 +222,72 @@ func (p *Properties) Length() int {
 	return len(*p)
 }
 
-func (p *Properties) Matches(that Properties) bool {
-	// The only Properties that a nil Properties matches is an empty one
+func (p *Properties) String() string {
 	if p == nil {
-		return that.Length() == 0
+		return "<nil>"
 	}
 
-	// Matching on cluster, namespace, controller, controller kind, pod,
-	// and container are simple string equality comparisons. By default,
-	// we assume a match. For each Property given to match, we say that the
-	// match fails if we don't have that Property, or if we have it but the
-	// strings are not equal.
-
-	if thatCluster, thatErr := that.GetCluster(); thatErr == nil {
-		if thisCluster, thisErr := p.GetCluster(); thisErr != nil || thisCluster != thatCluster {
-			return false
-		}
+	strs := []string{}
+	for key, prop := range *p {
+		strs = append(strs, fmt.Sprintf("%s:%s", key, prop))
 	}
+	return fmt.Sprintf("{%s}", strings.Join(strs, "; "))
+}
 
-	if thatNode, thatErr := that.GetNode(); thatErr == nil {
-		if thisNode, thisErr := p.GetNode(); thisErr != nil || thisNode != thatNode {
-			return false
-		}
+// AggregationStrings converts a Properties object into a slice of strings
+// representing a request to aggregate by certain properties.
+// NOTE: today, the ordering of the properties *has to match the ordering
+// of the allocaiton function generateKey*
+func (p *Properties) AggregationStrings() []string {
+	if p == nil {
+		return []string{}
 	}
 
-	if thatNamespace, thatErr := that.GetNamespace(); thatErr == nil {
-		if thisNamespace, thisErr := p.GetNamespace(); thisErr != nil || thisNamespace != thatNamespace {
-			return false
-		}
+	aggStrs := []string{}
+	if p.HasCluster() {
+		aggStrs = append(aggStrs, ClusterProp.String())
 	}
-
-	if thatController, thatErr := that.GetController(); thatErr == nil {
-		if thisController, thisErr := p.GetController(); thisErr != nil || thisController != thatController {
-			return false
-		}
+	if p.HasNode() {
+		aggStrs = append(aggStrs, NodeProp.String())
 	}
-
-	if thatControllerKind, thatErr := that.GetControllerKind(); thatErr == nil {
-		if thisControllerKind, thisErr := p.GetControllerKind(); thisErr != nil || thisControllerKind != thatControllerKind {
-			return false
-		}
+	if p.HasNamespace() {
+		aggStrs = append(aggStrs, NamespaceProp.String())
 	}
-
-	if thatPod, thatErr := that.GetPod(); thatErr == nil {
-		if thisPod, thisErr := p.GetPod(); thisErr != nil || thisPod != thatPod {
-			return false
-		}
+	if p.HasControllerKind() {
+		aggStrs = append(aggStrs, ControllerKindProp.String())
 	}
-
-	if thatContainer, thatErr := that.GetContainer(); thatErr == nil {
-		if thisContainer, thisErr := p.GetContainer(); thisErr != nil || thisContainer != thatContainer {
-			return false
-		}
+	if p.HasController() {
+		aggStrs = append(aggStrs, ControllerProp.String())
 	}
-
-	// Matching on Services only occurs if a non-zero length slice of strings
-	// is given. The comparison fails if there exists a string to match that is
-	// not present in our slice of services.
-	if thatServices, thatErr := that.GetServices(); thatErr == nil && len(thatServices) > 0 {
-		thisServices, thisErr := p.GetServices()
-		if thisErr != nil {
-			return false
-		}
-
-		for _, service := range thatServices {
-			match := false
-			for _, s := range thisServices {
-				if s == service {
-					match = true
-					break
-				}
-			}
-			if !match {
-				return false
-			}
-		}
+	if p.HasPod() {
+		aggStrs = append(aggStrs, PodProp.String())
 	}
-
-	// Matching on Labels only occurs if a non-zero length map of strings is
-	// given. The comparison fails if there exists a key/value pair to match
-	// that is not present in our set of labels.
-	if thatServices, thatErr := that.GetServices(); thatErr == nil && len(thatServices) > 0 {
-		thisServices, thisErr := p.GetServices()
-		if thisErr != nil {
-			return false
+	if p.HasContainer() {
+		aggStrs = append(aggStrs, ContainerProp.String())
+	}
+	if p.HasService() {
+		aggStrs = append(aggStrs, ServiceProp.String())
+	}
+	if p.HasLabel() {
+		// e.g. expect format map[string]string{
+		// 	 "env":""
+		// 	 "app":"",
+		// }
+		// for aggregating by "label:app,label:env"
+		labels, _ := p.GetLabels()
+		labelAggStrs := []string{}
+		for labelName := range labels {
+			labelAggStrs = append(labelAggStrs, fmt.Sprintf("label:%s", labelName))
 		}
-
-		for _, service := range thatServices {
-			match := false
-			for _, s := range thisServices {
-				if s == service {
-					match = true
-					break
-				}
-			}
-			if !match {
-				return false
+		if len(labelAggStrs) > 0 {
+			// Enforce alphabetical ordering, then append to aggStrs
+			sort.Strings(labelAggStrs)
+			for _, labelName := range labelAggStrs {
+				aggStrs = append(aggStrs, labelName)
 			}
 		}
 	}
-
-	return true
-}
-
-func (p *Properties) String() string {
-	if p == nil {
-		return "<nil>"
-	}
-
-	strs := []string{}
-	for key, prop := range *p {
-		strs = append(strs, fmt.Sprintf("%s:%s", key, prop))
-	}
-	return fmt.Sprintf("{%s}", strings.Join(strs, "; "))
+	return aggStrs
 }
 
 func (p *Properties) Get(prop Property) (string, error) {
@@ -707,8 +660,8 @@ func (p *Properties) UnmarshalBinary(data []byte) error {
 
 	// LabelProp
 	if buff.ReadUInt8() == 1 { // read nil byte
-		labels := map[string]string{}
 		length := buff.ReadInt() // read map len
+		labels := make(map[string]string, length)
 		for idx := 0; idx < length; idx++ {
 			key := buff.ReadString()
 			val := buff.ReadString()
@@ -719,8 +672,8 @@ func (p *Properties) UnmarshalBinary(data []byte) error {
 
 	// AnnotationProp
 	if buff.ReadUInt8() == 1 { // read nil byte
-		annotations := map[string]string{}
 		length := buff.ReadInt() // read map len
+		annotations := make(map[string]string, length)
 		for idx := 0; idx < length; idx++ {
 			key := buff.ReadString()
 			val := buff.ReadString()
@@ -731,11 +684,11 @@ func (p *Properties) UnmarshalBinary(data []byte) error {
 
 	// ServiceProp
 	if buff.ReadUInt8() == 1 { // read nil byte
-		services := []string{}
 		length := buff.ReadInt() // read map len
+		services := make([]string, length)
 		for idx := 0; idx < length; idx++ {
 			val := buff.ReadString()
-			services = append(services, val)
+			services[idx] = val
 		}
 		p.SetServices(services)
 	}

pkg/kubecost/properties_test.go

@@ -1,8 +1,6 @@
 package kubecost
 
-import (
-	"testing"
-)
+import "testing"
 
 // TODO niko/etl
 // func TestParseProperty(t *testing.T) {}
@@ -10,187 +8,63 @@ import (
 // TODO niko/etl
 // func TestProperty_String(t *testing.T) {}
 
-// TODO niko/etl
-// func TestProperties_Clone(t *testing.T) {}
-
-// TODO niko/etl
-// func TestProperties_Intersection(t *testing.T) {}
-
-func TestProperties_Matches(t *testing.T) {
-	// nil Properties should match empty Properties
-	var p *Properties
-	propsEmpty := Properties{}
-
-	if !p.Matches(propsEmpty) {
-		t.Fatalf("Properties.Matches: expect nil to match empty")
-	}
-
-	// Empty Properties should match empty Properties
-	p = &Properties{}
-	if !p.Matches(propsEmpty) {
-		t.Fatalf("Properties.Matches: expect nil to match empty")
-	}
-
-	p.SetCluster("cluster-one")
-	p.SetNamespace("kubecost")
-	p.SetController("kubecost-deployment")
-	p.SetControllerKind("deployment")
-	p.SetPod("kubecost-deployment-abc123")
-	p.SetContainer("kubecost-cost-model")
-	p.SetServices([]string{"kubecost-frontend"})
-	p.SetLabels(map[string]string{
-		"app":  "kubecost",
-		"tier": "frontend",
-	})
-
-	// Non-empty Properties should match empty Properties, but not vice-a-versa
-	if !p.Matches(propsEmpty) {
-		t.Fatalf("Properties.Matches: expect nil to match empty")
-	}
-	if propsEmpty.Matches(*p) {
-		t.Fatalf("Properties.Matches: expect empty to not match non-empty")
-	}
-
-	// Non-empty Properties should match itself
-	if !p.Matches(*p) {
-		t.Fatalf("Properties.Matches: expect non-empty to match itself")
-	}
-
-	// Match on all
-	if !p.Matches(Properties{
-		ClusterProp:        "cluster-one",
-		NamespaceProp:      "kubecost",
-		ControllerProp:     "kubecost-deployment",
-		ControllerKindProp: "deployment",
-		PodProp:            "kubecost-deployment-abc123",
-		ContainerProp:      "kubecost-cost-model",
-		ServiceProp:        []string{"kubecost-frontend"},
-		LabelProp: map[string]string{
-			"app":  "kubecost",
-			"tier": "frontend",
-		},
-	}) {
-		t.Fatalf("Properties.Matches: expect match on all")
-	}
-
-	// Match on cluster
-	if !p.Matches(Properties{
-		ClusterProp: "cluster-one",
-	}) {
-		t.Fatalf("Properties.Matches: expect match on cluster")
-	}
-
-	// No match on cluster
-	if p.Matches(Properties{
-		ClusterProp: "miss",
-	}) {
-		t.Fatalf("Properties.Matches: expect no match on cluster")
-	}
-
-	// Match on namespace
-	if !p.Matches(Properties{
-		NamespaceProp: "kubecost",
-	}) {
-		t.Fatalf("Properties.Matches: expect match on namespace")
-	}
+func TestProperties_AggregationString(t *testing.T) {
+	var props *Properties
+	var aggStrs []string
 
-	// No match on namespace
-	if p.Matches(Properties{
-		NamespaceProp: "miss",
-	}) {
-		t.Fatalf("Properties.Matches: expect no match on namespace")
+	// nil Properties should produce and empty slice
+	aggStrs = props.AggregationStrings()
+	if aggStrs == nil || len(aggStrs) > 0 {
+		t.Fatalf("expected empty slice; got %v", aggStrs)
 	}
 
-	// Match on controller
-	if !p.Matches(Properties{
-		ControllerProp: "kubecost-deployment",
-	}) {
-		t.Fatalf("Properties.Matches: expect match on controller")
+	// empty Properties should product an empty slice
+	props = &Properties{}
+	aggStrs = props.AggregationStrings()
+	if aggStrs == nil || len(aggStrs) > 0 {
+		t.Fatalf("expected empty slice; got %v", aggStrs)
 	}
 
-	// No match on controller
-	if p.Matches(Properties{
-		ControllerProp: "miss",
-	}) {
-		t.Fatalf("Properties.Matches: expect no match on controller")
+	// Properties with single, simple property set
+	props = &Properties{}
+	props.SetNamespace("")
+	aggStrs = props.AggregationStrings()
+	if len(aggStrs) != 1 || aggStrs[0] != "namespace" {
+		t.Fatalf("expected [\"namespace\"]; got %v", aggStrs)
 	}
 
-	// Match on controller kind
-	if !p.Matches(Properties{
-		ControllerKindProp: "deployment",
-	}) {
-		t.Fatalf("Properties.Matches: expect match on controller kind")
-	}
-
-	// No match on controller kind
-	if p.Matches(Properties{
-		ControllerKindProp: "miss",
-	}) {
-		t.Fatalf("Properties.Matches: expect no match on controller kind")
-	}
-
-	// Match on pod
-	if !p.Matches(Properties{
-		PodProp: "kubecost-deployment-abc123",
-	}) {
-		t.Fatalf("Properties.Matches: expect match on pod")
-	}
-
-	// No match on pod
-	if p.Matches(Properties{
-		PodProp: "miss",
-	}) {
-		t.Fatalf("Properties.Matches: expect no match on pod")
+	// Properties with mutiple properties, including labels
+	// Note: order matters!
+	props = &Properties{}
+	props.SetNamespace("")
+	props.SetLabels(map[string]string{
+		"env": "",
+		"app": "",
+	})
+	props.SetCluster("")
+	aggStrs = props.AggregationStrings()
+	if len(aggStrs) != 4 {
+		t.Fatalf("expected length %d; got lenfth %d", 4, len(aggStrs))
 	}
-
-	// Match on container
-	if !p.Matches(Properties{
-		ContainerProp: "kubecost-cost-model",
-	}) {
-		t.Fatalf("Properties.Matches: expect match on container")
+	if aggStrs[0] != "cluster" {
+		t.Fatalf("expected aggStrs[0] == \"%s\"; got \"%s\"", "cluster", aggStrs[0])
 	}
-
-	// No match on container
-	if p.Matches(Properties{
-		ContainerProp: "miss",
-	}) {
-		t.Fatalf("Properties.Matches: expect no match on container")
+	if aggStrs[1] != "namespace" {
+		t.Fatalf("expected aggStrs[1] == \"%s\"; got \"%s\"", "namespace", aggStrs[1])
 	}
-
-	// Match on single service
-	if !p.Matches(Properties{
-		ServiceProp: []string{"kubecost-frontend"},
-	}) {
-		t.Fatalf("Properties.Matches: expect match on service")
+	if aggStrs[2] != "label:app" {
+		t.Fatalf("expected aggStrs[2] == \"%s\"; got \"%s\"", "label:app", aggStrs[2])
 	}
-
-	// No match on one missing service
-	if p.Matches(Properties{
-		ServiceProp: []string{"missing-service", "kubecost-frontend"},
-	}) {
-		t.Fatalf("Properties.Matches: expect no match on 1 of 2 services")
+	if aggStrs[3] != "label:env" {
+		t.Fatalf("expected aggStrs[3] == \"%s\"; got \"%s\"", "label:env", aggStrs[3])
 	}
+}
 
-	// Match on single label
-	if !p.Matches(Properties{
-		LabelProp: map[string]string{
-			"app": "kubecost",
-		},
-	}) {
-		t.Fatalf("Properties.Matches: expect match on label")
-	}
+// TODO niko/etl
+// func TestProperties_Clone(t *testing.T) {}
 
-	// No match on one missing label
-	if !p.Matches(Properties{
-		LabelProp: map[string]string{
-			"app":   "kubecost",
-			"tier":  "frontend",
-			"label": "missing",
-		},
-	}) {
-		t.Fatalf("Properties.Matches: expect no match on 2 of 3 labels")
-	}
-}
+// TODO niko/etl
+// func TestProperties_Intersection(t *testing.T) {}
 
 // TODO niko/etl
 // func TestProperties_GetCluster(t *testing.T) {}

pkg/prom/result.go

@@ -240,7 +240,7 @@ func (qr *QueryResult) GetLabels() map[string]string {
 			continue
 		}
 
-		label := k[6:]
+		label := strings.TrimPrefix(k, "label_")
 		value, ok := v.(string)
 		if !ok {
 			log.Warningf("Failed to parse label value for label: '%s'", label)
@@ -253,6 +253,29 @@ func (qr *QueryResult) GetLabels() map[string]string {
 	return result
 }
 
+// GetAnnotations returns all annotations and their values from the query result
+func (qr *QueryResult) GetAnnotations() map[string]string {
+	result := make(map[string]string)
+
+	// Find All keys with prefix annotation_, remove prefix, add to annotations
+	for k, v := range qr.Metric {
+		if !strings.HasPrefix(k, "annotation_") {
+			continue
+		}
+
+		annotations := strings.TrimPrefix(k, "annotation_")
+		value, ok := v.(string)
+		if !ok {
+			log.Warningf("Failed to parse label value for label: '%s'", annotations)
+			continue
+		}
+
+		result[annotations] = value
+	}
+
+	return result
+}
+
 // parseDataPoint parses a data point from raw prometheus query results and returns
 // a new Vector instance containing the parsed data along with any warnings or errors.
 func parseDataPoint(query string, dataPoint interface{}) (*util.Vector, warning, error) {

pkg/util/math.go

@@ -0,0 +1,15 @@
+package util
+
+import "math"
+
+// IsApproximately returns true is a approximately equals b, within
+// a delta computed as a function of the size of a and b.
+func IsApproximately(a, b float64) bool {
+	delta := 0.000001 * math.Max(math.Abs(a), math.Abs(b))
+	return math.Abs(a-b) <= delta
+}
+
+// IsWithin returns true if a and b are within delta of each other
+func IsWithin(a, b, delta float64) bool {
+	return math.Abs(a-b) <= delta
+}