Merge branch 'develop' of https://github.com/opencost/opencost into develop

.github/ISSUE_TEMPLATE/opencost-bug-report.md

@@ -0,0 +1,30 @@
+---
+name: OpenCost Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the OpenCost bug is. Please ensure this is an issue related to the OpenCost cost model, API, UI or specification. Public Kubecost bugs may be opened at https://github.com/kubecost/cost-analyzer-helm-chart/ 
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Which version of OpenCost are you using?**
+This may be the Kubecost release.
+
+**Additional context**
+Add any other context about the problem here. Kubernetes versions and which public clouds you are working with are especially important.

.github/ISSUE_TEMPLATE/opencost-feature-request.md

@@ -0,0 +1,20 @@
+---
+name: OpenCost Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context, documentation links, or screenshots about the feature request here.

.github/dependabot.yml

@@ -0,0 +1,13 @@
+version: 2
+updates:
+  # Dependencies listed in go.mod
+  - package-ecosystem: "gomod"
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"
+
+  # Dependencies listed in .github/workflows/*.yml
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"

.github/workflows/pr.yaml

@@ -7,6 +7,13 @@ on:
 
 jobs:
   build:
+    strategy:
+      matrix:
+        include:
+          - component: Frontend
+            location: ui
+          - component: Backend
+            location: .
     runs-on: ubuntu-latest
 
     steps:
@@ -17,9 +24,9 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v1
 
-      - name: Build
+      - name: Build ${{ matrix.component }}
         uses: docker/build-push-action@v2
         with:
-          context: ./
-          file: ./Dockerfile
+          context: ${{ matrix.location }}/
+          file: ${{ matrix.location }}/Dockerfile
           push: false

.gitignore

@@ -1,5 +1,9 @@
+# Jetbrains project files
 .idea
+*.iml
+
 ui/.cache
 ui/dist
 ui/node_modules/
-cmd/costmodel/costmodel
+cmd/costmodel/costmodel
+pkg/cloud/azureorphan_test.go

CODE_OF_CONDUCT.md

@@ -1,7 +1,7 @@
 # Contributor Code of Conduct
 
-As contributors and maintainers in the Kubecost community, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
-We are committed to making participation in the Kubecost community a harassment-free experience for everyone.
+As contributors and maintainers in the OpenCost community, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
+We are committed to making participation in the OpenCost community a harassment-free experience for everyone.
 
 # Scope
 
@@ -32,13 +32,13 @@ Project maintainers who do not follow or enforce the Code of Conduct may be perm
 
 # Reporting
 
-For incidents occurring in the Kubecost community, contact the Kubecost Code of Conduct Committee via conduct@kubecost.com. You can expect a response within two business days.
-For other projects, please contact the Kubecost staff via conduct@kubecost.com. You can expect a response within three business days.
+For incidents occurring in the OpenCost community, contact the OpenCost Code of Conduct Committee via conduct@kubecost.com. You can expect a response within two business days.
+For other projects, please contact the OpenCost staff via conduct@kubecost.com. You can expect a response within three business days.
 
 # Enforcement
 
-The Kubecost project's Code of Conduct Committee enforces code of conduct issues. For all other projects, the Kubecost enforces code of conduct issues.
-Both bodies try to resolve incidents without punishment, but may remove people from the project or Kubecost communities at their discretion.
+The OpenCost project's Code of Conduct Committee enforces code of conduct issues. For all other projects, the OpenCost enforces code of conduct issues.
+Both bodies try to resolve incidents without punishment, but may remove people from the project or OpenCost communities at their discretion.
 
 # Acknowledgements
 This Code of Conduct is adapted from the Contributor Covenant (http://contributor-covenant.org), version 2.0 available at http://contributor-covenant.org/version/2/0/code_of_conduct/

CONTRIBUTING.md

@@ -3,14 +3,15 @@
 Thanks for your help improving the OpenCost project! There are many ways to contribute to the project, including the following:
 
 * contributing or providing feedback on the [OpenCost Spec](https://github.com/opencost/opencost/tree/develop/spec)
-* contributing documentation 
-* joining the discussion on [CNCF Slack](https://slack.cncf.io/) in the [#opencost](https://cloud-native.slack.com/archives/C03D56FPD4G) channel or in the [OpenCost community discussions](https://drive.google.com/drive/folders/1hXlcyFPePB7t3z6lyVzdxmdfrbzeT1Jz) folder
+* contributing documentation here or to the [OpenCost website](https://github.com/kubecost/opencost-website)
+* joining the discussion in the [CNCF Slack](https://slack.cncf.io/) in the [#opencost](https://cloud-native.slack.com/archives/C03D56FPD4G) channel
+* participating in the fortnightly [OpenCost Working Group](https://calendar.google.com/calendar/u/0/embed?src=c_c0f7q56e5eeod3j89bb320fvjg@group.calendar.google.com&ctz=America/Los_Angeles) meetings ([notes here](https://drive.google.com/drive/folders/1hXlcyFPePB7t3z6lyVzdxmdfrbzeT1Jz))
 * committing software via the workflow below
 
 ## Getting Help
 
 If you have a question about OpenCost or have encountered problems using it,
-you can start by asking a question on [CNCF Slack](https://slack.cncf.io/) in the [#opencost](https://cloud-native.slack.com/archives/C03D56FPD4G) channel or via email at [support@kubecost.com](support@kubecost.com)
+you can start by asking a question on [CNCF Slack](https://slack.cncf.io/) in the [#opencost](https://cloud-native.slack.com/archives/C03D56FPD4G) channel or via email at [opencost@kubecost.com](opencost@kubecost.com)
 
 ## Workflow
 
@@ -96,4 +97,4 @@ Please write a commit message with Fixes Issue # if there is an outstanding issu
 
 Please run go fmt on the project directory. Lint can be okay (for example, comments on exported functions are nice but not required on the server).
 
-Please email us [support@kubecost.com](support@kubecost.com) or reach out to us on [CNCF Slack](https://slack.cncf.io/) in the [#opencost](https://cloud-native.slack.com/archives/C03D56FPD4G) channel if you need help or have any questions!
+Please email us [opencost@kubecost.com](opencost@kubecost.com) or reach out to us on [CNCF Slack](https://slack.cncf.io/) in the [#opencost](https://cloud-native.slack.com/archives/C03D56FPD4G) channel if you need help or have any questions!

Dockerfile

@@ -37,5 +37,6 @@ ADD ./configs/default.json /models/default.json
 ADD ./configs/azure.json /models/azure.json
 ADD ./configs/aws.json /models/aws.json
 ADD ./configs/gcp.json /models/gcp.json
+ADD ./configs/alibaba.json /models/alibaba.json
 USER 1001
 ENTRYPOINT ["/go/bin/app"]

NOTICE

@@ -0,0 +1,9 @@
+OpenCost
+Copyright 2022 Cloud Native Computing Foundation
+
+This product includes software developed at
+The Cloud Native Computing Foundation (http://www.cncf.io).
+
+The Initial Developer of some parts of the specification and project is
+Kubecost (http://www.kubecost.com).
+Copyright 2019 - 2022 Stackwatch Incorporated. All Rights Reserved.

PROMETHEUS.md

@@ -1,67 +1 @@
-Kubecost allows you to export pricing data to Prometheus and then write custom queries for cost insights. Below are instructions for accomplishing this and a set of example queries to get you started.
-
-## Configuration
-
-After deploying the Kubecost model (see [README](README.md) for more info on installation), configure Prometheus to scrape the `/metrics` endpoint exposed by Kubecost. Below is a sample scrape config:
-
-```
-- job_name: kubecost
-  honor_labels: true
-  scrape_interval: 1m
-  scrape_timeout: 10s
-  metrics_path: /metrics
-  scheme: http
-  static_configs:
-  - targets:
-    - < address of cost-model service> # example: <service-name>.<namespace>:<port>
-``` 
-
-## Example queries
-
-Below are a set of sample queries that can be run after Prometheus begins ingesting Kubecost data:
-
-__Monthly cost of top 5 containers__
-
-```
-topk( 5, 
-  container_memory_allocation_bytes* on(instance) group_left() node_ram_hourly_cost  / 1024 / 1024 / 1024 * 730
-  + 
-  container_cpu_allocation * on(instance) group_left() node_cpu_hourly_cost * 730
-)
-```
-
-__Hourly memory cost for the *default* namespace__
-
-```
-sum(
-  avg(container_memory_allocation_bytes{namespace="default"}) by (instance) / 1024 / 1024 / 1024
-  * 
-  on(instance) group_left() avg(node_ram_hourly_cost) by (instance)
-)
-```
-
-__Monthly cost of currently provisioned nodes__
-
-```
-sum(node_total_hourly_cost) * 730
-```
-
-
-## Available Metrics
-
-**Note:** metrics today have both *instance* and *node* labels. The *instance* label will be deprecated in a future version.
-
-| Metric       | Description                                                                                            |
-| ------------ | ------------------------------------------------------------------------------------------------------ |
-| node_cpu_hourly_cost | Hourly cost per vCPU on this node  |
-| node_gpu_hourly_cost | Hourly cost per GPU on this node  |
-| node_ram_hourly_cost   | Hourly cost per Gb of memory on this node                       |
-| node_total_hourly_cost   | Total node cost per hour                       |
-| kubecost_load_balancer_cost   | Hourly cost of a load balancer                 |
-| kubecost_cluster_management_cost | Hourly management fee per cluster                 |
-| pv_hourly_cost   | Hourly cost per GP on a persistent volume                 |
-| node_gpu_count | Number of GPUs available on node |
-| container_cpu_allocation   | Average number of CPUs requested/used over last 1m                      |
-| container_gpu_allocation   | Average number of GPUs requested over last 1m                      |
-| container_memory_allocation_bytes   | Average bytes of RAM requested/used over last 1m                 |
-| pod_pvc_allocation   | Bytes provisioned for a PVC attached to a pod                      |
+Available at <https://www.opencost.io/docs/prometheus>

README.md

@@ -4,18 +4,16 @@
 
 OpenCost models give teams visibility into current and historical Kubernetes spend and resource allocation. These models provide cost transparency in Kubernetes environments that support multiple applications, teams, departments, etc.
 
-
 OpenCost was originally developed and open sourced by [Kubecost](https://kubecost.com). This project combines a [specification](/spec/) as well as a Golang implementation of these detailed requirements.
 
-![OpenCost allocation UI](/allocation-drilldown.gif)
+![OpenCost allocation UI](./ui/src/opencost-ui.png)
 
 To see the full functionality of OpenCost you can view [OpenCost features](https://opencost.io). Here is a summary of features enabled:
 
-- Real-time cost allocation by Kubernetes service, deployment, namespace, label, statefulset, daemonset, pod, and container
-- Dynamic asset pricing enabled by integrations with AWS, Azure, and GCP billing APIs
-- Supports on-prem k8s clusters with custom pricing sheets
+- Real-time cost allocation by Kubernetes cluster, node, namespace, controller kind, controller, service, or pod
+- Dynamic onDemand asset pricing enabled by integrations with AWS, Azure, and GCP billing APIs
+- Supports on-prem k8s clusters with custom CSV pricing
 - Allocation for in-cluster resources like CPU, GPU, memory, and persistent volumes.
-- Allocation for AWS & GCP out-of-cluster resources like RDS instances and S3 buckets with key (optional)
 - Easily export pricing data to Prometheus with /metrics endpoint ([learn more](PROMETHEUS.md))
 - Free and open source distribution (Apache2 license)
 
@@ -23,24 +21,24 @@ To see the full functionality of OpenCost you can view [OpenCost features](https
 
 You can deploy OpenCost on any Kubernetes 1.8+ cluster in a matter of minutes, if not seconds!
 
-Visit the full documentation for [recommended install options](https://www.opencost.io/docs/install). Compared to building from source, installing from Helm is faster and includes all necessary dependencies.
+Visit the full documentation for [recommended install options](https://www.opencost.io/docs/install).
 
 ## Usage
 
-- User interface
-- [Cost APIs](https://github.com/kubecost/docs/blob/master/apis.md)
-- [CLI / kubectl cost](https://github.com/kubecost/kubectl-cost)
-- [Prometheus metric exporter](kubecost-exporter.md)
+- [Cost APIs](https://www.opencost.io/docs/api)
+- [CLI / kubectl cost](https://www.opencost.io/docs/kubectl-cost)
+- [Prometheus Metrics](https://www.opencost.io/docs/prometheus)
+- Reference [User Interface](https://github.com/opencost/opencost/tree/develop/ui)
 
 ## Contributing
 
-We :heart: pull requests! See [`CONTRIBUTING.md`](CONTRIBUTING.md) for information on buiding the project from source
+We :heart: pull requests! See [`CONTRIBUTING.md`](CONTRIBUTING.md) for information on building the project from source
 and contributing changes.
 
 ## Community
 
-If you need any support or have any questions on contributing to the project, you can reach us on [CNCF Slack](https://slack.cncf.io/) in the [#opencost](https://cloud-native.slack.com/archives/C03D56FPD4G) channel or via email at [team@kubecost.com](team@kubecost.com).
+If you need any support or have any questions on contributing to the project, you can reach us on [CNCF Slack](https://slack.cncf.io/) in the [#opencost](https://cloud-native.slack.com/archives/C03D56FPD4G) channel or via email at [opencost@kubecost.com](opencost@kubecost.com).
 
 ## FAQ
 
-You can view [OpenCost documentation](https://www.opencost.io/docs/FAQ) for a list of commonly asked questions.
+You can view [OpenCost documentation](https://www.opencost.io/docs/FAQ) for a list of commonly asked questions.

ROADMAP.md

@@ -10,4 +10,4 @@ __2022 roadmap__
 * More robust API documentation
 * Expose carbon emission ratings
 
-Please contact us at team@kubecost.com if you're interest in more detail. 
+Please contact us at opencost@kubecost.com if you're interest in more detail.

configs/alibaba.json

@@ -0,0 +1,12 @@
+{
+    "provider": "Alibaba",
+    "description": "Default prices used to compute allocation between RAM and CPU. Alibaba Cloud pricing API data still used for total node cost.",
+    "alibabaServiceKeyName": "ABC",
+    "alibabaServiceKeySecret": "XYZ",
+    "CPU": "0.031611",
+    "spotCPU": "0.006655",
+    "RAM": "0.004237",
+    "GPU": "0.95",
+    "spotRAM": "0.000892",
+    "storage": "0.00005479452"
+}

configs/pricing_schema.csv

@@ -1,2 +1,4 @@
 EndTimestamp,InstanceID,Region,AssetClass,InstanceIDField,InstanceType,MarketPriceHourly,Version
 2019-04-17 23:34:22 UTC,gke-standard-cluster-1-pool-1-91dc432d-cg69,,node,metadata.name,,0.1337,
+2019-04-17 23:34:22 UTC,Quadro_RTX_4000,,gpu,nvidia.com/gpu_type,,0.75,
+2019-04-17 23:34:22 UTC,Quadro_RTX_4001,,gpu,gpu.nvidia.com/class,,0.80,

deploying-as-a-pod.md

@@ -1,14 +0,0 @@
-## Deploying as a pod
-
-See this page for all [Kubecost install options](http://docs.kubecost.com/install).
-
-If you would like to deploy the cost model (w/o dashboards) directly a pod on your cluster, complete the steps listed below.
-
-1. Set [this environment variable](https://github.com/opencost/opencost/blob/c211fbc1244a9da9667c7180a9e4c7f988d7978a/kubernetes/deployment.yaml#L33) to the address of your prometheus server
-2. `kubectl create namespace cost-model`
-3. `kubectl apply -f kubernetes/ --namespace cost-model`
-4. `kubectl port-forward --namespace cost-model service/cost-model 9003`
-
-To test that the server is running, you can hit [http://localhost:9003/costDataModel?timeWindow=1d](http://localhost:9003/costDataModel?timeWindow=1d)
-
-**Note:** This approach provides less functionality compared to other install options referenced above. Also, Prometheus and kube-state-metrics are external dependencies for this installation path.

docs/README.md

@@ -0,0 +1 @@
+The docs are available at <https://www.opencost.io/docs/> and the source is at <https://github.com/opencost/opencost-website/>

docs/opencost.postman_collection.json

@@ -0,0 +1,156 @@
+{
+	"info": {
+		"_postman_id": "bba454a2-641d-4481-9ccd-0395a6054a20",
+		"name": "opencost",
+		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
+	},
+	"item": [
+		{
+			"name": "namespace costs/day",
+			"request": {
+				"method": "GET",
+				"header": [],
+				"url": {
+					"raw": "{{host}}/allocation/compute?aggregate=namespace&window=1d&accumulate=false&step=1d",
+					"host": [
+						"{{host}}"
+					],
+					"path": [
+						"allocation",
+						"compute"
+					],
+					"query": [
+						{
+							"key": "aggregate",
+							"value": "namespace"
+						},
+						{
+							"key": "window",
+							"value": "1d"
+						},
+						{
+							"key": "accumulate",
+							"value": "false"
+						},
+						{
+							"key": "step",
+							"value": "1d"
+						}
+					]
+				}
+			},
+			"response": []
+		},
+		{
+			"name": "cluster total",
+			"request": {
+				"method": "GET",
+				"header": [],
+				"url": {
+					"raw": "{{host}}/allocation/compute?aggregate=cluster&window=lastweek",
+					"host": [
+						"{{host}}"
+					],
+					"path": [
+						"allocation",
+						"compute"
+					],
+					"query": [
+						{
+							"key": "aggregate",
+							"value": "cluster"
+						},
+						{
+							"key": "window",
+							"value": "lastweek"
+						}
+					]
+				}
+			},
+			"response": []
+		},
+		{
+			"name": "namespace totals",
+			"request": {
+				"method": "GET",
+				"header": [],
+				"url": {
+					"raw": "{{host}}/allocation/compute?aggregate=namespace&window=7d",
+					"host": [
+						"{{host}}"
+					],
+					"path": [
+						"allocation",
+						"compute"
+					],
+					"query": [
+						{
+							"key": "aggregate",
+							"value": "namespace"
+						},
+						{
+							"key": "window",
+							"value": "7d"
+						}
+					]
+				}
+			},
+			"response": []
+		},
+		{
+			"name": "pod totals",
+			"request": {
+				"method": "GET",
+				"header": [],
+				"url": {
+					"raw": "{{host}}/allocation/compute?aggregate=pod&window=7d",
+					"host": [
+						"{{host}}"
+					],
+					"path": [
+						"allocation",
+						"compute"
+					],
+					"query": [
+						{
+							"key": "aggregate",
+							"value": "pod"
+						},
+						{
+							"key": "window",
+							"value": "7d"
+						}
+					]
+				}
+			},
+			"response": []
+		}
+	],
+	"event": [
+		{
+			"listen": "prerequest",
+			"script": {
+				"type": "text/javascript",
+				"exec": [
+					""
+				]
+			}
+		},
+		{
+			"listen": "test",
+			"script": {
+				"type": "text/javascript",
+				"exec": [
+					""
+				]
+			}
+		}
+	],
+	"variable": [
+		{
+			"key": "host",
+			"value": "http://localhost:9003",
+			"type": "default"
+		}
+	]
+}

go.mod

@@ -12,7 +12,8 @@ require (
 	github.com/Azure/go-autorest/autorest v0.11.27
 	github.com/Azure/go-autorest/autorest/adal v0.9.18
 	github.com/Azure/go-autorest/autorest/azure/auth v0.5.11
-	github.com/aws/aws-sdk-go v1.28.9
+	github.com/aliyun/alibaba-cloud-sdk-go v1.62.3
+	github.com/aws/aws-sdk-go v1.44.153
 	github.com/aws/aws-sdk-go-v2 v1.13.0
 	github.com/aws/aws-sdk-go-v2/config v1.13.1
 	github.com/aws/aws-sdk-go-v2/credentials v1.8.0
@@ -40,12 +41,14 @@ require (
 	github.com/prometheus/common v0.32.1
 	github.com/rs/cors v1.7.0
 	github.com/rs/zerolog v1.26.1
+	github.com/scaleway/scaleway-sdk-go v1.0.0-beta.9
 	github.com/spf13/cobra v1.2.1
 	github.com/spf13/viper v1.8.1
 	go.etcd.io/bbolt v1.3.5
 	golang.org/x/exp v0.0.0-20220609121020-a51bd0440498
 	golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c
-	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
+	golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
+	golang.org/x/text v0.4.0
 	google.golang.org/api v0.44.0
 	gopkg.in/yaml.v2 v2.4.0
 	k8s.io/api v0.20.4
@@ -81,7 +84,7 @@ require (
 	github.com/go-logr/logr v0.2.0 // indirect
 	github.com/gofrs/uuid v4.2.0+incompatible // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
-	github.com/golang-jwt/jwt/v4 v4.4.1 // indirect
+	github.com/golang-jwt/jwt/v4 v4.4.2 // indirect
 	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
 	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/google/go-cmp v0.5.6 // indirect
@@ -107,10 +110,10 @@ require (
 	github.com/mitchellh/mapstructure v1.4.1 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/opentracing/opentracing-go v1.2.1-0.20220228012449-10b1cf09e00b // indirect
 	github.com/pelletier/go-toml v1.9.3 // indirect
 	github.com/prometheus/procfs v0.7.3 // indirect
 	github.com/rs/xid v1.3.0 // indirect
-	github.com/scaleway/scaleway-sdk-go v1.0.0-beta.9 // indirect
 	github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24 // indirect
 	github.com/sirupsen/logrus v1.8.1 // indirect
 	github.com/spf13/afero v1.6.0 // indirect
@@ -119,25 +122,23 @@ require (
 	github.com/spf13/pflag v1.0.5 // indirect
 	github.com/subosito/gotenv v1.2.0 // indirect
 	go.opencensus.io v0.23.0 // indirect
-	golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4 // indirect
+	golang.org/x/crypto v0.3.0 // indirect
 	golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 // indirect
-	golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 // indirect
-	golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2 // indirect
-	golang.org/x/sys v0.0.0-20220114195835-da31bd327af9 // indirect
-	golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 // indirect
-	golang.org/x/text v0.3.7 // indirect
+	golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 // indirect
+	golang.org/x/net v0.2.0 // indirect
+	golang.org/x/sys v0.2.0 // indirect
+	golang.org/x/term v0.2.0 // indirect
 	golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e // indirect
-	golang.org/x/tools v0.1.10 // indirect
-	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+	golang.org/x/tools v0.1.12 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c // indirect
 	google.golang.org/grpc v1.38.0 // indirect
 	google.golang.org/protobuf v1.26.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
-	gopkg.in/ini.v1 v1.62.0 // indirect
+	gopkg.in/ini.v1 v1.67.0 // indirect
 	k8s.io/klog/v2 v2.4.0 // indirect
 	k8s.io/utils v0.0.0-20201110183641-67b214c5f920 // indirect
 	sigs.k8s.io/structured-merge-diff/v4 v4.0.2 // indirect
 )
 
-go 1.18
+go 1.19

go.sum

@@ -93,14 +93,16 @@ github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuy
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
+github.com/aliyun/alibaba-cloud-sdk-go v1.62.3 h1:kWY5c/9JOhSYBogi3mtNG7G9TxXS0CddtQ6RKOI3mvY=
+github.com/aliyun/alibaba-cloud-sdk-go v1.62.3/go.mod h1:Api2AkmMgGaSUAhmk76oaFObkoeCPc/bKAqcyplPODs=
 github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
 github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
 github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
 github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
 github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
-github.com/aws/aws-sdk-go v1.28.9 h1:grIuBQc+p3dTRXerh5+2OxSuWFi0iXuxbFdTSg0jaW0=
-github.com/aws/aws-sdk-go v1.28.9/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
+github.com/aws/aws-sdk-go v1.44.153 h1:KfN5URb9O/Fk48xHrAinrPV2DzPcLa0cd9yo1ax5KGg=
+github.com/aws/aws-sdk-go v1.44.153/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
 github.com/aws/aws-sdk-go-v2 v1.13.0 h1:1XIXAfxsEmbhbj5ry3D3vX+6ZcUYvIqSm4CWWEuGZCA=
 github.com/aws/aws-sdk-go-v2 v1.13.0/go.mod h1:L6+ZpqHaLbAaxsqV0L4cvxZY7QupWJB4fhkf8LXvC7w=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.2.0 h1:scBthy70MB3m4LCMFaBcmYCyR2XWOz6MxSfdSu/+fQo=
@@ -173,6 +175,7 @@ github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZm
 github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
 github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
 github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
+github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
 github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
 github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
 github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
@@ -240,10 +243,11 @@ github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7a
 github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
 github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
+github.com/goji/httpauth v0.0.0-20160601135302-2da839ab0f4d/go.mod h1:nnjvkQ9ptGaCkuDUx6wNykzzlUixGxvkme+H/lnzb+A=
 github.com/golang-jwt/jwt/v4 v4.0.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
 github.com/golang-jwt/jwt/v4 v4.2.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
-github.com/golang-jwt/jwt/v4 v4.4.1 h1:pC5DB52sCeK48Wlb9oPcdhnjkz1TKt1D/P7WKJ0kUcQ=
-github.com/golang-jwt/jwt/v4 v4.4.1/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
+github.com/golang-jwt/jwt/v4 v4.4.2 h1:rcc4lwaZgFMCZ5jxF9ABolDcIHdBytAFgqFPbSJQAYs=
+github.com/golang-jwt/jwt/v4 v4.4.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
 github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
@@ -322,7 +326,6 @@ github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+
 github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
 github.com/googleapis/gnostic v0.4.1 h1:DLJCy1n/vrD4HPjOvYcT8aYQXpPIzoRZONaYwyycI+I=
 github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg=
-github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
 github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
 github.com/gorilla/css v1.0.0 h1:BQqNyPTi50JCFMTw/b67hByjMVXZRwGha6wxVGkeihY=
 github.com/gorilla/css v1.0.0/go.mod h1:Dn721qIggHpt4+EFCcTLTU/vk5ySda2ReITrtgBl60c=
@@ -373,6 +376,7 @@ github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGw
 github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
 github.com/jpillora/backoff v1.0.0 h1:uvFg412JmmHBHw7iwprIxkPMI+sGQ4kzOWsMeHnm2EA=
 github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4=
+github.com/json-iterator/go v1.1.5/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
 github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
@@ -383,7 +387,6 @@ github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfE
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
 github.com/jszwec/csvutil v1.2.1 h1:9+vmGqMdYxIbeDmVbTrVryibx2izwHAfKdPwl4GPNHM=
 github.com/jszwec/csvutil v1.2.1/go.mod h1:8YHz6C3KVdIeCxLMvwbbIVDCTA/Wi2df93AZlQNaE2U=
-github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
 github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/juju/errors v0.0.0-20181118221551-089d3ea4e4d5/go.mod h1:W54LbzXuIE0boCoNJfwqpmkKJ1O4TCTZMetAt6jGk7Q=
 github.com/juju/loggo v0.0.0-20180524022052-584905176618/go.mod h1:vgyd7OREkbtVEN/8IXZe5Ooef3LQePvuBm9UWj6ZL8U=
@@ -487,6 +490,8 @@ github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+
 github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
 github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
 github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
+github.com/opentracing/opentracing-go v1.2.1-0.20220228012449-10b1cf09e00b h1:FfH+VrHHk6Lxt9HdVS0PXzSXFyS2NbZKXv33FYPol0A=
+github.com/opentracing/opentracing-go v1.2.1-0.20220228012449-10b1cf09e00b/go.mod h1:AC62GU6hc0BrNm+9RK9VSiwa/EUe1bkIeFORAMcHvJU=
 github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
 github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
 github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
@@ -551,9 +556,7 @@ github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6Mwd
 github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
 github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
 github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
-github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
 github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
-github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
 github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
 github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
@@ -587,6 +590,10 @@ github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5Cc
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
 github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
+github.com/uber/jaeger-client-go v2.30.0+incompatible h1:D6wyKGCecFaSRUpo8lCVbaOOb6ThwMmTEbhRwtKR97o=
+github.com/uber/jaeger-client-go v2.30.0+incompatible/go.mod h1:WVhlPFC8FDjOFMMWRy2pZqQJSXxYSwNYOkTr/Z6d3Kk=
+github.com/uber/jaeger-lib v2.4.1+incompatible h1:td4jdvLcExb4cBISKIpHuGoVXh+dVKhn2Um6rjCsSsg=
+github.com/uber/jaeger-lib v2.4.1+incompatible/go.mod h1:ComeNDZlWwrWnDv8aPp0Ba6+uUTzImX/AauajbLI56U=
 github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
 github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
 github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
@@ -610,6 +617,7 @@ github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
 github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
+github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
 go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0=
 go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
 go.etcd.io/etcd/api/v3 v3.5.0/go.mod h1:cbVKeC6lCfl7j/8jBhAK6aIYO9XOjdptoxU/nLQcPvs=
@@ -624,6 +632,8 @@ go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
 go.opencensus.io v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M=
 go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E=
 go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
+go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
+go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
 go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
 go.uber.org/zap v1.17.0/go.mod h1:MXVU+bhUf/A7Xi2HNOnopQOrmycQ5Ih87HtOu4q5SSo=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
@@ -643,8 +653,8 @@ golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWP
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/crypto v0.0.0-20211215165025-cf75a172585e/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8=
-golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4 h1:kUhD7nTDoI3fVd9G4ORWrbV5NY0liEs/Jg2pv5f+bBA=
-golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/crypto v0.3.0 h1:a06MkbcxBrEFc0w0QIZWXrH/9cCX6KJyWbBOIwAn+7A=
+golang.org/x/crypto v0.3.0/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -683,8 +693,8 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=
-golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY=
+golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
+golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -732,8 +742,11 @@ golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qx
 golang.org/x/net v0.0.0-20210610132358-84b48f89b13b/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
-golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2 h1:CIJ76btIcR3eFI5EgSo6k1qKw9KJexJuRLI9G7Hp5wE=
 golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco=
+golang.org/x/net v0.2.0 h1:sZfSu1wtKLGlWI4ZZayP0ck9Y73K1ynO6gqzTdBVdPU=
+golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -758,8 +771,9 @@ golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
 golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 h1:uVc8UZUe6tr40fFVnUP5Oj+veunVezqYl9z7DYw9xzw=
+golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -821,11 +835,18 @@ golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220114195835-da31bd327af9 h1:XfKQ4OlFl8okEOr5UvAqFRVj8pY/4yfcXrddB8qAbU0=
 golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.2.0 h1:ljd4t30dBnAvMZaQCevtY0xLLD0A+bRZXbgLMLU1F/A=
+golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
-golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/term v0.2.0 h1:z85xZCsEl7bi/KwbNADeBYoOP0++7W1ipu+aGnpwzRM=
+golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -834,8 +855,9 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/text v0.4.0 h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg=
+golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -897,8 +919,8 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f
 golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
 golang.org/x/tools v0.1.2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
 golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo=
-golang.org/x/tools v0.1.10 h1:QjFRCZxdOhBJ/UNgnBZLbNV13DlbnK0quyivTnXJM20=
-golang.org/x/tools v0.1.10/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E=
+golang.org/x/tools v0.1.12 h1:VveCTK38A2rkS8ZqFY25HIDFscX5X9OoEhJd3quQmXU=
+golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -1024,8 +1046,10 @@ gopkg.in/go-playground/validator.v8 v8.18.2/go.mod h1:RX2a/7Ha8BgOhfk7j780h4/u/R
 gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
 gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
-gopkg.in/ini.v1 v1.62.0 h1:duBzk771uxoUuOlyRLkHsygud9+5lrlGjdFBb4mSKDU=
 gopkg.in/ini.v1 v1.62.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/ini.v1 v1.66.2/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
+gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
 gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

kubecost-exporter.md

@@ -1,134 +0,0 @@
-# Running Kubecost as a Prometheus metric exporter
-
-Running Kubecost as a Prometheus metric exporter allows you to export various cost metrics to Prometheus without setting up any other Kubecost dependencies. Doing so lets you write PromQL queries to calculate the cost and efficiency of any Kubernetes concept, e.g. namespace, service, label, deployment, etc. You can also calculate the cost of different Kubernetes resources, e.g. nodes, PVs, LoadBalancers, and more. Finally, you can do other interesting things like create custom alerts via AlertManager and custom dashboards via Grafana.
-
-## Installing
-
-> Note: all deployments of Kubecost function as a Prometheus metric exporter. We strongly recommend helm as an install path to take advantage of Kubecost’s full potential. [View recommended install](http://docs.kubecost.com/install).
-
-If you would prefer to not use the recommended install option and just deploy the Kubecost open source cost model as a metric exporter, you can follow these steps:
-
-
-1. Apply the combined YAML:
-
-    1.a.
-
-      ```
-      wget https://raw.githubusercontent.com/kubecost/cost-model/develop/kubernetes/exporter/exporter.yaml
-      ```
-
-    1.b.
-      On the line
-
-      ```
-      value: "{{prometheusEndpoint}}" # The endpoint should have the form http://<service-name>.<namespace-name>.svc
-      ```
-
-      of `exporter.yaml`, substitute your own Prometheus URI for `{{prometheusEndpoint}}`
-
-    1.c.
-
-      ```
-      kubectl apply -f exporter.yaml --namespace cost-model
-      ```
-
-    > If you want to use a namespace other than `cost-model`, you will have to edit the `ClusterRoleBinding` after applying the YAML to change `subjects[0].namespace`. You can do this with `kubectl edit clusterrolebinding cost-model`.
-
-2. To verify that metrics are available:
-
-    ```
-    kubectl port-forward --namespace cost-model service/cost-model 9003
-    ```
-
-    Visit [http://localhost:9003/metrics](http://localhost:9003/metrics) to see exported metrics
-
-Add Kubecost scrape config to Prom ([more info](https://prometheus.io/docs/introduction/first_steps/#configuring-prometheus))
-```
-- job_name: cost-model
-  scrape_interval: 1m
-  scrape_timeout: 10s
-  metrics_path: /metrics
-  scheme: http
-  static_configs:
-    - targets: ['cost-model.cost-model.:9003']
-```
-
-Done! Kubecost is now exporting cost metrics. See the following sections for different metrics available and query examples.
-
-## Available Prometheus Metrics
-
-| Metric       | Description                                                                                            |
-| ------------ | ------------------------------------------------------------------------------------------------------ |
-| node_cpu_hourly_cost | Hourly cost per vCPU on this node  |
-| node_gpu_hourly_cost | Hourly cost per GPU on this node  |
-| node_ram_hourly_cost   | Hourly cost per Gb of memory on this node                       |
-| node_total_hourly_cost   | Total node cost per hour                       |
-| kubecost_load_balancer_cost   | Hourly cost of a load balancer                 |
-| kubecost_cluster_management_cost | Hourly management fee per cluster                 |
-| container_cpu_allocation   | Average number of CPUs requested over last 1m                      |
-| container_memory_allocation_bytes   | Average bytes of RAM requested over last 1m                 |
-
-By default, all cost metrics are based on public billing APIs. See the Limitations section below about reflecting your precise billing information. Supported platforms are AWS, Azure, and GCP. For on-prem clusters, prices are based on configurable defaults.
-
-More metrics are available in the recommended install path and are described in [PROMETHEUS.md](PROMETHEUS.md).
-
-## Dashboard examples
-
-Here’s an example dashboard using Kubecost Prometheus metrics:
-
-![sample dashboard](https://grafana.com/api/dashboards/8670/images/5480/image)
-
-You can find other example dashboards at https://grafana.com/orgs/kubecost
-
-## Example Queries
-
-Once Kubecost’s cost model is running in your cluster and you have added it in your Prometheus scrape configuration, you can hit Prometheus with useful queries like these:
-
-#### Monthly cost of all nodes
-
-```
-sum(node_total_hourly_cost) * 730
-```
-
-#### Hourly cost of all load balancers broken down by namespace
-
-```
-sum(kubecost_load_balancer_cost) by (namespace)
-```
-
-#### Monthly rate of each namespace’s CPU request
-
-```
-sum(container_cpu_allocation * on (node) group_left node_cpu_hourly_cost) by (namespace) * 730
-```
-
-#### Historical memory request spend for all `fluentd` pods in the `kube-system` namespace
-
-```
-avg_over_time(container_memory_allocation_bytes{namespace="kube-system",pod=~"fluentd.*"}[1d])
-  * on (pod,node) group_left
-avg(count_over_time(container_memory_allocation_bytes{namespace="kube-system"}[1d:1m])/60) by (pod,node)
-  * on (node) group_left
-avg(avg_over_time(node_ram_hourly_cost[1d] )) by (node)
-```
-
-
-## Setting Cost Alerts
-
-Custom cost alerts can be implemented with a set of Prometheus queries and can be used for alerting with AlertManager or Grafana alerts. Below are example alerting rules.
-
-#### Determine in real-time if the monthly cost of all nodes is > $1000
-
-```
-sum(node_total_hourly_cost) * 730 > 1000
-```
-
-## Limitations
-
-Running Kubecost in exporter-only mode by definition limits functionality. The following limitations of this install method are addressed by the [recommended install path](http://docs.kubecost.com/install).
-
-- Persistent volume metrics not available (coming soon!)
-- For large clusters, these Prometheus queries might not scale well over large time windows. We recommend using [Kubecost APIs](https://github.com/kubecost/docs/blob/master/apis.md) for these scenarios.
-- Allocation metrics, like `container_cpu_allocation` only contain _requests_ and do not take usage into account.
-- Related to the previous point, efficiency metrics are not available.
-- Public billing costs on default. The standard Kubecost install and a cloud integration gives you accurate pricing based on your bill.

kubernetes/README.md

@@ -0,0 +1 @@
+<https://www.opencost.io/docs/>

kubernetes/cluster-role-binding.yaml

@@ -1,12 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: cost-model
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: cost-model
-subjects:
-  - kind: ServiceAccount
-    name: cost-model
-    namespace: cost-model

kubernetes/cluster-role.yaml

@@ -1,77 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: cost-model 
-rules:
-  - apiGroups:
-      - ''
-    resources:
-      - configmaps
-      - deployments
-      - nodes
-      - pods
-      - services
-      - resourcequotas
-      - replicationcontrollers
-      - limitranges
-      - persistentvolumeclaims
-      - persistentvolumes
-      - namespaces
-      - endpoints
-    verbs:
-      - get
-      - list
-      - watch
-  - apiGroups:
-      - extensions
-    resources:
-      - daemonsets
-      - deployments
-      - replicasets
-    verbs:
-      - get
-      - list
-      - watch
-  - apiGroups:
-      - apps
-    resources:
-      - statefulsets
-      - deployments
-      - daemonsets
-      - replicasets
-    verbs:
-      - list
-      - watch
-  - apiGroups:
-      - batch
-    resources:
-      - cronjobs
-      - jobs
-    verbs:
-      - get
-      - list
-      - watch
-  - apiGroups:
-      - autoscaling
-    resources:
-      - horizontalpodautoscalers
-    verbs:
-      - get
-      - list
-      - watch
-  - apiGroups:
-      - policy
-    resources:
-      - poddisruptionbudgets
-    verbs:
-      - get
-      - list
-      - watch
-  - apiGroups: 
-      - storage.k8s.io
-    resources: 
-      - storageclasses
-    verbs:
-      - get
-      - list
-      - watch

kubernetes/deployment.yaml

@@ -1,36 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: cost-model
-  labels:
-    app: cost-model
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: cost-model
-  strategy:
-    rollingUpdate:
-      maxSurge: 1
-      maxUnavailable: 1
-    type: RollingUpdate
-  template:
-    metadata:
-      labels:
-        app: cost-model
-    spec:
-      restartPolicy: Always
-      serviceAccountName: cost-model
-      containers:
-        - image: quay.io/kubecost1/kubecost-cost-model:latest
-          name: cost-model
-          resources:
-            requests:
-              cpu: "10m"
-              memory: "55M"
-          env:
-            - name: PROMETHEUS_SERVER_ENDPOINT
-              value: "{{prometheusEndpoint}}"  #The endpoint should have the form http://<service-name>.<namespace-name>.svc
-            - name: CLOUD_PROVIDER_API_KEY
-              value: "AIzaSyD29bGxmHAVEOBYtgd8sYM2gM2ekfxQX4U" # The GCP Pricing API requires a key. This is supplied just for evaluation.
-          imagePullPolicy: Always

kubernetes/exporter/README.md

@@ -1,11 +1 @@
-# Exporter Deployment
-
-This is the one YAML file that is the aggregation of the regular deployment files. This is done for easy distribution, allowing users to `kubectl apply` an exporter-only deployment without cloning the whole repository. Apply on the parent directory won't apply anything in this directory unless `kubectl apply --recursive=True` is used.
-
-## Usage
-
-Please be aware, you will have to change both the `Namespace` and `ClusterRoleBinding` resource if you want to deploy to a namespace other than `cost-model`.
-
-``` sh
-kubectl apply -f exporter.yaml --namespace cost-model
-```
+<https://www.opencost.io/docs/>

kubernetes/exporter/exporter.yaml → kubernetes/opencost.yaml

@@ -1,29 +1,26 @@
-# Based on the split YAML files, this is aggregated for convenience of deployment.
-
+# <https://www.opencost.io/docs/>
 ---
 
-# The namespace cost-model will run in
+# The namespace opencost will run in
 apiVersion: v1
 kind: Namespace
 metadata:
-    name: cost-model
-
+    name: opencost
 ---
 
 # Service account for permissions
 apiVersion: v1
 kind: ServiceAccount
 metadata:
-  name: cost-model
-
+  name: opencost
 ---
 
-# Cluster role so cost model can gather data about the cluster
-# No write permissions are allowed
+# Cluster role giving opencost to get, list, watch required recources
+# No write permissions are required
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
-  name: cost-model 
+  name: opencost
 rules:
   - apiGroups:
       - ''
@@ -89,9 +86,9 @@ rules:
       - get
       - list
       - watch
-  - apiGroups: 
+  - apiGroups:
       - storage.k8s.io
-    resources: 
+    resources:
       - storageclasses
     verbs:
       - get
@@ -104,33 +101,32 @@ rules:
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
-  name: cost-model
+  name: opencost
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
-  name: cost-model
+  name: opencost
 subjects:
   - kind: ServiceAccount
-    name: cost-model
-    namespace: cost-model
-
+    name: opencost
+    namespace: opencost
 ---
 
 # Create a deployment for a single cost model pod
-# 
+#
 # See environment variables if you would like to add a Prometheus for
 # cost model to read from for full functionality.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: cost-model
+  name: opencost
   labels:
-    app: cost-model
+    app: opencost
 spec:
   replicas: 1
   selector:
     matchLabels:
-      app: cost-model
+      app: opencost
   strategy:
     rollingUpdate:
       maxSurge: 1
@@ -139,44 +135,45 @@ spec:
   template:
     metadata:
       labels:
-        app: cost-model
+        app: opencost
     spec:
       restartPolicy: Always
-      serviceAccountName: cost-model
+      serviceAccountName: opencost
       containers:
         - image: quay.io/kubecost1/kubecost-cost-model:latest
-          name: cost-model
+          name: opencost
           resources:
             requests:
               cpu: "10m"
               memory: "55M"
+            limits:
+              cpu: "999m"
+              memory: "1G"
           env:
             - name: PROMETHEUS_SERVER_ENDPOINT
-              value: "{{prometheusEndpoint}}" # The endpoint should have the form http://<service-name>.<namespace-name>.svc
+              value: "http://my-prometheus-server.prometheus.svc" # The endpoint should have the form http://<service-name>.<namespace-name>.svc
             - name: CLOUD_PROVIDER_API_KEY
               value: "AIzaSyD29bGxmHAVEOBYtgd8sYM2gM2ekfxQX4U" # The GCP Pricing API requires a key. This is supplied just for evaluation.
             - name: CLUSTER_ID
               value: "cluster-one" # Default cluster ID to use if cluster_id is not set in Prometheus metrics.
           imagePullPolicy: Always
-
 ---
 
 # Expose the cost model with a service
-# 
+#
 # Without a Prometheus endpoint configured in the deployment,
-# only cost-model/metrics will have useful data as it is intended
+# only opencost/metrics will have useful data as it is intended
 # to be used as just an exporter.
 kind: Service
 apiVersion: v1
 metadata:
-  name: cost-model
+  name: opencost
 spec:
   selector:
-    app: cost-model
+    app: opencost
   type: ClusterIP
   ports:
-    - name: cost-model
+    - name: opencost
       port: 9003
       targetPort: 9003
-
 ---

kubernetes/prometheus/extraScrapeConfigs.yaml

@@ -0,0 +1,12 @@
+extraScrapeConfigs: |
+  - job_name: opencost
+    honor_labels: true
+    scrape_interval: 1m
+    scrape_timeout: 10s
+    metrics_path: /metrics
+    scheme: http
+    dns_sd_configs:
+    - names:
+      - opencost.opencost
+      type: 'A'
+      port: 9003

kubernetes/service-account.yaml

@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: cost-model

kubernetes/service.yaml

@@ -1,12 +0,0 @@
-kind: Service
-apiVersion: v1
-metadata:
-  name: cost-model
-spec:
-  selector:
-    app: cost-model
-  type: ClusterIP
-  ports:
-    - name: cost-model
-      port: 9003
-      targetPort: 9003

pkg/cloud/aliyunprovider.go

@@ -0,0 +1,1298 @@
+package cloud
+
+import (
+	"errors"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"regexp"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/aliyun/alibaba-cloud-sdk-go/sdk"
+	"github.com/aliyun/alibaba-cloud-sdk-go/sdk/auth/credentials"
+	"github.com/aliyun/alibaba-cloud-sdk-go/sdk/auth/signers"
+	"github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
+	"github.com/opencost/opencost/pkg/clustercache"
+	"github.com/opencost/opencost/pkg/env"
+	"github.com/opencost/opencost/pkg/kubecost"
+	"github.com/opencost/opencost/pkg/log"
+	"github.com/opencost/opencost/pkg/util/fileutil"
+	"github.com/opencost/opencost/pkg/util/json"
+	"github.com/opencost/opencost/pkg/util/stringutil"
+	"golang.org/x/exp/slices"
+	v1 "k8s.io/api/core/v1"
+)
+
+const (
+	ALIBABA_ECS_PRODUCT_CODE                   = "ecs"
+	ALIBABA_ECS_VERSION                        = "2014-05-26"
+	ALIBABA_ECS_DOMAIN                         = "ecs.aliyuncs.com"
+	ALIBABA_DESCRIBE_PRICE_API_ACTION          = "DescribePrice"
+	ALIBABA_DESCRIBE_DISK_API_ACTION           = "DescribeDisks"
+	ALIBABA_INSTANCE_RESOURCE_TYPE             = "instance"
+	ALIBABA_DISK_RESOURCE_TYPE                 = "disk"
+	ALIBABA_PAY_AS_YOU_GO_BILLING              = "Pay-As-You-Go"
+	ALIBABA_SUBSCRIPTION_BILLING               = "Subscription"
+	ALIBABA_PREEMPTIBLE_BILLING                = "Preemptible"
+	ALIBABA_OPTIMIZE_KEYWORD                   = "optimize"
+	ALIBABA_NON_OPTIMIZE_KEYWORD               = "nonoptimize"
+	ALIBABA_HOUR_PRICE_UNIT                    = "Hour"
+	ALIBABA_MONTH_PRICE_UNIT                   = "Month"
+	ALIBABA_YEAR_PRICE_UNIT                    = "Year"
+	ALIBABA_UNKNOWN_INSTANCE_FAMILY_TYPE       = "unknown"
+	ALIBABA_NOT_SUPPORTED_INSTANCE_FAMILY_TYPE = "unsupported"
+	ALIBABA_DISK_CLOUD_ESSD_CATEGORY           = "cloud_essd"
+	ALIBABA_DISK_CLOUD_CATEGORY                = "cloud"
+	ALIBABA_DATA_DISK_CATEGORY                 = "data"
+	ALIBABA_SYSTEM_DISK_CATEGORY               = "system"
+	ALIBABA_DATA_DISK_PREFIX                   = "DataDisk"
+	ALIBABA_PV_CLOUD_DISK_TYPE                 = "CloudDisk"
+	ALIBABA_PV_NAS_TYPE                        = "NAS"
+	ALIBABA_PV_OSS_TYPE                        = "OSS"
+	ALIBABA_DEFAULT_DATADISK_SIZE              = "2000"
+	ALIBABA_DISK_TOPOLOGY_REGION_LABEL         = "topology.diskplugin.csi.alibabacloud.com/region"
+	ALIBABA_DISK_TOPOLOGY_ZONE_LABEL           = "topology.diskplugin.csi.alibabacloud.com/zone"
+)
+
+var (
+	// Regular expression to get the numerical value of PV suffix with GiB from *v1.PersistentVolume.
+	sizeRegEx = regexp.MustCompile("(.*?)Gi")
+)
+
+// Variable to keep track of instance families that fail in DescribePrice API due improper defaulting of systemDisk if the information is not available
+var alibabaDefaultToCloudEssd = []string{"g6e", "r6e", "r7", "g7", "g7a", "r7a"}
+
+// Why predefined and dependency on code? Can be converted to API call - https://www.alibabacloud.com/help/en/elastic-compute-service/latest/regions-describeregions
+var alibabaRegions = []string{
+	"cn-qingdao",
+	"cn-beijing",
+	"cn-zhangjiakou",
+	"cn-huhehaote",
+	"cn-wulanchabu",
+	"cn-hangzhou",
+	"cn-shanghai",
+	"cn-nanjing",
+	"cn-fuzhou",
+	"cn-shenzhen",
+	"cn-guangzhou",
+	"cn-chengdu",
+	"cn-hongkong",
+	"ap-southeast-1",
+	"ap-southeast-2",
+	"ap-southeast-3",
+	"ap-southeast-5",
+	"ap-southeast-6",
+	"ap-southeast-7",
+	"ap-south-1",
+	"ap-northeast-1",
+	"ap-northeast-2",
+	"us-west-1",
+	"us-east-1",
+	"eu-central-1",
+	"me-east-1",
+}
+
+// To-Do: Convert to API call - https://www.alibabacloud.com/help/en/elastic-compute-service/latest/describeinstancetypefamilies
+// Also first pass only completely tested pricing API for General pupose instances families & memory optimized instance families
+var alibabaInstanceFamilies = []string{
+	"g7",
+	"g7a",
+	"g6e",
+	"g6",
+	"g5",
+	"sn2",
+	"sn2ne",
+	"r7",
+	"r7a",
+	"r6e",
+	"r6a",
+	"r6",
+	"r5",
+	"se1",
+	"se1ne",
+	"re6",
+	"re6p",
+	"re4",
+	"se1",
+}
+
+// AlibabaAccessKey holds Alibaba credentials parsing from the service-key.json file.
+type AlibabaAccessKey struct {
+	AccessKeyID     string `json:"alibaba_access_key_id"`
+	SecretAccessKey string `json:"alibaba_secret_access_key"`
+}
+
+// Slim Version of k8s disk assigned to a node or PV.
+type SlimK8sDisk struct {
+	DiskType         string
+	RegionID         string
+	PriceUnit        string
+	SizeInGiB        string
+	DiskCategory     string
+	PerformanceLevel string
+	ProviderID       string
+	StorageClass     string
+}
+
+func NewSlimK8sDisk(diskType, regionID, priceUnit, diskCategory, performanceLevel, providerID, storageClass, sizeInGiB string) *SlimK8sDisk {
+	return &SlimK8sDisk{
+		DiskType:         diskType,
+		RegionID:         regionID,
+		PriceUnit:        priceUnit,
+		SizeInGiB:        sizeInGiB,
+		DiskCategory:     diskCategory,
+		PerformanceLevel: performanceLevel,
+		ProviderID:       providerID,
+		StorageClass:     storageClass,
+	}
+}
+
+// Slim version of a k8s v1.node just to pass along the object of this struct instead of constant getting the labels from within v1.Node & unit testing.
+type SlimK8sNode struct {
+	InstanceType       string
+	RegionID           string
+	PriceUnit          string
+	MemorySizeInKiB    string // TO-DO : Possible to convert to float?
+	IsIoOptimized      bool
+	OSType             string
+	ProviderID         string
+	SystemDisk         *SlimK8sDisk
+	InstanceTypeFamily string // Bug in DescribePrice, doesn't default to enhanced type correctly and you get an error in DescribePrice to get around need the family of the InstanceType.
+}
+
+func NewSlimK8sNode(instanceType, regionID, priceUnit, memorySizeInKiB, osType, providerID, instanceTypeFamily string, isIOOptimized bool, systemDiskInfo *SlimK8sDisk) *SlimK8sNode {
+	return &SlimK8sNode{
+		InstanceType:       instanceType,
+		RegionID:           regionID,
+		PriceUnit:          priceUnit,
+		MemorySizeInKiB:    memorySizeInKiB,
+		IsIoOptimized:      isIOOptimized,
+		OSType:             osType,
+		SystemDisk:         systemDiskInfo,
+		ProviderID:         providerID,
+		InstanceTypeFamily: instanceTypeFamily,
+	}
+}
+
+// AlibabaNodeAttributes represents metadata about the Node in its pricing information.
+// Basic Attributes needed atleast to get the key, Some attributes from k8s Node response
+// be populated directly into *Node object.
+type AlibabaNodeAttributes struct {
+	// InstanceType represents the type of instance.
+	InstanceType string `json:"instanceType"`
+	// MemorySizeInKiB represents the size of memory of instance.
+	MemorySizeInKiB string `json:"memorySizeInKiB"`
+	// IsIoOptimized represents the if instance is I/O optimized.
+	IsIoOptimized bool `json:"isIoOptimized"`
+	// OSType represents the OS installed in the Instance.
+	OSType string `json:"osType"`
+	// SystemDiskCategory represents the exact category of the system disk attached to the node.
+	SystemDiskCategory string `json:"systemDiskCategory"`
+	// SystemDiskSizeInGiB represents the size of the system disk attached to the node.
+	SystemDiskSizeInGiB string `json:"systemDiskSizeInGiB"`
+	// SystemDiskPerformanceLevel represents the performance level of the system disk attached to the node.
+	SystemDiskPerformanceLevel string `json:"systemPerformanceLevel"`
+}
+
+func NewAlibabaNodeAttributes(node *SlimK8sNode) *AlibabaNodeAttributes {
+	if node == nil {
+		return nil
+	}
+	var diskCategory, sizeInGiB, performanceLevel string
+	if node.SystemDisk != nil {
+		diskCategory = node.SystemDisk.DiskCategory
+		sizeInGiB = node.SystemDisk.SizeInGiB
+		performanceLevel = node.SystemDisk.PerformanceLevel
+	}
+	return &AlibabaNodeAttributes{
+		InstanceType:               node.InstanceType,
+		MemorySizeInKiB:            node.MemorySizeInKiB,
+		IsIoOptimized:              node.IsIoOptimized,
+		OSType:                     node.OSType,
+		SystemDiskCategory:         diskCategory,
+		SystemDiskSizeInGiB:        sizeInGiB,
+		SystemDiskPerformanceLevel: performanceLevel,
+	}
+}
+
+// AlibabaPVAttributes represents metadata the PV in its pricing information.
+// Basic Attributes needed atleast to get the keys. Some attributes from k8s PV response
+// be populated directly into *PV object.
+type AlibabaPVAttributes struct {
+	// PVType can be Cloud Disk, NetWork Attached Storage(NAS) or Object Storage Service (OSS).
+	// Represents the way the PV was attached
+	PVType string `json:"pvType"`
+	// PVSubType represent the sub category of PVType. This is Data in case of Cloud Disk.
+	PVSubType string `json:"pvSubType"`
+	// Example for PVCategory with cloudDisk PVType are cloud, cloud_efficiency, cloud_ssd,
+	// ephemeral_ssd and cloud_essd. If not present returns empty.
+	PVCategory string `json:"pvCategory"`
+	// Example for PerformanceLevel with cloudDisk PVType are PL0,PL1,PL2 &PL3. If not present returns empty.
+	PVPerformanceLevel string `json:"performanceLevel"`
+	// The Size of the PV in terms of GiB
+	SizeInGiB string `json:"sizeInGiB"`
+}
+
+// TO-Do: next iteration of Alibaba provider support NetWork Attached Storage(NAS) and Object Storage Service (OSS type PVs).
+// Currently defaulting to cloudDisk with provision to add work in future.
+func NewAlibabaPVAttributes(disk *SlimK8sDisk) *AlibabaPVAttributes {
+	if disk == nil {
+		return nil
+	}
+	return &AlibabaPVAttributes{
+		PVType:             ALIBABA_PV_CLOUD_DISK_TYPE,
+		PVSubType:          disk.DiskType,
+		PVCategory:         disk.DiskCategory,
+		PVPerformanceLevel: disk.PerformanceLevel,
+		SizeInGiB:          disk.SizeInGiB,
+	}
+}
+
+// Stage 1 support will be Pay-As-You-Go with HourlyPrice equal to TradePrice with PriceUnit as Hour
+// TO-DO: Subscription and Premptible support, Information can be gathered from describing instance for subscription type
+// and spotprice can be gather from DescribeSpotPriceHistory API.
+// TO-DO: how would you calculate hourly price for subscription type, is it PRICE_YEARLY/HOURS_IN_THE_YEAR|MONTH?
+type AlibabaPricingDetails struct {
+	// Represents hourly price for the given Alibaba cloud Product.
+	HourlyPrice float32 `json:"hourlyPrice"`
+	// Represents the unit in which Alibaba Product is billed can be Hour, Month or Year based on the billingMethod.
+	PriceUnit string `json:"priceUnit"`
+	// Original Price paid to acquire the Alibaba Product.
+	TradePrice float32 `json:"tradePrice"`
+	// Represents the currency unit of the price for billing Alibaba Product.
+	CurrencyCode string `json:"currencyCode"`
+}
+
+func NewAlibabaPricingDetails(hourlyPrice float32, priceUnit string, tradePrice float32, currencyCode string) *AlibabaPricingDetails {
+	return &AlibabaPricingDetails{
+		HourlyPrice:  hourlyPrice,
+		PriceUnit:    priceUnit,
+		TradePrice:   tradePrice,
+		CurrencyCode: currencyCode,
+	}
+}
+
+// AlibabaPricingTerms can have three types of supported billing method Pay-As-You-Go, Subscription and Premptible
+type AlibabaPricingTerms struct {
+	BillingMethod  string                 `json:"billingMethod"`
+	PricingDetails *AlibabaPricingDetails `json:"pricingDetails"`
+}
+
+func NewAlibabaPricingTerms(billingMethod string, pricingDetails *AlibabaPricingDetails) *AlibabaPricingTerms {
+	return &AlibabaPricingTerms{
+		BillingMethod:  billingMethod,
+		PricingDetails: pricingDetails,
+	}
+}
+
+// Alibaba Pricing struct carry the Attributes and pricing information for Node or PV
+type AlibabaPricing struct {
+	NodeAttributes *AlibabaNodeAttributes
+	PVAttributes   *AlibabaPVAttributes
+	PricingTerms   *AlibabaPricingTerms
+	Node           *Node
+	PV             *PV
+}
+
+// Alibaba cloud's Provider struct
+type Alibaba struct {
+	// Data to store Alibaba cloud's pricing struct, key in the map represents exact match to
+	// node.features() or pv.features for easy lookup
+	Pricing map[string]*AlibabaPricing
+	// Lock Needed to provide thread safe
+	DownloadPricingDataLock sync.RWMutex
+	Clientset               clustercache.ClusterCache
+	Config                  *ProviderConfig
+	*CustomProvider
+
+	// The following fields are unexported because of avoiding any leak of secrets of these keys.
+	// Alibaba Access key used specifically in signer interface used to sign API calls
+	serviceAccountChecks *ServiceAccountChecks
+	clusterAccountId     string
+	clusterRegion        string
+	accessKey            *credentials.AccessKeyCredential
+	// Map of regionID to sdk.client to call API for that region
+	clients map[string]*sdk.Client
+}
+
+// GetAlibabaAccessKey return the Access Key used to interact with the Alibaba cloud, if not set it
+// set it first by looking at env variables else load it from secret files.
+func (alibaba *Alibaba) GetAlibabaAccessKey() (*credentials.AccessKeyCredential, error) {
+	if alibaba.accessKeyisLoaded() {
+		return alibaba.accessKey, nil
+	}
+
+	config, err := alibaba.GetConfig()
+	if err != nil {
+		return nil, fmt.Errorf("error getting the default config for Alibaba Cloud provider: %w", err)
+	}
+
+	if config.AlibabaServiceKeyName == "" {
+		config.AlibabaServiceKeyName = env.GetAlibabaAccessKeyID()
+	}
+	if config.AlibabaServiceKeySecret == "" {
+		config.AlibabaServiceKeySecret = env.GetAlibabaAccessKeySecret()
+	}
+
+	if config.AlibabaServiceKeyName == "" && config.AlibabaServiceKeySecret == "" {
+		log.Debugf("missing service key values for Alibaba cloud integration attempting to use service account integration")
+		err := alibaba.loadAlibabaAuthSecretAndSetEnv(true)
+		if err != nil {
+			return nil, fmt.Errorf("unable to set the Alibaba Cloud key/secret from config file %w", err)
+		}
+		config.AlibabaServiceKeyName = env.GetAlibabaAccessKeyID()
+		config.AlibabaServiceKeySecret = env.GetAlibabaAccessKeySecret()
+	}
+
+	if config.AlibabaServiceKeyName == "" && config.AlibabaServiceKeySecret == "" {
+		return nil, fmt.Errorf("failed to get the access key for the current alibaba account")
+	}
+
+	alibaba.accessKey = &credentials.AccessKeyCredential{AccessKeyId: env.GetAlibabaAccessKeyID(), AccessKeySecret: env.GetAlibabaAccessKeySecret()}
+
+	return alibaba.accessKey, nil
+}
+
+// DownloadPricingData satisfies the provider interface and downloads the prices for Node instances and PVs.
+func (alibaba *Alibaba) DownloadPricingData() error {
+	alibaba.DownloadPricingDataLock.Lock()
+	defer alibaba.DownloadPricingDataLock.Unlock()
+
+	var aak *credentials.AccessKeyCredential
+	var err error
+
+	if !alibaba.accessKeyisLoaded() {
+		aak, err = alibaba.GetAlibabaAccessKey()
+		if err != nil {
+			return fmt.Errorf("unable to get the access key information: %w", err)
+		}
+	} else {
+		aak = alibaba.accessKey
+	}
+
+	c, err := alibaba.Config.GetCustomPricingData()
+	if err != nil {
+		return fmt.Errorf("error downloading default pricing data: %w", err)
+	}
+
+	// Get all the nodes from Alibaba cluster.
+	nodeList := alibaba.Clientset.GetAllNodes()
+
+	var client *sdk.Client
+	var signer *signers.AccessKeySigner
+	var ok bool
+	var lookupKey string
+	alibaba.clients = make(map[string]*sdk.Client)
+	alibaba.Pricing = make(map[string]*AlibabaPricing)
+
+	for _, node := range nodeList {
+		pricingObj := &AlibabaPricing{}
+		slimK8sNode := generateSlimK8sNodeFromV1Node(node)
+
+		if client, ok = alibaba.clients[slimK8sNode.RegionID]; !ok {
+			client, err = sdk.NewClientWithAccessKey(slimK8sNode.RegionID, aak.AccessKeyId, aak.AccessKeySecret)
+			if err != nil {
+				return fmt.Errorf("unable to initiate alibaba cloud sdk client for region %s : %w", slimK8sNode.RegionID, err)
+			}
+			alibaba.clients[slimK8sNode.RegionID] = client
+		}
+		signer = signers.NewAccessKeySigner(aak)
+
+		// Adjust the system Disk information of a Node by retrieving the details of associated disk. If unable to retrieve set it to empty
+		// system disk to pass through and use defaults with Alibaba pricing API.
+		instanceID := getInstanceIDFromProviderID(slimK8sNode.ProviderID)
+		slimK8sNode.SystemDisk = getSystemDiskInfoOfANode(instanceID, slimK8sNode.RegionID, client, signer)
+
+		lookupKey, err = determineKeyForPricing(slimK8sNode)
+		if _, ok := alibaba.Pricing[lookupKey]; ok {
+			log.Debugf("Pricing information for node with same features %s already exists hence skipping", lookupKey)
+			continue
+		}
+
+		pricingObj, err = processDescribePriceAndCreateAlibabaPricing(client, slimK8sNode, signer, c)
+
+		if err != nil {
+			return fmt.Errorf("failed to create pricing information for node with type %s with error: %w", slimK8sNode.InstanceType, err)
+		}
+		alibaba.Pricing[lookupKey] = pricingObj
+	}
+
+	// set the first occurance of region from the node
+	if alibaba.clusterRegion == "" {
+		for _, node := range nodeList {
+			if regionID, ok := node.Labels["topology.kubernetes.io/region"]; ok {
+				alibaba.clusterRegion = regionID
+				break
+			}
+		}
+	}
+
+	// PV pricing for only Cloud Disk for now.
+	// TO-DO: Support both NAS(Network Attached storage) and OSS(Object Storage Service) type PVs
+
+	pvList := alibaba.Clientset.GetAllPersistentVolumes()
+
+	for _, pv := range pvList {
+		pvRegion := determinePVRegion(pv)
+		if pvRegion == "" {
+			pvRegion = alibaba.clusterRegion
+		}
+		pricingObj := &AlibabaPricing{}
+		slimK8sDisk := generateSlimK8sDiskFromV1PV(pv, pvRegion)
+		lookupKey, err = determineKeyForPricing(slimK8sDisk)
+		if _, ok := alibaba.Pricing[lookupKey]; ok {
+			log.Debugf("Pricing information for pv with same features %s already exists hence skipping", lookupKey)
+			continue
+		}
+		if client, ok = alibaba.clients[slimK8sDisk.RegionID]; !ok {
+			client, err = sdk.NewClientWithAccessKey(slimK8sDisk.RegionID, aak.AccessKeyId, aak.AccessKeySecret)
+			if err != nil {
+				return fmt.Errorf("unable to initiate alibaba cloud sdk client for region %s : %w", slimK8sDisk.RegionID, err)
+			}
+			alibaba.clients[slimK8sDisk.RegionID] = client
+		}
+		signer = signers.NewAccessKeySigner(aak)
+		pricingObj, err = processDescribePriceAndCreateAlibabaPricing(client, slimK8sDisk, signer, c)
+		if err != nil {
+			return fmt.Errorf("failed to create pricing information for pv with category %s with error: %w", slimK8sDisk.DiskCategory, err)
+		}
+		alibaba.Pricing[lookupKey] = pricingObj
+	}
+
+	return nil
+}
+
+// AllNodePricing returns all the pricing data for all nodes and pvs
+func (alibaba *Alibaba) AllNodePricing() (interface{}, error) {
+	alibaba.DownloadPricingDataLock.RLock()
+	defer alibaba.DownloadPricingDataLock.RUnlock()
+	return alibaba.Pricing, nil
+}
+
+// NodePricing gives pricing information of a specific node given by the key
+func (alibaba *Alibaba) NodePricing(key Key) (*Node, error) {
+	alibaba.DownloadPricingDataLock.RLock()
+	defer alibaba.DownloadPricingDataLock.RUnlock()
+
+	// Get node features for the key
+	keyFeature := key.Features()
+
+	pricing, ok := alibaba.Pricing[keyFeature]
+	if !ok {
+		log.Errorf("Node pricing information not found for node with feature: %s", keyFeature)
+		return nil, fmt.Errorf("Node pricing information not found for node with feature: %s letting it use default values", keyFeature)
+	}
+
+	log.Debugf("returning the node price for the node with feature: %s", keyFeature)
+	returnNode := pricing.Node
+
+	return returnNode, nil
+}
+
+// PVPricing gives a pricing information of a specific PV given by PVkey
+func (alibaba *Alibaba) PVPricing(pvk PVKey) (*PV, error) {
+	alibaba.DownloadPricingDataLock.RLock()
+	defer alibaba.DownloadPricingDataLock.RUnlock()
+
+	keyFeature := pvk.Features()
+
+	pricing, ok := alibaba.Pricing[keyFeature]
+
+	if !ok {
+		log.Errorf("Persistent Volume pricing not found for PV with feature: %s", keyFeature)
+		return nil, fmt.Errorf("Persistent Volume pricing not found for PV with feature: %s letting it use default values", keyFeature)
+	}
+
+	log.Debugf("returning the PV price for the node with feature: %s", keyFeature)
+	return pricing.PV, nil
+}
+
+// Stubbed NetworkPricing for Alibaba Cloud. Will look at this in Next PR
+func (alibaba *Alibaba) NetworkPricing() (*Network, error) {
+	return &Network{
+		ZoneNetworkEgressCost:     0.0,
+		RegionNetworkEgressCost:   0.0,
+		InternetNetworkEgressCost: 0.0,
+	}, nil
+}
+
+// Stubbed LoadBalancerPricing for Alibaba Cloud. Will look at this in Next PR
+func (alibaba *Alibaba) LoadBalancerPricing() (*LoadBalancer, error) {
+	return &LoadBalancer{
+		Cost: 0.0,
+	}, nil
+}
+
+func (alibaba *Alibaba) GetConfig() (*CustomPricing, error) {
+	c, err := alibaba.Config.GetCustomPricingData()
+	if err != nil {
+		return nil, err
+	}
+	if c.Discount == "" {
+		c.Discount = "0%"
+	}
+	if c.NegotiatedDiscount == "" {
+		c.NegotiatedDiscount = "0%"
+	}
+	if c.ShareTenancyCosts == "" {
+		c.ShareTenancyCosts = defaultShareTenancyCost
+	}
+
+	return c, nil
+}
+
+// Load once and cache the result (even on failure). This is an install time secret, so
+// we don't expect the secret to change. If it does, however, we can force reload using
+// the input parameter.
+func (alibaba *Alibaba) loadAlibabaAuthSecretAndSetEnv(force bool) error {
+	if !force && alibaba.accessKeyisLoaded() {
+		return nil
+	}
+
+	exists, err := fileutil.FileExists(authSecretPath)
+	if !exists || err != nil {
+		return fmt.Errorf("failed to locate service account file: %s with err: %w", authSecretPath, err)
+	}
+
+	result, err := ioutil.ReadFile(authSecretPath)
+	if err != nil {
+		return fmt.Errorf("failed to read service account file: %s with err: %w", authSecretPath, err)
+	}
+
+	var ak *AlibabaAccessKey
+	err = json.Unmarshal(result, &ak)
+	if err != nil {
+		return fmt.Errorf("failed to unmarshall access key id and access key secret with err: %w", err)
+	}
+
+	err = env.Set(env.AlibabaAccessKeyIDEnvVar, ak.AccessKeyID)
+	if err != nil {
+		return fmt.Errorf("failed to set environment variable: %s with err: %w", env.AlibabaAccessKeyIDEnvVar, err)
+	}
+	err = env.Set(env.AlibabaAccessKeySecretEnvVar, ak.SecretAccessKey)
+	if err != nil {
+		return fmt.Errorf("failed to set environment variable: %s with err: %w", env.AlibabaAccessKeySecretEnvVar, err)
+	}
+
+	alibaba.accessKey = &credentials.AccessKeyCredential{
+		AccessKeyId:     ak.AccessKeyID,
+		AccessKeySecret: ak.SecretAccessKey,
+	}
+	return nil
+}
+
+// Regions returns a current supported list of Alibaba regions
+func (alibaba *Alibaba) Regions() []string {
+	return alibabaRegions
+}
+
+// ClusterInfo returns information about Alibaba Cloud cluster, as provided by metadata.
+func (alibaba *Alibaba) ClusterInfo() (map[string]string, error) {
+
+	c, err := alibaba.GetConfig()
+	if err != nil {
+		return nil, fmt.Errorf("failed to getConfig with err: %w", err)
+	}
+
+	var clusterName string
+	if c.ClusterName != "" {
+		clusterName = c.ClusterName
+	}
+
+	// Set it to environment clusterID if not set at this point
+	if clusterName == "" {
+		clusterName = env.GetClusterID()
+	}
+
+	m := make(map[string]string)
+	m["name"] = clusterName
+	m["provider"] = kubecost.AlibabaProvider
+	m["project"] = alibaba.clusterAccountId
+	m["region"] = alibaba.clusterRegion
+	m["id"] = env.GetClusterID()
+	return m, nil
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) GetAddresses() ([]byte, error) {
+	return nil, nil
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) GetDisks() ([]byte, error) {
+	return nil, nil
+}
+
+func (alibaba *Alibaba) GetOrphanedResources() ([]OrphanedResource, error) {
+	return nil, errors.New("not implemented")
+}
+
+func (alibaba *Alibaba) UpdateConfig(r io.Reader, updateType string) (*CustomPricing, error) {
+	return alibaba.Config.Update(func(c *CustomPricing) error {
+		if updateType != "" {
+			return fmt.Errorf("UpdateConfig for Alibaba Provider doesn't support updateType %s at this time", updateType)
+
+		} else {
+			a := make(map[string]interface{})
+			err := json.NewDecoder(r).Decode(&a)
+			if err != nil {
+				return err
+			}
+			for k, v := range a {
+				kUpper := strings.Title(k) // Just so we consistently supply / receive the same values, uppercase the first letter.
+				vstr, ok := v.(string)
+				if ok {
+					err := SetCustomPricingField(c, kUpper, vstr)
+					if err != nil {
+						return err
+					}
+				} else {
+					return fmt.Errorf("type error while updating config for %s", kUpper)
+				}
+			}
+		}
+
+		if env.IsRemoteEnabled() {
+			err := UpdateClusterMeta(env.GetClusterID(), c.ClusterName)
+			if err != nil {
+				return err
+			}
+		}
+		return nil
+	})
+}
+
+func (alibaba *Alibaba) UpdateConfigFromConfigMap(cm map[string]string) (*CustomPricing, error) {
+	return alibaba.Config.UpdateFromMap(cm)
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) GetManagementPlatform() (string, error) {
+	return "", nil
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) GetLocalStorageQuery(window, offset time.Duration, rate bool, used bool) string {
+	return ""
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) ApplyReservedInstancePricing(nodes map[string]*Node) {
+
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) ServiceAccountStatus() *ServiceAccountStatus {
+	return &ServiceAccountStatus{}
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) PricingSourceStatus() map[string]*PricingSource {
+	return map[string]*PricingSource{}
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) ClusterManagementPricing() (string, float64, error) {
+	return "", 0.0, nil
+}
+
+// Will look at this in Next PR if needed
+func (alibaba *Alibaba) CombinedDiscountForNode(string, bool, float64, float64) float64 {
+	return 0.0
+}
+
+func (alibaba *Alibaba) accessKeyisLoaded() bool {
+	return alibaba.accessKey != nil
+}
+
+type AlibabaNodeKey struct {
+	ProviderID                 string
+	RegionID                   string
+	InstanceType               string
+	OSType                     string
+	OptimizedKeyword           string //If IsIoOptimized is true use the word optimize in the Node key and if its not optimized use the word nonoptimize
+	SystemDiskCategory         string
+	SystemDiskSizeInGiB        string
+	SystemDiskPerformanceLevel string
+}
+
+func NewAlibabaNodeKey(node *SlimK8sNode, optimizedKeyword, systemDiskCategory, systemDiskSizeInGiB, systemDiskPerfromanceLevel string) *AlibabaNodeKey {
+	var providerID, regionID, instanceType, osType string
+	if node != nil {
+		providerID = node.ProviderID
+		regionID = node.RegionID
+		instanceType = node.InstanceType
+		osType = node.OSType
+	}
+	return &AlibabaNodeKey{
+		ProviderID:                 providerID,
+		RegionID:                   regionID,
+		InstanceType:               instanceType,
+		OSType:                     osType,
+		OptimizedKeyword:           optimizedKeyword,
+		SystemDiskCategory:         systemDiskCategory,
+		SystemDiskSizeInGiB:        systemDiskSizeInGiB,
+		SystemDiskPerformanceLevel: systemDiskPerfromanceLevel,
+	}
+}
+
+func (alibabaNodeKey *AlibabaNodeKey) ID() string {
+	return alibabaNodeKey.ProviderID
+}
+
+func (alibabaNodeKey *AlibabaNodeKey) Features() string {
+	keyLookup := stringutil.DeleteEmptyStringsFromArray([]string{alibabaNodeKey.RegionID, alibabaNodeKey.InstanceType, alibabaNodeKey.OSType,
+		alibabaNodeKey.OptimizedKeyword, alibabaNodeKey.SystemDiskCategory, alibabaNodeKey.SystemDiskSizeInGiB, alibabaNodeKey.SystemDiskPerformanceLevel})
+	return strings.Join(keyLookup, "::")
+}
+
+func (alibabaNodeKey *AlibabaNodeKey) GPUType() string {
+	return ""
+}
+
+func (alibabaNodeKey *AlibabaNodeKey) GPUCount() int {
+	return 0
+}
+
+// Get's the key for the k8s node input
+func (alibaba *Alibaba) GetKey(mapValue map[string]string, node *v1.Node) Key {
+	slimK8sNode := generateSlimK8sNodeFromV1Node(node)
+
+	var aak *credentials.AccessKeyCredential
+	var err error
+	var ok bool
+	var client *sdk.Client
+	var signer *signers.AccessKeySigner
+
+	optimizedKeyword := ""
+	if slimK8sNode.IsIoOptimized {
+		optimizedKeyword = ALIBABA_OPTIMIZE_KEYWORD
+	} else {
+		optimizedKeyword = ALIBABA_NON_OPTIMIZE_KEYWORD
+	}
+
+	var diskCategory, diskSizeInGiB, diskPerformanceLevel string
+
+	if !alibaba.accessKeyisLoaded() {
+		aak, err = alibaba.GetAlibabaAccessKey()
+		if err != nil {
+			log.Warnf("unable to set the signer for node with providerID %s to retrieve the key skipping SystemDisk Retrieval with err: %v", slimK8sNode.ProviderID, err)
+			return NewAlibabaNodeKey(slimK8sNode, optimizedKeyword, diskCategory, diskSizeInGiB, diskPerformanceLevel)
+		}
+	} else {
+		aak = alibaba.accessKey
+	}
+
+	signer = signers.NewAccessKeySigner(aak)
+
+	if aak == nil {
+		log.Warnf("unable to retrieve the Alibaba API keys for node with providerID %s hence skipping SystemDisk Retrieval", slimK8sNode.ProviderID)
+		return NewAlibabaNodeKey(slimK8sNode, optimizedKeyword, diskCategory, diskSizeInGiB, diskPerformanceLevel)
+	}
+
+	if client, ok = alibaba.clients[slimK8sNode.RegionID]; !ok {
+		client, err = sdk.NewClientWithAccessKey(slimK8sNode.RegionID, aak.AccessKeyId, aak.AccessKeySecret)
+		if err != nil {
+			log.Warnf("unable to set the client  for node with providerID %s to retrieve the key skipping SystemDisk Retrieval with err: %v", slimK8sNode.ProviderID, err)
+			return NewAlibabaNodeKey(slimK8sNode, optimizedKeyword, diskCategory, diskSizeInGiB, diskPerformanceLevel)
+		}
+		alibaba.clients[slimK8sNode.RegionID] = client
+	}
+
+	instanceID := getInstanceIDFromProviderID(slimK8sNode.ProviderID)
+	slimK8sNode.SystemDisk = getSystemDiskInfoOfANode(instanceID, slimK8sNode.RegionID, client, signer)
+
+	if slimK8sNode.SystemDisk != nil {
+		diskCategory = slimK8sNode.SystemDisk.DiskCategory
+		diskSizeInGiB = slimK8sNode.SystemDisk.SizeInGiB
+		diskPerformanceLevel = slimK8sNode.SystemDisk.PerformanceLevel
+	}
+	return NewAlibabaNodeKey(slimK8sNode, optimizedKeyword, diskCategory, diskSizeInGiB, diskPerformanceLevel)
+}
+
+type AlibabaPVKey struct {
+	ProviderID        string
+	RegionID          string
+	PVType            string
+	PVSubType         string
+	PVCategory        string
+	PVPerformaceLevel string
+	StorageClassName  string
+	SizeInGiB         string
+}
+
+func (alibaba *Alibaba) GetPVKey(pv *v1.PersistentVolume, parameters map[string]string, defaultRegion string) PVKey {
+	regionID := defaultRegion
+	// If default Region is not passed default it to cluster region ID.
+	if defaultRegion == "" {
+		regionID = alibaba.clusterRegion
+	}
+	slimK8sDisk := generateSlimK8sDiskFromV1PV(pv, defaultRegion)
+	return &AlibabaPVKey{
+		ProviderID:        slimK8sDisk.ProviderID,
+		RegionID:          regionID,
+		PVType:            ALIBABA_PV_CLOUD_DISK_TYPE,
+		PVSubType:         slimK8sDisk.DiskType,
+		PVCategory:        slimK8sDisk.DiskCategory,
+		PVPerformaceLevel: slimK8sDisk.PerformanceLevel,
+		StorageClassName:  pv.Spec.StorageClassName,
+		SizeInGiB:         slimK8sDisk.SizeInGiB,
+	}
+}
+
+func (alibabaPVKey *AlibabaPVKey) Features() string {
+	keyLookup := stringutil.DeleteEmptyStringsFromArray([]string{alibabaPVKey.RegionID, alibabaPVKey.PVSubType, alibabaPVKey.PVCategory, alibabaPVKey.PVPerformaceLevel, alibabaPVKey.SizeInGiB})
+	return strings.Join(keyLookup, "::")
+}
+
+func (alibabaPVKey *AlibabaPVKey) ID() string {
+	return alibabaPVKey.ProviderID
+}
+
+// Get storage class information for PV.
+func (alibabaPVKey *AlibabaPVKey) GetStorageClass() string {
+	return alibabaPVKey.StorageClassName
+}
+
+// Helper functions for alibabaprovider.go
+
+// createDescribePriceACSRequest creates the HTTP GET request for the required resources' Price information,
+// When supporting subscription and Premptible resources this HTTP call needs to be modified with PriceUnit information
+// When supporting different new type of instances like Compute Optimized, Memory Optimized etc make sure you add the instance type
+// in unit test and check if it works or not to create the ack request and processDescribePriceAndCreateAlibabaPricing function
+// else more paramters need to be pulled from kubernetes node response or gather infromation from elsewhere and function modified.
+func createDescribePriceACSRequest(i interface{}) (*requests.CommonRequest, error) {
+	request := requests.NewCommonRequest()
+	request.Method = requests.GET
+	request.Product = ALIBABA_ECS_PRODUCT_CODE
+	request.Domain = ALIBABA_ECS_DOMAIN
+	request.Version = ALIBABA_ECS_VERSION
+	request.Scheme = requests.HTTPS
+	request.ApiName = ALIBABA_DESCRIBE_PRICE_API_ACTION
+	switch i.(type) {
+	case *SlimK8sNode:
+		node := i.(*SlimK8sNode)
+		request.QueryParams["RegionId"] = node.RegionID
+		request.QueryParams["ResourceType"] = ALIBABA_INSTANCE_RESOURCE_TYPE
+		request.QueryParams["InstanceType"] = node.InstanceType
+		request.QueryParams["PriceUnit"] = node.PriceUnit
+		if node.SystemDisk != nil {
+			// Only if the required information is present it should be overridden else default it via the API
+			if node.SystemDisk.DiskCategory != "" {
+				request.QueryParams["SystemDisk.Category"] = node.SystemDisk.DiskCategory
+			}
+			if node.SystemDisk.SizeInGiB != "" {
+				request.QueryParams["SystemDisk.Size"] = node.SystemDisk.SizeInGiB
+			}
+			if node.SystemDisk.PerformanceLevel != "" {
+				request.QueryParams["SystemDisk.PerformanceLevel"] = node.SystemDisk.PerformanceLevel
+			}
+		} else {
+			// When System Disk information is not available for instance family g6e, r7 and r6e the defaults in
+			// DescribePrice dont default rightly to cloud_essd for these instances.
+			if slices.Contains(alibabaDefaultToCloudEssd, node.InstanceTypeFamily) {
+				request.QueryParams["SystemDisk.Category"] = ALIBABA_DISK_CLOUD_ESSD_CATEGORY
+			}
+		}
+		request.TransToAcsRequest()
+		return request, nil
+	case *SlimK8sDisk:
+		disk := i.(*SlimK8sDisk)
+		request.QueryParams["RegionId"] = disk.RegionID
+		request.QueryParams["PriceUnit"] = disk.PriceUnit
+		request.QueryParams["ResourceType"] = ALIBABA_DISK_RESOURCE_TYPE
+		request.QueryParams[fmt.Sprintf("%s.%d.Size", ALIBABA_DATA_DISK_PREFIX, 1)] = disk.SizeInGiB
+		request.QueryParams[fmt.Sprintf("%s.%d.Category", ALIBABA_DATA_DISK_PREFIX, 1)] = disk.DiskCategory
+		// Performance level defaults to PL1 if not present in volume attribute.
+		if disk.PerformanceLevel != "" {
+			request.QueryParams[fmt.Sprintf("%s.%d.PerformanceLevel", ALIBABA_DATA_DISK_PREFIX, 1)] = disk.PerformanceLevel
+		}
+		request.TransToAcsRequest()
+		return request, nil
+	default:
+		return nil, fmt.Errorf("unsupported ECS type (%T) for DescribePrice at this time", i)
+	}
+}
+
+// createDescribeDisksCSRequest creates the HTTP GET Request to map the system disk to the InstanceID
+func createDescribeDisksACSRequest(instanceID, regionID, diskType string) (*requests.CommonRequest, error) {
+	request := requests.NewCommonRequest()
+	request.Method = requests.GET
+	request.Product = ALIBABA_ECS_PRODUCT_CODE
+	request.Domain = ALIBABA_ECS_DOMAIN
+	request.Version = ALIBABA_ECS_VERSION
+	request.Scheme = requests.HTTPS
+	request.ApiName = ALIBABA_DESCRIBE_DISK_API_ACTION
+	request.QueryParams["RegionId"] = regionID
+	request.QueryParams["InstanceId"] = instanceID
+	request.QueryParams["DiskType"] = diskType
+	request.TransToAcsRequest()
+	return request, nil
+}
+
+// determineKeyForPricing generate a unique key from SlimK8sNode object that is constructed from v1.Node object and
+// SlimK8sDisk that is constructed from v1.PersistentVolume.
+func determineKeyForPricing(i interface{}) (string, error) {
+	if i == nil {
+		return "", fmt.Errorf("nil component passed to determine key")
+	}
+	switch i.(type) {
+	case *SlimK8sNode:
+		node := i.(*SlimK8sNode)
+		var diskCategory, diskSizeInGiB, diskPerformanceLevel string
+		if node.SystemDisk != nil {
+			diskCategory = node.SystemDisk.DiskCategory
+			diskSizeInGiB = node.SystemDisk.SizeInGiB
+			diskPerformanceLevel = node.SystemDisk.PerformanceLevel
+		}
+		if node.IsIoOptimized {
+			keyLookup := stringutil.DeleteEmptyStringsFromArray([]string{node.RegionID, node.InstanceType, node.OSType, ALIBABA_OPTIMIZE_KEYWORD, diskCategory, diskSizeInGiB, diskPerformanceLevel})
+			return strings.Join(keyLookup, "::"), nil
+		} else {
+			keyLookup := stringutil.DeleteEmptyStringsFromArray([]string{node.RegionID, node.InstanceType, node.OSType, ALIBABA_NON_OPTIMIZE_KEYWORD, diskCategory, diskSizeInGiB, diskPerformanceLevel})
+			return strings.Join(keyLookup, "::"), nil
+		}
+	case *SlimK8sDisk:
+		disk := i.(*SlimK8sDisk)
+		keyLookup := stringutil.DeleteEmptyStringsFromArray([]string{disk.RegionID, disk.DiskType, disk.DiskCategory, disk.PerformanceLevel, disk.SizeInGiB})
+		return strings.Join(keyLookup, "::"), nil
+	default:
+		return "", fmt.Errorf("unsupported ECS type (%T) at this time", i)
+	}
+}
+
+// Below structs are used to unmarshal json response of Alibaba cloud's API DescribePrice
+type Price struct {
+	OriginalPrice             float32 `json:"OriginalPrice"`
+	ReservedInstanceHourPrice float32 `json:"ReservedInstanceHourPrice"`
+	DiscountPrice             float32 `json:"DiscountPrice"`
+	Currency                  string  `json:"Currency"`
+	TradePrice                float32 `json:"TradePrice"`
+}
+
+type PriceInfo struct {
+	Price Price `json:"Price"`
+}
+
+type DescribePriceResponse struct {
+	RequestId string    `json:"RequestId"`
+	PriceInfo PriceInfo `json:"PriceInfo"`
+}
+
+// processDescribePriceAndCreateAlibabaPricing processes the DescribePrice API and generates the pricing information for alibaba node resource and alibaba pv resource that's backed by cloud disk.
+func processDescribePriceAndCreateAlibabaPricing(client *sdk.Client, i interface{}, signer *signers.AccessKeySigner, custom *CustomPricing) (pricing *AlibabaPricing, err error) {
+	pricing = &AlibabaPricing{}
+	var response DescribePriceResponse
+
+	if i == nil {
+		return nil, fmt.Errorf("nil component passed to process the pricing information")
+	}
+	switch i.(type) {
+	case *SlimK8sNode:
+		node := i.(*SlimK8sNode)
+		req, err := createDescribePriceACSRequest(node)
+		if err != nil {
+			return nil, err
+		}
+		resp, err := client.ProcessCommonRequestWithSigner(req, signer)
+		pricing.NodeAttributes = NewAlibabaNodeAttributes(node)
+		if err != nil || resp.GetHttpStatus() != 200 {
+			// Can be defaulted to some value here?
+			return nil, fmt.Errorf("unable to fetch information for node with InstanceType: %v", node.InstanceType)
+		} else {
+			// This is where population of Pricing happens
+			err = json.Unmarshal(resp.GetHttpContentBytes(), &response)
+			if err != nil {
+				return nil, fmt.Errorf("unable to unmarshall json response to custom struct with err: %w", err)
+			}
+			// TO-DO : Ask in PR How to get the defaults is it equal to AWS/GCP defaults? And what needs to be returned
+			pricing.Node = &Node{
+				Cost:         fmt.Sprintf("%f", response.PriceInfo.Price.TradePrice),
+				BaseCPUPrice: custom.CPU,
+				BaseRAMPrice: custom.RAM,
+				BaseGPUPrice: custom.GPU,
+			}
+			// TO-DO : Currently with Pay-As-You-go Offering TradePrice = HourlyPrice , When support happens to other type HourlyPrice Need to be determined.
+			pricing.PricingTerms = NewAlibabaPricingTerms(ALIBABA_PAY_AS_YOU_GO_BILLING, NewAlibabaPricingDetails(response.PriceInfo.Price.TradePrice, ALIBABA_HOUR_PRICE_UNIT, response.PriceInfo.Price.TradePrice, response.PriceInfo.Price.Currency))
+		}
+	case *SlimK8sDisk:
+		disk := i.(*SlimK8sDisk)
+		req, err := createDescribePriceACSRequest(disk)
+		if err != nil {
+			return nil, err
+		}
+		resp, err := client.ProcessCommonRequestWithSigner(req, signer)
+		if err != nil || resp.GetHttpStatus() != 200 {
+			return nil, fmt.Errorf("unable to fetch information for disk with DiskType: %v with err: %w", disk.DiskCategory, err)
+		} else {
+			// This is where population of Pricing happens
+			err = json.Unmarshal(resp.GetHttpContentBytes(), &response)
+			if err != nil {
+				return nil, fmt.Errorf("unable to unmarshall json response to custom struct with err: %w", err)
+			}
+			pricing.PVAttributes = NewAlibabaPVAttributes(disk)
+			pricing.PV = &PV{
+				Cost: fmt.Sprintf("%f", response.PriceInfo.Price.TradePrice),
+			}
+			// TO-DO : Disk has support for Hour and Month but pricing API is failing for month for disk(Research why?) and same challenge as node pricing no prepaid/postpaid distinction in v1.PersistentVolume object have to look at APIs for th information.
+			pricing.PricingTerms = NewAlibabaPricingTerms(ALIBABA_PAY_AS_YOU_GO_BILLING, NewAlibabaPricingDetails(response.PriceInfo.Price.TradePrice, ALIBABA_HOUR_PRICE_UNIT, response.PriceInfo.Price.TradePrice, response.PriceInfo.Price.Currency))
+		}
+	default:
+		return nil, fmt.Errorf("unsupported ECS Pricing component of type (%T) at this time", i)
+	}
+
+	return pricing, nil
+}
+
+// This function is to get the InstanceFamily from the InstanceType , convention followed in
+// instance type is ecs.[FamilyName].[DifferentSize], it gets the familyName , if it is unable to get it
+// it lists the instance family name as Unknown.
+func getInstanceFamilyFromType(instanceType string) string {
+	splitinstanceType := strings.Split(instanceType, ".")
+	if len(splitinstanceType) != 3 {
+		log.Warnf("unable to find the family of the instance type %s, returning its family type unknown", instanceType)
+		return ALIBABA_UNKNOWN_INSTANCE_FAMILY_TYPE
+	}
+	if !slices.Contains(alibabaInstanceFamilies, splitinstanceType[1]) {
+		log.Warnf("currently the instance family type %s is not valid or not tested completely for pricing API", instanceType)
+		return ALIBABA_NOT_SUPPORTED_INSTANCE_FAMILY_TYPE
+	}
+	return splitinstanceType[1]
+}
+
+// getInstanceIDFromProviderID returns the instance ID associated with the Node. A *v1.Node providerID in Alibaba cloud
+// is of <REGION-ID>.<INSTANCE-ID>. This function returns the Instance ID for the given ProviderID. if its unable to interpret
+// it defaults to empty string.
+func getInstanceIDFromProviderID(providerID string) string {
+	if providerID == "" {
+		return ""
+	}
+	splitStrings := strings.Split(providerID, ".")
+	if len(splitStrings) < 2 {
+		return ""
+	}
+	return splitStrings[1]
+}
+
+type Disk struct {
+	Category         string `json:"Category"`
+	Size             int    `json:"Size"`
+	PerformanceLevel string `json:"PerformanceLevel"`
+	Type             string `json:"Type"`
+	RegionId         string `json:"RegionId"`
+	DiskId           string `json:"DiskId"`
+	DiskChargeType   string `json:"DiskChargeType"`
+}
+
+type Disks struct {
+	Disk []*Disk `json:"Disk"`
+}
+
+type DescribeDiskResponse struct {
+	TotalCount int    `json:"TotalCount"`
+	Disks      *Disks `json:"Disks"`
+}
+
+// getSystemDiskInfoOfANode gets the relevant System disk information associated with the Node given by the instanceID
+// in form of a SlimK8sDisk with only relevant information that can adjust the node pricing. If any error occurs return
+// an empty disk to not impact any default set at the price retrieval of the node.
+func getSystemDiskInfoOfANode(instanceID, regionID string, client *sdk.Client, signer *signers.AccessKeySigner) (systemDisk *SlimK8sDisk) {
+	systemDisk = &SlimK8sDisk{}
+	var response DescribeDiskResponse
+	// if instanceID is empty string return an empty k8s
+	if instanceID == "" {
+		return
+	}
+	req, err := createDescribeDisksACSRequest(instanceID, regionID, ALIBABA_SYSTEM_DISK_CATEGORY)
+	// if any error occurs return an empty disk to not impact default pricing.
+	if err != nil {
+		log.Warnf("Unable to create Describe Disk Request with err: %v for node with InstanceID: %s, hence defaulting it to an empty system disk to pass through to defaults", err, instanceID)
+		return
+	}
+
+	resp, err := client.ProcessCommonRequestWithSigner(req, signer)
+	if err != nil || resp.GetHttpStatus() != 200 {
+		log.Warnf("Unable to process Describe Disk request with err: %v and errcode: %d for the node with InstanceID: %s, hence defaulting it to an empty system disk to pass through to defaults", err, resp.GetHttpStatus(), instanceID)
+		return
+	} else {
+		// This is where population of Pricing happens
+		err = json.Unmarshal(resp.GetHttpContentBytes(), &response)
+		if err != nil {
+			log.Warnf("Unable to unmarshall Describe Disk response with err: %v for the node with InstanceID: %s, hence defaulting it to an empty system disk to pass through to defaults", err, instanceID)
+			return
+		}
+		// Every instance should only have one system disk per Alibaba Cloud documentation https://www.alibabacloud.com/help/en/elastic-compute-service/latest/block-storage-overview-disks,
+		// if TotalCount is not 1 just return empty and let it not impact default pricing.
+		if response.TotalCount != 1 {
+			log.Warnf("Total count of system disk for node with InstanceID: %s is not 1, hence defaulting it to an empty system disk to pass through to defaults", instanceID)
+			return
+		}
+
+		if response.Disks == nil {
+			log.Warnf("Disks information missing for node with InstanceID: %s, hence defaulting it to an empty system disk to pass through to defaults", instanceID)
+			return
+		}
+
+		if len(response.Disks.Disk) < 1 {
+			log.Warnf("Total number of system disk for node with InstanceID: %s is less than 1, hence defaulting it to an empty system disk to pass through to defaults", instanceID)
+			return
+		}
+
+		// TO-DO: When supporting Subscription type disk, you can leverge the disk.DiskChargeType here to map it to subscription type.
+		systemDisk := response.Disks.Disk[0]
+		return NewSlimK8sDisk(systemDisk.Type, systemDisk.RegionId, ALIBABA_HOUR_PRICE_UNIT, systemDisk.Category, systemDisk.PerformanceLevel, systemDisk.DiskId, "", fmt.Sprintf("%d", systemDisk.Size))
+	}
+}
+
+// generateSlimK8sNodeFromV1Node generates SlimK8sNode struct from v1.Node to fetch pricing information and call alibaba API.
+func generateSlimK8sNodeFromV1Node(node *v1.Node) *SlimK8sNode {
+	var regionID, osType, instanceType, providerID, priceUnit, instanceFamily string
+	var memorySizeInKiB string // TO-DO: try to convert it into float
+	var ok, IsIoOptimized bool
+	if regionID, ok = node.Labels["topology.kubernetes.io/region"]; !ok {
+		// HIGHLY UNLIKELY THAT THIS LABEL WONT BE THERE.
+		log.Debugf("No RegionID label for the node: %s", node.Name)
+	}
+	if osType, ok = node.Labels["beta.kubernetes.io/os"]; !ok {
+		// HIGHLY UNLIKELY THAT THIS LABEL WONT BE THERE.
+		log.Debugf("OS type undetected for the node: %s", node.Name)
+	}
+	if instanceType, ok = node.Labels["node.kubernetes.io/instance-type"]; !ok {
+		// HIGHLY UNLIKELY THAT THIS LABEL WONT BE THERE.
+		log.Debugf("Instance Type undetected for the node: %s", node.Name)
+	}
+
+	instanceFamily = getInstanceFamilyFromType(instanceType)
+	memorySizeInKiB = fmt.Sprintf("%s", node.Status.Capacity.Memory())
+	providerID = node.Spec.ProviderID // Alibaba Cloud provider doesnt follow convention of prefix with cloud provider name
+
+	// Looking at current Instance offering , all of the Instances seem to be I/O optimized - https://www.alibabacloud.com/help/en/elastic-compute-service/latest/instance-family
+	// Basic price Json has it as part of the key so defaulting to true.
+	IsIoOptimized = true
+	priceUnit = ALIBABA_HOUR_PRICE_UNIT
+
+	systemDisk := &SlimK8sDisk{}
+	return NewSlimK8sNode(instanceType, regionID, priceUnit, memorySizeInKiB, osType, providerID, instanceFamily, IsIoOptimized, systemDisk)
+}
+
+// getNumericalValueFromResourceQuantity returns the numericalValue of the resourceQuantity
+// An example is: 20Gi returns to 20. If any error occurs it returns the default value used in describePrice API which is 2000.
+func getNumericalValueFromResourceQuantity(quantity string) (value string) {
+	// defaulting when any panic or empty string occurs.
+	defer func() {
+		log.Debugf("unable to determine the size of the PV so defaulting the size to %s", ALIBABA_DEFAULT_DATADISK_SIZE)
+		if err := recover(); err != nil {
+			value = ALIBABA_DEFAULT_DATADISK_SIZE
+		}
+		if value == "" {
+			value = ALIBABA_DEFAULT_DATADISK_SIZE
+		}
+	}()
+	res := sizeRegEx.FindAllStringSubmatch(quantity, 1)
+	value = res[0][1]
+	return
+}
+
+// generateSlimK8sDiskFromV1PV function generates SlimK8sDisk from v1.PersistentVolume
+// to generate slim disk type that can be used to fetch pricing information for Data disk type.
+func generateSlimK8sDiskFromV1PV(pv *v1.PersistentVolume, regionID string) *SlimK8sDisk {
+
+	// All PVs are data disks while local disk are categorized as system disk
+	diskType := ALIBABA_DATA_DISK_CATEGORY
+
+	//TO-DO: Disk supports month and hour prices , defaulting to hour
+	priceUnit := ALIBABA_HOUR_PRICE_UNIT
+
+	sizeQuantity := fmt.Sprintf("%s", pv.Spec.Capacity.Storage())
+
+	// res := sizeRegEx.FindAllStringSubmatch(sizeQuantity, 1)
+
+	sizeInGiB := getNumericalValueFromResourceQuantity(sizeQuantity)
+
+	providerID := ""
+	if pv.Spec.CSI != nil {
+		providerID = pv.Spec.CSI.VolumeHandle
+	} else {
+		providerID = pv.Name // Looks like pv name is same as providerID in Alibaba k8s cluster
+	}
+
+	// Performance level being empty string gets defaulted in describePrice to PL1.
+	performanceLevel := ""
+	diskCategory := ""
+	if pv.Spec.CSI != nil {
+		if val, ok := pv.Spec.CSI.VolumeAttributes["performanceLevel"]; ok {
+			performanceLevel = val
+		}
+		if val, ok := pv.Spec.CSI.VolumeAttributes["type"]; ok {
+			diskCategory = val
+		}
+	}
+
+	// Highly unlikely that label pv.Spec.CSI.VolumeAttributes["type"] doesn't exist but if occured default to cloud (most basic disk type)
+	if diskCategory == "" {
+		diskCategory = ALIBABA_DISK_CLOUD_CATEGORY
+	}
+
+	return NewSlimK8sDisk(diskType, regionID, priceUnit, diskCategory, performanceLevel, providerID, pv.Spec.StorageClassName, sizeInGiB)
+}
+
+// determinePVRegion determines associated region for a particular PV based on the following priority, which can be changed and any other path to determine region can be added!
+// if topology.diskplugin.csi.alibabacloud.com/region label/annotation is passed during PV creation return that as the PV region.
+// if topology.diskplugin.csi.alibabacloud.com/zone label/annotation is passed during PV creation determine the region based on this pv label.
+// if neither of the above label/annotation is present check node affinity for the zone affinity and determine the region based on this zone.
+// if nether of the above yields a region , return empty string to default it to cluster region.
+func determinePVRegion(pv *v1.PersistentVolume) string {
+	// if "topology.diskplugin.csi.alibabacloud.com/region" is present as a label or annotation return that as the PV region
+	if val, ok := pv.Labels[ALIBABA_DISK_TOPOLOGY_REGION_LABEL]; ok {
+		log.Debugf("determinePVRegion returned a region value of: %s through label: %s for PV name: %s", val, ALIBABA_DISK_TOPOLOGY_REGION_LABEL, pv.Name)
+		return val
+	}
+	if val, ok := pv.Annotations[ALIBABA_DISK_TOPOLOGY_REGION_LABEL]; ok {
+		log.Debugf("determinePVRegion returned a region value of: %s through annotation: %s for PV name: %s", val, ALIBABA_DISK_TOPOLOGY_REGION_LABEL, pv.Name)
+		return val
+	}
+
+	// if "topology.diskplugin.csi.alibabacloud.com/zone" is present as a label or annotation set it as the PV zone before looking at node affinity to determine the region PV belongs too
+	var pvZone string
+
+	if val, ok := pv.Labels[ALIBABA_DISK_TOPOLOGY_ZONE_LABEL]; ok {
+		log.Debugf("determinePVRegion will set zone value to: %s through label: %s for PV name: %s", val, ALIBABA_DISK_TOPOLOGY_ZONE_LABEL, pv.Name)
+		pvZone = val
+	}
+
+	if pvZone == "" {
+		if val, ok := pv.Annotations[ALIBABA_DISK_TOPOLOGY_ZONE_LABEL]; ok {
+			log.Debugf("determinePVRegion will set zone value to: %s through annotation: %s for PV name: %s", val, ALIBABA_DISK_TOPOLOGY_ZONE_LABEL, pv.Name)
+			pvZone = val
+		}
+	}
+
+	if pvZone == "" {
+		// zone and regionID labels are optional in Alibaba PV creation, while PV through UI creation put's a zone PV is associated with and the region
+		// can be determined from this information. If pv is provision via yaml and the block is missing that's the only time it gets defaulted to clusterRegion.
+		if pv.Spec.NodeAffinity != nil {
+			nodeAffinity := pv.Spec.NodeAffinity
+			if nodeAffinity.Required != nil && nodeAffinity.Required.NodeSelectorTerms != nil {
+				for _, nodeSelectorTerm := range nodeAffinity.Required.NodeSelectorTerms {
+					matchExpression := nodeSelectorTerm.MatchExpressions
+					for _, nodeSelectorRequirement := range matchExpression {
+						if nodeSelectorRequirement.Key == ALIBABA_DISK_TOPOLOGY_ZONE_LABEL {
+							log.Debugf("determinePVRegion will set zone value to: %s through node affinity label: %s for PV name: %s", nodeSelectorRequirement.Values[0], ALIBABA_DISK_TOPOLOGY_ZONE_LABEL, pv.Name)
+							pvZone = nodeSelectorRequirement.Values[0]
+						}
+					}
+				}
+			}
+		}
+	}
+
+	for _, region := range alibabaRegions {
+		if strings.Contains(pvZone, region) {
+			log.Debugf("determinePVRegion determined region of %s through zone affiliation of the PV %s\n", region, pvZone)
+			return region
+		}
+	}
+	return ""
+}

pkg/cloud/aliyunprovider_test.go

@@ -0,0 +1,838 @@
+package cloud
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/aliyun/alibaba-cloud-sdk-go/sdk"
+	"github.com/aliyun/alibaba-cloud-sdk-go/sdk/auth/credentials"
+	"github.com/aliyun/alibaba-cloud-sdk-go/sdk/auth/signers"
+	v1 "k8s.io/api/core/v1"
+	resource "k8s.io/apimachinery/pkg/api/resource"
+)
+
+func TestCreateDescribePriceACSRequest(t *testing.T) {
+	node := &SlimK8sNode{
+		InstanceType:       "ecs.g6.large",
+		RegionID:           "cn-hangzhou",
+		PriceUnit:          "Hour",
+		MemorySizeInKiB:    "16KiB",
+		IsIoOptimized:      true,
+		OSType:             "Linux",
+		ProviderID:         "Ali-XXX-node-01",
+		InstanceTypeFamily: "g6",
+	}
+
+	disk := &SlimK8sDisk{
+		DiskType:         "data",
+		RegionID:         "cn-hangzhou",
+		PriceUnit:        "Hour",
+		SizeInGiB:        "20",
+		DiskCategory:     "diskCategory",
+		PerformanceLevel: "cloud_essd",
+		ProviderID:       "d-Ali-XXX-01",
+		StorageClass:     "testStorageClass",
+	}
+
+	cases := []struct {
+		name          string
+		testStruct    interface{}
+		expectedError error
+	}{
+		{
+			name:          "test CreateDescribePriceACSRequest with SlimK8sNode struct Object",
+			testStruct:    node,
+			expectedError: nil,
+		},
+		{
+			name:          "test CreateDescribePriceACSRequest with SlimK8sDisk struct Object",
+			testStruct:    disk,
+			expectedError: nil,
+		},
+	}
+
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			_, err := createDescribePriceACSRequest(c.testStruct)
+			if err != nil && c.expectedError == nil {
+				t.Fatalf("Case name %s: Error converting to Alibaba cloud request", c.name)
+			}
+		})
+	}
+}
+
+func TestProcessDescribePriceAndCreateAlibabaPricing(t *testing.T) {
+	// Skipping this test case since it exposes secret but a good test case to verify when
+	// supporting a new family of instances, steps to perform are
+	// STEP 1: Comment the t.Skip() line and then replace XXX_KEY_ID with the alibaba key id of your account and XXX_SECRET_ID with alibaba cloud secret of your account.
+	// STEP 2: Once you verify describePrice is working and no change needed in processDescribePriceAndCreateAlibabaPricing, you can go ahead and revert the step 1 changes.
+
+	// This test case was use to test all general puprose instances
+
+	t.Skip()
+
+	client, err := sdk.NewClientWithAccessKey("cn-hangzhou", "XXX_KEY_ID", "XXX_SECRET_ID")
+	if err != nil {
+		t.Errorf("Error connecting to the Alibaba cloud")
+	}
+	aak := credentials.NewAccessKeyCredential("XXX_KEY_ID", "XXX_SECRET_ID")
+	signer := signers.NewAccessKeySigner(aak)
+
+	cases := []struct {
+		name          string
+		teststruct    interface{}
+		expectedError error
+	}{
+		{
+			name: "test General Purpose Type g7 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.g7.4xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "16777216KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-01a",
+				InstanceTypeFamily: "g7",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test General Purpose Type g7a instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.g7a.8xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-01b",
+				InstanceTypeFamily: "g7a",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Enhanced General Purpose Type g6e instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.g6e.xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "16777216KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-01",
+				InstanceTypeFamily: "g6e",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test General Purpose Type g6 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.g6.3xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "50331648KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-02",
+				InstanceTypeFamily: "g6",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test General Purpose Type g5 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.g5.2xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-03",
+				InstanceTypeFamily: "g5",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test General Purpose Type sn2 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.sn2.large",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "16777216KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-04",
+				InstanceTypeFamily: "sn2",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test General Purpose Type with Enhanced Network Performance sn2ne instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.sn2ne.2xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-05",
+				InstanceTypeFamily: "sn2ne",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory Optmized instance type r7 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.r7.6xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "2013265592KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-06",
+				InstanceTypeFamily: "r7",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory Optmized instance type r7a instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.r7a.8xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-06a",
+				InstanceTypeFamily: "r7a",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Enhanced Memory Optmized instance type r6e instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.r6e.4xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "2013265592KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-07",
+				InstanceTypeFamily: "r6e",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory Optmized instance type r6a instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.r6a.8xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-07a",
+				InstanceTypeFamily: "r6a",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory Optmized instance type r6 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.r6.8xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-08",
+				InstanceTypeFamily: "r6",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory type instance and r5 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.r5.xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-09",
+				InstanceTypeFamily: "r5",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory Optmized instance type with se1 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.se1.4xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "16777216KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-10",
+				InstanceTypeFamily: "se1",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory Optmized instance type with Enhanced Network Performance se1ne instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.se1ne.3xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "100663296KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-11",
+				InstanceTypeFamily: "se1ne",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test High Memory type with re6 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.re6.8xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-12",
+				InstanceTypeFamily: "re6",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Persistent Memory Optimized type with re6p instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.re6p.4xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-13",
+				InstanceTypeFamily: "re6p",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory type with re4 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.re4.10xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "41943040KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-14",
+				InstanceTypeFamily: "re4",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Memory optimized type with se1 instance family",
+			teststruct: &SlimK8sNode{
+				InstanceType:       "ecs.se1.8xlarge",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "33554432KiB",
+				IsIoOptimized:      true,
+				OSType:             "Linux",
+				ProviderID:         "cn-hangzhou.i-test-15",
+				InstanceTypeFamily: "se1",
+			},
+			expectedError: nil,
+		},
+		{
+			name:          "test for a nil information",
+			teststruct:    nil,
+			expectedError: fmt.Errorf("unsupported ECS pricing component at this time"),
+		},
+		{
+			name: "test Cloud Disk with Category cloud representing basic disk",
+			teststruct: &SlimK8sDisk{
+				DiskType:     "data",
+				RegionID:     "cn-hangzhou",
+				PriceUnit:    "Hour",
+				SizeInGiB:    "20",
+				DiskCategory: "cloud",
+				ProviderID:   "d-Ali-cloud-XXX-01",
+				StorageClass: "temp",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Cloud Disk with Category cloud_efficiency representing ultra disk",
+			teststruct: &SlimK8sDisk{
+				DiskType:     "data",
+				RegionID:     "cn-hangzhou",
+				PriceUnit:    "Hour",
+				SizeInGiB:    "40",
+				DiskCategory: "cloud_efficiency",
+				ProviderID:   "d-Ali-cloud-XXX-02",
+				StorageClass: "temp",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Cloud Disk with Category cloud_ssd representing standard SSD",
+			teststruct: &SlimK8sDisk{
+				DiskType:     "data",
+				RegionID:     "cn-hangzhou",
+				PriceUnit:    "Hour",
+				SizeInGiB:    "40",
+				DiskCategory: "cloud_efficiency",
+				ProviderID:   "d-Ali-cloud-XXX-02",
+				StorageClass: "temp",
+			},
+			expectedError: nil,
+		},
+		{
+			name: "test Cloud Disk with Category cloud_essd representing Enhanced SSD with PL2 performance level",
+			teststruct: &SlimK8sDisk{
+				DiskType:         "data",
+				RegionID:         "cn-hangzhou",
+				PriceUnit:        "Hour",
+				SizeInGiB:        "80",
+				DiskCategory:     "cloud_ssd",
+				PerformanceLevel: "PL2",
+				ProviderID:       "d-Ali-cloud-XXX-04",
+				StorageClass:     "temp",
+			},
+			expectedError: nil,
+		},
+	}
+	custom := &CustomPricing{}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			pricingObj, err := processDescribePriceAndCreateAlibabaPricing(client, c.teststruct, signer, custom)
+			if err != nil && c.expectedError == nil {
+				t.Fatalf("Case name %s: got an error %s", c.name, err)
+			}
+			if c.teststruct != nil {
+				if pricingObj == nil {
+					t.Fatalf("Case name %s: got a nil pricing object", c.name)
+				}
+				t.Logf("Case name %s: Pricing Information gathered for instanceType is %v", c.name, pricingObj.PricingTerms.PricingDetails.TradePrice)
+			}
+		})
+	}
+}
+
+func TestGetInstanceFamilyFromType(t *testing.T) {
+	cases := []struct {
+		name                   string
+		instanceType           string
+		expectedInstanceFamily string
+	}{
+		{
+			name:                   "test if ecs.[instance-family].[different-type] work",
+			instanceType:           "ecs.sn2ne.2xlarge",
+			expectedInstanceFamily: "sn2ne",
+		},
+		{
+			name:                   "test if random word gives you ALIBABA_UNKNOWN_INSTANCE_FAMILY_TYPE value ",
+			instanceType:           "random.value",
+			expectedInstanceFamily: ALIBABA_UNKNOWN_INSTANCE_FAMILY_TYPE,
+		},
+		{
+			name:                   "test if random instance family gives you ALIBABA_NOT_SUPPORTED_INSTANCE_FAMILY_TYPE value ",
+			instanceType:           "ecs.g7e.2xlarge",
+			expectedInstanceFamily: ALIBABA_NOT_SUPPORTED_INSTANCE_FAMILY_TYPE,
+		},
+	}
+
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnValue := getInstanceFamilyFromType(c.instanceType)
+			if returnValue != c.expectedInstanceFamily {
+				t.Fatalf("Case name %s: expected instance family of type %s but got %s", c.name, c.expectedInstanceFamily, returnValue)
+			}
+		})
+	}
+}
+
+func TestDetermineKeyForPricing(t *testing.T) {
+	type randomK8sStruct struct {
+		name string
+	}
+	cases := []struct {
+		name          string
+		testVar       interface{}
+		expectedKey   string
+		expectedError error
+	}{
+		{
+			name: "test when all RegionID, InstanceType, OSType & ALIBABA_OPTIMIZE_KEYWORD words are used in Node key",
+			testVar: &SlimK8sNode{
+				InstanceType:       "ecs.sn2.large",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "16777216KiB",
+				IsIoOptimized:      true,
+				OSType:             "linux",
+				ProviderID:         "cn-hangzhou.i-test-04",
+				InstanceTypeFamily: "sn2",
+			},
+			expectedKey:   "cn-hangzhou::ecs.sn2.large::linux::optimize",
+			expectedError: nil,
+		},
+		{
+			name: "test missing InstanceType to create Node key",
+			testVar: &SlimK8sNode{
+				RegionID:        "cn-hangzhou",
+				PriceUnit:       "Hour",
+				MemorySizeInKiB: "16777216KiB",
+				IsIoOptimized:   true,
+				OSType:          "linux",
+				ProviderID:      "cn-hangzhou.i-test-04",
+			},
+			expectedKey:   "cn-hangzhou::linux::optimize",
+			expectedError: nil,
+		},
+		{
+			name: "test when node has a systemDisk Information with missing Performance level",
+			testVar: &SlimK8sNode{
+				InstanceType:       "ecs.sn2.large",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "16777216KiB",
+				IsIoOptimized:      true,
+				OSType:             "linux",
+				ProviderID:         "cn-hangzhou.i-test-04",
+				InstanceTypeFamily: "sn2",
+				SystemDisk: &SlimK8sDisk{
+					DiskType:     "system",
+					RegionID:     "cn-hangzhou",
+					PriceUnit:    "Hour",
+					SizeInGiB:    "40",
+					DiskCategory: "cloud_efficiency",
+					ProviderID:   "d-Ali-cloud-XXX-i1",
+					StorageClass: "",
+				},
+			},
+			expectedKey:   "cn-hangzhou::ecs.sn2.large::linux::optimize::cloud_efficiency::40",
+			expectedError: nil,
+		},
+		{
+			name: "test when node has a systemDisk Information with all information",
+			testVar: &SlimK8sNode{
+				InstanceType:       "ecs.sn2.large",
+				RegionID:           "cn-hangzhou",
+				PriceUnit:          "Hour",
+				MemorySizeInKiB:    "16777216KiB",
+				IsIoOptimized:      true,
+				OSType:             "linux",
+				ProviderID:         "cn-hangzhou.i-test-04",
+				InstanceTypeFamily: "sn2",
+				SystemDisk: &SlimK8sDisk{
+					DiskType:         "data",
+					RegionID:         "cn-hangzhou",
+					PriceUnit:        "Hour",
+					SizeInGiB:        "80",
+					DiskCategory:     "cloud_ssd",
+					PerformanceLevel: "PL2",
+					ProviderID:       "d-Ali-cloud-XXX-04",
+					StorageClass:     "",
+				},
+			},
+			expectedKey:   "cn-hangzhou::ecs.sn2.large::linux::optimize::cloud_ssd::80::PL2",
+			expectedError: nil,
+		},
+		{
+			name: "test random k8s struct should return unsupported error",
+			testVar: &randomK8sStruct{
+				name: "test struct",
+			},
+			expectedKey:   "",
+			expectedError: fmt.Errorf("unsupported ECS type randomK8sStruct for DescribePrice at this time"),
+		},
+		{
+			name:          "test for nil check",
+			testVar:       nil,
+			expectedKey:   "",
+			expectedError: fmt.Errorf("unsupported ECS type randomK8sStruct for DescribePrice at this time"),
+		},
+		{
+			name: "test when all RegionID, InstanceType, OSType & ALIBABA_OPTIMIZE_KEYWORD words are used to key",
+			testVar: &SlimK8sDisk{
+				DiskType:     "data",
+				RegionID:     "cn-hangzhou",
+				PriceUnit:    "Hour",
+				SizeInGiB:    "40",
+				DiskCategory: "cloud_efficiency",
+				ProviderID:   "d-Ali-cloud-XXX-02",
+				StorageClass: "temp",
+			},
+			expectedKey:   "cn-hangzhou::data::cloud_efficiency::40",
+			expectedError: nil,
+		},
+		{
+			name: "test missing InstanceType to create key",
+			testVar: &SlimK8sDisk{
+				DiskType:         "data",
+				RegionID:         "cn-hangzhou",
+				PriceUnit:        "Hour",
+				SizeInGiB:        "80",
+				DiskCategory:     "cloud_ssd",
+				PerformanceLevel: "PL2",
+				ProviderID:       "d-Ali-cloud-XXX-04",
+				StorageClass:     "temp",
+			},
+			expectedKey:   "cn-hangzhou::data::cloud_ssd::PL2::80",
+			expectedError: nil,
+		},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnString, returnErr := determineKeyForPricing(c.testVar)
+			if c.expectedError == nil && returnErr != nil {
+				t.Fatalf("Case name %s: expected error was nil but recieved error %v", c.name, returnErr)
+			}
+			if returnString != c.expectedKey {
+				t.Fatalf("Case name %s: determineKeyForPricing recieved %s but expected %s", c.name, returnString, c.expectedKey)
+			}
+		})
+	}
+}
+
+func TestGenerateSlimK8sNodeFromV1Node(t *testing.T) {
+	testv1Node := &v1.Node{}
+	testv1Node.Labels = make(map[string]string)
+	testv1Node.Labels["topology.kubernetes.io/region"] = "us-east-1"
+	testv1Node.Labels["beta.kubernetes.io/os"] = "linux"
+	testv1Node.Labels["node.kubernetes.io/instance-type"] = "ecs.sn2ne.2xlarge"
+	testv1Node.Status.Capacity = v1.ResourceList{
+		v1.ResourceMemory: *resource.NewQuantity(16, resource.BinarySI),
+	}
+	cases := []struct {
+		name             string
+		testNode         *v1.Node
+		expectedSlimNode *SlimK8sNode
+	}{
+		{
+			name:     "test a generic *v1.Node to *SlimK8sNode Conversion",
+			testNode: testv1Node,
+			expectedSlimNode: &SlimK8sNode{
+				InstanceType:       "ecs.sn2ne.2xlarge",
+				RegionID:           "us-east-1",
+				PriceUnit:          ALIBABA_HOUR_PRICE_UNIT,
+				MemorySizeInKiB:    "16",
+				IsIoOptimized:      true,
+				OSType:             "linux",
+				InstanceTypeFamily: "sn2ne",
+			},
+		},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnSlimK8sNode := generateSlimK8sNodeFromV1Node(c.testNode)
+			if returnSlimK8sNode.InstanceType != c.expectedSlimNode.InstanceType {
+				t.Fatalf("unexpected conversion in function generateSlimK8sNodeFromV1Node expected InstanceType: %s , recieved InstanceType: %s", c.expectedSlimNode.InstanceType, returnSlimK8sNode.InstanceType)
+			}
+			if returnSlimK8sNode.RegionID != c.expectedSlimNode.RegionID {
+				t.Fatalf("unexpected conversion in function generateSlimK8sNodeFromV1Node expected RegionID: %s , recieved RegionID: %s", c.expectedSlimNode.RegionID, returnSlimK8sNode.RegionID)
+			}
+			if returnSlimK8sNode.PriceUnit != c.expectedSlimNode.PriceUnit {
+				t.Fatalf("unexpected conversion in function generateSlimK8sNodeFromV1Node expected PriceUnit: %s , recieved PriceUnit: %s", c.expectedSlimNode.PriceUnit, returnSlimK8sNode.PriceUnit)
+			}
+			if returnSlimK8sNode.MemorySizeInKiB != c.expectedSlimNode.MemorySizeInKiB {
+				t.Fatalf("unexpected conversion in function generateSlimK8sNodeFromV1Node expected MemorySizeInKiB: %s , recieved MemorySizeInKiB: %s", c.expectedSlimNode.MemorySizeInKiB, returnSlimK8sNode.MemorySizeInKiB)
+			}
+			if returnSlimK8sNode.OSType != c.expectedSlimNode.OSType {
+				t.Fatalf("unexpected conversion in function generateSlimK8sNodeFromV1Node expected OSType: %s , recieved OSType: %s", c.expectedSlimNode.OSType, returnSlimK8sNode.OSType)
+			}
+			if returnSlimK8sNode.InstanceTypeFamily != c.expectedSlimNode.InstanceTypeFamily {
+				t.Fatalf("unexpected conversion in function generateSlimK8sNodeFromV1Node expected InstanceTypeFamily: %s , recieved InstanceTypeFamily: %s", c.expectedSlimNode.InstanceTypeFamily, returnSlimK8sNode.InstanceTypeFamily)
+			}
+		})
+	}
+}
+
+func TestGenerateSlimK8sDiskFromV1PV(t *testing.T) {
+	testv1PV := &v1.PersistentVolume{}
+	testv1PV.Spec.Capacity = v1.ResourceList{
+		v1.ResourceStorage: *resource.NewQuantity(16*1024*1024*1024, resource.BinarySI),
+	}
+	testv1PV.Spec.CSI = &v1.CSIPersistentVolumeSource{}
+	testv1PV.Spec.CSI.VolumeHandle = "testPV"
+	testv1PV.Spec.CSI.VolumeAttributes = map[string]string{
+		"performanceLevel": "PL2",
+		"type":             "cloud_essd",
+	}
+	testv1PV.Spec.CSI.VolumeHandle = "testPV"
+	testv1PV.Spec.StorageClassName = "testStorageClass"
+	cases := []struct {
+		name             string
+		testPV           *v1.PersistentVolume
+		expectedSlimDisk *SlimK8sDisk
+		inpRegionID      string
+	}{
+		{
+			name:   "test a generic *v1.Node to *SlimK8sNode Conversion",
+			testPV: testv1PV,
+			expectedSlimDisk: &SlimK8sDisk{
+				DiskType:         ALIBABA_DATA_DISK_CATEGORY,
+				RegionID:         "us-east-1",
+				PriceUnit:        ALIBABA_HOUR_PRICE_UNIT,
+				SizeInGiB:        "16",
+				DiskCategory:     "cloud_essd",
+				PerformanceLevel: "PL2",
+				ProviderID:       "testPV",
+				StorageClass:     "testStorageClass",
+			},
+			inpRegionID: "us-east-1",
+		},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnSlimK8sDisk := generateSlimK8sDiskFromV1PV(c.testPV, c.inpRegionID)
+			if returnSlimK8sDisk.DiskType != c.expectedSlimDisk.DiskType {
+				t.Fatalf("unexpected conversion in function generateSlimK8sDiskFromV1PV expected DiskType: %s , recieved DiskType: %s", c.expectedSlimDisk.DiskType, returnSlimK8sDisk.DiskType)
+			}
+			if returnSlimK8sDisk.RegionID != c.expectedSlimDisk.RegionID {
+				t.Fatalf("unexpected conversion in function generateSlimK8sDiskFromV1PV expected RegionID: %s , recieved RegionID Type: %s", c.expectedSlimDisk.RegionID, returnSlimK8sDisk.RegionID)
+			}
+			if returnSlimK8sDisk.PriceUnit != c.expectedSlimDisk.PriceUnit {
+				t.Fatalf("unexpected conversion in function generateSlimK8sDiskFromV1PV expected PriceUnit: %s , recieved PriceUnit Type: %s", c.expectedSlimDisk.PriceUnit, returnSlimK8sDisk.PriceUnit)
+			}
+			if returnSlimK8sDisk.SizeInGiB != c.expectedSlimDisk.SizeInGiB {
+				t.Fatalf("unexpected conversion in function generateSlimK8sDiskFromV1PV expected SizeInGiB: %s , recieved SizeInGiB Type: %s", c.expectedSlimDisk.SizeInGiB, returnSlimK8sDisk.SizeInGiB)
+			}
+			if returnSlimK8sDisk.DiskCategory != c.expectedSlimDisk.DiskCategory {
+				t.Fatalf("unexpected conversion in function generateSlimK8sDiskFromV1PV expected DiskCategory: %s , recieved DiskCategory Type: %s", c.expectedSlimDisk.DiskCategory, returnSlimK8sDisk.DiskCategory)
+			}
+			if returnSlimK8sDisk.PerformanceLevel != c.expectedSlimDisk.PerformanceLevel {
+				t.Fatalf("unexpected conversion in function generateSlimK8sDiskFromV1PV expected PerformanceLevel: %s , recieved PerformanceLevel Type: %s", c.expectedSlimDisk.PerformanceLevel, returnSlimK8sDisk.PerformanceLevel)
+			}
+			if returnSlimK8sDisk.ProviderID != c.expectedSlimDisk.ProviderID {
+				t.Fatalf("unexpected conversion in function generateSlimK8sDiskFromV1PV expected ProviderID: %s , recieved ProviderID Type: %s", c.expectedSlimDisk.ProviderID, returnSlimK8sDisk.ProviderID)
+			}
+			if returnSlimK8sDisk.StorageClass != c.expectedSlimDisk.StorageClass {
+				t.Fatalf("unexpected conversion in function generateSlimK8sDiskFromV1PV expected StorageClass: %s , recieved StorageClass Type: %s", c.expectedSlimDisk.StorageClass, returnSlimK8sDisk.StorageClass)
+			}
+		})
+	}
+}
+
+func TestGetNumericalValueFromResourceQuantity(t *testing.T) {
+	cases := []struct {
+		name                 string
+		inputResourceQuanity string
+		expectedValue        string
+	}{
+		{
+			name:                 "positive scenario: when inputResourceQuantity is 10Gi",
+			inputResourceQuanity: "10Gi",
+			expectedValue:        "10",
+		},
+		{
+			name:                 "negative scenario: when inputResourceQuantity is Gi",
+			inputResourceQuanity: "Gi",
+			expectedValue:        ALIBABA_DEFAULT_DATADISK_SIZE,
+		},
+		{
+			name:                 "negative scenario: when inputResourceQuantity is 10",
+			inputResourceQuanity: "10",
+			expectedValue:        ALIBABA_DEFAULT_DATADISK_SIZE,
+		},
+		{
+			name:                 "negative scenario: when inputResourceQuantity is empty string",
+			inputResourceQuanity: "",
+			expectedValue:        ALIBABA_DEFAULT_DATADISK_SIZE,
+		},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnValue := getNumericalValueFromResourceQuantity(c.inputResourceQuanity)
+			if c.expectedValue != returnValue {
+				t.Fatalf("Case name %s: getNumericalValueFromResourceQuantity recieved %s but expected %s", c.name, returnValue, c.expectedValue)
+			}
+		})
+	}
+}
+
+func TestDeterminePVRegion(t *testing.T) {
+	genericNodeAffinityTestStruct := v1.NodeSelectorTerm{
+		MatchExpressions: []v1.NodeSelectorRequirement{
+			{
+				Key:      "topology.diskplugin.csi.alibabacloud.com/zone",
+				Operator: v1.NodeSelectorOpIn,
+				Values:   []string{"us-east-1a"},
+			},
+		},
+		MatchFields: []v1.NodeSelectorRequirement{},
+	}
+
+	// testPV1 contains the Label with region information as well as node affinity in spec
+	testPV1 := &v1.PersistentVolume{}
+	testPV1.Name = "testPV1"
+	testPV1.Labels = make(map[string]string)
+	testPV1.Labels[ALIBABA_DISK_TOPOLOGY_REGION_LABEL] = "us-east-1"
+	testPV1.Spec.NodeAffinity = &v1.VolumeNodeAffinity{
+		Required: &v1.NodeSelector{
+			NodeSelectorTerms: []v1.NodeSelectorTerm{genericNodeAffinityTestStruct},
+		},
+	}
+
+	// testPV2 contains the only zone label
+	testPV2 := &v1.PersistentVolume{}
+	testPV2.Name = "testPV2"
+	testPV2.Labels = make(map[string]string)
+	testPV2.Labels[ALIBABA_DISK_TOPOLOGY_ZONE_LABEL] = "us-east-1a"
+
+	// testPV3 contains only node affinity in spec
+	testPV3 := &v1.PersistentVolume{}
+	testPV3.Name = "testPV3"
+	testPV3.Spec.NodeAffinity = &v1.VolumeNodeAffinity{
+		Required: &v1.NodeSelector{
+			NodeSelectorTerms: []v1.NodeSelectorTerm{genericNodeAffinityTestStruct},
+		},
+	}
+
+	// testPV4 contains no label/annotation or any node affinity
+	testPV4 := &v1.PersistentVolume{}
+	testPV4.Name = "testPV4"
+
+	cases := []struct {
+		name           string
+		inputPV        *v1.PersistentVolume
+		expectedRegion string
+	}{
+		{
+			name:           "When Region label topology.diskplugin.csi.alibabacloud.com/region is present along with node affinity details",
+			inputPV:        testPV1,
+			expectedRegion: "us-east-1",
+		},
+		{
+			name:           "When zone label topology.diskplugin.csi.alibabacloud.com/zone is present function has to determine region",
+			inputPV:        testPV2,
+			expectedRegion: "us-east-1",
+		},
+		{
+			name:           "When only node affinity detail is present function has to determine the region",
+			inputPV:        testPV3,
+			expectedRegion: "us-east-1",
+		},
+		{
+			name:           "When no region/zone information is present function returns empty to default to cluster region",
+			inputPV:        testPV4,
+			expectedRegion: "",
+		},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnRegion := determinePVRegion(c.inputPV)
+			if c.expectedRegion != returnRegion {
+				t.Fatalf("Case name %s: determinePVRegion recieved region :%s but expected region: %s", c.name, returnRegion, c.expectedRegion)
+			}
+		})
+	}
+
+}

pkg/cloud/awsprovider.go

@@ -5,10 +5,11 @@ import (
 	"compress/gzip"
 	"context"
 	"encoding/csv"
+	"errors"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"net/http"
+	"os"
 	"regexp"
 	"strconv"
 	"strings"
@@ -19,7 +20,7 @@ import (
 
 	"github.com/opencost/opencost/pkg/clustercache"
 	"github.com/opencost/opencost/pkg/env"
-	"github.com/opencost/opencost/pkg/errors"
+	errs "github.com/opencost/opencost/pkg/errors"
 	"github.com/opencost/opencost/pkg/log"
 	"github.com/opencost/opencost/pkg/util"
 	"github.com/opencost/opencost/pkg/util/fileutil"
@@ -306,12 +307,14 @@ const HourlyRateCodeCn = ".Q7UJUT2CE6"
 // name and the EC2 API.
 var volTypes = map[string]string{
 	"EBS:VolumeUsage.gp2":    "gp2",
+	"EBS:VolumeUsage.gp3":    "gp3",
 	"EBS:VolumeUsage":        "standard",
 	"EBS:VolumeUsage.sc1":    "sc1",
 	"EBS:VolumeP-IOPS.piops": "io1",
 	"EBS:VolumeUsage.st1":    "st1",
 	"EBS:VolumeUsage.piops":  "io1",
 	"gp2":                    "EBS:VolumeUsage.gp2",
+	"gp3":                    "EBS:VolumeUsage.gp3",
 	"standard":               "EBS:VolumeUsage",
 	"sc1":                    "EBS:VolumeUsage.sc1",
 	"io1":                    "EBS:VolumeUsage.piops",
@@ -610,6 +613,10 @@ type awsKey struct {
 	ProviderID     string
 }
 
+func (k *awsKey) GPUCount() int {
+	return 0
+}
+
 func (k *awsKey) GPUType() string {
 	return ""
 }
@@ -760,6 +767,10 @@ func (aws *AWS) getRegionPricing(nodeList []*v1.Node) (*http.Response, string, e
 
 	pricingURL += "index.json"
 
+	if env.GetAWSPricingURL() != "" { // Allow override of pricing URL
+		pricingURL = env.GetAWSPricingURL()
+	}
+
 	log.Infof("starting download of \"%s\", which is quite large ...", pricingURL)
 	resp, err := http.Get(pricingURL)
 	if err != nil {
@@ -841,7 +852,7 @@ func (aws *AWS) DownloadPricingData() error {
 		pvkeys[key.Features()] = key
 	}
 
-	// RIDataRunning establishes the existance of the goroutine. Since it's possible we
+	// RIDataRunning establishes the existence of the goroutine. Since it's possible we
 	// run multiple downloads, we don't want to create multiple go routines if one already exists
 	if !aws.RIDataRunning {
 		err = aws.GetReservationDataFromAthena() // Block until one run has completed.
@@ -849,7 +860,7 @@ func (aws *AWS) DownloadPricingData() error {
 			log.Errorf("Failed to lookup reserved instance data: %s", err.Error())
 		} else { // If we make one successful run, check on new reservation data every hour
 			go func() {
-				defer errors.HandlePanic()
+				defer errs.HandlePanic()
 				aws.RIDataRunning = true
 
 				for {
@@ -869,7 +880,7 @@ func (aws *AWS) DownloadPricingData() error {
 			log.Errorf("Failed to lookup savings plan data: %s", err.Error())
 		} else {
 			go func() {
-				defer errors.HandlePanic()
+				defer errs.HandlePanic()
 				aws.SavingsPlanDataRunning = true
 				for {
 					log.Infof("Savings Plan watcher running... next update in 1h")
@@ -1046,7 +1057,7 @@ func (aws *AWS) DownloadPricingData() error {
 		aws.SpotRefreshRunning = true
 
 		go func() {
-			defer errors.HandlePanic()
+			defer errs.HandlePanic()
 
 			for {
 				log.Infof("Spot Pricing Refresh scheduled in %.2f minutes.", SpotRefreshDuration.Minutes())
@@ -1412,7 +1423,7 @@ func (aws *AWS) loadAWSAuthSecret(force bool) (*AWSAccessKey, error) {
 		return nil, fmt.Errorf("Failed to locate service account file: %s", authSecretPath)
 	}
 
-	result, err := ioutil.ReadFile(authSecretPath)
+	result, err := os.ReadFile(authSecretPath)
 	if err != nil {
 		return nil, err
 	}
@@ -1457,7 +1468,7 @@ func (aws *AWS) GetAddresses() ([]byte, error) {
 		// respective channels
 		go func(region string) {
 			defer wg.Done()
-			defer errors.HandlePanic()
+			defer errs.HandlePanic()
 
 			// Query for first page of volume results
 			resp, err := aws.getAddressesForRegion(context.TODO(), region)
@@ -1471,7 +1482,7 @@ func (aws *AWS) GetAddresses() ([]byte, error) {
 
 	// Close the result channels after everything has been sent
 	go func() {
-		defer errors.HandlePanic()
+		defer errs.HandlePanic()
 
 		wg.Wait()
 		close(errorCh)
@@ -1540,7 +1551,7 @@ func (aws *AWS) GetDisks() ([]byte, error) {
 		// respective channels
 		go func(region string) {
 			defer wg.Done()
-			defer errors.HandlePanic()
+			defer errs.HandlePanic()
 
 			// Query for first page of volume results
 			resp, err := aws.getDisksForRegion(context.TODO(), region, 1000, nil)
@@ -1565,7 +1576,7 @@ func (aws *AWS) GetDisks() ([]byte, error) {
 
 	// Close the result channels after everything has been sent
 	go func() {
-		defer errors.HandlePanic()
+		defer errs.HandlePanic()
 
 		wg.Wait()
 		close(errorCh)
@@ -1599,6 +1610,10 @@ func (aws *AWS) GetDisks() ([]byte, error) {
 	})
 }
 
+func (*AWS) GetOrphanedResources() ([]OrphanedResource, error) {
+	return nil, errors.New("not implemented")
+}
+
 // QueryAthenaPaginated executes athena query and processes results.
 func (aws *AWS) QueryAthenaPaginated(ctx context.Context, query string, fn func(*athena.GetQueryResultsOutput) bool) error {
 	awsAthenaInfo, err := aws.GetAWSAthenaInfo()

pkg/cloud/azureprovider.go

@@ -4,22 +4,28 @@ import (
 	"context"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"net/http"
 	"net/url"
+	"os"
 	"regexp"
 	"strconv"
 	"strings"
 	"sync"
 	"time"
 
+	"github.com/opencost/opencost/pkg/kubecost"
+
 	"github.com/opencost/opencost/pkg/clustercache"
 	"github.com/opencost/opencost/pkg/env"
 	"github.com/opencost/opencost/pkg/log"
 	"github.com/opencost/opencost/pkg/util"
 	"github.com/opencost/opencost/pkg/util/fileutil"
 	"github.com/opencost/opencost/pkg/util/json"
+	"github.com/opencost/opencost/pkg/util/timeutil"
+	"golang.org/x/text/cases"
+	"golang.org/x/text/language"
 
+	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2021-11-01/compute"
 	"github.com/Azure/azure-sdk-for-go/services/preview/commerce/mgmt/2015-06-01-preview/commerce"
 	"github.com/Azure/azure-sdk-for-go/services/resources/mgmt/2016-06-01/subscriptions"
 	"github.com/Azure/azure-sdk-for-go/services/resources/mgmt/2018-05-01/resources"
@@ -40,6 +46,8 @@ const (
 	AzureStorageUpdateType           = "AzureStorage"
 )
 
+var toTitle = cases.Title(language.Und, cases.NoLower)
+
 var (
 	regionCodeMappings = map[string]string{
 		"ap": "asia",
@@ -271,7 +279,7 @@ func getRetailPrice(region string, skuName string, currencyCode string, spot boo
 
 	pricingPayload := AzureRetailPricing{}
 
-	body, err := ioutil.ReadAll(resp.Body)
+	body, err := io.ReadAll(resp.Body)
 	if err != nil {
 		return "", fmt.Errorf("Error getting response: %v", err)
 	}
@@ -355,7 +363,7 @@ type AzureRetailPricing struct {
 	Count              int                            `json:"Count"`
 }
 
-//AzureRetailPricingAttributes struct for unmarshalling Azure Retail pricing api JSON response
+// AzureRetailPricingAttributes struct for unmarshalling Azure Retail pricing api JSON response
 type AzureRetailPricingAttributes struct {
 	CurrencyCode         string     `json:"currencyCode"`
 	TierMinimumUnits     float32    `json:"tierMinimumUnits"`
@@ -415,6 +423,10 @@ func (k *azureKey) Features() string {
 	return fmt.Sprintf("%s,%s,%s", region, instance, usageType)
 }
 
+func (k *azureKey) GPUCount() int {
+	return 0
+}
+
 // GPUType returns value of GPULabel if present
 func (k *azureKey) GPUType() string {
 	if t, ok := k.Labels[k.GPULabel]; ok {
@@ -531,9 +543,15 @@ func (az *Azure) getAzureRateCardAuth(forceReload bool, cp *CustomPricing) (subs
 		tenantID = cp.AzureTenantID
 		return
 	}
-
-	// 3. Empty values
+	// 3. Check if AzureSubscriptionID is set in config (set though endpoint)
+	// MSI credentials will be attempted if the subscription ID is set, but clientID, clientSecret and tenantID are not
+	if cp.AzureSubscriptionID != "" {
+		subscriptionID = cp.AzureSubscriptionID
+		return
+	}
+	// 4. Empty values
 	return "", "", "", ""
+
 }
 
 // GetAzureStorageConfig retrieves storage config from secret and sets default values
@@ -571,12 +589,12 @@ func (az *Azure) GetAzureStorageConfig(forceReload bool, cp *CustomPricing) (*Az
 			asc.SubscriptionId = defaultSubscriptionID
 		}
 		// check for required fields
-		if asc.AccessKey != "" && asc.AccountName != "" && asc.ContainerName != "" && asc.SubscriptionId == "" {
+		if asc.AccessKey != "" && asc.AccountName != "" && asc.ContainerName != "" && asc.SubscriptionId != "" {
 			az.serviceAccountChecks.set("hasStorage", &ServiceAccountCheck{
 				Message: "Azure Storage Config exists",
 				Status:  true,
 			})
-			
+
 			return asc, nil
 		}
 	}
@@ -603,7 +621,7 @@ func (az *Azure) loadAzureAuthSecret(force bool) (*AzureServiceKey, error) {
 		return nil, fmt.Errorf("Failed to locate service account file: %s", authSecretPath)
 	}
 
-	result, err := ioutil.ReadFile(authSecretPath)
+	result, err := os.ReadFile(authSecretPath)
 	if err != nil {
 		return nil, err
 	}
@@ -632,7 +650,7 @@ func (az *Azure) loadAzureStorageConfig(force bool) (*AzureStorageConfig, error)
 		return nil, fmt.Errorf("Failed to locate azure storage config file: %s", storageConfigSecretPath)
 	}
 
-	result, err := ioutil.ReadFile(storageConfigSecretPath)
+	result, err := os.ReadFile(storageConfigSecretPath)
 	if err != nil {
 		return nil, err
 	}
@@ -1155,8 +1173,150 @@ func (*Azure) GetAddresses() ([]byte, error) {
 	return nil, nil
 }
 
-func (*Azure) GetDisks() ([]byte, error) {
-	return nil, nil
+func (az *Azure) GetDisks() ([]byte, error) {
+	disks, err := az.getDisks()
+	if err != nil {
+		return nil, err
+	}
+
+	return json.Marshal(disks)
+}
+
+func (az *Azure) getDisks() ([]*compute.Disk, error) {
+	config, err := az.GetConfig()
+	if err != nil {
+		return nil, err
+	}
+
+	// Load the service provider keys
+	subscriptionID, clientID, clientSecret, tenantID := az.getAzureRateCardAuth(false, config)
+	config.AzureSubscriptionID = subscriptionID
+	config.AzureClientID = clientID
+	config.AzureClientSecret = clientSecret
+	config.AzureTenantID = tenantID
+
+	var authorizer autorest.Authorizer
+
+	azureEnv := determineCloudByRegion(az.clusterRegion)
+
+	if config.AzureClientID != "" && config.AzureClientSecret != "" && config.AzureTenantID != "" {
+		credentialsConfig := NewClientCredentialsConfig(config.AzureClientID, config.AzureClientSecret, config.AzureTenantID, azureEnv)
+		a, err := credentialsConfig.Authorizer()
+		if err != nil {
+			az.RateCardPricingError = err
+			return nil, err
+		}
+		authorizer = a
+	}
+
+	if authorizer == nil {
+		a, err := auth.NewAuthorizerFromEnvironment()
+		authorizer = a
+		if err != nil {
+			a, err := auth.NewAuthorizerFromFile(azureEnv.ResourceManagerEndpoint)
+			if err != nil {
+				az.RateCardPricingError = err
+				return nil, err
+			}
+			authorizer = a
+		}
+	}
+	client := compute.NewDisksClient(config.AzureSubscriptionID)
+	client.Authorizer = authorizer
+
+	ctx := context.TODO()
+
+	var disks []*compute.Disk
+
+	diskPage, err := client.List(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("error getting disks: %v", err)
+	}
+
+	for diskPage.NotDone() {
+		for _, d := range diskPage.Values() {
+			d := d
+			disks = append(disks, &d)
+		}
+		err := diskPage.Next()
+		if err != nil {
+			return nil, fmt.Errorf("error getting next page: %v", err)
+		}
+	}
+
+	return disks, nil
+}
+
+func isDiskOrphaned(disk *compute.Disk) bool {
+	//TODO: needs better algorithm
+	return disk.DiskState == "Unattached" || disk.DiskState == "Reserved"
+}
+
+func (az *Azure) GetOrphanedResources() ([]OrphanedResource, error) {
+	disks, err := az.getDisks()
+	if err != nil {
+		return nil, err
+	}
+
+	var orphanedResources []OrphanedResource
+
+	for _, d := range disks {
+		if isDiskOrphaned(d) {
+			cost, err := az.findCostForDisk(d)
+			if err != nil {
+				return nil, err
+			}
+
+			diskName := ""
+			if d.Name != nil {
+				diskName = *d.Name
+			}
+
+			diskRegion := ""
+			if d.Location != nil {
+				diskRegion = *d.Location
+			}
+
+			or := OrphanedResource{
+				Kind:   "disk",
+				Region: diskRegion,
+				Description: map[string]string{
+					"diskState":   string(d.DiskState),
+					"timeCreated": d.TimeCreated.String(),
+				},
+				Size:        d.DiskSizeGB,
+				DiskName:    diskName,
+				MonthlyCost: &cost,
+			}
+			orphanedResources = append(orphanedResources, or)
+		}
+	}
+
+	return orphanedResources, nil
+}
+
+func (az *Azure) findCostForDisk(d *compute.Disk) (float64, error) {
+	if d == nil {
+		return 0.0, fmt.Errorf("disk is empty")
+	}
+	storageClass := string(d.Sku.Name)
+	if strings.EqualFold(storageClass, "Premium_LRS") {
+		storageClass = AzureDiskPremiumSSDStorageClass
+	} else if strings.EqualFold(storageClass, "StandardSSD_LRS") {
+		storageClass = AzureDiskStandardSSDStorageClass
+	} else if strings.EqualFold(storageClass, "Standard_LRS") {
+		storageClass = AzureDiskStandardStorageClass
+	}
+
+	key := *d.Location + "," + storageClass
+
+	diskPricePerGBHour, err := strconv.ParseFloat(az.Pricing[key].PV.Cost, 64)
+	if err != nil {
+		return 0.0, fmt.Errorf("error converting to float: %s", err)
+	}
+	cost := diskPricePerGBHour * timeutil.HoursPerMonth * float64(*d.DiskSizeGB)
+
+	return cost, nil
 }
 
 func (az *Azure) ClusterInfo() (map[string]string, error) {
@@ -1171,7 +1331,7 @@ func (az *Azure) ClusterInfo() (map[string]string, error) {
 	if c.ClusterName != "" {
 		m["name"] = c.ClusterName
 	}
-	m["provider"] = "Azure"
+	m["provider"] = kubecost.AzureProvider
 	m["account"] = az.clusterAccountId
 	m["region"] = az.clusterRegion
 	m["remoteReadEnabled"] = strconv.FormatBool(remoteEnabled)
@@ -1190,7 +1350,7 @@ func (az *Azure) UpdateConfig(r io.Reader, updateType string) (*CustomPricing, e
 			asc := &AzureStorageConfig{}
 			err := json.NewDecoder(r).Decode(&asc)
 			if err != nil {
-				return err
+				return fmt.Errorf("error decoding AzureStorageConfig: %s", err)
 			}
 
 			c.AzureStorageSubscriptionID = asc.SubscriptionId
@@ -1202,35 +1362,46 @@ func (az *Azure) UpdateConfig(r io.Reader, updateType string) (*CustomPricing, e
 			c.AzureContainerPath = asc.ContainerPath
 			c.AzureCloud = asc.AzureCloud
 		} else {
-			defer az.DownloadPricingData()
+			// This will block if not in a goroutine. It calls GetConfig(), which
+			// in turn calls GetCustomPricingData, which acquires the same lock
+			// that is acquired by az.Config.Update, which is the function to
+			// which this function gets passed, and subsequently called. Booo.
+			defer func() {
+				go az.DownloadPricingData()
+			}()
+
 			a := make(map[string]interface{})
 			err := json.NewDecoder(r).Decode(&a)
 			if err != nil {
-				return err
+				return fmt.Errorf("error decoding AzureStorageConfig: %s", err)
 			}
+
 			for k, v := range a {
-				kUpper := strings.Title(k) // Just so we consistently supply / receive the same values, uppercase the first letter.
+				// Just so we consistently supply / receive the same values, uppercase the first letter.
+				kUpper := toTitle.String(k)
 				vstr, ok := v.(string)
 				if ok {
 					err := SetCustomPricingField(c, kUpper, vstr)
 					if err != nil {
-						return err
+						return fmt.Errorf("error setting custom pricing field on AzureStorageConfig: %s", err)
 					}
 				} else {
 					return fmt.Errorf("type error while updating config for %s", kUpper)
 				}
 			}
 		}
+
 		if env.IsRemoteEnabled() {
 			err := UpdateClusterMeta(env.GetClusterID(), c.ClusterName)
 			if err != nil {
-				return err
+				return fmt.Errorf("error updating cluster metadata: %s", err)
 			}
 		}
 
 		return nil
 	})
 }
+
 func (az *Azure) GetConfig() (*CustomPricing, error) {
 	c, err := az.Config.GetCustomPricingData()
 	if err != nil {

pkg/cloud/csvprovider.go

@@ -33,6 +33,8 @@ type CSVProvider struct {
 	NodeMapField            string
 	PricingPV               map[string]*price
 	PVMapField              string
+	GPUClassPricing         map[string]*price
+	GPUMapFields            []string // Fields in a node's labels that represent the GPU class.
 	UsesRegion              bool
 	DownloadPricingDataLock sync.RWMutex
 }
@@ -59,6 +61,8 @@ func (c *CSVProvider) DownloadPricingData() error {
 	nodeclasspricing := make(map[string]float64)
 	nodeclasscount := make(map[string]float64)
 	pvpricing := make(map[string]*price)
+	gpupricing := make(map[string]*price)
+	c.GPUMapFields = make([]string, 0, 1)
 	header, err := csvutil.Header(price{}, "csv")
 	if err != nil {
 		return err
@@ -87,6 +91,7 @@ func (c *CSVProvider) DownloadPricingData() error {
 			c.NodeClassPricing = nodeclasspricing
 			c.NodeClassCount = nodeclasscount
 			c.PricingPV = pvpricing
+			c.GPUClassPricing = gpupricing
 			return fmt.Errorf("Invalid s3 URI: %s", c.CSVLocation)
 		}
 	} else {
@@ -98,6 +103,7 @@ func (c *CSVProvider) DownloadPricingData() error {
 		c.NodeClassPricing = nodeclasspricing
 		c.NodeClassCount = nodeclasscount
 		c.PricingPV = pvpricing
+		c.GPUClassPricing = gpupricing
 		return nil
 	}
 	csvReader := csv.NewReader(csvr)
@@ -110,6 +116,7 @@ func (c *CSVProvider) DownloadPricingData() error {
 		c.NodeClassPricing = nodeclasspricing
 		c.NodeClassCount = nodeclasscount
 		c.PricingPV = pvpricing
+		c.GPUClassPricing = gpupricing
 		return err
 	}
 	for {
@@ -163,6 +170,9 @@ func (c *CSVProvider) DownloadPricingData() error {
 			}
 
 			c.NodeMapField = p.InstanceIDField
+		} else if p.AssetClass == "gpu" {
+			gpupricing[key] = &p
+			c.GPUMapFields = append(c.GPUMapFields, strings.ToLower(p.InstanceIDField))
 		} else {
 			log.Infof("Unrecognized asset class %s, defaulting to node", p.AssetClass)
 			pricing[key] = &p
@@ -174,6 +184,7 @@ func (c *CSVProvider) DownloadPricingData() error {
 		c.NodeClassPricing = nodeclasspricing
 		c.NodeClassCount = nodeclasscount
 		c.PricingPV = pvpricing
+		c.GPUClassPricing = gpupricing
 	} else {
 		log.DedupedWarningf(5, "No data received from csv at %s", c.CSVLocation)
 	}
@@ -183,6 +194,8 @@ func (c *CSVProvider) DownloadPricingData() error {
 type csvKey struct {
 	Labels     map[string]string
 	ProviderID string
+	GPULabel   []string
+	GPU        int64
 }
 
 func (k *csvKey) Features() string {
@@ -192,7 +205,17 @@ func (k *csvKey) Features() string {
 
 	return region + "," + instanceType + "," + class
 }
+
+func (k *csvKey) GPUCount() int {
+	return int(k.GPU)
+}
+
 func (k *csvKey) GPUType() string {
+	for _, label := range k.GPULabel {
+		if val, ok := k.Labels[label]; ok {
+			return val
+		}
+	}
 	return ""
 }
 func (k *csvKey) ID() string {
@@ -202,30 +225,56 @@ func (k *csvKey) ID() string {
 func (c *CSVProvider) NodePricing(key Key) (*Node, error) {
 	c.DownloadPricingDataLock.RLock()
 	defer c.DownloadPricingDataLock.RUnlock()
+	var node *Node
 	if p, ok := c.Pricing[key.ID()]; ok {
-		return &Node{
+		node = &Node{
 			Cost:        p.MarketPriceHourly,
 			PricingType: CsvExact,
-		}, nil
+		}
 	}
 	s := strings.Split(key.ID(), ",") // Try without a region to be sure
 	if len(s) == 2 {
 		if p, ok := c.Pricing[s[1]]; ok {
-			return &Node{
+			node = &Node{
 				Cost:        p.MarketPriceHourly,
 				PricingType: CsvExact,
-			}, nil
+			}
 		}
 	}
 	classKey := key.Features() // Use node attributes to try and do a class match
 	if cost, ok := c.NodeClassPricing[classKey]; ok {
 		log.Infof("Unable to find provider ID `%s`, using features:`%s`", key.ID(), key.Features())
-		return &Node{
+		node = &Node{
 			Cost:        fmt.Sprintf("%f", cost),
 			PricingType: CsvClass,
-		}, nil
+		}
+	}
+
+	if node != nil {
+		if t := key.GPUType(); t != "" {
+			t = strings.ToLower(t)
+			count := key.GPUCount()
+			node.GPU = strconv.Itoa(count)
+			hourly := 0.0
+			if p, ok := c.GPUClassPricing[t]; ok {
+				var err error
+				hourly, err = strconv.ParseFloat(p.MarketPriceHourly, 64)
+				if err != nil {
+					log.Errorf("Unable to parse %s as float", p.MarketPriceHourly)
+				}
+			}
+			totalCost := hourly * float64(count)
+			node.GPUCost = fmt.Sprintf("%f", totalCost)
+			nc, err := strconv.ParseFloat(node.Cost, 64)
+			if err != nil {
+				log.Errorf("Unable to parse %s as float", node.Cost)
+			}
+			node.Cost = fmt.Sprintf("%f", nc+totalCost)
+		}
+		return node, nil
+	} else {
+		return nil, fmt.Errorf("Unable to find Node matching `%s`:`%s`", key.ID(), key.Features())
 	}
-	return nil, fmt.Errorf("Unable to find Node matching `%s`:`%s`", key.ID(), key.Features())
 }
 
 func NodeValueFromMapField(m string, n *v1.Node, useRegion bool) string {
@@ -299,9 +348,16 @@ func PVValueFromMapField(m string, n *v1.PersistentVolume) string {
 
 func (c *CSVProvider) GetKey(l map[string]string, n *v1.Node) Key {
 	id := NodeValueFromMapField(c.NodeMapField, n, c.UsesRegion)
+	var gpuCount int64
+	gpuCount = 0
+	if gpuc, ok := n.Status.Capacity["nvidia.com/gpu"]; ok { // TODO: support non-nvidia GPUs
+		gpuCount = gpuc.Value()
+	}
 	return &csvKey{
 		ProviderID: id,
 		Labels:     l,
+		GPULabel:   c.GPUMapFields,
+		GPU:        gpuCount,
 	}
 }
 
@@ -368,3 +424,4 @@ func (c *CSVProvider) CombinedDiscountForNode(instanceType string, isPreemptible
 func (c *CSVProvider) Regions() []string {
 	return []string{}
 }
+

pkg/cloud/customprovider.go

@@ -1,7 +1,9 @@
 package cloud
 
 import (
+	"errors"
 	"fmt"
+	"github.com/opencost/opencost/pkg/kubecost"
 	"io"
 	"strconv"
 	"strings"
@@ -107,7 +109,7 @@ func (cp *CustomProvider) ClusterInfo() (map[string]string, error) {
 	if conf.ClusterName != "" {
 		m["name"] = conf.ClusterName
 	}
-	m["provider"] = "custom"
+	m["provider"] = kubecost.CustomProvider
 	m["id"] = env.GetClusterID()
 	return m, nil
 }
@@ -120,6 +122,10 @@ func (*CustomProvider) GetDisks() ([]byte, error) {
 	return nil, nil
 }
 
+func (*CustomProvider) GetOrphanedResources() ([]OrphanedResource, error) {
+	return nil, errors.New("not implemented")
+}
+
 func (cp *CustomProvider) AllNodePricing() (interface{}, error) {
 	cp.DownloadPricingDataLock.RLock()
 	defer cp.DownloadPricingDataLock.RUnlock()
@@ -277,6 +283,10 @@ func (*CustomProvider) GetPVKey(pv *v1.PersistentVolume, parameters map[string]s
 	}
 }
 
+func (k *customProviderKey) GPUCount() int {
+	return 0
+}
+
 func (cpk *customProviderKey) GPUType() string {
 	if t, ok := cpk.Labels[cpk.GPULabel]; ok {
 		return t

pkg/cloud/gcpprovider.go

@@ -2,11 +2,12 @@ package cloud
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"math"
 	"net/http"
+	"os"
 	"regexp"
 	"strconv"
 	"strings"
@@ -76,16 +77,6 @@ var (
 	gceRegex = regexp.MustCompile("gce://([^/]*)/*")
 )
 
-type userAgentTransport struct {
-	userAgent string
-	base      http.RoundTripper
-}
-
-func (t userAgentTransport) RoundTrip(req *http.Request) (*http.Response, error) {
-	req.Header.Set("User-Agent", t.userAgent)
-	return t.base.RoundTrip(req)
-}
-
 // GCP implements a provider interface for GCP
 type GCP struct {
 	Pricing                 map[string]*GCPPricing
@@ -99,6 +90,7 @@ type GCP struct {
 	Config                  *ProviderConfig
 	ServiceKeyProvided      bool
 	ValidPricingKeys        map[string]bool
+	metadataClient          *metadata.Client
 	clusterManagementPrice  float64
 	clusterProjectId        string
 	clusterRegion           string
@@ -218,13 +210,13 @@ func (*GCP) loadGCPAuthSecret() {
 		return
 	}
 
-	result, err := ioutil.ReadFile(authSecretPath)
+	result, err := os.ReadFile(authSecretPath)
 	if err != nil {
 		log.Warnf("Failed to load auth secret, or was not mounted: %s", err.Error())
 		return
 	}
 
-	err = ioutil.WriteFile(keyPath, result, 0644)
+	err = os.WriteFile(keyPath, result, 0644)
 	if err != nil {
 		log.Warnf("Failed to copy auth secret to %s: %s", keyPath, err.Error())
 	}
@@ -255,7 +247,7 @@ func (gcp *GCP) UpdateConfig(r io.Reader, updateType string) (*CustomPricing, er
 				path := env.GetConfigPathWithDefault("/models/")
 
 				keyPath := path + "key.json"
-				err = ioutil.WriteFile(keyPath, j, 0644)
+				err = os.WriteFile(keyPath, j, 0644)
 				if err != nil {
 					return err
 				}
@@ -310,12 +302,7 @@ func (gcp *GCP) UpdateConfig(r io.Reader, updateType string) (*CustomPricing, er
 func (gcp *GCP) ClusterInfo() (map[string]string, error) {
 	remoteEnabled := env.IsRemoteEnabled()
 
-	metadataClient := metadata.NewClient(&http.Client{Transport: userAgentTransport{
-		userAgent: "kubecost",
-		base:      http.DefaultTransport,
-	}})
-
-	attribute, err := metadataClient.InstanceAttributeValue("cluster-name")
+	attribute, err := gcp.metadataClient.InstanceAttributeValue("cluster-name")
 	if err != nil {
 		log.Infof("Error loading metadata cluster-name: %s", err.Error())
 	}
@@ -348,13 +335,8 @@ func (gcp *GCP) ClusterManagementPricing() (string, float64, error) {
 	return gcp.clusterProvisioner, gcp.clusterManagementPrice, nil
 }
 
-func (*GCP) GetAddresses() ([]byte, error) {
-	// metadata API setup
-	metadataClient := metadata.NewClient(&http.Client{Transport: userAgentTransport{
-		userAgent: "kubecost",
-		base:      http.DefaultTransport,
-	}})
-	projID, err := metadataClient.ProjectID()
+func (gcp *GCP) GetAddresses() ([]byte, error) {
+	projID, err := gcp.metadataClient.ProjectID()
 	if err != nil {
 		return nil, err
 	}
@@ -377,13 +359,8 @@ func (*GCP) GetAddresses() ([]byte, error) {
 }
 
 // GetDisks returns the GCP disks backing PVs. Useful because sometimes k8s will not clean up PVs correctly. Requires a json config in /var/configs with key region.
-func (*GCP) GetDisks() ([]byte, error) {
-	// metadata API setup
-	metadataClient := metadata.NewClient(&http.Client{Transport: userAgentTransport{
-		userAgent: "kubecost",
-		base:      http.DefaultTransport,
-	}})
-	projID, err := metadataClient.ProjectID()
+func (gcp *GCP) GetDisks() ([]byte, error) {
+	projID, err := gcp.metadataClient.ProjectID()
 	if err != nil {
 		return nil, err
 	}
@@ -406,6 +383,10 @@ func (*GCP) GetDisks() ([]byte, error) {
 
 }
 
+func (*GCP) GetOrphanedResources() ([]OrphanedResource, error) {
+	return nil, errors.New("not implemented")
+}
+
 // GCPPricing represents GCP pricing data for a SKU
 type GCPPricing struct {
 	Name                string           `json:"name"`
@@ -1261,6 +1242,10 @@ func (gcp *gcpKey) ID() string {
 	return ""
 }
 
+func (k *gcpKey) GPUCount() int {
+	return 0
+}
+
 func (gcp *gcpKey) GPUType() string {
 	if t, ok := gcp.Labels[GKE_GPU_TAG]; ok {
 		usageType := getUsageType(gcp.Labels)

pkg/cloud/provider.go

@@ -5,13 +5,13 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"net/http"
 	"regexp"
 	"strconv"
 	"strings"
 	"sync"
 	"time"
 
-
 	"github.com/opencost/opencost/pkg/kubecost"
 
 	"github.com/opencost/opencost/pkg/util"
@@ -22,6 +22,7 @@ import (
 	"github.com/opencost/opencost/pkg/config"
 	"github.com/opencost/opencost/pkg/env"
 	"github.com/opencost/opencost/pkg/log"
+	"github.com/opencost/opencost/pkg/util/httputil"
 	"github.com/opencost/opencost/pkg/util/watcher"
 
 	v1 "k8s.io/api/core/v1"
@@ -103,6 +104,16 @@ type Network struct {
 	InternetNetworkEgressCost float64
 }
 
+type OrphanedResource struct {
+	Kind        string            `json:"resourceKind"`
+	Region      string            `json:"region"`
+	Description map[string]string `json:"description"`
+	Size        *int32            `json:"diskSizeInGB,omitempty"`
+	DiskName    string            `json:"diskName,omitempty"`
+	Address     string            `json:"ipAddress,omitempty"`
+	MonthlyCost *float64          `json:"monthlyCost"`
+}
+
 // PV is the interface by which the provider and cost model communicate PV prices.
 // The provider will best-effort try to fill out this struct.
 type PV struct {
@@ -119,7 +130,8 @@ type PV struct {
 type Key interface {
 	ID() string       // ID represents an exact match
 	Features() string // Features are a comma separated string of node metadata that could match pricing
-	GPUType() string  // GPUType returns "" if no GPU exists, but the name of the GPU otherwise
+	GPUType() string  // GPUType returns "" if no GPU exists or GPUs, but the name of the GPU otherwise
+	GPUCount() int    // GPUCount returns 0 if no GPU exists or GPUs, but the number of attached GPUs otherwise
 }
 
 type PVKey interface {
@@ -159,6 +171,8 @@ type CustomPricing struct {
 	GpuLabelValue                string `json:"gpuLabelValue,omitempty"`
 	ServiceKeyName               string `json:"awsServiceKeyName,omitempty"`
 	ServiceKeySecret             string `json:"awsServiceKeySecret,omitempty"`
+	AlibabaServiceKeyName        string `json:"alibabaServiceKeyName,omitempty"`
+	AlibabaServiceKeySecret      string `json:"alibabaServiceKeySecret,omitempty"`
 	SpotDataRegion               string `json:"awsSpotDataRegion,omitempty"`
 	SpotDataBucket               string `json:"awsSpotDataBucket,omitempty"`
 	SpotDataPrefix               string `json:"awsSpotDataPrefix,omitempty"`
@@ -295,6 +309,7 @@ type Provider interface {
 	ClusterInfo() (map[string]string, error)
 	GetAddresses() ([]byte, error)
 	GetDisks() ([]byte, error)
+	GetOrphanedResources() ([]OrphanedResource, error)
 	NodePricing(Key) (*Node, error)
 	PVPricing(PVKey) (*PV, error)
 	NetworkPricing() (*Network, error)           // TODO: add key interface arg for dynamic price fetching
@@ -464,6 +479,9 @@ func NewProvider(cache clustercache.ClusterCache, apiKey string, config *config.
 			Config:           NewProviderConfig(config, cp.configFileName),
 			clusterRegion:    cp.region,
 			clusterProjectId: cp.projectID,
+			metadataClient: metadata.NewClient(&http.Client{
+				Transport: httputil.NewUserAgentTransport("kubecost", http.DefaultTransport),
+			}),
 		}, nil
 	case kubecost.AWSProvider:
 		log.Info("Found ProviderID starting with \"aws\", using AWS Provider")
@@ -483,6 +501,15 @@ func NewProvider(cache clustercache.ClusterCache, apiKey string, config *config.
 			clusterAccountId:     cp.accountID,
 			serviceAccountChecks: NewServiceAccountChecks(),
 		}, nil
+	case kubecost.AlibabaProvider:
+		log.Info("Found ProviderID starting with \"alibaba\", using Alibaba Cloud Provider")
+		return &Alibaba{
+			Clientset:            cache,
+			Config:               NewProviderConfig(config, cp.configFileName),
+			clusterRegion:        cp.region,
+			clusterAccountId:     cp.accountID,
+			serviceAccountChecks: NewServiceAccountChecks(),
+		}, nil
 	case kubecost.ScalewayProvider:
 		log.Info("Found ProviderID starting with \"scaleway\", using Scaleway Provider")
 		return &Scaleway{
@@ -531,6 +558,9 @@ func getClusterProperties(node *v1.Node) clusterProperties {
 	} else if strings.HasPrefix(providerID, "scaleway") { // the scaleway provider ID looks like scaleway://instance/<instance_id>
 		cp.provider = kubecost.ScalewayProvider
 		cp.configFileName = "scaleway.json"
+	} else if strings.Contains(node.Status.NodeInfo.KubeletVersion, "aliyun") { // provider ID is not prefix with any distinct keyword like other providers
+		cp.provider = kubecost.AlibabaProvider
+		cp.configFileName = "alibaba.json"
 	}
 	if env.IsUseCSVProvider() {
 		cp.provider = kubecost.CSVProvider

pkg/cloud/scalewayprovider.go

@@ -1,7 +1,9 @@
 package cloud
 
 import (
+	"errors"
 	"fmt"
+	"github.com/opencost/opencost/pkg/kubecost"
 	"io"
 	"strconv"
 	"strings"
@@ -104,6 +106,10 @@ func (k *scalewayKey) Features() string {
 	return zone + "," + instanceType
 }
 
+func (k *scalewayKey) GPUCount() int {
+	return 0
+}
+
 func (k *scalewayKey) GPUType() string {
 	instanceType, _ := util.GetInstanceType(k.Labels)
 	if strings.HasPrefix(instanceType, "RENDER") || strings.HasPrefix(instanceType, "GPU") {
@@ -246,6 +252,10 @@ func (*Scaleway) GetDisks() ([]byte, error) {
 	return nil, nil
 }
 
+func (*Scaleway) GetOrphanedResources() ([]OrphanedResource, error) {
+	return nil, errors.New("not implemented")
+}
+
 func (scw *Scaleway) ClusterInfo() (map[string]string, error) {
 	remoteEnabled := env.IsRemoteEnabled()
 
@@ -258,7 +268,7 @@ func (scw *Scaleway) ClusterInfo() (map[string]string, error) {
 	if c.ClusterName != "" {
 		m["name"] = c.ClusterName
 	}
-	m["provider"] = "Scaleway"
+	m["provider"] = kubecost.ScalewayProvider
 	m["remoteReadEnabled"] = strconv.FormatBool(remoteEnabled)
 	m["id"] = env.GetClusterID()
 	return m, nil

pkg/clustercache/clustercache.go

@@ -7,7 +7,6 @@ import (
 	"github.com/opencost/opencost/pkg/log"
 
 	appsv1 "k8s.io/api/apps/v1"
-	autoscaling "k8s.io/api/autoscaling/v2beta1"
 	batchv1 "k8s.io/api/batch/v1"
 	v1 "k8s.io/api/core/v1"
 	"k8s.io/api/policy/v1beta1"
@@ -61,9 +60,6 @@ type ClusterCache interface {
 	// GetAllJobs returns all the cached jobs
 	GetAllJobs() []*batchv1.Job
 
-	// GetAllHorizontalPodAutoscalers returns all cached horizontal pod autoscalers
-	GetAllHorizontalPodAutoscalers() []*autoscaling.HorizontalPodAutoscaler
-
 	// GetAllPodDisruptionBudgets returns all cached pod disruption budgets
 	GetAllPodDisruptionBudgets() []*v1beta1.PodDisruptionBudget
 
@@ -91,7 +87,6 @@ type KubernetesClusterCache struct {
 	pvcWatch                   WatchController
 	storageClassWatch          WatchController
 	jobsWatch                  WatchController
-	hpaWatch                   WatchController
 	pdbWatch                   WatchController
 	replicationControllerWatch WatchController
 	stop                       chan struct{}
@@ -107,7 +102,6 @@ func NewKubernetesClusterCache(client kubernetes.Interface) ClusterCache {
 	appsRestClient := client.AppsV1().RESTClient()
 	storageRestClient := client.StorageV1().RESTClient()
 	batchClient := client.BatchV1().RESTClient()
-	autoscalingClient := client.AutoscalingV2beta1().RESTClient()
 	pdbClient := client.PolicyV1beta1().RESTClient()
 
 	kubecostNamespace := env.GetKubecostNamespace()
@@ -128,7 +122,6 @@ func NewKubernetesClusterCache(client kubernetes.Interface) ClusterCache {
 		pvcWatch:                   NewCachingWatcher(coreRestClient, "persistentvolumeclaims", &v1.PersistentVolumeClaim{}, "", fields.Everything()),
 		storageClassWatch:          NewCachingWatcher(storageRestClient, "storageclasses", &stv1.StorageClass{}, "", fields.Everything()),
 		jobsWatch:                  NewCachingWatcher(batchClient, "jobs", &batchv1.Job{}, "", fields.Everything()),
-		hpaWatch:                   NewCachingWatcher(autoscalingClient, "horizontalpodautoscalers", &autoscaling.HorizontalPodAutoscaler{}, "", fields.Everything()),
 		pdbWatch:                   NewCachingWatcher(pdbClient, "poddisruptionbudgets", &v1beta1.PodDisruptionBudget{}, "", fields.Everything()),
 		replicationControllerWatch: NewCachingWatcher(coreRestClient, "replicationcontrollers", &v1.ReplicationController{}, "", fields.Everything()),
 	}
@@ -140,7 +133,7 @@ func NewKubernetesClusterCache(client kubernetes.Interface) ClusterCache {
 		wg.Add(1)
 		go initializeCache(kcc.kubecostConfigMapWatch, &wg, cancel)
 	} else {
-		wg.Add(16)
+		wg.Add(15)
 		go initializeCache(kcc.kubecostConfigMapWatch, &wg, cancel)
 		go initializeCache(kcc.namespaceWatch, &wg, cancel)
 		go initializeCache(kcc.nodeWatch, &wg, cancel)
@@ -154,7 +147,6 @@ func NewKubernetesClusterCache(client kubernetes.Interface) ClusterCache {
 		go initializeCache(kcc.pvcWatch, &wg, cancel)
 		go initializeCache(kcc.storageClassWatch, &wg, cancel)
 		go initializeCache(kcc.jobsWatch, &wg, cancel)
-		go initializeCache(kcc.hpaWatch, &wg, cancel)
 		go initializeCache(kcc.podWatch, &wg, cancel)
 		go initializeCache(kcc.replicationControllerWatch, &wg, cancel)
 	}
@@ -185,7 +177,6 @@ func (kcc *KubernetesClusterCache) Run() {
 	go kcc.pvcWatch.Run(1, stopCh)
 	go kcc.storageClassWatch.Run(1, stopCh)
 	go kcc.jobsWatch.Run(1, stopCh)
-	go kcc.hpaWatch.Run(1, stopCh)
 	go kcc.pdbWatch.Run(1, stopCh)
 	go kcc.replicationControllerWatch.Run(1, stopCh)
 
@@ -309,15 +300,6 @@ func (kcc *KubernetesClusterCache) GetAllJobs() []*batchv1.Job {
 	return jobs
 }
 
-func (kcc *KubernetesClusterCache) GetAllHorizontalPodAutoscalers() []*autoscaling.HorizontalPodAutoscaler {
-	var hpas []*autoscaling.HorizontalPodAutoscaler
-	items := kcc.hpaWatch.GetAll()
-	for _, hpa := range items {
-		hpas = append(hpas, hpa.(*autoscaling.HorizontalPodAutoscaler))
-	}
-	return hpas
-}
-
 func (kcc *KubernetesClusterCache) GetAllPodDisruptionBudgets() []*v1beta1.PodDisruptionBudget {
 	var pdbs []*v1beta1.PodDisruptionBudget
 	items := kcc.pdbWatch.GetAll()

pkg/clustercache/clusterexporter.go

@@ -9,7 +9,6 @@ import (
 	"github.com/opencost/opencost/pkg/util/json"
 
 	appsv1 "k8s.io/api/apps/v1"
-	autoscaling "k8s.io/api/autoscaling/v2beta1"
 	batchv1 "k8s.io/api/batch/v1"
 	v1 "k8s.io/api/core/v1"
 	"k8s.io/api/policy/v1beta1"
@@ -18,21 +17,20 @@ import (
 
 // clusterEncoding is used to represent the cluster objects in the encoded states.
 type clusterEncoding struct {
-	Namespaces               []*v1.Namespace                        `json:"namespaces,omitempty"`
-	Nodes                    []*v1.Node                             `json:"nodes,omitempty"`
-	Pods                     []*v1.Pod                              `json:"pods,omitempty"`
-	Services                 []*v1.Service                          `json:"services,omitempty"`
-	DaemonSets               []*appsv1.DaemonSet                    `json:"daemonSets,omitempty"`
-	Deployments              []*appsv1.Deployment                   `json:"deployments,omitempty"`
-	StatefulSets             []*appsv1.StatefulSet                  `json:"statefulSets,omitempty"`
-	ReplicaSets              []*appsv1.ReplicaSet                   `json:"replicaSets,omitempty"`
-	PersistentVolumes        []*v1.PersistentVolume                 `json:"persistentVolumes,omitempty"`
-	PersistentVolumeClaims   []*v1.PersistentVolumeClaim            `json:"persistentVolumeClaims,omitempty"`
-	StorageClasses           []*stv1.StorageClass                   `json:"storageClasses,omitempty"`
-	Jobs                     []*batchv1.Job                         `json:"jobs,omitempty"`
-	HorizontalPodAutoscalers []*autoscaling.HorizontalPodAutoscaler `json:"horizontalPodAutoscalers,omitempty"`
-	PodDisruptionBudgets     []*v1beta1.PodDisruptionBudget         `json:"podDisruptionBudgets,omitEmpty"`
-	ReplicationControllers   []*v1.ReplicationController            `json:"replicationController,omitEmpty"`
+	Namespaces             []*v1.Namespace                `json:"namespaces,omitempty"`
+	Nodes                  []*v1.Node                     `json:"nodes,omitempty"`
+	Pods                   []*v1.Pod                      `json:"pods,omitempty"`
+	Services               []*v1.Service                  `json:"services,omitempty"`
+	DaemonSets             []*appsv1.DaemonSet            `json:"daemonSets,omitempty"`
+	Deployments            []*appsv1.Deployment           `json:"deployments,omitempty"`
+	StatefulSets           []*appsv1.StatefulSet          `json:"statefulSets,omitempty"`
+	ReplicaSets            []*appsv1.ReplicaSet           `json:"replicaSets,omitempty"`
+	PersistentVolumes      []*v1.PersistentVolume         `json:"persistentVolumes,omitempty"`
+	PersistentVolumeClaims []*v1.PersistentVolumeClaim    `json:"persistentVolumeClaims,omitempty"`
+	StorageClasses         []*stv1.StorageClass           `json:"storageClasses,omitempty"`
+	Jobs                   []*batchv1.Job                 `json:"jobs,omitempty"`
+	PodDisruptionBudgets   []*v1beta1.PodDisruptionBudget `json:"podDisruptionBudgets,omitEmpty"`
+	ReplicationControllers []*v1.ReplicationController    `json:"replicationController,omitEmpty"`
 }
 
 // ClusterExporter manages and runs an file export process which dumps the local kubernetes cluster to a target location.
@@ -90,21 +88,20 @@ func (ce *ClusterExporter) Stop() {
 func (ce *ClusterExporter) Export() error {
 	c := ce.cluster
 	encoding := &clusterEncoding{
-		Namespaces:               c.GetAllNamespaces(),
-		Nodes:                    c.GetAllNodes(),
-		Pods:                     c.GetAllPods(),
-		Services:                 c.GetAllServices(),
-		DaemonSets:               c.GetAllDaemonSets(),
-		Deployments:              c.GetAllDeployments(),
-		StatefulSets:             c.GetAllStatefulSets(),
-		ReplicaSets:              c.GetAllReplicaSets(),
-		PersistentVolumes:        c.GetAllPersistentVolumes(),
-		PersistentVolumeClaims:   c.GetAllPersistentVolumeClaims(),
-		StorageClasses:           c.GetAllStorageClasses(),
-		Jobs:                     c.GetAllJobs(),
-		HorizontalPodAutoscalers: c.GetAllHorizontalPodAutoscalers(),
-		PodDisruptionBudgets:     c.GetAllPodDisruptionBudgets(),
-		ReplicationControllers:   c.GetAllReplicationControllers(),
+		Namespaces:             c.GetAllNamespaces(),
+		Nodes:                  c.GetAllNodes(),
+		Pods:                   c.GetAllPods(),
+		Services:               c.GetAllServices(),
+		DaemonSets:             c.GetAllDaemonSets(),
+		Deployments:            c.GetAllDeployments(),
+		StatefulSets:           c.GetAllStatefulSets(),
+		ReplicaSets:            c.GetAllReplicaSets(),
+		PersistentVolumes:      c.GetAllPersistentVolumes(),
+		PersistentVolumeClaims: c.GetAllPersistentVolumeClaims(),
+		StorageClasses:         c.GetAllStorageClasses(),
+		Jobs:                   c.GetAllJobs(),
+		PodDisruptionBudgets:   c.GetAllPodDisruptionBudgets(),
+		ReplicationControllers: c.GetAllReplicationControllers(),
 	}
 
 	data, err := json.Marshal(encoding)

pkg/clustercache/clusterimporter.go

@@ -7,7 +7,6 @@ import (
 	"github.com/opencost/opencost/pkg/log"
 	"github.com/opencost/opencost/pkg/util/json"
 	appsv1 "k8s.io/api/apps/v1"
-	autoscaling "k8s.io/api/autoscaling/v2beta1"
 	batchv1 "k8s.io/api/batch/v1"
 	v1 "k8s.io/api/core/v1"
 	"k8s.io/api/policy/v1beta1"
@@ -271,21 +270,6 @@ func (ci *ClusterImporter) GetAllJobs() []*batchv1.Job {
 	return cloneList
 }
 
-// GetAllHorizontalPodAutoscalers() returns all cached horizontal pod autoscalers
-func (ci *ClusterImporter) GetAllHorizontalPodAutoscalers() []*autoscaling.HorizontalPodAutoscaler {
-	ci.dataLock.Lock()
-	defer ci.dataLock.Unlock()
-
-	// Deep copy here to avoid callers from corrupting the cache
-	// This also mimics the behavior of the default cluster cache impl.
-	hpas := ci.data.HorizontalPodAutoscalers
-	cloneList := make([]*autoscaling.HorizontalPodAutoscaler, 0, len(hpas))
-	for _, v := range hpas {
-		cloneList = append(cloneList, v.DeepCopy())
-	}
-	return cloneList
-}
-
 // GetAllPodDisruptionBudgets returns all cached pod disruption budgets
 func (ci *ClusterImporter) GetAllPodDisruptionBudgets() []*v1beta1.PodDisruptionBudget {
 	ci.dataLock.Lock()

pkg/config/configmanager.go

@@ -1,7 +1,7 @@
 package config
 
 import (
-	"io/ioutil"
+	"os"
 	"sync"
 
 	"github.com/opencost/opencost/pkg/log"
@@ -59,7 +59,7 @@ func NewConfigFileManager(opts *ConfigFileManagerOpts) *ConfigFileManager {
 
 	var configStore storage.Storage
 	if opts.IsBucketStorageEnabled() {
-		bucketConfig, err := ioutil.ReadFile(opts.BucketStoreConfig)
+		bucketConfig, err := os.ReadFile(opts.BucketStoreConfig)
 		if err != nil {
 			log.Warnf("Failed to initialize config bucket storage: %s", err)
 		} else {

pkg/costmodel/aggregation.go

@@ -1012,38 +1012,38 @@ func compressVectorSeries(vs []*util.Vector, resolutionHours float64) []*util.Ve
 }
 
 type AggregateQueryOpts struct {
-	Rate                  string
-	Filters               map[string]string
-	SharedResources       *SharedResourceInfo
-	ShareSplit            string
-	AllocateIdle          bool
-	IncludeTimeSeries     bool
-	IncludeEfficiency     bool
-	DisableCache          bool
-	ClearCache            bool
-	NoCache               bool
-	NoExpireCache         bool
-	RemoteEnabled         bool
-	DisableSharedOverhead bool
-	UseETLAdapter         bool
+	Rate                           string
+	Filters                        map[string]string
+	SharedResources                *SharedResourceInfo
+	ShareSplit                     string
+	AllocateIdle                   bool
+	IncludeTimeSeries              bool
+	IncludeEfficiency              bool
+	DisableAggregateCostModelCache bool
+	ClearCache                     bool
+	NoCache                        bool
+	NoExpireCache                  bool
+	RemoteEnabled                  bool
+	DisableSharedOverhead          bool
+	UseETLAdapter                  bool
 }
 
 func DefaultAggregateQueryOpts() *AggregateQueryOpts {
 	return &AggregateQueryOpts{
-		Rate:                  "",
-		Filters:               map[string]string{},
-		SharedResources:       nil,
-		ShareSplit:            SplitTypeWeighted,
-		AllocateIdle:          false,
-		IncludeTimeSeries:     true,
-		IncludeEfficiency:     true,
-		DisableCache:          false,
-		ClearCache:            false,
-		NoCache:               false,
-		NoExpireCache:         false,
-		RemoteEnabled:         env.IsRemoteEnabled(),
-		DisableSharedOverhead: false,
-		UseETLAdapter:         false,
+		Rate:                           "",
+		Filters:                        map[string]string{},
+		SharedResources:                nil,
+		ShareSplit:                     SplitTypeWeighted,
+		AllocateIdle:                   false,
+		IncludeTimeSeries:              true,
+		IncludeEfficiency:              true,
+		DisableAggregateCostModelCache: env.IsAggregateCostModelCacheDisabled(),
+		ClearCache:                     false,
+		NoCache:                        false,
+		NoExpireCache:                  false,
+		RemoteEnabled:                  env.IsRemoteEnabled(),
+		DisableSharedOverhead:          false,
+		UseETLAdapter:                  false,
 	}
 }
 
@@ -1095,7 +1095,7 @@ func (a *Accesses) ComputeAggregateCostModel(promClient prometheusClient.Client,
 	allocateIdle := opts.AllocateIdle
 	includeTimeSeries := opts.IncludeTimeSeries
 	includeEfficiency := opts.IncludeEfficiency
-	disableCache := opts.DisableCache
+	disableAggregateCostModelCache := opts.DisableAggregateCostModelCache
 	clearCache := opts.ClearCache
 	noCache := opts.NoCache
 	noExpireCache := opts.NoExpireCache
@@ -1377,7 +1377,7 @@ func (a *Accesses) ComputeAggregateCostModel(promClient prometheusClient.Client,
 	cacheMessage := fmt.Sprintf("ComputeAggregateCostModel: L1 cache miss: %s L2 cache miss: %s", aggKey, key)
 
 	// check the cache for aggregated response; if cache is hit and not disabled, return response
-	if value, found := a.AggregateCache.Get(aggKey); found && !disableCache && !noCache {
+	if value, found := a.AggregateCache.Get(aggKey); found && !disableAggregateCostModelCache && !noCache {
 		result, ok := value.(map[string]*Aggregation)
 		if !ok {
 			// disable cache and recompute if type cast fails
@@ -1393,14 +1393,14 @@ func (a *Accesses) ComputeAggregateCostModel(promClient prometheusClient.Client,
 		window.Set(&start, window.End())
 	} else {
 		// don't cache requests for durations of less than one hour
-		disableCache = true
+		disableAggregateCostModelCache = true
 	}
 
 	// attempt to retrieve cost data from cache
 	var costData map[string]*CostData
 	var err error
 	cacheData, found := a.CostDataCache.Get(key)
-	if found && !disableCache && !noCache {
+	if found && !disableAggregateCostModelCache && !noCache {
 		ok := false
 		costData, ok = cacheData.(map[string]*CostData)
 		cacheMessage = fmt.Sprintf("ComputeAggregateCostModel: L1 cache miss: %s, L2 cost data cache hit: %s", aggKey, key)
@@ -1408,7 +1408,7 @@ func (a *Accesses) ComputeAggregateCostModel(promClient prometheusClient.Client,
 			log.Errorf("ComputeAggregateCostModel: caching error: failed to cast cost data to struct: %s", key)
 		}
 	} else {
-		log.Infof("ComputeAggregateCostModel: missed cache: %s (found %t, disableCache %t, noCache %t)", key, found, disableCache, noCache)
+		log.Infof("ComputeAggregateCostModel: missed cache: %s (found %t, disableAggregateCostModelCache %t, noCache %t)", key, found, disableAggregateCostModelCache, noCache)
 
 		costData, err = a.Model.ComputeCostDataRange(promClient, a.CloudProvider, window, resolution, "", "", remoteEnabled)
 		if err != nil {
@@ -1761,7 +1761,7 @@ func (a *Accesses) warmAggregateCostModelCache() {
 		aggOpts.Filters = map[string]string{}
 		aggOpts.IncludeTimeSeries = false
 		aggOpts.IncludeEfficiency = true
-		aggOpts.DisableCache = true
+		aggOpts.DisableAggregateCostModelCache = true
 		aggOpts.ClearCache = false
 		aggOpts.NoCache = false
 		aggOpts.NoExpireCache = false
@@ -1990,7 +1990,7 @@ func (a *Accesses) AggregateCostModelHandler(w http.ResponseWriter, r *http.Requ
 	// TODO niko/caching rename "recomputeCache"
 	// disableCache, if set to "true", tells this function to recompute and
 	// cache the requested data
-	opts.DisableCache = r.URL.Query().Get("disableCache") == "true"
+	opts.DisableAggregateCostModelCache = r.URL.Query().Get("disableCache") == "true"
 
 	// clearCache, if set to "true", tells this function to flush the cache,
 	// then recompute and cache the requested data

pkg/costmodel/allocation.go

@@ -2,19 +2,14 @@ package costmodel
 
 import (
 	"fmt"
-	"math"
-	"strconv"
-	"strings"
 	"time"
 
 	"github.com/opencost/opencost/pkg/util/timeutil"
 
-	"github.com/opencost/opencost/pkg/cloud"
 	"github.com/opencost/opencost/pkg/env"
 	"github.com/opencost/opencost/pkg/kubecost"
 	"github.com/opencost/opencost/pkg/log"
 	"github.com/opencost/opencost/pkg/prom"
-	"k8s.io/apimachinery/pkg/labels"
 )
 
 const (
@@ -35,9 +30,10 @@ const (
 	queryFmtNodeCostPerGPUHr         = `avg(avg_over_time(node_gpu_hourly_cost[%s])) by (node, %s, instance_type, provider_id)`
 	queryFmtNodeIsSpot               = `avg_over_time(kubecost_node_is_spot[%s])`
 	queryFmtPVCInfo                  = `avg(kube_persistentvolumeclaim_info{volumename != ""}) by (persistentvolumeclaim, storageclass, volumename, namespace, %s)[%s:%s]`
-	queryFmtPVBytes                  = `avg(avg_over_time(kube_persistentvolume_capacity_bytes[%s])) by (persistentvolume, %s)`
 	queryFmtPodPVCAllocation         = `avg(avg_over_time(pod_pvc_allocation[%s])) by (persistentvolume, persistentvolumeclaim, pod, namespace, %s)`
 	queryFmtPVCBytesRequested        = `avg(avg_over_time(kube_persistentvolumeclaim_resource_requests_storage_bytes{}[%s])) by (persistentvolumeclaim, namespace, %s)`
+	queryFmtPVActiveMins             = `count(kube_persistentvolume_capacity_bytes) by (persistentvolume, %s)[%s:%s]`
+	queryFmtPVBytes                  = `avg(avg_over_time(kube_persistentvolume_capacity_bytes[%s])) by (persistentvolume, %s)`
 	queryFmtPVCostPerGiBHour         = `avg(avg_over_time(pv_hourly_cost[%s])) by (volumename, %s)`
 	queryFmtNetZoneGiB               = `sum(increase(kubecost_pod_network_egress_bytes_total{internet="false", sameZone="false", sameRegion="true"}[%s])) by (pod_name, namespace, %s) / 1024 / 1024 / 1024`
 	queryFmtNetZoneCostPerGiB        = `avg(avg_over_time(kubecost_network_zone_egress_cost{}[%s])) by (%s)`
@@ -62,10 +58,6 @@ const (
 	queryFmtLBActiveMins             = `count(kubecost_load_balancer_cost) by (namespace, service_name, %s)[%s:%s]`
 )
 
-// This is a bit of a hack to work around garbage data from cadvisor
-// Ideally you cap each pod to the max CPU on its node, but that involves a bit more complexity, as it it would need to be done when allocations joins with asset data.
-const MAX_CPU_CAP = 512
-
 // CanCompute should return true if CostModel can act as a valid source for the
 // given time range. In the case of CostModel we want to attempt to compute as
 // long as the range starts in the past. If the CostModel ends up not having
@@ -137,8 +129,8 @@ func (cm *CostModel) ComputeAllocation(start, end time.Time, resolution time.Dur
 	errors := []string{}
 	warnings := []string{}
 
-	asr.Each(func(i int, as *kubecost.AllocationSet) {
-		as.Each(func(k string, a *kubecost.Allocation) {
+	for _, as := range asr.Allocations {
+		for k, a := range as.Allocations {
 			if len(a.Properties.Annotations) > 0 {
 				if _, ok := allocationAnnotations[k]; !ok {
 					allocationAnnotations[k] = map[string]string{}
@@ -165,11 +157,11 @@ func (cm *CostModel) ComputeAllocation(start, end time.Time, resolution time.Dur
 					allocationServices[k][val] = true
 				}
 			}
-		})
+		}
 
 		errors = append(errors, as.Errors...)
 		warnings = append(warnings, as.Warnings...)
-	})
+	}
 
 	// Accumulate to yield the result AllocationSet. After this step, we will
 	// be nearly complete, but without the raw allocation data, which must be
@@ -181,7 +173,7 @@ func (cm *CostModel) ComputeAllocation(start, end time.Time, resolution time.Dur
 
 	// Apply the annotations, labels, and services to the post-accumulation
 	// results. (See above for why this is necessary.)
-	result.Each(func(k string, a *kubecost.Allocation) {
+	for k, a := range result.Allocations {
 		if annotations, ok := allocationAnnotations[k]; ok {
 			a.Properties.Annotations = annotations
 		}
@@ -201,15 +193,15 @@ func (cm *CostModel) ComputeAllocation(start, end time.Time, resolution time.Dur
 		// to match the Window of the AllocationSet, which gets expanded
 		// at the end of this function.
 		a.Window = a.Window.ExpandStart(start).ExpandEnd(end)
-	})
+	}
 
 	// Maintain RAM and CPU max usage values by iterating over the range,
 	// computing maximums on a rolling basis, and setting on the result set.
-	asr.Each(func(i int, as *kubecost.AllocationSet) {
-		as.Each(func(key string, alloc *kubecost.Allocation) {
+	for _, as := range asr.Allocations {
+		for key, alloc := range as.Allocations {
 			resultAlloc := result.Get(key)
 			if resultAlloc == nil {
-				return
+				continue
 			}
 
 			if resultAlloc.RawAllocationOnly == nil {
@@ -222,7 +214,7 @@ func (cm *CostModel) ComputeAllocation(start, end time.Time, resolution time.Dur
 				if !alloc.IsUnmounted() {
 					log.DedupedWarningf(10, "ComputeAllocation: raw allocation data missing for %s", key)
 				}
-				return
+				continue
 			}
 
 			if alloc.RawAllocationOnly.CPUCoreUsageMax > resultAlloc.RawAllocationOnly.CPUCoreUsageMax {
@@ -232,8 +224,8 @@ func (cm *CostModel) ComputeAllocation(start, end time.Time, resolution time.Dur
 			if alloc.RawAllocationOnly.RAMBytesUsageMax > resultAlloc.RawAllocationOnly.RAMBytesUsageMax {
 				resultAlloc.RawAllocationOnly.RAMBytesUsageMax = alloc.RawAllocationOnly.RAMBytesUsageMax
 			}
-		})
-	})
+		}
+	}
 
 	// Expand the window to match the queried time range.
 	result.Window = result.Window.ExpandStart(start).ExpandEnd(end)
@@ -264,7 +256,7 @@ func (cm *CostModel) computeAllocation(start, end time.Time, resolution time.Dur
 	// underlying-Allocation instance, starting with (start, end) so that we
 	// begin with minutes, from which we compute resource allocation and cost
 	// totals from measured rate data.
-	podMap := map[podKey]*Pod{}
+	podMap := map[podKey]*pod{}
 
 	// clusterStarts and clusterEnds record the earliest start and latest end
 	// times, respectively, on a cluster-basis. These are used for unmounted
@@ -355,15 +347,18 @@ func (cm *CostModel) computeAllocation(start, end time.Time, resolution time.Dur
 	queryPVCInfo := fmt.Sprintf(queryFmtPVCInfo, env.GetPromClusterLabel(), durStr, resStr)
 	resChPVCInfo := ctx.QueryAtTime(queryPVCInfo, end)
 
-	queryPVBytes := fmt.Sprintf(queryFmtPVBytes, durStr, env.GetPromClusterLabel())
-	resChPVBytes := ctx.QueryAtTime(queryPVBytes, end)
-
 	queryPodPVCAllocation := fmt.Sprintf(queryFmtPodPVCAllocation, durStr, env.GetPromClusterLabel())
 	resChPodPVCAllocation := ctx.QueryAtTime(queryPodPVCAllocation, end)
 
 	queryPVCBytesRequested := fmt.Sprintf(queryFmtPVCBytesRequested, durStr, env.GetPromClusterLabel())
 	resChPVCBytesRequested := ctx.QueryAtTime(queryPVCBytesRequested, end)
 
+	queryPVActiveMins := fmt.Sprintf(queryFmtPVActiveMins, env.GetPromClusterLabel(), durStr, resStr)
+	resChPVActiveMins := ctx.QueryAtTime(queryPVActiveMins, end)
+
+	queryPVBytes := fmt.Sprintf(queryFmtPVBytes, durStr, env.GetPromClusterLabel())
+	resChPVBytes := ctx.QueryAtTime(queryPVBytes, end)
+
 	queryPVCostPerGiBHour := fmt.Sprintf(queryFmtPVCostPerGiBHour, durStr, env.GetPromClusterLabel())
 	resChPVCostPerGiBHour := ctx.QueryAtTime(queryPVCostPerGiBHour, end)
 
@@ -446,6 +441,7 @@ func (cm *CostModel) computeAllocation(start, end time.Time, resolution time.Dur
 	resNodeCostPerGPUHr, _ := resChNodeCostPerGPUHr.Await()
 	resNodeIsSpot, _ := resChNodeIsSpot.Await()
 
+	resPVActiveMins, _ := resChPVActiveMins.Await()
 	resPVBytes, _ := resChPVBytes.Await()
 	resPVCostPerGiBHour, _ := resChPVCostPerGiBHour.Await()
 
@@ -516,10 +512,6 @@ func (cm *CostModel) computeAllocation(start, end time.Time, resolution time.Dur
 	applyLabels(podMap, namespaceLabels, podLabels)
 	applyAnnotations(podMap, namespaceAnnotations, podAnnotations)
 
-	serviceLabels := getServiceLabels(resServiceLabels)
-	allocsByService := map[serviceKey][]*kubecost.Allocation{}
-	applyServicesToPods(podMap, podLabels, allocsByService, serviceLabels)
-
 	podDeploymentMap := labelsToPodControllerMap(podLabels, resToDeploymentLabels(resDeploymentLabels))
 	podStatefulSetMap := labelsToPodControllerMap(podLabels, resToStatefulSetLabels(resStatefulSetLabels))
 	podDaemonSetMap := resToPodDaemonSetMap(resDaemonSetLabels, podUIDKeyMap, ingestPodUID)
@@ -531,2252 +523,71 @@ func (cm *CostModel) computeAllocation(start, end time.Time, resolution time.Dur
 	applyControllersToPods(podMap, podJobMap)
 	applyControllersToPods(podMap, podReplicaSetMap)
 
-	// TODO breakdown network costs?
-
-	// Build out a map of Nodes with resource costs, discounts, and node types
-	// for converting resource allocation data to cumulative costs.
-	nodeMap := map[nodeKey]*NodePricing{}
+	serviceLabels := getServiceLabels(resServiceLabels)
+	allocsByService := map[serviceKey][]*kubecost.Allocation{}
+	applyServicesToPods(podMap, podLabels, allocsByService, serviceLabels)
 
-	applyNodeCostPerCPUHr(nodeMap, resNodeCostPerCPUHr)
-	applyNodeCostPerRAMGiBHr(nodeMap, resNodeCostPerRAMGiBHr)
-	applyNodeCostPerGPUHr(nodeMap, resNodeCostPerGPUHr)
-	applyNodeSpot(nodeMap, resNodeIsSpot)
-	applyNodeDiscount(nodeMap, cm)
+	// TODO breakdown network costs?
 
 	// Build out the map of all PVs with class, size and cost-per-hour.
 	// Note: this does not record time running, which we may want to
 	// include later for increased PV precision. (As long as the PV has
 	// a PVC, we get time running there, so this is only inaccurate
 	// for short-lived, unmounted PVs.)
-	pvMap := map[pvKey]*PV{}
-	buildPVMap(pvMap, resPVCostPerGiBHour)
+	pvMap := map[pvKey]*pv{}
+	buildPVMap(resolution, pvMap, resPVCostPerGiBHour, resPVActiveMins)
 	applyPVBytes(pvMap, resPVBytes)
 
 	// Build out the map of all PVCs with time running, bytes requested,
 	// and connect to the correct PV from pvMap. (If no PV exists, that
 	// is noted, but does not result in any allocation/cost.)
-	pvcMap := map[pvcKey]*PVC{}
-	buildPVCMap(window, pvcMap, pvMap, resPVCInfo)
+	pvcMap := map[pvcKey]*pvc{}
+	buildPVCMap(resolution, pvcMap, pvMap, resPVCInfo)
 	applyPVCBytesRequested(pvcMap, resPVCBytesRequested)
 
 	// Build out the relationships of pods to their PVCs. This step
-	// populates the PVC.Count field so that PVC allocation can be
+	// populates the pvc.Count field so that pvc allocation can be
 	// split appropriately among each pod's container allocation.
-	podPVCMap := map[podKey][]*PVC{}
+	podPVCMap := map[podKey][]*pvc{}
 	buildPodPVCMap(podPVCMap, pvMap, pvcMap, podMap, resPodPVCAllocation, podUIDKeyMap, ingestPodUID)
+	applyPVCsToPods(window, podMap, podPVCMap, pvcMap)
 
-	// Because PVCs can be shared among pods, the respective PV cost
-	// needs to be evenly distributed to those pods based on time
-	// running, as well as the amount of time the PVC was shared.
-
-	// Build a relation between every PVC to the pods that mount it
-	// and a window representing the interval during which they
-	// were associated.
-	pvcPodIntervalMap := make(map[pvcKey]map[podKey]kubecost.Window)
-
-	for _, pod := range podMap {
-
-		for _, alloc := range pod.Allocations {
-
-			cluster := alloc.Properties.Cluster
-			namespace := alloc.Properties.Namespace
-			pod := alloc.Properties.Pod
-			thisPodKey := newPodKey(cluster, namespace, pod)
-
-			if pvcs, ok := podPVCMap[thisPodKey]; ok {
-				for _, pvc := range pvcs {
-
-					// Determine the (start, end) of the relationship between the
-					// given PVC and the associated Allocation so that a precise
-					// number of hours can be used to compute cumulative cost.
-					s, e := alloc.Start, alloc.End
-					if pvc.Start.After(alloc.Start) {
-						s = pvc.Start
-					}
-					if pvc.End.Before(alloc.End) {
-						e = pvc.End
-					}
-
-					thisPVCKey := newPVCKey(cluster, namespace, pvc.Name)
-					if pvcPodIntervalMap[thisPVCKey] == nil {
-						pvcPodIntervalMap[thisPVCKey] = make(map[podKey]kubecost.Window)
-					}
-
-					pvcPodIntervalMap[thisPVCKey][thisPodKey] = kubecost.NewWindow(&s, &e)
-				}
-			}
-
-			// We only need to look at one alloc per pod
-			break
-		}
-
-	}
-
-	// Build out a PV price coefficient for each pod with a PVC. Each
-	// PVC-pod relation needs a coefficient which modifies the PV cost
-	// such that PV costs can be shared between all pods using that PVC.
-	sharedPVCCostCoefficientMap := make(map[pvcKey]map[podKey][]CoefficientComponent)
-	for pvcKey, podIntervalMap := range pvcPodIntervalMap {
-
-		// Get single-point intervals from alloc-PVC relation windows.
-		intervals := getIntervalPointsFromWindows(podIntervalMap)
+	// Identify PVCs without pods and add pv costs to the unmounted Allocation for the pvc's cluster
+	applyUnmountedPVCs(window, podMap, pvcMap)
 
-		// Determine coefficients for each PVC-pod relation.
-		sharedPVCCostCoefficientMap[pvcKey] = getPVCCostCoefficients(intervals, podIntervalMap)
+	// Identify PVs without PVCs and add PV costs to the unmounted Allocation for the PV's cluster
+	applyUnmountedPVs(window, podMap, pvMap, pvcMap)
 
-	}
+	lbMap := make(map[serviceKey]*lbCost)
+	getLoadBalancerCosts(lbMap, resLBCostPerHr, resLBActiveMins, resolution)
+	applyLoadBalancersToPods(window, podMap, lbMap, allocsByService)
 
-	// Identify unmounted PVs (PVs without PVCs) and add one Allocation per
-	// cluster representing each cluster's unmounted PVs (if necessary).
-	applyUnmountedPVs(window, podMap, pvMap, pvcMap)
+	// Build out a map of Nodes with resource costs, discounts, and node types
+	// for converting resource allocation data to cumulative costs.
+	nodeMap := map[nodeKey]*nodePricing{}
 
-	lbMap := getLoadBalancerCosts(resLBCostPerHr, resLBActiveMins, resolution)
-	applyLoadBalancersToPods(lbMap, allocsByService)
+	applyNodeCostPerCPUHr(nodeMap, resNodeCostPerCPUHr)
+	applyNodeCostPerRAMGiBHr(nodeMap, resNodeCostPerRAMGiBHr)
+	applyNodeCostPerGPUHr(nodeMap, resNodeCostPerGPUHr)
+	applyNodeSpot(nodeMap, resNodeIsSpot)
+	applyNodeDiscount(nodeMap, cm)
+	cm.applyNodesToPod(podMap, nodeMap)
 
 	// (3) Build out AllocationSet from Pod map
-
 	for _, pod := range podMap {
 		for _, alloc := range pod.Allocations {
 			cluster := alloc.Properties.Cluster
 			nodeName := alloc.Properties.Node
 			namespace := alloc.Properties.Namespace
-			pod := alloc.Properties.Pod
+			podName := alloc.Properties.Pod
 			container := alloc.Properties.Container
 
-			podKey := newPodKey(cluster, namespace, pod)
-			nodeKey := newNodeKey(cluster, nodeName)
-
-			node := cm.getNodePricing(nodeMap, nodeKey)
-			alloc.Properties.ProviderID = node.ProviderID
-			alloc.CPUCost = alloc.CPUCoreHours * node.CostPerCPUHr
-			alloc.RAMCost = (alloc.RAMByteHours / 1024 / 1024 / 1024) * node.CostPerRAMGiBHr
-			alloc.GPUCost = alloc.GPUHours * node.CostPerGPUHr
-
-			if pvcs, ok := podPVCMap[podKey]; ok {
-				for _, pvc := range pvcs {
-
-					pvcKey := newPVCKey(cluster, namespace, pvc.Name)
-
-					s, e := alloc.Start, alloc.End
-					if pvcInterval, ok := pvcPodIntervalMap[pvcKey][podKey]; ok {
-						s, e = *pvcInterval.Start(), *pvcInterval.End()
-					} else {
-						log.Warnf("CostModel.ComputeAllocation: allocation %s and PVC %s have no associated active window", alloc.Name, pvc.Name)
-					}
-
-					minutes := e.Sub(s).Minutes()
-					hrs := minutes / 60.0
-
-					count := float64(pvc.Count)
-					if pvc.Count < 1 {
-						count = 1
-					}
-
-					gib := pvc.Bytes / 1024 / 1024 / 1024
-					cost := pvc.Volume.CostPerGiBHour * gib * hrs
-
-					// Scale PV cost by PVC sharing coefficient.
-					if coeffComponents, ok := sharedPVCCostCoefficientMap[pvcKey][podKey]; ok {
-						cost *= getCoefficientFromComponents(coeffComponents)
-					} else {
-						log.Warnf("CostModel.ComputeAllocation: allocation %s and PVC %s have relation but no coeff", alloc.Name, pvc.Name)
-					}
-
-					// Apply the size and cost of the PV to the allocation, each
-					// weighted by count (i.e. the number of containers in the pod)
-					// record the amount of total PVBytes Hours attributable to a given PV
-					if alloc.PVs == nil {
-						alloc.PVs = kubecost.PVAllocations{}
-					}
-					pvKey := kubecost.PVKey{
-						Cluster: pvc.Cluster,
-						Name:    pvc.Volume.Name,
-					}
-					alloc.PVs[pvKey] = &kubecost.PVAllocation{
-						ByteHours: pvc.Bytes * hrs / count,
-						Cost:      cost / count,
-					}
-				}
-			}
-
 			// Make sure that the name is correct (node may not be present at this
 			// point due to it missing from queryMinutes) then insert.
-			alloc.Name = fmt.Sprintf("%s/%s/%s/%s/%s", cluster, nodeName, namespace, pod, container)
+			alloc.Name = fmt.Sprintf("%s/%s/%s/%s/%s", cluster, nodeName, namespace, podName, container)
 			allocSet.Set(alloc)
 		}
 	}
 
 	return allocSet, nil
 }
-
-func (cm *CostModel) buildPodMap(window kubecost.Window, resolution, maxBatchSize time.Duration, podMap map[podKey]*Pod, clusterStart, clusterEnd map[string]time.Time, ingestPodUID bool, podUIDKeyMap map[podKey][]podKey) error {
-	// Assumes that window is positive and closed
-	start, end := *window.Start(), *window.End()
-
-	// Convert resolution duration to a query-ready string
-	resStr := timeutil.DurationString(resolution)
-
-	ctx := prom.NewNamedContext(cm.PrometheusClient, prom.AllocationContextName)
-
-	// Query for (start, end) by (pod, namespace, cluster) over the given
-	// window, using the given resolution, and if necessary in batches no
-	// larger than the given maximum batch size. If working in batches, track
-	// overall progress by starting with (window.start, window.start) and
-	// querying in batches no larger than maxBatchSize from start-to-end,
-	// folding each result set into podMap as the results come back.
-	coverage := kubecost.NewWindow(&start, &start)
-
-	numQuery := 1
-	for coverage.End().Before(end) {
-		// Determine the (start, end) of the current batch
-		batchStart := *coverage.End()
-		batchEnd := coverage.End().Add(maxBatchSize)
-		if batchEnd.After(end) {
-			batchEnd = end
-		}
-
-		var resPods []*prom.QueryResult
-		var err error
-		maxTries := 3
-		numTries := 0
-		for resPods == nil && numTries < maxTries {
-			numTries++
-
-			// Query for the duration between start and end
-			durStr := timeutil.DurationString(batchEnd.Sub(batchStart))
-			if durStr == "" {
-				// Negative duration, so set empty results and don't query
-				resPods = []*prom.QueryResult{}
-				err = nil
-				break
-			}
-
-			// Submit and profile query
-
-			var queryPods string
-			// If ingesting UIDs, avg on them
-			if ingestPodUID {
-				queryPods = fmt.Sprintf(queryFmtPodsUID, env.GetPromClusterLabel(), durStr, resStr)
-			} else {
-				queryPods = fmt.Sprintf(queryFmtPods, env.GetPromClusterLabel(), durStr, resStr)
-			}
-
-			queryProfile := time.Now()
-			resPods, err = ctx.QueryAtTime(queryPods, batchEnd).Await()
-			if err != nil {
-				log.Profile(queryProfile, fmt.Sprintf("CostModel.ComputeAllocation: pod query %d try %d failed: %s", numQuery, numTries, queryPods))
-				resPods = nil
-			}
-		}
-
-		if err != nil {
-			return err
-		}
-
-		// queryFmtPodsUID will return both UID-containing results, and non-UID-containing results,
-		// so filter out the non-containing results so we don't duplicate pods. This is due to the
-		// default setup of Kubecost having replicated kube_pod_container_status_running and
-		// included KSM kube_pod_container_status_running. Querying w/ UID will return both.
-		if ingestPodUID {
-			var resPodsUID []*prom.QueryResult
-
-			for _, res := range resPods {
-				_, err := res.GetString("uid")
-				if err == nil {
-					resPodsUID = append(resPodsUID, res)
-				}
-			}
-
-			if len(resPodsUID) > 0 {
-				resPods = resPodsUID
-			} else {
-				log.DedupedWarningf(5, "CostModel.ComputeAllocation: UID ingestion enabled, but query did not return any results with UID")
-			}
-		}
-
-		applyPodResults(window, resolution, podMap, clusterStart, clusterEnd, resPods, ingestPodUID, podUIDKeyMap)
-
-		coverage = coverage.ExpandEnd(batchEnd)
-		numQuery++
-	}
-
-	return nil
-}
-
-func applyPodResults(window kubecost.Window, resolution time.Duration, podMap map[podKey]*Pod, clusterStart, clusterEnd map[string]time.Time, resPods []*prom.QueryResult, ingestPodUID bool, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resPods {
-		if len(res.Values) == 0 {
-			log.Warnf("CostModel.ComputeAllocation: empty minutes result")
-			continue
-		}
-
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		labels, err := res.GetStrings("namespace", "pod")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: minutes query result missing field: %s", err)
-			continue
-		}
-
-		namespace := labels["namespace"]
-		pod := labels["pod"]
-		key := newPodKey(cluster, namespace, pod)
-
-		// If pod UIDs are being used to ID pods, append them to the pod name in
-		// the podKey.
-		if ingestPodUID {
-
-			uid, err := res.GetString("uid")
-			if err != nil {
-				log.Warnf("CostModel.ComputeAllocation: UID ingestion enabled, but query result missing field: %s", err)
-			} else {
-
-				newKey := newPodKey(cluster, namespace, pod+" "+uid)
-				podUIDKeyMap[key] = append(podUIDKeyMap[key], newKey)
-
-				key = newKey
-
-			}
-
-		}
-
-		// allocStart and allocEnd are the timestamps of the first and last
-		// minutes the pod was running, respectively. We subtract one resolution
-		// from allocStart because this point will actually represent the end
-		// of the first minute. We don't subtract from allocEnd because it
-		// already represents the end of the last minute.
-		var allocStart, allocEnd time.Time
-		startAdjustmentCoeff, endAdjustmentCoeff := 1.0, 1.0
-		for _, datum := range res.Values {
-			t := time.Unix(int64(datum.Timestamp), 0)
-
-			if allocStart.IsZero() && datum.Value > 0 && window.Contains(t) {
-				// Set the start timestamp to the earliest non-zero timestamp
-				allocStart = t
-
-				// Record adjustment coefficient, i.e. the portion of the start
-				// timestamp to "ignore". That is, sometimes the value will be
-				// 0.5, meaning that we should discount the time running by
-				// half of the resolution the timestamp stands for.
-				startAdjustmentCoeff = (1.0 - datum.Value)
-			}
-
-			if datum.Value > 0 && window.Contains(t) {
-				// Set the end timestamp to the latest non-zero timestamp
-				allocEnd = t
-
-				// Record adjustment coefficient, i.e. the portion of the end
-				// timestamp to "ignore". (See explanation above for start.)
-				endAdjustmentCoeff = (1.0 - datum.Value)
-			}
-		}
-
-		if allocStart.IsZero() || allocEnd.IsZero() {
-			continue
-		}
-
-		// Adjust timestamps according to the resolution and the adjustment
-		// coefficients, as described above. That is, count the start timestamp
-		// from the beginning of the resolution, not the end. Then "reduce" the
-		// start and end by the correct amount, in the case that the "running"
-		// value of the first or last timestamp was not a full 1.0.
-		allocStart = allocStart.Add(-resolution)
-		// Note: the *100 and /100 are necessary because Duration is an int, so
-		// 0.5, for instance, will be truncated, resulting in no adjustment.
-		allocStart = allocStart.Add(time.Duration(startAdjustmentCoeff*100) * resolution / time.Duration(100))
-		allocEnd = allocEnd.Add(-time.Duration(endAdjustmentCoeff*100) * resolution / time.Duration(100))
-
-		// Ensure that the allocStart is always within the window, adjusting
-		// for the occasions where start falls 1m before the query window.
-		// NOTE: window here will always be closed (so no need to nil check
-		// "start").
-		// TODO:CLEANUP revisit query methodology to figure out why this is
-		// happening on occasion
-		if allocStart.Before(*window.Start()) {
-			allocStart = *window.Start()
-		}
-
-		// If there is only one point with a value <= 0.5 that the start and
-		// end timestamps both share, then we will enter this case because at
-		// least half of a resolution will be subtracted from both the start
-		// and the end. If that is the case, then add back half of each side
-		// so that the pod is said to run for half a resolution total.
-		// e.g. For resolution 1m and a value of 0.5 at one timestamp, we'll
-		//      end up with allocEnd == allocStart and each coeff == 0.5. In
-		//      that case, add 0.25m to each side, resulting in 0.5m duration.
-		if !allocEnd.After(allocStart) {
-			allocStart = allocStart.Add(-time.Duration(50*startAdjustmentCoeff) * resolution / time.Duration(100))
-			allocEnd = allocEnd.Add(time.Duration(50*endAdjustmentCoeff) * resolution / time.Duration(100))
-		}
-
-		// Ensure that the allocEnf is always within the window, adjusting
-		// for the occasions where end falls 1m after the query window. This
-		// has not ever happened, but is symmetrical with the start check
-		// above.
-		// NOTE: window here will always be closed (so no need to nil check
-		// "end").
-		// TODO:CLEANUP revisit query methodology to figure out why this is
-		// happening on occasion
-		if allocEnd.After(*window.End()) {
-			allocEnd = *window.End()
-		}
-
-		// Set start if unset or this datum's start time is earlier than the
-		// current earliest time.
-		if _, ok := clusterStart[cluster]; !ok || allocStart.Before(clusterStart[cluster]) {
-			clusterStart[cluster] = allocStart
-		}
-
-		// Set end if unset or this datum's end time is later than the
-		// current latest time.
-		if _, ok := clusterEnd[cluster]; !ok || allocEnd.After(clusterEnd[cluster]) {
-			clusterEnd[cluster] = allocEnd
-		}
-
-		if pod, ok := podMap[key]; ok {
-			// Pod has already been recorded, so update it accordingly
-			if allocStart.Before(pod.Start) {
-				pod.Start = allocStart
-			}
-			if allocEnd.After(pod.End) {
-				pod.End = allocEnd
-			}
-		} else {
-			// Pod has not been recorded yet, so insert it
-			podMap[key] = &Pod{
-				Window:      window.Clone(),
-				Start:       allocStart,
-				End:         allocEnd,
-				Key:         key,
-				Allocations: map[string]*kubecost.Allocation{},
-			}
-		}
-	}
-}
-
-func applyCPUCoresAllocated(podMap map[podKey]*Pod, resCPUCoresAllocated []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resCPUCoresAllocated {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU allocation result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU allocation query result missing 'container': %s", key)
-			continue
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			cpuCores := res.Values[0].Value
-			if cpuCores > MAX_CPU_CAP {
-				log.Infof("[WARNING] Very large cpu allocation, clamping to %f", res.Values[0].Value*(pod.Allocations[container].Minutes()/60.0))
-				cpuCores = 0.0
-			}
-			hours := pod.Allocations[container].Minutes() / 60.0
-			pod.Allocations[container].CPUCoreHours = cpuCores * hours
-
-			node, err := res.GetString("node")
-			if err != nil {
-				log.Warnf("CostModel.ComputeAllocation: CPU allocation query result missing 'node': %s", key)
-				continue
-			}
-			pod.Allocations[container].Properties.Node = node
-		}
-	}
-}
-
-func applyCPUCoresRequested(podMap map[podKey]*Pod, resCPUCoresRequested []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resCPUCoresRequested {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU request result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU request query result missing 'container': %s", key)
-			continue
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			pod.Allocations[container].CPUCoreRequestAverage = res.Values[0].Value
-
-			// If CPU allocation is less than requests, set CPUCoreHours to
-			// request level.
-			if pod.Allocations[container].CPUCores() < res.Values[0].Value {
-				pod.Allocations[container].CPUCoreHours = res.Values[0].Value * (pod.Allocations[container].Minutes() / 60.0)
-			}
-			if pod.Allocations[container].CPUCores() > MAX_CPU_CAP {
-				log.Infof("[WARNING] Very large cpu allocation, clamping! to %f", res.Values[0].Value*(pod.Allocations[container].Minutes()/60.0))
-				pod.Allocations[container].CPUCoreHours = res.Values[0].Value * (pod.Allocations[container].Minutes() / 60.0)
-			}
-			node, err := res.GetString("node")
-			if err != nil {
-				log.Warnf("CostModel.ComputeAllocation: CPU request query result missing 'node': %s", key)
-				continue
-			}
-			pod.Allocations[container].Properties.Node = node
-		}
-	}
-}
-
-func applyCPUCoresUsedAvg(podMap map[podKey]*Pod, resCPUCoresUsedAvg []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resCPUCoresUsedAvg {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU usage avg result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if container == "" || err != nil {
-			container, err = res.GetString("container_name")
-			if err != nil {
-				log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU usage avg query result missing 'container': %s", key)
-				continue
-			}
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			pod.Allocations[container].CPUCoreUsageAverage = res.Values[0].Value
-			if res.Values[0].Value > MAX_CPU_CAP {
-				log.Infof("[WARNING] Very large cpu USAGE, dropping outlier")
-				pod.Allocations[container].CPUCoreUsageAverage = 0.0
-			}
-		}
-	}
-}
-
-func applyCPUCoresUsedMax(podMap map[podKey]*Pod, resCPUCoresUsedMax []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resCPUCoresUsedMax {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU usage max result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if container == "" || err != nil {
-			container, err = res.GetString("container_name")
-			if err != nil {
-				log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU usage max query result missing 'container': %s", key)
-				continue
-			}
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			if pod.Allocations[container].RawAllocationOnly == nil {
-				pod.Allocations[container].RawAllocationOnly = &kubecost.RawAllocationOnlyData{
-					CPUCoreUsageMax: res.Values[0].Value,
-				}
-			} else {
-				pod.Allocations[container].RawAllocationOnly.CPUCoreUsageMax = res.Values[0].Value
-			}
-		}
-	}
-}
-
-func applyRAMBytesAllocated(podMap map[podKey]*Pod, resRAMBytesAllocated []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resRAMBytesAllocated {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM allocation result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM allocation query result missing 'container': %s", key)
-			continue
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			ramBytes := res.Values[0].Value
-			hours := pod.Allocations[container].Minutes() / 60.0
-			pod.Allocations[container].RAMByteHours = ramBytes * hours
-
-			node, err := res.GetString("node")
-			if err != nil {
-				log.Warnf("CostModel.ComputeAllocation: RAM allocation query result missing 'node': %s", key)
-				continue
-			}
-			pod.Allocations[container].Properties.Node = node
-		}
-	}
-}
-
-func applyRAMBytesRequested(podMap map[podKey]*Pod, resRAMBytesRequested []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resRAMBytesRequested {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM request result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM request query result missing 'container': %s", key)
-			continue
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			pod.Allocations[container].RAMBytesRequestAverage = res.Values[0].Value
-
-			// If RAM allocation is less than requests, set RAMByteHours to
-			// request level.
-			if pod.Allocations[container].RAMBytes() < res.Values[0].Value {
-				pod.Allocations[container].RAMByteHours = res.Values[0].Value * (pod.Allocations[container].Minutes() / 60.0)
-			}
-
-			node, err := res.GetString("node")
-			if err != nil {
-				log.Warnf("CostModel.ComputeAllocation: RAM request query result missing 'node': %s", key)
-				continue
-			}
-			pod.Allocations[container].Properties.Node = node
-		}
-	}
-}
-
-func applyRAMBytesUsedAvg(podMap map[podKey]*Pod, resRAMBytesUsedAvg []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resRAMBytesUsedAvg {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM avg usage result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if container == "" || err != nil {
-			container, err = res.GetString("container_name")
-			if err != nil {
-				log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM usage avg query result missing 'container': %s", key)
-				continue
-			}
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			pod.Allocations[container].RAMBytesUsageAverage = res.Values[0].Value
-		}
-	}
-}
-
-func applyRAMBytesUsedMax(podMap map[podKey]*Pod, resRAMBytesUsedMax []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resRAMBytesUsedMax {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM usage max result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if container == "" || err != nil {
-			container, err = res.GetString("container_name")
-			if err != nil {
-				log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM usage max query result missing 'container': %s", key)
-				continue
-			}
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			if pod.Allocations[container].RawAllocationOnly == nil {
-				pod.Allocations[container].RawAllocationOnly = &kubecost.RawAllocationOnlyData{
-					RAMBytesUsageMax: res.Values[0].Value,
-				}
-			} else {
-				pod.Allocations[container].RawAllocationOnly.RAMBytesUsageMax = res.Values[0].Value
-			}
-		}
-	}
-}
-
-func applyGPUsAllocated(podMap map[podKey]*Pod, resGPUsRequested []*prom.QueryResult, resGPUsAllocated []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	if len(resGPUsAllocated) > 0 { // Use the new query, when it's become available in a window
-		resGPUsRequested = resGPUsAllocated
-	}
-	for _, res := range resGPUsRequested {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: GPU request result missing field: %s", err)
-			continue
-		}
-
-		container, err := res.GetString("container")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: GPU request query result missing 'container': %s", key)
-			continue
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[key]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-
-			if _, ok := pod.Allocations[container]; !ok {
-				pod.AppendContainer(container)
-			}
-
-			hrs := pod.Allocations[container].Minutes() / 60.0
-			pod.Allocations[container].GPUHours = res.Values[0].Value * hrs
-		}
-	}
-}
-
-func applyNetworkTotals(podMap map[podKey]*Pod, resNetworkTransferBytes []*prom.QueryResult, resNetworkReceiveBytes []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	for _, res := range resNetworkTransferBytes {
-		podKey, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: Network Transfer Bytes query result missing field: %s", err)
-			continue
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[podKey]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[podKey]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-			for _, alloc := range pod.Allocations {
-				alloc.NetworkTransferBytes = res.Values[0].Value / float64(len(pod.Allocations)) / float64(len(pods))
-			}
-		}
-	}
-	for _, res := range resNetworkReceiveBytes {
-		podKey, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: Network Receive Bytes query result missing field: %s", err)
-			continue
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[podKey]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[podKey]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-			for _, alloc := range pod.Allocations {
-				alloc.NetworkReceiveBytes = res.Values[0].Value / float64(len(pod.Allocations)) / float64(len(pods))
-			}
-		}
-	}
-}
-
-func applyNetworkAllocation(podMap map[podKey]*Pod, resNetworkGiB []*prom.QueryResult, resNetworkCostPerGiB []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
-	costPerGiBByCluster := map[string]float64{}
-
-	for _, res := range resNetworkCostPerGiB {
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		costPerGiBByCluster[cluster] = res.Values[0].Value
-	}
-
-	for _, res := range resNetworkGiB {
-		podKey, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: Network allocation query result missing field: %s", err)
-			continue
-		}
-
-		var pods []*Pod
-
-		pod, ok := podMap[podKey]
-		if !ok {
-			if uidKeys, ok := podUIDKeyMap[podKey]; ok {
-				for _, uidKey := range uidKeys {
-					pod, ok = podMap[uidKey]
-					if ok {
-						pods = append(pods, pod)
-					}
-				}
-			} else {
-				continue
-			}
-		} else {
-			pods = []*Pod{pod}
-		}
-
-		for _, pod := range pods {
-			for _, alloc := range pod.Allocations {
-				gib := res.Values[0].Value / float64(len(pod.Allocations))
-				costPerGiB := costPerGiBByCluster[podKey.Cluster]
-				alloc.NetworkCost = gib * costPerGiB / float64(len(pods))
-			}
-		}
-	}
-}
-
-func resToNamespaceLabels(resNamespaceLabels []*prom.QueryResult) map[namespaceKey]map[string]string {
-	namespaceLabels := map[namespaceKey]map[string]string{}
-
-	for _, res := range resNamespaceLabels {
-		nsKey, err := resultNamespaceKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			continue
-		}
-
-		if _, ok := namespaceLabels[nsKey]; !ok {
-			namespaceLabels[nsKey] = map[string]string{}
-		}
-
-		for k, l := range res.GetLabels() {
-			namespaceLabels[nsKey][k] = l
-		}
-	}
-
-	return namespaceLabels
-}
-
-func resToPodLabels(resPodLabels []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]map[string]string {
-	podLabels := map[podKey]map[string]string{}
-
-	for _, res := range resPodLabels {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			continue
-		}
-
-		var keys []podKey
-
-		if ingestPodUID {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-
-				keys = append(keys, uidKeys...)
-
-			}
-		} else {
-			keys = []podKey{key}
-		}
-
-		for _, key := range keys {
-			if _, ok := podLabels[key]; !ok {
-				podLabels[key] = map[string]string{}
-			}
-
-			for k, l := range res.GetLabels() {
-				podLabels[key][k] = l
-			}
-		}
-	}
-
-	return podLabels
-}
-
-func resToNamespaceAnnotations(resNamespaceAnnotations []*prom.QueryResult) map[string]map[string]string {
-	namespaceAnnotations := map[string]map[string]string{}
-
-	for _, res := range resNamespaceAnnotations {
-		namespace, err := res.GetString("namespace")
-		if err != nil {
-			continue
-		}
-
-		if _, ok := namespaceAnnotations[namespace]; !ok {
-			namespaceAnnotations[namespace] = map[string]string{}
-		}
-
-		for k, l := range res.GetAnnotations() {
-			namespaceAnnotations[namespace][k] = l
-		}
-	}
-
-	return namespaceAnnotations
-}
-
-func resToPodAnnotations(resPodAnnotations []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]map[string]string {
-	podAnnotations := map[podKey]map[string]string{}
-
-	for _, res := range resPodAnnotations {
-		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
-		if err != nil {
-			continue
-		}
-
-		var keys []podKey
-
-		if ingestPodUID {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-
-				keys = append(keys, uidKeys...)
-
-			}
-		} else {
-			keys = []podKey{key}
-		}
-
-		for _, key := range keys {
-			if _, ok := podAnnotations[key]; !ok {
-				podAnnotations[key] = map[string]string{}
-			}
-
-			for k, l := range res.GetAnnotations() {
-				podAnnotations[key][k] = l
-			}
-		}
-	}
-
-	return podAnnotations
-}
-
-func applyLabels(podMap map[podKey]*Pod, namespaceLabels map[namespaceKey]map[string]string, podLabels map[podKey]map[string]string) {
-	for podKey, pod := range podMap {
-		for _, alloc := range pod.Allocations {
-			allocLabels := alloc.Properties.Labels
-			if allocLabels == nil {
-				allocLabels = make(map[string]string)
-			}
-			// Apply namespace labels first, then pod labels so that pod labels
-			// overwrite namespace labels.
-			nsKey := podKey.namespaceKey // newNamespaceKey(podKey.Cluster, podKey.Namespace)
-			if labels, ok := namespaceLabels[nsKey]; ok {
-				for k, v := range labels {
-					allocLabels[k] = v
-				}
-			}
-			if labels, ok := podLabels[podKey]; ok {
-				for k, v := range labels {
-					allocLabels[k] = v
-				}
-			}
-
-			alloc.Properties.Labels = allocLabels
-		}
-	}
-}
-
-func applyAnnotations(podMap map[podKey]*Pod, namespaceAnnotations map[string]map[string]string, podAnnotations map[podKey]map[string]string) {
-	for key, pod := range podMap {
-		for _, alloc := range pod.Allocations {
-			allocAnnotations := alloc.Properties.Annotations
-			if allocAnnotations == nil {
-				allocAnnotations = make(map[string]string)
-			}
-			// Apply namespace annotations first, then pod annotations so that
-			// pod labels overwrite namespace labels.
-			if labels, ok := namespaceAnnotations[key.Namespace]; ok {
-				for k, v := range labels {
-					allocAnnotations[k] = v
-				}
-			}
-			if labels, ok := podAnnotations[key]; ok {
-				for k, v := range labels {
-					allocAnnotations[k] = v
-				}
-			}
-
-			alloc.Properties.Annotations = allocAnnotations
-		}
-	}
-}
-
-func getServiceLabels(resServiceLabels []*prom.QueryResult) map[serviceKey]map[string]string {
-	serviceLabels := map[serviceKey]map[string]string{}
-
-	for _, res := range resServiceLabels {
-		serviceKey, err := resultServiceKey(res, env.GetPromClusterLabel(), "namespace", "service")
-		if err != nil {
-			continue
-		}
-
-		if _, ok := serviceLabels[serviceKey]; !ok {
-			serviceLabels[serviceKey] = map[string]string{}
-		}
-
-		for k, l := range res.GetLabels() {
-			serviceLabels[serviceKey][k] = l
-		}
-	}
-
-	// Prune duplicate services. That is, if the same service exists with
-	// hyphens instead of underscores, keep the one that uses hyphens.
-	for key := range serviceLabels {
-		if strings.Contains(key.Service, "_") {
-			duplicateService := strings.Replace(key.Service, "_", "-", -1)
-			duplicateKey := newServiceKey(key.Cluster, key.Namespace, duplicateService)
-			if _, ok := serviceLabels[duplicateKey]; ok {
-				delete(serviceLabels, key)
-			}
-		}
-	}
-
-	return serviceLabels
-}
-
-func resToDeploymentLabels(resDeploymentLabels []*prom.QueryResult) map[controllerKey]map[string]string {
-	deploymentLabels := map[controllerKey]map[string]string{}
-
-	for _, res := range resDeploymentLabels {
-		controllerKey, err := resultDeploymentKey(res, env.GetPromClusterLabel(), "namespace", "deployment")
-		if err != nil {
-			continue
-		}
-
-		if _, ok := deploymentLabels[controllerKey]; !ok {
-			deploymentLabels[controllerKey] = map[string]string{}
-		}
-
-		for k, l := range res.GetLabels() {
-			deploymentLabels[controllerKey][k] = l
-		}
-	}
-
-	// Prune duplicate deployments. That is, if the same deployment exists with
-	// hyphens instead of underscores, keep the one that uses hyphens.
-	for key := range deploymentLabels {
-		if strings.Contains(key.Controller, "_") {
-			duplicateController := strings.Replace(key.Controller, "_", "-", -1)
-			duplicateKey := newControllerKey(key.Cluster, key.Namespace, key.ControllerKind, duplicateController)
-			if _, ok := deploymentLabels[duplicateKey]; ok {
-				delete(deploymentLabels, key)
-			}
-		}
-	}
-
-	return deploymentLabels
-}
-
-func resToStatefulSetLabels(resStatefulSetLabels []*prom.QueryResult) map[controllerKey]map[string]string {
-	statefulSetLabels := map[controllerKey]map[string]string{}
-
-	for _, res := range resStatefulSetLabels {
-		controllerKey, err := resultStatefulSetKey(res, env.GetPromClusterLabel(), "namespace", "statefulSet")
-		if err != nil {
-			continue
-		}
-
-		if _, ok := statefulSetLabels[controllerKey]; !ok {
-			statefulSetLabels[controllerKey] = map[string]string{}
-		}
-
-		for k, l := range res.GetLabels() {
-			statefulSetLabels[controllerKey][k] = l
-		}
-	}
-
-	// Prune duplicate stateful sets. That is, if the same stateful set exists
-	// with hyphens instead of underscores, keep the one that uses hyphens.
-	for key := range statefulSetLabels {
-		if strings.Contains(key.Controller, "_") {
-			duplicateController := strings.Replace(key.Controller, "_", "-", -1)
-			duplicateKey := newControllerKey(key.Cluster, key.Namespace, key.ControllerKind, duplicateController)
-			if _, ok := statefulSetLabels[duplicateKey]; ok {
-				delete(statefulSetLabels, key)
-			}
-		}
-	}
-
-	return statefulSetLabels
-}
-
-func labelsToPodControllerMap(podLabels map[podKey]map[string]string, controllerLabels map[controllerKey]map[string]string) map[podKey]controllerKey {
-	podControllerMap := map[podKey]controllerKey{}
-
-	// For each controller, turn the labels into a selector and attempt to
-	// match it with each set of pod labels. A match indicates that the pod
-	// belongs to the controller.
-	for cKey, cLabels := range controllerLabels {
-		selector := labels.Set(cLabels).AsSelectorPreValidated()
-
-		for pKey, pLabels := range podLabels {
-			// If the pod is in a different cluster or namespace, there is
-			// no need to compare the labels.
-			if cKey.Cluster != pKey.Cluster || cKey.Namespace != pKey.Namespace {
-				continue
-			}
-
-			podLabelSet := labels.Set(pLabels)
-			if selector.Matches(podLabelSet) {
-				if _, ok := podControllerMap[pKey]; ok {
-					log.DedupedWarningf(5, "CostModel.ComputeAllocation: PodControllerMap match already exists: %s matches %s and %s", pKey, podControllerMap[pKey], cKey)
-				}
-				podControllerMap[pKey] = cKey
-			}
-		}
-	}
-
-	return podControllerMap
-}
-
-func resToPodDaemonSetMap(resDaemonSetLabels []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]controllerKey {
-	daemonSetLabels := map[podKey]controllerKey{}
-
-	for _, res := range resDaemonSetLabels {
-		controllerKey, err := resultDaemonSetKey(res, env.GetPromClusterLabel(), "namespace", "owner_name")
-		if err != nil {
-			continue
-		}
-
-		pod, err := res.GetString("pod")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: DaemonSetLabel result without pod: %s", controllerKey)
-		}
-
-		key := newPodKey(controllerKey.Cluster, controllerKey.Namespace, pod)
-
-		var keys []podKey
-
-		if ingestPodUID {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-
-				keys = append(keys, uidKeys...)
-
-			}
-		} else {
-			keys = []podKey{key}
-		}
-
-		for _, key := range keys {
-			daemonSetLabels[key] = controllerKey
-		}
-	}
-
-	return daemonSetLabels
-}
-
-func resToPodJobMap(resJobLabels []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]controllerKey {
-	jobLabels := map[podKey]controllerKey{}
-
-	for _, res := range resJobLabels {
-		controllerKey, err := resultJobKey(res, env.GetPromClusterLabel(), "namespace", "owner_name")
-		if err != nil {
-			continue
-		}
-
-		// Convert the name of Jobs generated by CronJobs to the name of the
-		// CronJob by stripping the timestamp off the end.
-		match := isCron.FindStringSubmatch(controllerKey.Controller)
-		if match != nil {
-			controllerKey.Controller = match[1]
-		}
-
-		pod, err := res.GetString("pod")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: JobLabel result without pod: %s", controllerKey)
-		}
-
-		key := newPodKey(controllerKey.Cluster, controllerKey.Namespace, pod)
-
-		var keys []podKey
-
-		if ingestPodUID {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-
-				keys = append(keys, uidKeys...)
-
-			}
-		} else {
-			keys = []podKey{key}
-		}
-
-		for _, key := range keys {
-			jobLabels[key] = controllerKey
-		}
-	}
-
-	return jobLabels
-}
-
-func resToPodReplicaSetMap(resPodsWithReplicaSetOwner []*prom.QueryResult, resReplicaSetsWithoutOwners []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]controllerKey {
-	// Build out set of ReplicaSets that have no owners, themselves, such that
-	// the ReplicaSet should be used as the owner of the Pods it controls.
-	// (This should exclude, for example, ReplicaSets that are controlled by
-	// Deployments, in which case the Deployment should be the Pod's owner.)
-	replicaSets := map[controllerKey]struct{}{}
-
-	for _, res := range resReplicaSetsWithoutOwners {
-		controllerKey, err := resultReplicaSetKey(res, env.GetPromClusterLabel(), "namespace", "replicaset")
-		if err != nil {
-			continue
-		}
-
-		replicaSets[controllerKey] = struct{}{}
-	}
-
-	// Create the mapping of Pods to ReplicaSets, ignoring any ReplicaSets that
-	// to not appear in the set of uncontrolled ReplicaSets above.
-	podToReplicaSet := map[podKey]controllerKey{}
-
-	for _, res := range resPodsWithReplicaSetOwner {
-		controllerKey, err := resultReplicaSetKey(res, env.GetPromClusterLabel(), "namespace", "owner_name")
-		if err != nil {
-			continue
-		}
-		if _, ok := replicaSets[controllerKey]; !ok {
-			continue
-		}
-
-		pod, err := res.GetString("pod")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: ReplicaSet result without pod: %s", controllerKey)
-		}
-
-		key := newPodKey(controllerKey.Cluster, controllerKey.Namespace, pod)
-
-		var keys []podKey
-
-		if ingestPodUID {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-
-				keys = append(keys, uidKeys...)
-
-			}
-		} else {
-			keys = []podKey{key}
-		}
-
-		for _, key := range keys {
-
-			podToReplicaSet[key] = controllerKey
-
-		}
-	}
-
-	return podToReplicaSet
-}
-
-func applyServicesToPods(podMap map[podKey]*Pod, podLabels map[podKey]map[string]string, allocsByService map[serviceKey][]*kubecost.Allocation, serviceLabels map[serviceKey]map[string]string) {
-	podServicesMap := map[podKey][]serviceKey{}
-
-	// For each service, turn the labels into a selector and attempt to
-	// match it with each set of pod labels. A match indicates that the pod
-	// belongs to the service.
-	for sKey, sLabels := range serviceLabels {
-		selector := labels.Set(sLabels).AsSelectorPreValidated()
-
-		for pKey, pLabels := range podLabels {
-			// If the pod is in a different cluster or namespace, there is
-			// no need to compare the labels.
-			if sKey.Cluster != pKey.Cluster || sKey.Namespace != pKey.Namespace {
-				continue
-			}
-
-			podLabelSet := labels.Set(pLabels)
-			if selector.Matches(podLabelSet) {
-				if _, ok := podServicesMap[pKey]; !ok {
-					podServicesMap[pKey] = []serviceKey{}
-				}
-				podServicesMap[pKey] = append(podServicesMap[pKey], sKey)
-			}
-		}
-	}
-
-	// For each allocation in each pod, attempt to find and apply the list of
-	// services associated with the allocation's pod.
-	for key, pod := range podMap {
-		for _, alloc := range pod.Allocations {
-			if sKeys, ok := podServicesMap[key]; ok {
-				services := []string{}
-				for _, sKey := range sKeys {
-					services = append(services, sKey.Service)
-					allocsByService[sKey] = append(allocsByService[sKey], alloc)
-				}
-				alloc.Properties.Services = services
-
-			}
-		}
-	}
-}
-
-func applyControllersToPods(podMap map[podKey]*Pod, podControllerMap map[podKey]controllerKey) {
-	for key, pod := range podMap {
-		for _, alloc := range pod.Allocations {
-			if controllerKey, ok := podControllerMap[key]; ok {
-				alloc.Properties.ControllerKind = controllerKey.ControllerKind
-				alloc.Properties.Controller = controllerKey.Controller
-			}
-		}
-	}
-}
-
-func applyNodeCostPerCPUHr(nodeMap map[nodeKey]*NodePricing, resNodeCostPerCPUHr []*prom.QueryResult) {
-	for _, res := range resNodeCostPerCPUHr {
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		node, err := res.GetString("node")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node CPU cost query result missing field: %s", err)
-			continue
-		}
-
-		instanceType, err := res.GetString("instance_type")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node CPU cost query result missing field: %s", err)
-			continue
-		}
-
-		providerID, err := res.GetString("provider_id")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node CPU cost query result missing field: %s", err)
-			continue
-		}
-
-		key := newNodeKey(cluster, node)
-		if _, ok := nodeMap[key]; !ok {
-			nodeMap[key] = &NodePricing{
-				Name:       node,
-				NodeType:   instanceType,
-				ProviderID: cloud.ParseID(providerID),
-			}
-		}
-
-		nodeMap[key].CostPerCPUHr = res.Values[0].Value
-	}
-}
-
-func applyNodeCostPerRAMGiBHr(nodeMap map[nodeKey]*NodePricing, resNodeCostPerRAMGiBHr []*prom.QueryResult) {
-	for _, res := range resNodeCostPerRAMGiBHr {
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		node, err := res.GetString("node")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node RAM cost query result missing field: %s", err)
-			continue
-		}
-
-		instanceType, err := res.GetString("instance_type")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node RAM cost query result missing field: %s", err)
-			continue
-		}
-
-		providerID, err := res.GetString("provider_id")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node RAM cost query result missing field: %s", err)
-			continue
-		}
-
-		key := newNodeKey(cluster, node)
-		if _, ok := nodeMap[key]; !ok {
-			nodeMap[key] = &NodePricing{
-				Name:       node,
-				NodeType:   instanceType,
-				ProviderID: cloud.ParseID(providerID),
-			}
-		}
-
-		nodeMap[key].CostPerRAMGiBHr = res.Values[0].Value
-	}
-}
-
-func applyNodeCostPerGPUHr(nodeMap map[nodeKey]*NodePricing, resNodeCostPerGPUHr []*prom.QueryResult) {
-	for _, res := range resNodeCostPerGPUHr {
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		node, err := res.GetString("node")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node GPU cost query result missing field: %s", err)
-			continue
-		}
-
-		instanceType, err := res.GetString("instance_type")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node GPU cost query result missing field: %s", err)
-			continue
-		}
-
-		providerID, err := res.GetString("provider_id")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node GPU cost query result missing field: %s", err)
-			continue
-		}
-
-		key := newNodeKey(cluster, node)
-		if _, ok := nodeMap[key]; !ok {
-			nodeMap[key] = &NodePricing{
-				Name:       node,
-				NodeType:   instanceType,
-				ProviderID: cloud.ParseID(providerID),
-			}
-		}
-
-		nodeMap[key].CostPerGPUHr = res.Values[0].Value
-	}
-}
-
-func applyNodeSpot(nodeMap map[nodeKey]*NodePricing, resNodeIsSpot []*prom.QueryResult) {
-	for _, res := range resNodeIsSpot {
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		node, err := res.GetString("node")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: Node spot query result missing field: %s", err)
-			continue
-		}
-
-		key := newNodeKey(cluster, node)
-		if _, ok := nodeMap[key]; !ok {
-			log.Warnf("CostModel.ComputeAllocation: Node spot  query result for missing node: %s", key)
-			continue
-		}
-
-		nodeMap[key].Preemptible = res.Values[0].Value > 0
-	}
-}
-
-func applyNodeDiscount(nodeMap map[nodeKey]*NodePricing, cm *CostModel) {
-	if cm == nil {
-		return
-	}
-
-	c, err := cm.Provider.GetConfig()
-	if err != nil {
-		log.Errorf("CostModel.ComputeAllocation: applyNodeDiscount: %s", err)
-		return
-	}
-
-	discount, err := ParsePercentString(c.Discount)
-	if err != nil {
-		log.Errorf("CostModel.ComputeAllocation: applyNodeDiscount: %s", err)
-		return
-	}
-
-	negotiatedDiscount, err := ParsePercentString(c.NegotiatedDiscount)
-	if err != nil {
-		log.Errorf("CostModel.ComputeAllocation: applyNodeDiscount: %s", err)
-		return
-	}
-
-	for _, node := range nodeMap {
-		// TODO GKE Reserved Instances into account
-		node.Discount = cm.Provider.CombinedDiscountForNode(node.NodeType, node.Preemptible, discount, negotiatedDiscount)
-		node.CostPerCPUHr *= (1.0 - node.Discount)
-		node.CostPerRAMGiBHr *= (1.0 - node.Discount)
-	}
-}
-
-func buildPVMap(pvMap map[pvKey]*PV, resPVCostPerGiBHour []*prom.QueryResult) {
-	for _, res := range resPVCostPerGiBHour {
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		name, err := res.GetString("volumename")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: PV cost without volumename")
-			continue
-		}
-
-		key := newPVKey(cluster, name)
-
-		pvMap[key] = &PV{
-			Cluster:        cluster,
-			Name:           name,
-			CostPerGiBHour: res.Values[0].Value,
-		}
-	}
-}
-
-func applyPVBytes(pvMap map[pvKey]*PV, resPVBytes []*prom.QueryResult) {
-	for _, res := range resPVBytes {
-		key, err := resultPVKey(res, env.GetPromClusterLabel(), "persistentvolume")
-		if err != nil {
-			log.Warnf("CostModel.ComputeAllocation: PV bytes query result missing field: %s", err)
-			continue
-		}
-
-		if _, ok := pvMap[key]; !ok {
-			log.Warnf("CostModel.ComputeAllocation: PV bytes result for missing PV: %s", err)
-			continue
-		}
-
-		pvMap[key].Bytes = res.Values[0].Value
-	}
-}
-
-func buildPVCMap(window kubecost.Window, pvcMap map[pvcKey]*PVC, pvMap map[pvKey]*PV, resPVCInfo []*prom.QueryResult) {
-	for _, res := range resPVCInfo {
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		values, err := res.GetStrings("persistentvolumeclaim", "storageclass", "volumename", "namespace")
-		if err != nil {
-			log.DedupedWarningf(10, "CostModel.ComputeAllocation: PVC info query result missing field: %s", err)
-			continue
-		}
-
-		namespace := values["namespace"]
-		name := values["persistentvolumeclaim"]
-		volume := values["volumename"]
-		storageClass := values["storageclass"]
-
-		pvKey := newPVKey(cluster, volume)
-		pvcKey := newPVCKey(cluster, namespace, name)
-
-		// pvcStart and pvcEnd are the timestamps of the first and last minutes
-		// the PVC was running, respectively. We subtract 1m from pvcStart
-		// because this point will actually represent the end of the first
-		// minute. We don't subtract from pvcEnd because it already represents
-		// the end of the last minute.
-		var pvcStart, pvcEnd time.Time
-		for _, datum := range res.Values {
-			t := time.Unix(int64(datum.Timestamp), 0)
-			if pvcStart.IsZero() && datum.Value > 0 && window.Contains(t) {
-				pvcStart = t
-			}
-			if datum.Value > 0 && window.Contains(t) {
-				pvcEnd = t
-			}
-		}
-		if pvcStart.IsZero() || pvcEnd.IsZero() {
-			log.Warnf("CostModel.ComputeAllocation: PVC %s has no running time", pvcKey)
-		}
-		pvcStart = pvcStart.Add(-time.Minute)
-
-		if _, ok := pvMap[pvKey]; !ok {
-			continue
-		}
-
-		pvMap[pvKey].StorageClass = storageClass
-
-		if _, ok := pvcMap[pvcKey]; !ok {
-			pvcMap[pvcKey] = &PVC{}
-		}
-
-		pvcMap[pvcKey].Name = name
-		pvcMap[pvcKey].Namespace = namespace
-		pvcMap[pvcKey].Volume = pvMap[pvKey]
-		pvcMap[pvcKey].Start = pvcStart
-		pvcMap[pvcKey].End = pvcEnd
-	}
-}
-
-func applyPVCBytesRequested(pvcMap map[pvcKey]*PVC, resPVCBytesRequested []*prom.QueryResult) {
-	for _, res := range resPVCBytesRequested {
-		key, err := resultPVCKey(res, env.GetPromClusterLabel(), "namespace", "persistentvolumeclaim")
-		if err != nil {
-			continue
-		}
-
-		if _, ok := pvcMap[key]; !ok {
-			continue
-		}
-
-		pvcMap[key].Bytes = res.Values[0].Value
-	}
-}
-
-func buildPodPVCMap(podPVCMap map[podKey][]*PVC, pvMap map[pvKey]*PV, pvcMap map[pvcKey]*PVC, podMap map[podKey]*Pod, resPodPVCAllocation []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) {
-	for _, res := range resPodPVCAllocation {
-		cluster, err := res.GetString(env.GetPromClusterLabel())
-		if err != nil {
-			cluster = env.GetClusterID()
-		}
-
-		values, err := res.GetStrings("persistentvolume", "persistentvolumeclaim", "pod", "namespace")
-		if err != nil {
-			log.DedupedWarningf(5, "CostModel.ComputeAllocation: PVC allocation query result missing field: %s", err)
-			continue
-		}
-
-		namespace := values["namespace"]
-		pod := values["pod"]
-		name := values["persistentvolumeclaim"]
-		volume := values["persistentvolume"]
-
-		key := newPodKey(cluster, namespace, pod)
-		pvKey := newPVKey(cluster, volume)
-		pvcKey := newPVCKey(cluster, namespace, name)
-
-		var keys []podKey
-
-		if ingestPodUID {
-			if uidKeys, ok := podUIDKeyMap[key]; ok {
-
-				keys = append(keys, uidKeys...)
-
-			}
-		} else {
-			keys = []podKey{key}
-		}
-
-		for _, key := range keys {
-
-			if _, ok := pvMap[pvKey]; !ok {
-				log.DedupedWarningf(5, "CostModel.ComputeAllocation: PV missing for PVC allocation query result: %s", pvKey)
-				continue
-			}
-
-			if _, ok := podPVCMap[key]; !ok {
-				podPVCMap[key] = []*PVC{}
-			}
-
-			pvc, ok := pvcMap[pvcKey]
-			if !ok {
-				log.DedupedWarningf(5, "CostModel.ComputeAllocation: PVC missing for PVC allocation query: %s", pvcKey)
-				continue
-			}
-
-			count := 1
-			if pod, ok := podMap[key]; ok && len(pod.Allocations) > 0 {
-				count = len(pod.Allocations)
-			} else {
-				log.DedupedWarningf(10, "CostModel.ComputeAllocation: PVC %s for missing pod %s", pvcKey, key)
-			}
-
-			pvc.Count = count
-			pvc.Mounted = true
-
-			podPVCMap[key] = append(podPVCMap[key], pvc)
-		}
-	}
-}
-
-func applyUnmountedPVs(window kubecost.Window, podMap map[podKey]*Pod, pvMap map[pvKey]*PV, pvcMap map[pvcKey]*PVC) {
-	unmountedPVBytes := map[string]float64{}
-	unmountedPVCost := map[string]float64{}
-
-	for _, pv := range pvMap {
-		mounted := false
-		for _, pvc := range pvcMap {
-			if pvc.Volume == nil {
-				continue
-			}
-			if pvc.Volume == pv {
-				mounted = true
-				break
-			}
-		}
-
-		if !mounted {
-			gib := pv.Bytes / 1024 / 1024 / 1024
-			hrs := window.Minutes() / 60.0 // TODO improve with PV hours, not window hours
-			cost := pv.CostPerGiBHour * gib * hrs
-			unmountedPVCost[pv.Cluster] += cost
-			unmountedPVBytes[pv.Cluster] += pv.Bytes
-		}
-	}
-
-	for cluster, amount := range unmountedPVCost {
-		container := kubecost.UnmountedSuffix
-		pod := kubecost.UnmountedSuffix
-		namespace := kubecost.UnmountedSuffix
-		node := ""
-
-		key := newPodKey(cluster, namespace, pod)
-		podMap[key] = &Pod{
-			Window:      window.Clone(),
-			Start:       *window.Start(),
-			End:         *window.End(),
-			Key:         key,
-			Allocations: map[string]*kubecost.Allocation{},
-		}
-
-		podMap[key].AppendContainer(container)
-		podMap[key].Allocations[container].Properties.Cluster = cluster
-		podMap[key].Allocations[container].Properties.Node = node
-		podMap[key].Allocations[container].Properties.Namespace = namespace
-		podMap[key].Allocations[container].Properties.Pod = pod
-		podMap[key].Allocations[container].Properties.Container = container
-		pvKey := kubecost.PVKey{
-			Cluster: cluster,
-			Name:    kubecost.UnmountedSuffix,
-		}
-		unmountedPVs := kubecost.PVAllocations{
-			pvKey: {
-				ByteHours: unmountedPVBytes[cluster] * window.Minutes() / 60.0,
-				Cost:      amount,
-			},
-		}
-		podMap[key].Allocations[container].PVs = podMap[key].Allocations[container].PVs.Add(unmountedPVs)
-	}
-}
-
-func applyUnmountedPVCs(window kubecost.Window, podMap map[podKey]*Pod, pvcMap map[pvcKey]*PVC) {
-	unmountedPVCBytes := map[namespaceKey]float64{}
-	unmountedPVCCost := map[namespaceKey]float64{}
-
-	for _, pvc := range pvcMap {
-		if !pvc.Mounted && pvc.Volume != nil {
-			key := newNamespaceKey(pvc.Cluster, pvc.Namespace)
-
-			gib := pvc.Volume.Bytes / 1024 / 1024 / 1024
-			hrs := pvc.Minutes() / 60.0
-			cost := pvc.Volume.CostPerGiBHour * gib * hrs
-			unmountedPVCCost[key] += cost
-			unmountedPVCBytes[key] += pvc.Volume.Bytes
-		}
-	}
-
-	for key, amount := range unmountedPVCCost {
-		container := kubecost.UnmountedSuffix
-		pod := kubecost.UnmountedSuffix
-		namespace := key.Namespace
-		node := ""
-		cluster := key.Cluster
-
-		podKey := newPodKey(cluster, namespace, pod)
-		podMap[podKey] = &Pod{
-			Window:      window.Clone(),
-			Start:       *window.Start(),
-			End:         *window.End(),
-			Key:         podKey,
-			Allocations: map[string]*kubecost.Allocation{},
-		}
-
-		podMap[podKey].AppendContainer(container)
-		podMap[podKey].Allocations[container].Properties.Cluster = cluster
-		podMap[podKey].Allocations[container].Properties.Node = node
-		podMap[podKey].Allocations[container].Properties.Namespace = namespace
-		podMap[podKey].Allocations[container].Properties.Pod = pod
-		podMap[podKey].Allocations[container].Properties.Container = container
-		pvKey := kubecost.PVKey{
-			Cluster: cluster,
-			Name:    kubecost.UnmountedSuffix,
-		}
-		unmountedPVs := kubecost.PVAllocations{
-			pvKey: {
-				ByteHours: unmountedPVCBytes[key] * window.Minutes() / 60.0,
-				Cost:      amount,
-			},
-		}
-		podMap[podKey].Allocations[container].PVs = podMap[podKey].Allocations[container].PVs.Add(unmountedPVs)
-
-	}
-}
-
-// LB describes the start and end time of a Load Balancer along with cost
-type LB struct {
-	TotalCost float64
-	Start     time.Time
-	End       time.Time
-}
-
-func getLoadBalancerCosts(resLBCost, resLBActiveMins []*prom.QueryResult, resolution time.Duration) map[serviceKey]*LB {
-	lbMap := make(map[serviceKey]*LB)
-
-	for _, res := range resLBActiveMins {
-		serviceKey, err := resultServiceKey(res, env.GetPromClusterLabel(), "namespace", "service_name")
-		if err != nil || len(res.Values) == 0 {
-			continue
-		}
-
-		s := time.Unix(int64(res.Values[0].Timestamp), 0)
-		// subtract resolution from start time to cover full time period
-		s = s.Add(-resolution)
-		e := time.Unix(int64(res.Values[len(res.Values)-1].Timestamp), 0)
-
-		lbMap[serviceKey] = &LB{
-			Start: s,
-			End:   e,
-		}
-	}
-
-	for _, res := range resLBCost {
-		serviceKey, err := resultServiceKey(res, env.GetPromClusterLabel(), "namespace", "service_name")
-		if err != nil {
-			continue
-		}
-		// Apply cost as price-per-hour * hours
-		if lb, ok := lbMap[serviceKey]; ok {
-			lbPricePerHr := res.Values[0].Value
-			hours := lb.End.Sub(lb.Start).Hours()
-			lb.TotalCost += lbPricePerHr * hours
-		} else {
-			log.DedupedWarningf(20, "CostModel: found minutes for key that does not exist: %s", serviceKey)
-		}
-	}
-	return lbMap
-}
-
-func applyLoadBalancersToPods(lbMap map[serviceKey]*LB, allocsByService map[serviceKey][]*kubecost.Allocation) {
-	for sKey, lb := range lbMap {
-		totalHours := 0.0
-		allocHours := make(map[*kubecost.Allocation]float64)
-		// Add portion of load balancing cost to each allocation
-		// proportional to the total number of hours allocations used the load balancer
-		for _, alloc := range allocsByService[sKey] {
-			// Determine the (start, end) of the relationship between the
-			// given LB and the associated Allocation so that a precise
-			// number of hours can be used to compute cumulative cost.
-			s, e := alloc.Start, alloc.End
-			if lb.Start.After(alloc.Start) {
-				s = lb.Start
-			}
-			if lb.End.Before(alloc.End) {
-				e = lb.End
-			}
-			hours := e.Sub(s).Hours()
-			// A negative number of hours signifies no overlap between the windows
-			if hours > 0 {
-				totalHours += hours
-				allocHours[alloc] = hours
-			}
-		}
-
-		// Distribute cost of service once total hours is calculated
-		for alloc, hours := range allocHours {
-			alloc.LoadBalancerCost += lb.TotalCost * hours / totalHours
-		}
-	}
-}
-
-// getNodePricing determines node pricing, given a key and a mapping from keys
-// to their NodePricing instances, as well as the custom pricing configuration
-// inherent to the CostModel instance. If custom pricing is set, use that. If
-// not, use the pricing defined by the given key. If that doesn't exist, fall
-// back on custom pricing as a default.
-func (cm *CostModel) getNodePricing(nodeMap map[nodeKey]*NodePricing, nodeKey nodeKey) *NodePricing {
-	// Find the relevant NodePricing, if it exists. If not, substitute the
-	// custom NodePricing as a default.
-	node, ok := nodeMap[nodeKey]
-	if !ok || node == nil {
-		if nodeKey.Node != "" {
-			log.DedupedWarningf(5, "CostModel: failed to find node for %s", nodeKey)
-		}
-		return cm.getCustomNodePricing(false)
-	}
-
-	// If custom pricing is enabled and can be retrieved, override detected
-	// node pricing with the custom values.
-	customPricingConfig, err := cm.Provider.GetConfig()
-	if err != nil {
-		log.Warnf("CostModel: failed to load custom pricing: %s", err)
-	}
-	if cloud.CustomPricesEnabled(cm.Provider) && customPricingConfig != nil {
-		return cm.getCustomNodePricing(node.Preemptible)
-	}
-
-	node.Source = "prometheus"
-
-	// If any of the values are NaN or zero, replace them with the custom
-	// values as default.
-	// TODO:CLEANUP can't we parse these custom prices once? why do we store
-	// them as strings like this?
-
-	if node.CostPerCPUHr == 0 || math.IsNaN(node.CostPerCPUHr) {
-		log.Warnf("CostModel: node pricing has illegal CostPerCPUHr; replacing with custom pricing: %s", nodeKey)
-		cpuCostStr := customPricingConfig.CPU
-		if node.Preemptible {
-			cpuCostStr = customPricingConfig.SpotCPU
-		}
-		costPerCPUHr, err := strconv.ParseFloat(cpuCostStr, 64)
-		if err != nil {
-			log.Warnf("CostModel: custom pricing has illegal CPU cost: %s", cpuCostStr)
-		}
-		node.CostPerCPUHr = costPerCPUHr
-		node.Source += "/customCPU"
-	}
-
-	if math.IsNaN(node.CostPerGPUHr) {
-		log.Warnf("CostModel: node pricing has illegal CostPerGPUHr; replacing with custom pricing: %s", nodeKey)
-		gpuCostStr := customPricingConfig.GPU
-		if node.Preemptible {
-			gpuCostStr = customPricingConfig.SpotGPU
-		}
-		costPerGPUHr, err := strconv.ParseFloat(gpuCostStr, 64)
-		if err != nil {
-			log.Warnf("CostModel: custom pricing has illegal GPU cost: %s", gpuCostStr)
-		}
-		node.CostPerGPUHr = costPerGPUHr
-		node.Source += "/customGPU"
-	}
-
-	if node.CostPerRAMGiBHr == 0 || math.IsNaN(node.CostPerRAMGiBHr) {
-		log.Warnf("CostModel: node pricing has illegal CostPerRAMHr; replacing with custom pricing: %s", nodeKey)
-		ramCostStr := customPricingConfig.RAM
-		if node.Preemptible {
-			ramCostStr = customPricingConfig.SpotRAM
-		}
-		costPerRAMHr, err := strconv.ParseFloat(ramCostStr, 64)
-		if err != nil {
-			log.Warnf("CostModel: custom pricing has illegal RAM cost: %s", ramCostStr)
-		}
-		node.CostPerRAMGiBHr = costPerRAMHr
-		node.Source += "/customRAM"
-	}
-
-	return node
-}
-
-// getCustomNodePricing converts the CostModel's configured custom pricing
-// values into a NodePricing instance.
-func (cm *CostModel) getCustomNodePricing(spot bool) *NodePricing {
-	customPricingConfig, err := cm.Provider.GetConfig()
-	if err != nil {
-		return nil
-	}
-
-	cpuCostStr := customPricingConfig.CPU
-	gpuCostStr := customPricingConfig.GPU
-	ramCostStr := customPricingConfig.RAM
-	if spot {
-		cpuCostStr = customPricingConfig.SpotCPU
-		gpuCostStr = customPricingConfig.SpotGPU
-		ramCostStr = customPricingConfig.SpotRAM
-	}
-
-	node := &NodePricing{Source: "custom"}
-
-	costPerCPUHr, err := strconv.ParseFloat(cpuCostStr, 64)
-	if err != nil {
-		log.Warnf("CostModel: custom pricing has illegal CPU cost: %s", cpuCostStr)
-	}
-	node.CostPerCPUHr = costPerCPUHr
-
-	costPerGPUHr, err := strconv.ParseFloat(gpuCostStr, 64)
-	if err != nil {
-		log.Warnf("CostModel: custom pricing has illegal GPU cost: %s", gpuCostStr)
-	}
-	node.CostPerGPUHr = costPerGPUHr
-
-	costPerRAMHr, err := strconv.ParseFloat(ramCostStr, 64)
-	if err != nil {
-		log.Warnf("CostModel: custom pricing has illegal RAM cost: %s", ramCostStr)
-	}
-	node.CostPerRAMGiBHr = costPerRAMHr
-
-	return node
-}
-
-// NodePricing describes the resource costs associated with a given node, as
-// well as the source of the information (e.g. prometheus, custom)
-type NodePricing struct {
-	Name            string
-	NodeType        string
-	ProviderID      string
-	Preemptible     bool
-	CostPerCPUHr    float64
-	CostPerRAMGiBHr float64
-	CostPerGPUHr    float64
-	Discount        float64
-	Source          string
-}
-
-// Pod describes a running pod's start and end time within a Window and
-// all the Allocations (i.e. containers) contained within it.
-type Pod struct {
-	Window      kubecost.Window
-	Start       time.Time
-	End         time.Time
-	Key         podKey
-	Allocations map[string]*kubecost.Allocation
-}
-
-// AppendContainer adds an entry for the given container name to the Pod.
-func (p Pod) AppendContainer(container string) {
-	name := fmt.Sprintf("%s/%s/%s/%s", p.Key.Cluster, p.Key.Namespace, p.Key.Pod, container)
-
-	alloc := &kubecost.Allocation{
-		Name:       name,
-		Properties: &kubecost.AllocationProperties{},
-		Window:     p.Window.Clone(),
-		Start:      p.Start,
-		End:        p.End,
-	}
-	alloc.Properties.Container = container
-	alloc.Properties.Pod = p.Key.Pod
-	alloc.Properties.Namespace = p.Key.Namespace
-	alloc.Properties.Cluster = p.Key.Cluster
-
-	p.Allocations[container] = alloc
-}
-
-// PVC describes a PersistentVolumeClaim
-// TODO:CLEANUP move to pkg/kubecost?
-// TODO:CLEANUP add PersistentVolumeClaims field to type Allocation?
-type PVC struct {
-	Bytes     float64   `json:"bytes"`
-	Count     int       `json:"count"`
-	Name      string    `json:"name"`
-	Cluster   string    `json:"cluster"`
-	Namespace string    `json:"namespace"`
-	Volume    *PV       `json:"persistentVolume"`
-	Mounted   bool      `json:"mounted"`
-	Start     time.Time `json:"start"`
-	End       time.Time `json:"end"`
-}
-
-// Cost computes the cumulative cost of the PVC
-func (pvc *PVC) Cost() float64 {
-	if pvc == nil || pvc.Volume == nil {
-		return 0.0
-	}
-
-	gib := pvc.Bytes / 1024 / 1024 / 1024
-	hrs := pvc.Minutes() / 60.0
-
-	return pvc.Volume.CostPerGiBHour * gib * hrs
-}
-
-// Minutes computes the number of minutes over which the PVC is defined
-func (pvc *PVC) Minutes() float64 {
-	if pvc == nil {
-		return 0.0
-	}
-
-	return pvc.End.Sub(pvc.Start).Minutes()
-}
-
-// String returns a string representation of the PVC
-func (pvc *PVC) String() string {
-	if pvc == nil {
-		return "<nil>"
-	}
-	return fmt.Sprintf("%s/%s/%s{Bytes:%.2f, Cost:%.6f, Start,End:%s}", pvc.Cluster, pvc.Namespace, pvc.Name, pvc.Bytes, pvc.Cost(), kubecost.NewWindow(&pvc.Start, &pvc.End))
-}
-
-// PV describes a PersistentVolume
-// TODO:CLEANUP move to pkg/kubecost?
-type PV struct {
-	Bytes          float64 `json:"bytes"`
-	CostPerGiBHour float64 `json:"costPerGiBHour"`
-	Cluster        string  `json:"cluster"`
-	Name           string  `json:"name"`
-	StorageClass   string  `json:"storageClass"`
-}
-
-// String returns a string representation of the PV
-func (pv *PV) String() string {
-	if pv == nil {
-		return "<nil>"
-	}
-	return fmt.Sprintf("%s/%s{Bytes:%.2f, Cost/GiB*Hr:%.6f, StorageClass:%s}", pv.Cluster, pv.Name, pv.Bytes, pv.CostPerGiBHour, pv.StorageClass)
-}

pkg/costmodel/allocation_helpers.go

@@ -0,0 +1,2130 @@
+package costmodel
+
+import (
+	"fmt"
+	"math"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/opencost/opencost/pkg/cloud"
+	"github.com/opencost/opencost/pkg/env"
+	"github.com/opencost/opencost/pkg/kubecost"
+	"github.com/opencost/opencost/pkg/log"
+	"github.com/opencost/opencost/pkg/prom"
+	"github.com/opencost/opencost/pkg/util/timeutil"
+	"k8s.io/apimachinery/pkg/labels"
+)
+
+// This is a bit of a hack to work around garbage data from cadvisor
+// Ideally you cap each pod to the max CPU on its node, but that involves a bit more complexity, as it it would need to be done when allocations joins with asset data.
+const MAX_CPU_CAP = 512
+
+/* Pod Helpers */
+
+func (cm *CostModel) buildPodMap(window kubecost.Window, resolution, maxBatchSize time.Duration, podMap map[podKey]*pod, clusterStart, clusterEnd map[string]time.Time, ingestPodUID bool, podUIDKeyMap map[podKey][]podKey) error {
+	// Assumes that window is positive and closed
+	start, end := *window.Start(), *window.End()
+
+	// Convert resolution duration to a query-ready string
+	resStr := timeutil.DurationString(resolution)
+
+	ctx := prom.NewNamedContext(cm.PrometheusClient, prom.AllocationContextName)
+
+	// Query for (start, end) by (pod, namespace, cluster) over the given
+	// window, using the given resolution, and if necessary in batches no
+	// larger than the given maximum batch size. If working in batches, track
+	// overall progress by starting with (window.start, window.start) and
+	// querying in batches no larger than maxBatchSize from start-to-end,
+	// folding each result set into podMap as the results come back.
+	coverage := kubecost.NewWindow(&start, &start)
+
+	numQuery := 1
+	for coverage.End().Before(end) {
+		// Determine the (start, end) of the current batch
+		batchStart := *coverage.End()
+		batchEnd := coverage.End().Add(maxBatchSize)
+		if batchEnd.After(end) {
+			batchEnd = end
+		}
+
+		var resPods []*prom.QueryResult
+		var err error
+		maxTries := 3
+		numTries := 0
+		for resPods == nil && numTries < maxTries {
+			numTries++
+
+			// Query for the duration between start and end
+			durStr := timeutil.DurationString(batchEnd.Sub(batchStart))
+			if durStr == "" {
+				// Negative duration, so set empty results and don't query
+				resPods = []*prom.QueryResult{}
+				err = nil
+				break
+			}
+
+			// Submit and profile query
+
+			var queryPods string
+			// If ingesting UIDs, avg on them
+			if ingestPodUID {
+				queryPods = fmt.Sprintf(queryFmtPodsUID, env.GetPromClusterLabel(), durStr, resStr)
+			} else {
+				queryPods = fmt.Sprintf(queryFmtPods, env.GetPromClusterLabel(), durStr, resStr)
+			}
+
+			queryProfile := time.Now()
+			resPods, err = ctx.QueryAtTime(queryPods, batchEnd).Await()
+			if err != nil {
+				log.Profile(queryProfile, fmt.Sprintf("CostModel.ComputeAllocation: pod query %d try %d failed: %s", numQuery, numTries, queryPods))
+				resPods = nil
+			}
+		}
+
+		if err != nil {
+			return err
+		}
+
+		// queryFmtPodsUID will return both UID-containing results, and non-UID-containing results,
+		// so filter out the non-containing results so we don't duplicate pods. This is due to the
+		// default setup of Kubecost having replicated kube_pod_container_status_running and
+		// included KSM kube_pod_container_status_running. Querying w/ UID will return both.
+		if ingestPodUID {
+			var resPodsUID []*prom.QueryResult
+
+			for _, res := range resPods {
+				_, err := res.GetString("uid")
+				if err == nil {
+					resPodsUID = append(resPodsUID, res)
+				}
+			}
+
+			if len(resPodsUID) > 0 {
+				resPods = resPodsUID
+			} else {
+				log.DedupedWarningf(5, "CostModel.ComputeAllocation: UID ingestion enabled, but query did not return any results with UID")
+			}
+		}
+
+		applyPodResults(window, resolution, podMap, clusterStart, clusterEnd, resPods, ingestPodUID, podUIDKeyMap)
+
+		coverage = coverage.ExpandEnd(batchEnd)
+		numQuery++
+	}
+
+	return nil
+}
+
+func applyPodResults(window kubecost.Window, resolution time.Duration, podMap map[podKey]*pod, clusterStart, clusterEnd map[string]time.Time, resPods []*prom.QueryResult, ingestPodUID bool, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resPods {
+		if len(res.Values) == 0 {
+			log.Warnf("CostModel.ComputeAllocation: empty minutes result")
+			continue
+		}
+
+		cluster, err := res.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		labels, err := res.GetStrings("namespace", "pod")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: minutes query result missing field: %s", err)
+			continue
+		}
+
+		namespace := labels["namespace"]
+		podName := labels["pod"]
+		key := newPodKey(cluster, namespace, podName)
+
+		// If thisPod UIDs are being used to ID pods, append them to the thisPod name in
+		// the podKey.
+		if ingestPodUID {
+
+			uid, err := res.GetString("uid")
+			if err != nil {
+				log.Warnf("CostModel.ComputeAllocation: UID ingestion enabled, but query result missing field: %s", err)
+			} else {
+
+				newKey := newPodKey(cluster, namespace, podName+" "+uid)
+				podUIDKeyMap[key] = append(podUIDKeyMap[key], newKey)
+
+				key = newKey
+
+			}
+
+		}
+
+		allocStart, allocEnd := calculateStartEndFromIsRunning(res, resolution, window)
+		if allocStart.IsZero() || allocEnd.IsZero() {
+			continue
+		}
+
+		// Set start if unset or this datum's start time is earlier than the
+		// current earliest time.
+		if _, ok := clusterStart[cluster]; !ok || allocStart.Before(clusterStart[cluster]) {
+			clusterStart[cluster] = allocStart
+		}
+
+		// Set end if unset or this datum's end time is later than the
+		// current latest time.
+		if _, ok := clusterEnd[cluster]; !ok || allocEnd.After(clusterEnd[cluster]) {
+			clusterEnd[cluster] = allocEnd
+		}
+
+		if thisPod, ok := podMap[key]; ok {
+			// Pod has already been recorded, so update it accordingly
+			if allocStart.Before(thisPod.Start) {
+				thisPod.Start = allocStart
+			}
+			if allocEnd.After(thisPod.End) {
+				thisPod.End = allocEnd
+			}
+		} else {
+			// pod has not been recorded yet, so insert it
+			podMap[key] = &pod{
+				Window:      window.Clone(),
+				Start:       allocStart,
+				End:         allocEnd,
+				Key:         key,
+				Allocations: map[string]*kubecost.Allocation{},
+			}
+		}
+	}
+}
+
+func applyCPUCoresAllocated(podMap map[podKey]*pod, resCPUCoresAllocated []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resCPUCoresAllocated {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU allocation result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU allocation query result missing 'container': %s", key)
+			continue
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+
+			if _, ok := thisPod.Allocations[container]; !ok {
+				thisPod.appendContainer(container)
+			}
+
+			cpuCores := res.Values[0].Value
+			if cpuCores > MAX_CPU_CAP {
+				log.Infof("[WARNING] Very large cpu allocation, clamping to %f", res.Values[0].Value*(thisPod.Allocations[container].Minutes()/60.0))
+				cpuCores = 0.0
+			}
+			hours := thisPod.Allocations[container].Minutes() / 60.0
+			thisPod.Allocations[container].CPUCoreHours = cpuCores * hours
+
+			node, err := res.GetString("node")
+			if err != nil {
+				log.Warnf("CostModel.ComputeAllocation: CPU allocation query result missing 'node': %s", key)
+				continue
+			}
+			thisPod.Allocations[container].Properties.Node = node
+		}
+	}
+}
+
+func applyCPUCoresRequested(podMap map[podKey]*pod, resCPUCoresRequested []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resCPUCoresRequested {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU request result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU request query result missing 'container': %s", key)
+			continue
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+
+			if _, ok := thisPod.Allocations[container]; !ok {
+				thisPod.appendContainer(container)
+			}
+
+			thisPod.Allocations[container].CPUCoreRequestAverage = res.Values[0].Value
+
+			// If CPU allocation is less than requests, set CPUCoreHours to
+			// request level.
+			if thisPod.Allocations[container].CPUCores() < res.Values[0].Value {
+				thisPod.Allocations[container].CPUCoreHours = res.Values[0].Value * (thisPod.Allocations[container].Minutes() / 60.0)
+			}
+			if thisPod.Allocations[container].CPUCores() > MAX_CPU_CAP {
+				log.Infof("[WARNING] Very large cpu allocation, clamping! to %f", res.Values[0].Value*(thisPod.Allocations[container].Minutes()/60.0))
+				thisPod.Allocations[container].CPUCoreHours = res.Values[0].Value * (thisPod.Allocations[container].Minutes() / 60.0)
+			}
+			node, err := res.GetString("node")
+			if err != nil {
+				log.Warnf("CostModel.ComputeAllocation: CPU request query result missing 'node': %s", key)
+				continue
+			}
+			thisPod.Allocations[container].Properties.Node = node
+		}
+	}
+}
+
+func applyCPUCoresUsedAvg(podMap map[podKey]*pod, resCPUCoresUsedAvg []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resCPUCoresUsedAvg {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU usage avg result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if container == "" || err != nil {
+			container, err = res.GetString("container_name")
+			if err != nil {
+				log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU usage avg query result missing 'container': %s", key)
+				continue
+			}
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+
+			if _, ok := thisPod.Allocations[container]; !ok {
+				thisPod.appendContainer(container)
+			}
+
+			thisPod.Allocations[container].CPUCoreUsageAverage = res.Values[0].Value
+			if res.Values[0].Value > MAX_CPU_CAP {
+				log.Infof("[WARNING] Very large cpu USAGE, dropping outlier")
+				thisPod.Allocations[container].CPUCoreUsageAverage = 0.0
+			}
+		}
+	}
+}
+
+func applyCPUCoresUsedMax(podMap map[podKey]*pod, resCPUCoresUsedMax []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resCPUCoresUsedMax {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU usage max result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if container == "" || err != nil {
+			container, err = res.GetString("container_name")
+			if err != nil {
+				log.DedupedWarningf(10, "CostModel.ComputeAllocation: CPU usage max query result missing 'container': %s", key)
+				continue
+			}
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+
+			if _, ok := thisPod.Allocations[container]; !ok {
+				thisPod.appendContainer(container)
+			}
+
+			if thisPod.Allocations[container].RawAllocationOnly == nil {
+				thisPod.Allocations[container].RawAllocationOnly = &kubecost.RawAllocationOnlyData{
+					CPUCoreUsageMax: res.Values[0].Value,
+				}
+			} else {
+				thisPod.Allocations[container].RawAllocationOnly.CPUCoreUsageMax = res.Values[0].Value
+			}
+		}
+	}
+}
+
+func applyRAMBytesAllocated(podMap map[podKey]*pod, resRAMBytesAllocated []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resRAMBytesAllocated {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM allocation result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM allocation query result missing 'container': %s", key)
+			continue
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+
+			if _, ok := thisPod.Allocations[container]; !ok {
+				thisPod.appendContainer(container)
+			}
+
+			ramBytes := res.Values[0].Value
+			hours := thisPod.Allocations[container].Minutes() / 60.0
+			thisPod.Allocations[container].RAMByteHours = ramBytes * hours
+
+			node, err := res.GetString("node")
+			if err != nil {
+				log.Warnf("CostModel.ComputeAllocation: RAM allocation query result missing 'node': %s", key)
+				continue
+			}
+			thisPod.Allocations[container].Properties.Node = node
+		}
+	}
+}
+
+func applyRAMBytesRequested(podMap map[podKey]*pod, resRAMBytesRequested []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resRAMBytesRequested {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM request result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM request query result missing 'container': %s", key)
+			continue
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, pod := range pods {
+
+			if _, ok := pod.Allocations[container]; !ok {
+				pod.appendContainer(container)
+			}
+
+			pod.Allocations[container].RAMBytesRequestAverage = res.Values[0].Value
+
+			// If RAM allocation is less than requests, set RAMByteHours to
+			// request level.
+			if pod.Allocations[container].RAMBytes() < res.Values[0].Value {
+				pod.Allocations[container].RAMByteHours = res.Values[0].Value * (pod.Allocations[container].Minutes() / 60.0)
+			}
+
+			node, err := res.GetString("node")
+			if err != nil {
+				log.Warnf("CostModel.ComputeAllocation: RAM request query result missing 'node': %s", key)
+				continue
+			}
+			pod.Allocations[container].Properties.Node = node
+		}
+	}
+}
+
+func applyRAMBytesUsedAvg(podMap map[podKey]*pod, resRAMBytesUsedAvg []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resRAMBytesUsedAvg {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM avg usage result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if container == "" || err != nil {
+			container, err = res.GetString("container_name")
+			if err != nil {
+				log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM usage avg query result missing 'container': %s", key)
+				continue
+			}
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+
+			if _, ok := thisPod.Allocations[container]; !ok {
+				thisPod.appendContainer(container)
+			}
+
+			thisPod.Allocations[container].RAMBytesUsageAverage = res.Values[0].Value
+		}
+	}
+}
+
+func applyRAMBytesUsedMax(podMap map[podKey]*pod, resRAMBytesUsedMax []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resRAMBytesUsedMax {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM usage max result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if container == "" || err != nil {
+			container, err = res.GetString("container_name")
+			if err != nil {
+				log.DedupedWarningf(10, "CostModel.ComputeAllocation: RAM usage max query result missing 'container': %s", key)
+				continue
+			}
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+
+			if _, ok := thisPod.Allocations[container]; !ok {
+				thisPod.appendContainer(container)
+			}
+
+			if thisPod.Allocations[container].RawAllocationOnly == nil {
+				thisPod.Allocations[container].RawAllocationOnly = &kubecost.RawAllocationOnlyData{
+					RAMBytesUsageMax: res.Values[0].Value,
+				}
+			} else {
+				thisPod.Allocations[container].RawAllocationOnly.RAMBytesUsageMax = res.Values[0].Value
+			}
+		}
+	}
+}
+
+func applyGPUsAllocated(podMap map[podKey]*pod, resGPUsRequested []*prom.QueryResult, resGPUsAllocated []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	if len(resGPUsAllocated) > 0 { // Use the new query, when it's become available in a window
+		resGPUsRequested = resGPUsAllocated
+	}
+	for _, res := range resGPUsRequested {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: GPU request result missing field: %s", err)
+			continue
+		}
+
+		container, err := res.GetString("container")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: GPU request query result missing 'container': %s", key)
+			continue
+		}
+
+		var pods []*pod
+		if thisPod, ok := podMap[key]; !ok {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+
+			if _, ok := thisPod.Allocations[container]; !ok {
+				thisPod.appendContainer(container)
+			}
+
+			hrs := thisPod.Allocations[container].Minutes() / 60.0
+			thisPod.Allocations[container].GPUHours = res.Values[0].Value * hrs
+		}
+	}
+}
+
+func applyNetworkTotals(podMap map[podKey]*pod, resNetworkTransferBytes []*prom.QueryResult, resNetworkReceiveBytes []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	for _, res := range resNetworkTransferBytes {
+		podKey, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: Network Transfer Bytes query result missing field: %s", err)
+			continue
+		}
+
+		var pods []*pod
+
+		if thisPod, ok := podMap[podKey]; !ok {
+			if uidKeys, ok := podUIDKeyMap[podKey]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+			for _, alloc := range thisPod.Allocations {
+				alloc.NetworkTransferBytes = res.Values[0].Value / float64(len(thisPod.Allocations)) / float64(len(pods))
+			}
+		}
+	}
+	for _, res := range resNetworkReceiveBytes {
+		podKey, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: Network Receive Bytes query result missing field: %s", err)
+			continue
+		}
+
+		var pods []*pod
+
+		if thisPod, ok := podMap[podKey]; !ok {
+			if uidKeys, ok := podUIDKeyMap[podKey]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+			for _, alloc := range thisPod.Allocations {
+				alloc.NetworkReceiveBytes = res.Values[0].Value / float64(len(thisPod.Allocations)) / float64(len(pods))
+			}
+		}
+	}
+}
+
+func applyNetworkAllocation(podMap map[podKey]*pod, resNetworkGiB []*prom.QueryResult, resNetworkCostPerGiB []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey) {
+	costPerGiBByCluster := map[string]float64{}
+
+	for _, res := range resNetworkCostPerGiB {
+		cluster, err := res.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		costPerGiBByCluster[cluster] = res.Values[0].Value
+	}
+
+	for _, res := range resNetworkGiB {
+		podKey, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: Network allocation query result missing field: %s", err)
+			continue
+		}
+
+		var pods []*pod
+
+		if thisPod, ok := podMap[podKey]; !ok {
+			if uidKeys, ok := podUIDKeyMap[podKey]; ok {
+				for _, uidKey := range uidKeys {
+					thisPod, ok = podMap[uidKey]
+					if ok {
+						pods = append(pods, thisPod)
+					}
+				}
+			} else {
+				continue
+			}
+		} else {
+			pods = []*pod{thisPod}
+		}
+
+		for _, thisPod := range pods {
+			for _, alloc := range thisPod.Allocations {
+				gib := res.Values[0].Value / float64(len(thisPod.Allocations))
+				costPerGiB := costPerGiBByCluster[podKey.Cluster]
+				alloc.NetworkCost = gib * costPerGiB / float64(len(pods))
+			}
+		}
+	}
+}
+
+func resToNamespaceLabels(resNamespaceLabels []*prom.QueryResult) map[namespaceKey]map[string]string {
+	namespaceLabels := map[namespaceKey]map[string]string{}
+
+	for _, res := range resNamespaceLabels {
+		nsKey, err := resultNamespaceKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			continue
+		}
+
+		if _, ok := namespaceLabels[nsKey]; !ok {
+			namespaceLabels[nsKey] = map[string]string{}
+		}
+
+		for k, l := range res.GetLabels() {
+			namespaceLabels[nsKey][k] = l
+		}
+	}
+
+	return namespaceLabels
+}
+
+func resToPodLabels(resPodLabels []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]map[string]string {
+	podLabels := map[podKey]map[string]string{}
+
+	for _, res := range resPodLabels {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			continue
+		}
+
+		var keys []podKey
+
+		if ingestPodUID {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+
+				keys = append(keys, uidKeys...)
+
+			}
+		} else {
+			keys = []podKey{key}
+		}
+
+		for _, key := range keys {
+			if _, ok := podLabels[key]; !ok {
+				podLabels[key] = map[string]string{}
+			}
+
+			for k, l := range res.GetLabels() {
+				podLabels[key][k] = l
+			}
+		}
+	}
+
+	return podLabels
+}
+
+func resToNamespaceAnnotations(resNamespaceAnnotations []*prom.QueryResult) map[string]map[string]string {
+	namespaceAnnotations := map[string]map[string]string{}
+
+	for _, res := range resNamespaceAnnotations {
+		namespace, err := res.GetString("namespace")
+		if err != nil {
+			continue
+		}
+
+		if _, ok := namespaceAnnotations[namespace]; !ok {
+			namespaceAnnotations[namespace] = map[string]string{}
+		}
+
+		for k, l := range res.GetAnnotations() {
+			namespaceAnnotations[namespace][k] = l
+		}
+	}
+
+	return namespaceAnnotations
+}
+
+func resToPodAnnotations(resPodAnnotations []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]map[string]string {
+	podAnnotations := map[podKey]map[string]string{}
+
+	for _, res := range resPodAnnotations {
+		key, err := resultPodKey(res, env.GetPromClusterLabel(), "namespace")
+		if err != nil {
+			continue
+		}
+
+		var keys []podKey
+
+		if ingestPodUID {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+
+				keys = append(keys, uidKeys...)
+
+			}
+		} else {
+			keys = []podKey{key}
+		}
+
+		for _, key := range keys {
+			if _, ok := podAnnotations[key]; !ok {
+				podAnnotations[key] = map[string]string{}
+			}
+
+			for k, l := range res.GetAnnotations() {
+				podAnnotations[key][k] = l
+			}
+		}
+	}
+
+	return podAnnotations
+}
+
+func applyLabels(podMap map[podKey]*pod, namespaceLabels map[namespaceKey]map[string]string, podLabels map[podKey]map[string]string) {
+	for podKey, pod := range podMap {
+		for _, alloc := range pod.Allocations {
+			allocLabels := alloc.Properties.Labels
+			if allocLabels == nil {
+				allocLabels = make(map[string]string)
+			}
+			// Apply namespace labels first, then pod labels so that pod labels
+			// overwrite namespace labels.
+			nsKey := podKey.namespaceKey // newNamespaceKey(podKey.Cluster, podKey.Namespace)
+			if labels, ok := namespaceLabels[nsKey]; ok {
+				for k, v := range labels {
+					allocLabels[k] = v
+				}
+			}
+			if labels, ok := podLabels[podKey]; ok {
+				for k, v := range labels {
+					allocLabels[k] = v
+				}
+			}
+
+			alloc.Properties.Labels = allocLabels
+		}
+	}
+}
+
+func applyAnnotations(podMap map[podKey]*pod, namespaceAnnotations map[string]map[string]string, podAnnotations map[podKey]map[string]string) {
+	for key, pod := range podMap {
+		for _, alloc := range pod.Allocations {
+			allocAnnotations := alloc.Properties.Annotations
+			if allocAnnotations == nil {
+				allocAnnotations = make(map[string]string)
+			}
+			// Apply namespace annotations first, then pod annotations so that
+			// pod labels overwrite namespace labels.
+			if labels, ok := namespaceAnnotations[key.Namespace]; ok {
+				for k, v := range labels {
+					allocAnnotations[k] = v
+				}
+			}
+			if labels, ok := podAnnotations[key]; ok {
+				for k, v := range labels {
+					allocAnnotations[k] = v
+				}
+			}
+
+			alloc.Properties.Annotations = allocAnnotations
+		}
+	}
+}
+
+func resToDeploymentLabels(resDeploymentLabels []*prom.QueryResult) map[controllerKey]map[string]string {
+	deploymentLabels := map[controllerKey]map[string]string{}
+
+	for _, res := range resDeploymentLabels {
+		controllerKey, err := resultDeploymentKey(res, env.GetPromClusterLabel(), "namespace", "deployment")
+		if err != nil {
+			continue
+		}
+
+		if _, ok := deploymentLabels[controllerKey]; !ok {
+			deploymentLabels[controllerKey] = map[string]string{}
+		}
+
+		for k, l := range res.GetLabels() {
+			deploymentLabels[controllerKey][k] = l
+		}
+	}
+
+	// Prune duplicate deployments. That is, if the same deployment exists with
+	// hyphens instead of underscores, keep the one that uses hyphens.
+	for key := range deploymentLabels {
+		if strings.Contains(key.Controller, "_") {
+			duplicateController := strings.Replace(key.Controller, "_", "-", -1)
+			duplicateKey := newControllerKey(key.Cluster, key.Namespace, key.ControllerKind, duplicateController)
+			if _, ok := deploymentLabels[duplicateKey]; ok {
+				delete(deploymentLabels, key)
+			}
+		}
+	}
+
+	return deploymentLabels
+}
+
+func resToStatefulSetLabels(resStatefulSetLabels []*prom.QueryResult) map[controllerKey]map[string]string {
+	statefulSetLabels := map[controllerKey]map[string]string{}
+
+	for _, res := range resStatefulSetLabels {
+		controllerKey, err := resultStatefulSetKey(res, env.GetPromClusterLabel(), "namespace", "statefulSet")
+		if err != nil {
+			continue
+		}
+
+		if _, ok := statefulSetLabels[controllerKey]; !ok {
+			statefulSetLabels[controllerKey] = map[string]string{}
+		}
+
+		for k, l := range res.GetLabels() {
+			statefulSetLabels[controllerKey][k] = l
+		}
+	}
+
+	// Prune duplicate stateful sets. That is, if the same stateful set exists
+	// with hyphens instead of underscores, keep the one that uses hyphens.
+	for key := range statefulSetLabels {
+		if strings.Contains(key.Controller, "_") {
+			duplicateController := strings.Replace(key.Controller, "_", "-", -1)
+			duplicateKey := newControllerKey(key.Cluster, key.Namespace, key.ControllerKind, duplicateController)
+			if _, ok := statefulSetLabels[duplicateKey]; ok {
+				delete(statefulSetLabels, key)
+			}
+		}
+	}
+
+	return statefulSetLabels
+}
+
+func labelsToPodControllerMap(podLabels map[podKey]map[string]string, controllerLabels map[controllerKey]map[string]string) map[podKey]controllerKey {
+	podControllerMap := map[podKey]controllerKey{}
+
+	// For each controller, turn the labels into a selector and attempt to
+	// match it with each set of pod labels. A match indicates that the pod
+	// belongs to the controller.
+	for cKey, cLabels := range controllerLabels {
+		selector := labels.Set(cLabels).AsSelectorPreValidated()
+
+		for pKey, pLabels := range podLabels {
+			// If the pod is in a different cluster or namespace, there is
+			// no need to compare the labels.
+			if cKey.Cluster != pKey.Cluster || cKey.Namespace != pKey.Namespace {
+				continue
+			}
+
+			podLabelSet := labels.Set(pLabels)
+			if selector.Matches(podLabelSet) {
+				if _, ok := podControllerMap[pKey]; ok {
+					log.DedupedWarningf(5, "CostModel.ComputeAllocation: PodControllerMap match already exists: %s matches %s and %s", pKey, podControllerMap[pKey], cKey)
+				}
+				podControllerMap[pKey] = cKey
+			}
+		}
+	}
+
+	return podControllerMap
+}
+
+func resToPodDaemonSetMap(resDaemonSetLabels []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]controllerKey {
+	daemonSetLabels := map[podKey]controllerKey{}
+
+	for _, res := range resDaemonSetLabels {
+		controllerKey, err := resultDaemonSetKey(res, env.GetPromClusterLabel(), "namespace", "owner_name")
+		if err != nil {
+			continue
+		}
+
+		pod, err := res.GetString("pod")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: DaemonSetLabel result without pod: %s", controllerKey)
+		}
+
+		key := newPodKey(controllerKey.Cluster, controllerKey.Namespace, pod)
+
+		var keys []podKey
+
+		if ingestPodUID {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+
+				keys = append(keys, uidKeys...)
+
+			}
+		} else {
+			keys = []podKey{key}
+		}
+
+		for _, key := range keys {
+			daemonSetLabels[key] = controllerKey
+		}
+	}
+
+	return daemonSetLabels
+}
+
+func resToPodJobMap(resJobLabels []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]controllerKey {
+	jobLabels := map[podKey]controllerKey{}
+
+	for _, res := range resJobLabels {
+		controllerKey, err := resultJobKey(res, env.GetPromClusterLabel(), "namespace", "owner_name")
+		if err != nil {
+			continue
+		}
+
+		// Convert the name of Jobs generated by CronJobs to the name of the
+		// CronJob by stripping the timestamp off the end.
+		match := isCron.FindStringSubmatch(controllerKey.Controller)
+		if match != nil {
+			controllerKey.Controller = match[1]
+		}
+
+		pod, err := res.GetString("pod")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: JobLabel result without pod: %s", controllerKey)
+		}
+
+		key := newPodKey(controllerKey.Cluster, controllerKey.Namespace, pod)
+
+		var keys []podKey
+
+		if ingestPodUID {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+
+				keys = append(keys, uidKeys...)
+
+			}
+		} else {
+			keys = []podKey{key}
+		}
+
+		for _, key := range keys {
+			jobLabels[key] = controllerKey
+		}
+	}
+
+	return jobLabels
+}
+
+func resToPodReplicaSetMap(resPodsWithReplicaSetOwner []*prom.QueryResult, resReplicaSetsWithoutOwners []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) map[podKey]controllerKey {
+	// Build out set of ReplicaSets that have no owners, themselves, such that
+	// the ReplicaSet should be used as the owner of the Pods it controls.
+	// (This should exclude, for example, ReplicaSets that are controlled by
+	// Deployments, in which case the Deployment should be the pod's owner.)
+	replicaSets := map[controllerKey]struct{}{}
+
+	for _, res := range resReplicaSetsWithoutOwners {
+		controllerKey, err := resultReplicaSetKey(res, env.GetPromClusterLabel(), "namespace", "replicaset")
+		if err != nil {
+			continue
+		}
+
+		replicaSets[controllerKey] = struct{}{}
+	}
+
+	// Create the mapping of Pods to ReplicaSets, ignoring any ReplicaSets that
+	// to not appear in the set of uncontrolled ReplicaSets above.
+	podToReplicaSet := map[podKey]controllerKey{}
+
+	for _, res := range resPodsWithReplicaSetOwner {
+		controllerKey, err := resultReplicaSetKey(res, env.GetPromClusterLabel(), "namespace", "owner_name")
+		if err != nil {
+			continue
+		}
+		if _, ok := replicaSets[controllerKey]; !ok {
+			continue
+		}
+
+		pod, err := res.GetString("pod")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: ReplicaSet result without pod: %s", controllerKey)
+		}
+
+		key := newPodKey(controllerKey.Cluster, controllerKey.Namespace, pod)
+
+		var keys []podKey
+
+		if ingestPodUID {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+
+				keys = append(keys, uidKeys...)
+
+			}
+		} else {
+			keys = []podKey{key}
+		}
+
+		for _, key := range keys {
+
+			podToReplicaSet[key] = controllerKey
+
+		}
+	}
+
+	return podToReplicaSet
+}
+
+func applyControllersToPods(podMap map[podKey]*pod, podControllerMap map[podKey]controllerKey) {
+	for key, pod := range podMap {
+		for _, alloc := range pod.Allocations {
+			if controllerKey, ok := podControllerMap[key]; ok {
+				alloc.Properties.ControllerKind = controllerKey.ControllerKind
+				alloc.Properties.Controller = controllerKey.Controller
+			}
+		}
+	}
+}
+
+/* Service Helpers */
+
+func getServiceLabels(resServiceLabels []*prom.QueryResult) map[serviceKey]map[string]string {
+	serviceLabels := map[serviceKey]map[string]string{}
+
+	for _, res := range resServiceLabels {
+		serviceKey, err := resultServiceKey(res, env.GetPromClusterLabel(), "namespace", "service")
+		if err != nil {
+			continue
+		}
+
+		if _, ok := serviceLabels[serviceKey]; !ok {
+			serviceLabels[serviceKey] = map[string]string{}
+		}
+
+		for k, l := range res.GetLabels() {
+			serviceLabels[serviceKey][k] = l
+		}
+	}
+
+	// Prune duplicate services. That is, if the same service exists with
+	// hyphens instead of underscores, keep the one that uses hyphens.
+	for key := range serviceLabels {
+		if strings.Contains(key.Service, "_") {
+			duplicateService := strings.Replace(key.Service, "_", "-", -1)
+			duplicateKey := newServiceKey(key.Cluster, key.Namespace, duplicateService)
+			if _, ok := serviceLabels[duplicateKey]; ok {
+				delete(serviceLabels, key)
+			}
+		}
+	}
+
+	return serviceLabels
+}
+
+func applyServicesToPods(podMap map[podKey]*pod, podLabels map[podKey]map[string]string, allocsByService map[serviceKey][]*kubecost.Allocation, serviceLabels map[serviceKey]map[string]string) {
+	podServicesMap := map[podKey][]serviceKey{}
+
+	// For each service, turn the labels into a selector and attempt to
+	// match it with each set of pod labels. A match indicates that the pod
+	// belongs to the service.
+	for sKey, sLabels := range serviceLabels {
+		selector := labels.Set(sLabels).AsSelectorPreValidated()
+
+		for pKey, pLabels := range podLabels {
+			// If the pod is in a different cluster or namespace, there is
+			// no need to compare the labels.
+			if sKey.Cluster != pKey.Cluster || sKey.Namespace != pKey.Namespace {
+				continue
+			}
+
+			podLabelSet := labels.Set(pLabels)
+			if selector.Matches(podLabelSet) {
+				if _, ok := podServicesMap[pKey]; !ok {
+					podServicesMap[pKey] = []serviceKey{}
+				}
+				podServicesMap[pKey] = append(podServicesMap[pKey], sKey)
+			}
+		}
+	}
+
+	// For each allocation in each pod, attempt to find and apply the list of
+	// services associated with the allocation's pod.
+	for key, pod := range podMap {
+		for _, alloc := range pod.Allocations {
+			if sKeys, ok := podServicesMap[key]; ok {
+				services := []string{}
+				for _, sKey := range sKeys {
+					services = append(services, sKey.Service)
+					allocsByService[sKey] = append(allocsByService[sKey], alloc)
+				}
+				alloc.Properties.Services = services
+
+			}
+		}
+	}
+}
+
+func getLoadBalancerCosts(lbMap map[serviceKey]*lbCost, resLBCost, resLBActiveMins []*prom.QueryResult, resolution time.Duration) {
+	for _, res := range resLBActiveMins {
+		serviceKey, err := resultServiceKey(res, env.GetPromClusterLabel(), "namespace", "service_name")
+		if err != nil || len(res.Values) == 0 {
+			continue
+		}
+
+		lbStart, lbEnd := calculateStartAndEnd(res, resolution)
+		if lbStart.IsZero() || lbEnd.IsZero() {
+			log.Warnf("CostModel.ComputeAllocation: pvc %s has no running time", serviceKey)
+		}
+
+		lbMap[serviceKey] = &lbCost{
+			Start: lbStart,
+			End:   lbEnd,
+		}
+	}
+
+	for _, res := range resLBCost {
+		serviceKey, err := resultServiceKey(res, env.GetPromClusterLabel(), "namespace", "service_name")
+		if err != nil {
+			continue
+		}
+		// Apply cost as price-per-hour * hours
+		if lb, ok := lbMap[serviceKey]; ok {
+			lbPricePerHr := res.Values[0].Value
+			hours := lb.End.Sub(lb.Start).Hours()
+			lb.TotalCost += lbPricePerHr * hours
+		} else {
+			log.DedupedWarningf(20, "CostModel: found minutes for key that does not exist: %s", serviceKey)
+		}
+	}
+}
+
+func applyLoadBalancersToPods(window kubecost.Window, podMap map[podKey]*pod, lbMap map[serviceKey]*lbCost, allocsByService map[serviceKey][]*kubecost.Allocation) {
+	for sKey, lb := range lbMap {
+		totalHours := 0.0
+		allocHours := make(map[*kubecost.Allocation]float64)
+
+		allocs, ok := allocsByService[sKey]
+		// if there are no allocations using the service, add its cost to the Unmounted pod for its cluster
+		if !ok {
+			pod := getUnmountedPodForCluster(window, podMap, sKey.Cluster)
+			pod.Allocations[kubecost.UnmountedSuffix].LoadBalancerCost += lb.TotalCost
+			pod.Allocations[kubecost.UnmountedSuffix].Properties.Services = append(pod.Allocations[kubecost.UnmountedSuffix].Properties.Services, sKey.Service)
+		}
+		// Add portion of load balancing cost to each allocation
+		// proportional to the total number of hours allocations used the load balancer
+		for _, alloc := range allocs {
+			// Determine the (start, end) of the relationship between the
+			// given lbCost and the associated Allocation so that a precise
+			// number of hours can be used to compute cumulative cost.
+			s, e := alloc.Start, alloc.End
+			if lb.Start.After(alloc.Start) {
+				s = lb.Start
+			}
+			if lb.End.Before(alloc.End) {
+				e = lb.End
+			}
+			hours := e.Sub(s).Hours()
+			// A negative number of hours signifies no overlap between the windows
+			if hours > 0 {
+				totalHours += hours
+				allocHours[alloc] = hours
+			}
+		}
+
+		// Distribute cost of service once total hours is calculated
+		for alloc, hours := range allocHours {
+			alloc.LoadBalancerCost += lb.TotalCost * hours / totalHours
+		}
+
+		// If there was no overlap apply to Unmounted pod
+		if len(allocHours) == 0 {
+			pod := getUnmountedPodForCluster(window, podMap, sKey.Cluster)
+			pod.Allocations[kubecost.UnmountedSuffix].LoadBalancerCost += lb.TotalCost
+			pod.Allocations[kubecost.UnmountedSuffix].Properties.Services = append(pod.Allocations[kubecost.UnmountedSuffix].Properties.Services, sKey.Service)
+		}
+	}
+}
+
+/* Node Helpers */
+
+func applyNodeCostPerCPUHr(nodeMap map[nodeKey]*nodePricing, resNodeCostPerCPUHr []*prom.QueryResult) {
+	for _, res := range resNodeCostPerCPUHr {
+		cluster, err := res.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		node, err := res.GetString("node")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node CPU cost query result missing field: %s", err)
+			continue
+		}
+
+		instanceType, err := res.GetString("instance_type")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node CPU cost query result missing field: %s", err)
+			continue
+		}
+
+		providerID, err := res.GetString("provider_id")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node CPU cost query result missing field: %s", err)
+			continue
+		}
+
+		key := newNodeKey(cluster, node)
+		if _, ok := nodeMap[key]; !ok {
+			nodeMap[key] = &nodePricing{
+				Name:       node,
+				NodeType:   instanceType,
+				ProviderID: cloud.ParseID(providerID),
+			}
+		}
+
+		nodeMap[key].CostPerCPUHr = res.Values[0].Value
+	}
+}
+
+func applyNodeCostPerRAMGiBHr(nodeMap map[nodeKey]*nodePricing, resNodeCostPerRAMGiBHr []*prom.QueryResult) {
+	for _, res := range resNodeCostPerRAMGiBHr {
+		cluster, err := res.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		node, err := res.GetString("node")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node RAM cost query result missing field: %s", err)
+			continue
+		}
+
+		instanceType, err := res.GetString("instance_type")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node RAM cost query result missing field: %s", err)
+			continue
+		}
+
+		providerID, err := res.GetString("provider_id")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node RAM cost query result missing field: %s", err)
+			continue
+		}
+
+		key := newNodeKey(cluster, node)
+		if _, ok := nodeMap[key]; !ok {
+			nodeMap[key] = &nodePricing{
+				Name:       node,
+				NodeType:   instanceType,
+				ProviderID: cloud.ParseID(providerID),
+			}
+		}
+
+		nodeMap[key].CostPerRAMGiBHr = res.Values[0].Value
+	}
+}
+
+func applyNodeCostPerGPUHr(nodeMap map[nodeKey]*nodePricing, resNodeCostPerGPUHr []*prom.QueryResult) {
+	for _, res := range resNodeCostPerGPUHr {
+		cluster, err := res.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		node, err := res.GetString("node")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node GPU cost query result missing field: %s", err)
+			continue
+		}
+
+		instanceType, err := res.GetString("instance_type")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node GPU cost query result missing field: %s", err)
+			continue
+		}
+
+		providerID, err := res.GetString("provider_id")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node GPU cost query result missing field: %s", err)
+			continue
+		}
+
+		key := newNodeKey(cluster, node)
+		if _, ok := nodeMap[key]; !ok {
+			nodeMap[key] = &nodePricing{
+				Name:       node,
+				NodeType:   instanceType,
+				ProviderID: cloud.ParseID(providerID),
+			}
+		}
+
+		nodeMap[key].CostPerGPUHr = res.Values[0].Value
+	}
+}
+
+func applyNodeSpot(nodeMap map[nodeKey]*nodePricing, resNodeIsSpot []*prom.QueryResult) {
+	for _, res := range resNodeIsSpot {
+		cluster, err := res.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		node, err := res.GetString("node")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: Node spot query result missing field: %s", err)
+			continue
+		}
+
+		key := newNodeKey(cluster, node)
+		if _, ok := nodeMap[key]; !ok {
+			log.Warnf("CostModel.ComputeAllocation: Node spot  query result for missing node: %s", key)
+			continue
+		}
+
+		nodeMap[key].Preemptible = res.Values[0].Value > 0
+	}
+}
+
+func applyNodeDiscount(nodeMap map[nodeKey]*nodePricing, cm *CostModel) {
+	if cm == nil {
+		return
+	}
+
+	c, err := cm.Provider.GetConfig()
+	if err != nil {
+		log.Errorf("CostModel.ComputeAllocation: applyNodeDiscount: %s", err)
+		return
+	}
+
+	discount, err := ParsePercentString(c.Discount)
+	if err != nil {
+		log.Errorf("CostModel.ComputeAllocation: applyNodeDiscount: %s", err)
+		return
+	}
+
+	negotiatedDiscount, err := ParsePercentString(c.NegotiatedDiscount)
+	if err != nil {
+		log.Errorf("CostModel.ComputeAllocation: applyNodeDiscount: %s", err)
+		return
+	}
+
+	for _, node := range nodeMap {
+		// TODO GKE Reserved Instances into account
+		node.Discount = cm.Provider.CombinedDiscountForNode(node.NodeType, node.Preemptible, discount, negotiatedDiscount)
+		node.CostPerCPUHr *= (1.0 - node.Discount)
+		node.CostPerRAMGiBHr *= (1.0 - node.Discount)
+	}
+}
+
+func (cm *CostModel) applyNodesToPod(podMap map[podKey]*pod, nodeMap map[nodeKey]*nodePricing) {
+	for _, pod := range podMap {
+		for _, alloc := range pod.Allocations {
+			cluster := alloc.Properties.Cluster
+			nodeName := alloc.Properties.Node
+			thisNodeKey := newNodeKey(cluster, nodeName)
+
+			node := cm.getNodePricing(nodeMap, thisNodeKey)
+			alloc.Properties.ProviderID = node.ProviderID
+			alloc.CPUCost = alloc.CPUCoreHours * node.CostPerCPUHr
+			alloc.RAMCost = (alloc.RAMByteHours / 1024 / 1024 / 1024) * node.CostPerRAMGiBHr
+			alloc.GPUCost = alloc.GPUHours * node.CostPerGPUHr
+		}
+	}
+}
+
+// getCustomNodePricing converts the CostModel's configured custom pricing
+// values into a nodePricing instance.
+func (cm *CostModel) getCustomNodePricing(spot bool) *nodePricing {
+	customPricingConfig, err := cm.Provider.GetConfig()
+	if err != nil {
+		return nil
+	}
+
+	cpuCostStr := customPricingConfig.CPU
+	gpuCostStr := customPricingConfig.GPU
+	ramCostStr := customPricingConfig.RAM
+	if spot {
+		cpuCostStr = customPricingConfig.SpotCPU
+		gpuCostStr = customPricingConfig.SpotGPU
+		ramCostStr = customPricingConfig.SpotRAM
+	}
+
+	node := &nodePricing{Source: "custom"}
+
+	costPerCPUHr, err := strconv.ParseFloat(cpuCostStr, 64)
+	if err != nil {
+		log.Warnf("CostModel: custom pricing has illegal CPU cost: %s", cpuCostStr)
+	}
+	node.CostPerCPUHr = costPerCPUHr
+
+	costPerGPUHr, err := strconv.ParseFloat(gpuCostStr, 64)
+	if err != nil {
+		log.Warnf("CostModel: custom pricing has illegal GPU cost: %s", gpuCostStr)
+	}
+	node.CostPerGPUHr = costPerGPUHr
+
+	costPerRAMHr, err := strconv.ParseFloat(ramCostStr, 64)
+	if err != nil {
+		log.Warnf("CostModel: custom pricing has illegal RAM cost: %s", ramCostStr)
+	}
+	node.CostPerRAMGiBHr = costPerRAMHr
+
+	return node
+}
+
+// getNodePricing determines node pricing, given a key and a mapping from keys
+// to their nodePricing instances, as well as the custom pricing configuration
+// inherent to the CostModel instance. If custom pricing is set, use that. If
+// not, use the pricing defined by the given key. If that doesn't exist, fall
+// back on custom pricing as a default.
+func (cm *CostModel) getNodePricing(nodeMap map[nodeKey]*nodePricing, nodeKey nodeKey) *nodePricing {
+	// Find the relevant nodePricing, if it exists. If not, substitute the
+	// custom nodePricing as a default.
+	node, ok := nodeMap[nodeKey]
+	if !ok || node == nil {
+		if nodeKey.Node != "" {
+			log.DedupedWarningf(5, "CostModel: failed to find node for %s", nodeKey)
+		}
+		return cm.getCustomNodePricing(false)
+	}
+
+	// If custom pricing is enabled and can be retrieved, override detected
+	// node pricing with the custom values.
+	customPricingConfig, err := cm.Provider.GetConfig()
+	if err != nil {
+		log.Warnf("CostModel: failed to load custom pricing: %s", err)
+	}
+	if cloud.CustomPricesEnabled(cm.Provider) && customPricingConfig != nil {
+		return cm.getCustomNodePricing(node.Preemptible)
+	}
+
+	node.Source = "prometheus"
+
+	// If any of the values are NaN or zero, replace them with the custom
+	// values as default.
+	// TODO:CLEANUP can't we parse these custom prices once? why do we store
+	// them as strings like this?
+
+	if node.CostPerCPUHr == 0 || math.IsNaN(node.CostPerCPUHr) {
+		log.Warnf("CostModel: node pricing has illegal CostPerCPUHr; replacing with custom pricing: %s", nodeKey)
+		cpuCostStr := customPricingConfig.CPU
+		if node.Preemptible {
+			cpuCostStr = customPricingConfig.SpotCPU
+		}
+		costPerCPUHr, err := strconv.ParseFloat(cpuCostStr, 64)
+		if err != nil {
+			log.Warnf("CostModel: custom pricing has illegal CPU cost: %s", cpuCostStr)
+		}
+		node.CostPerCPUHr = costPerCPUHr
+		node.Source += "/customCPU"
+	}
+
+	if math.IsNaN(node.CostPerGPUHr) {
+		log.Warnf("CostModel: node pricing has illegal CostPerGPUHr; replacing with custom pricing: %s", nodeKey)
+		gpuCostStr := customPricingConfig.GPU
+		if node.Preemptible {
+			gpuCostStr = customPricingConfig.SpotGPU
+		}
+		costPerGPUHr, err := strconv.ParseFloat(gpuCostStr, 64)
+		if err != nil {
+			log.Warnf("CostModel: custom pricing has illegal GPU cost: %s", gpuCostStr)
+		}
+		node.CostPerGPUHr = costPerGPUHr
+		node.Source += "/customGPU"
+	}
+
+	if node.CostPerRAMGiBHr == 0 || math.IsNaN(node.CostPerRAMGiBHr) {
+		log.Warnf("CostModel: node pricing has illegal CostPerRAMHr; replacing with custom pricing: %s", nodeKey)
+		ramCostStr := customPricingConfig.RAM
+		if node.Preemptible {
+			ramCostStr = customPricingConfig.SpotRAM
+		}
+		costPerRAMHr, err := strconv.ParseFloat(ramCostStr, 64)
+		if err != nil {
+			log.Warnf("CostModel: custom pricing has illegal RAM cost: %s", ramCostStr)
+		}
+		node.CostPerRAMGiBHr = costPerRAMHr
+		node.Source += "/customRAM"
+	}
+
+	return node
+}
+
+/* PV/PVC Helpers */
+
+func buildPVMap(resolution time.Duration, pvMap map[pvKey]*pv, resPVCostPerGiBHour, resPVActiveMins []*prom.QueryResult) {
+	for _, result := range resPVActiveMins {
+		key, err := resultPVKey(result, env.GetPromClusterLabel(), "persistentvolume")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: pv bytes query result missing field: %s", err)
+			continue
+		}
+
+		pvStart, pvEnd := calculateStartAndEnd(result, resolution)
+		if pvStart.IsZero() || pvEnd.IsZero() {
+			log.Warnf("CostModel.ComputeAllocation: pv %s has no running time", key)
+		}
+
+		pvMap[key] = &pv{
+			Cluster: key.Cluster,
+			Name:    key.PersistentVolume,
+			Start:   pvStart,
+			End:     pvEnd,
+		}
+	}
+
+	for _, result := range resPVCostPerGiBHour {
+		key, err := resultPVKey(result, env.GetPromClusterLabel(), "volumename")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: thisPV bytes query result missing field: %s", err)
+			continue
+		}
+
+		if _, ok := pvMap[key]; !ok {
+			pvMap[key] = &pv{
+				Cluster: key.Cluster,
+				Name:    key.PersistentVolume,
+			}
+		}
+		pvMap[key].CostPerGiBHour = result.Values[0].Value
+
+	}
+}
+
+func applyPVBytes(pvMap map[pvKey]*pv, resPVBytes []*prom.QueryResult) {
+	for _, res := range resPVBytes {
+		key, err := resultPVKey(res, env.GetPromClusterLabel(), "persistentvolume")
+		if err != nil {
+			log.Warnf("CostModel.ComputeAllocation: pv bytes query result missing field: %s", err)
+			continue
+		}
+
+		if _, ok := pvMap[key]; !ok {
+			log.Warnf("CostModel.ComputeAllocation: pv bytes result for missing pv: %s", err)
+			continue
+		}
+
+		pvMap[key].Bytes = res.Values[0].Value
+	}
+}
+
+func buildPVCMap(resolution time.Duration, pvcMap map[pvcKey]*pvc, pvMap map[pvKey]*pv, resPVCInfo []*prom.QueryResult) {
+	for _, res := range resPVCInfo {
+		cluster, err := res.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		values, err := res.GetStrings("persistentvolumeclaim", "storageclass", "volumename", "namespace")
+		if err != nil {
+			log.DedupedWarningf(10, "CostModel.ComputeAllocation: pvc info query result missing field: %s", err)
+			continue
+		}
+
+		namespace := values["namespace"]
+		name := values["persistentvolumeclaim"]
+		volume := values["volumename"]
+		storageClass := values["storageclass"]
+
+		pvKey := newPVKey(cluster, volume)
+		pvcKey := newPVCKey(cluster, namespace, name)
+
+		pvcStart, pvcEnd := calculateStartAndEnd(res, resolution)
+		if pvcStart.IsZero() || pvcEnd.IsZero() {
+			log.Warnf("CostModel.ComputeAllocation: pvc %s has no running time", pvcKey)
+		}
+
+		if _, ok := pvMap[pvKey]; !ok {
+			continue
+		}
+
+		pvMap[pvKey].StorageClass = storageClass
+
+		if _, ok := pvcMap[pvcKey]; !ok {
+			pvcMap[pvcKey] = &pvc{}
+		}
+
+		pvcMap[pvcKey].Name = name
+		pvcMap[pvcKey].Namespace = namespace
+		pvcMap[pvcKey].Cluster = cluster
+		pvcMap[pvcKey].Volume = pvMap[pvKey]
+		pvcMap[pvcKey].Start = pvcStart
+		pvcMap[pvcKey].End = pvcEnd
+	}
+}
+
+func applyPVCBytesRequested(pvcMap map[pvcKey]*pvc, resPVCBytesRequested []*prom.QueryResult) {
+	for _, res := range resPVCBytesRequested {
+		key, err := resultPVCKey(res, env.GetPromClusterLabel(), "namespace", "persistentvolumeclaim")
+		if err != nil {
+			continue
+		}
+
+		if _, ok := pvcMap[key]; !ok {
+			continue
+		}
+
+		pvcMap[key].Bytes = res.Values[0].Value
+	}
+}
+
+func buildPodPVCMap(podPVCMap map[podKey][]*pvc, pvMap map[pvKey]*pv, pvcMap map[pvcKey]*pvc, podMap map[podKey]*pod, resPodPVCAllocation []*prom.QueryResult, podUIDKeyMap map[podKey][]podKey, ingestPodUID bool) {
+	for _, res := range resPodPVCAllocation {
+		cluster, err := res.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		values, err := res.GetStrings("persistentvolume", "persistentvolumeclaim", "pod", "namespace")
+		if err != nil {
+			log.DedupedWarningf(5, "CostModel.ComputeAllocation: pvc allocation query result missing field: %s", err)
+			continue
+		}
+
+		namespace := values["namespace"]
+		pod := values["pod"]
+		name := values["persistentvolumeclaim"]
+		volume := values["persistentvolume"]
+
+		key := newPodKey(cluster, namespace, pod)
+		pvKey := newPVKey(cluster, volume)
+		pvcKey := newPVCKey(cluster, namespace, name)
+
+		var keys []podKey
+
+		if ingestPodUID {
+			if uidKeys, ok := podUIDKeyMap[key]; ok {
+
+				keys = append(keys, uidKeys...)
+
+			}
+		} else {
+			keys = []podKey{key}
+		}
+
+		for _, key := range keys {
+
+			if _, ok := pvMap[pvKey]; !ok {
+				log.DedupedWarningf(5, "CostModel.ComputeAllocation: pv missing for pvc allocation query result: %s", pvKey)
+				continue
+			}
+
+			if _, ok := podPVCMap[key]; !ok {
+				podPVCMap[key] = []*pvc{}
+			}
+
+			pvc, ok := pvcMap[pvcKey]
+			if !ok {
+				log.DedupedWarningf(5, "CostModel.ComputeAllocation: pvc missing for pvc allocation query: %s", pvcKey)
+				continue
+			}
+
+			if pod, ok := podMap[key]; !ok || len(pod.Allocations) <= 0 {
+				log.DedupedWarningf(10, "CostModel.ComputeAllocation: pvc %s for missing pod %s", pvcKey, key)
+				continue
+			}
+
+			pvc.Mounted = true
+
+			podPVCMap[key] = append(podPVCMap[key], pvc)
+		}
+	}
+}
+func applyPVCsToPods(window kubecost.Window, podMap map[podKey]*pod, podPVCMap map[podKey][]*pvc, pvcMap map[pvcKey]*pvc) {
+	// Because PVCs can be shared among pods, the respective pv cost
+	// needs to be evenly distributed to those pods based on time
+	// running, as well as the amount of time the pvc was shared.
+
+	// Build a relation between every pvc to the pods that mount it
+	// and a window representing the interval during which they
+	// were associated.
+	pvcPodWindowMap := make(map[pvcKey]map[podKey]kubecost.Window)
+
+	for thisPodKey, thisPod := range podMap {
+		if pvcs, ok := podPVCMap[thisPodKey]; ok {
+			for _, thisPVC := range pvcs {
+
+				// Determine the (start, end) of the relationship between the
+				// given pvc and the associated Allocation so that a precise
+				// number of hours can be used to compute cumulative cost.
+				s, e := thisPod.Start, thisPod.End
+				if thisPVC.Start.After(thisPod.Start) {
+					s = thisPVC.Start
+				}
+				if thisPVC.End.Before(thisPod.End) {
+					e = thisPVC.End
+				}
+
+				thisPVCKey := thisPVC.key()
+				if pvcPodWindowMap[thisPVCKey] == nil {
+					pvcPodWindowMap[thisPVCKey] = make(map[podKey]kubecost.Window)
+				}
+
+				pvcPodWindowMap[thisPVCKey][thisPodKey] = kubecost.NewWindow(&s, &e)
+			}
+		}
+	}
+
+	for thisPVCKey, podWindowMap := range pvcPodWindowMap {
+		// Build out a pv price coefficient for each pod with a pvc. Each
+		// pvc-pod relation needs a coefficient which modifies the pv cost
+		// such that pv costs can be shared between all pods using that pvc.
+
+		// Get single-point intervals from alloc-pvc relation windows.
+		intervals := getIntervalPointsFromWindows(podWindowMap)
+
+		pvc, ok := pvcMap[thisPVCKey]
+		if !ok {
+			log.DedupedWarningf(5, "Missing pvc with key %s", thisPVCKey)
+			continue
+		}
+
+		// Determine coefficients for each pvc-pod relation.
+		sharedPVCCostCoefficients := getPVCCostCoefficients(intervals, pvc)
+
+		// Distribute pvc costs to Allocations
+		for thisPodKey, coeffComponents := range sharedPVCCostCoefficients {
+			pod, ok2 := podMap[thisPodKey]
+			// If pod does not exist or the pod does not have any allocations
+			// get unmounted pod for cluster
+			if !ok2 || len(pod.Allocations) == 0 {
+				// Get namespace unmounted pod, as pvc will have a namespace
+				pod = getUnmountedPodForNamespace(window, podMap, pvc.Cluster, pvc.Namespace)
+			}
+			for _, alloc := range pod.Allocations {
+				s, e := pod.Start, pod.End
+
+				minutes := e.Sub(s).Minutes()
+				hrs := minutes / 60.0
+
+				gib := pvc.Bytes / 1024 / 1024 / 1024
+				cost := pvc.Volume.CostPerGiBHour * gib * hrs
+				byteHours := pvc.Bytes * hrs
+				coef := getCoefficientFromComponents(coeffComponents)
+
+				// Apply the size and cost of the pv to the allocation, each
+				// weighted by count (i.e. the number of containers in the pod)
+				// record the amount of total PVBytes Hours attributable to a given pv
+				if alloc.PVs == nil {
+					alloc.PVs = kubecost.PVAllocations{}
+				}
+				pvKey := kubecost.PVKey{
+					Cluster: pvc.Volume.Cluster,
+					Name:    pvc.Volume.Name,
+				}
+				// Both Cost and byteHours should be multiplied by the coef and divided by count
+				// so that you if all allocations with a given pv key are summed the result of those
+				// would be equal to the values of the original pv
+				count := float64(len(pod.Allocations))
+				alloc.PVs[pvKey] = &kubecost.PVAllocation{
+					ByteHours: byteHours * coef / count,
+					Cost:      cost * coef / count,
+				}
+			}
+		}
+	}
+}
+
+func applyUnmountedPVs(window kubecost.Window, podMap map[podKey]*pod, pvMap map[pvKey]*pv, pvcMap map[pvcKey]*pvc) {
+	for _, pv := range pvMap {
+		mounted := false
+		for _, pvc := range pvcMap {
+			if pvc.Volume == nil {
+				continue
+			}
+			if pvc.Volume == pv {
+				mounted = true
+				break
+			}
+		}
+
+		if !mounted {
+
+			// a pv without a pvc will not have a namespace, so get the cluster unmounted pod
+			pod := getUnmountedPodForCluster(window, podMap, pv.Cluster)
+
+			// Calculate pv Cost
+
+			// Unmounted pv should have correct keyso it can still reconcile
+			thisPVKey := kubecost.PVKey{
+				Cluster: pv.Cluster,
+				Name:    pv.Name,
+			}
+			gib := pv.Bytes / 1024 / 1024 / 1024
+			hrs := pv.minutes() / 60.0
+			cost := pv.CostPerGiBHour * gib * hrs
+			unmountedPVs := kubecost.PVAllocations{
+				thisPVKey: {
+					ByteHours: pv.Bytes * hrs,
+					Cost:      cost,
+				},
+			}
+			pod.Allocations[kubecost.UnmountedSuffix].PVs = pod.Allocations[kubecost.UnmountedSuffix].PVs.Add(unmountedPVs)
+		}
+	}
+}
+
+func applyUnmountedPVCs(window kubecost.Window, podMap map[podKey]*pod, pvcMap map[pvcKey]*pvc) {
+	for _, pvc := range pvcMap {
+		if !pvc.Mounted && pvc.Volume != nil {
+
+			// Get namespace unmounted pod, as pvc will have a namespace
+			pod := getUnmountedPodForNamespace(window, podMap, pvc.Cluster, pvc.Namespace)
+
+			// Calculate pv Cost
+
+			// Unmounted pv should have correct key so it can still reconcile
+			thisPVKey := kubecost.PVKey{
+				Cluster: pvc.Volume.Cluster,
+				Name:    pvc.Volume.Name,
+			}
+
+			// Use the Volume Bytes here because pvc bytes could be different,
+			// however the pv bytes are what are going to determine cost
+			gib := pvc.Volume.Bytes / 1024 / 1024 / 1024
+			hrs := pvc.Volume.minutes() / 60.0
+			cost := pvc.Volume.CostPerGiBHour * gib * hrs
+			unmountedPVs := kubecost.PVAllocations{
+				thisPVKey: {
+					ByteHours: pvc.Volume.Bytes * hrs,
+					Cost:      cost,
+				},
+			}
+			pod.Allocations[kubecost.UnmountedSuffix].PVs = pod.Allocations[kubecost.UnmountedSuffix].PVs.Add(unmountedPVs)
+		}
+	}
+}
+
+/* Helper Helpers */
+
+// getUnmountedPodForCluster retrieve the unmounted pod for a cluster and create it if it does not exist
+func getUnmountedPodForCluster(window kubecost.Window, podMap map[podKey]*pod, cluster string) *pod {
+	container := kubecost.UnmountedSuffix
+	podName := kubecost.UnmountedSuffix
+	namespace := kubecost.UnmountedSuffix
+	node := ""
+
+	thisPodKey := getUnmountedPodKey(cluster)
+	// Initialize pod and container if they do not already exist
+	thisPod, ok := podMap[thisPodKey]
+	if !ok {
+		thisPod = &pod{
+			Window:      window.Clone(),
+			Start:       *window.Start(),
+			End:         *window.End(),
+			Key:         thisPodKey,
+			Allocations: map[string]*kubecost.Allocation{},
+		}
+
+		thisPod.appendContainer(container)
+		thisPod.Allocations[container].Properties.Cluster = cluster
+		thisPod.Allocations[container].Properties.Node = node
+		thisPod.Allocations[container].Properties.Namespace = namespace
+		thisPod.Allocations[container].Properties.Pod = podName
+		thisPod.Allocations[container].Properties.Container = container
+
+		podMap[thisPodKey] = thisPod
+	}
+	return thisPod
+}
+
+// getUnmountedPodForNamespace is as getUnmountedPodForCluster, but keys allocation property pod/namespace field off namespace
+// This creates or adds allocations to an unmounted pod in the specified namespace, rather than in __unmounted__
+func getUnmountedPodForNamespace(window kubecost.Window, podMap map[podKey]*pod, cluster string, namespace string) *pod {
+	container := kubecost.UnmountedSuffix
+	podName := fmt.Sprintf("%s-unmounted-pvcs", namespace)
+	node := ""
+
+	thisPodKey := newPodKey(cluster, namespace, podName)
+	// Initialize pod and container if they do not already exist
+	thisPod, ok := podMap[thisPodKey]
+	if !ok {
+		thisPod = &pod{
+			Window:      window.Clone(),
+			Start:       *window.Start(),
+			End:         *window.End(),
+			Key:         thisPodKey,
+			Allocations: map[string]*kubecost.Allocation{},
+		}
+
+		thisPod.appendContainer(container)
+		thisPod.Allocations[container].Properties.Cluster = cluster
+		thisPod.Allocations[container].Properties.Node = node
+		thisPod.Allocations[container].Properties.Namespace = namespace
+		thisPod.Allocations[container].Properties.Pod = podName
+		thisPod.Allocations[container].Properties.Container = container
+
+		podMap[thisPodKey] = thisPod
+	}
+	return thisPod
+}
+
+func calculateStartAndEnd(result *prom.QueryResult, resolution time.Duration) (time.Time, time.Time) {
+	s := time.Unix(int64(result.Values[0].Timestamp), 0).UTC()
+	// subtract resolution from start time to cover full time period
+	s = s.Add(-resolution)
+	e := time.Unix(int64(result.Values[len(result.Values)-1].Timestamp), 0).UTC()
+	return s, e
+}
+
+// calculateStartEndFromIsRunning Calculates the start and end of a prom result when the values of the datum are 0 for not running and 1 for running
+// the coeffs are used to adjust the start and end when the value is not equal to 1 or 0, which means that pod came up or went down in that window.
+func calculateStartEndFromIsRunning(result *prom.QueryResult, resolution time.Duration, window kubecost.Window) (time.Time, time.Time) {
+	// start and end are the timestamps of the first and last
+	// minutes the pod was running, respectively. We subtract one resolution
+	// from start because this point will actually represent the end
+	// of the first minute. We don't subtract from end because it
+	// already represents the end of the last minute.
+	var start, end time.Time
+	startAdjustmentCoeff, endAdjustmentCoeff := 1.0, 1.0
+	for _, datum := range result.Values {
+		t := time.Unix(int64(datum.Timestamp), 0)
+
+		if start.IsZero() && datum.Value > 0 && window.Contains(t) {
+			// Set the start timestamp to the earliest non-zero timestamp
+			start = t
+
+			// Record adjustment coefficient, i.e. the portion of the start
+			// timestamp to "ignore". That is, sometimes the value will be
+			// 0.5, meaning that we should discount the time running by
+			// half of the resolution the timestamp stands for.
+			startAdjustmentCoeff = (1.0 - datum.Value)
+		}
+
+		if datum.Value > 0 && window.Contains(t) {
+			// Set the end timestamp to the latest non-zero timestamp
+			end = t
+
+			// Record adjustment coefficient, i.e. the portion of the end
+			// timestamp to "ignore". (See explanation above for start.)
+			endAdjustmentCoeff = (1.0 - datum.Value)
+		}
+	}
+
+	// Do not attempt to adjust start if it is zero
+	if !start.IsZero() {
+		// Adjust timestamps according to the resolution and the adjustment
+		// coefficients, as described above. That is, count the start timestamp
+		// from the beginning of the resolution, not the end. Then "reduce" the
+		// start and end by the correct amount, in the case that the "running"
+		// value of the first or last timestamp was not a full 1.0.
+		start = start.Add(-resolution)
+		// Note: the *100 and /100 are necessary because Duration is an int, so
+		// 0.5, for instance, will be truncated, resulting in no adjustment.
+		start = start.Add(time.Duration(startAdjustmentCoeff*100) * resolution / time.Duration(100))
+		end = end.Add(-time.Duration(endAdjustmentCoeff*100) * resolution / time.Duration(100))
+
+		// Ensure that the start is always within the window, adjusting
+		// for the occasions where start falls 1m before the query window.
+		// NOTE: window here will always be closed (so no need to nil check
+		// "start").
+		// TODO:CLEANUP revisit query methodology to figure out why this is
+		// happening on occasion
+		if start.Before(*window.Start()) {
+			start = *window.Start()
+		}
+	}
+
+	// do not attempt to adjust end if it is zero
+	if !end.IsZero() {
+		// If there is only one point with a value <= 0.5 that the start and
+		// end timestamps both share, then we will enter this case because at
+		// least half of a resolution will be subtracted from both the start
+		// and the end. If that is the case, then add back half of each side
+		// so that the pod is said to run for half a resolution total.
+		// e.g. For resolution 1m and a value of 0.5 at one timestamp, we'll
+		//      end up with end == start and each coeff == 0.5. In
+		//      that case, add 0.25m to each side, resulting in 0.5m duration.
+		if !end.After(start) {
+			start = start.Add(-time.Duration(50*startAdjustmentCoeff) * resolution / time.Duration(100))
+			end = end.Add(time.Duration(50*endAdjustmentCoeff) * resolution / time.Duration(100))
+		}
+
+		// Ensure that the allocEnf is always within the window, adjusting
+		// for the occasions where end falls 1m after the query window. This
+		// has not ever happened, but is symmetrical with the start check
+		// above.
+		// NOTE: window here will always be closed (so no need to nil check
+		// "end").
+		// TODO:CLEANUP revisit query methodology to figure out why this is
+		// happening on occasion
+		if end.After(*window.End()) {
+			end = *window.End()
+		}
+	}
+	return start, end
+}

pkg/costmodel/allocation_helpers_test.go

@@ -0,0 +1,510 @@
+package costmodel
+
+import (
+	"fmt"
+	"github.com/opencost/opencost/pkg/kubecost"
+	"github.com/opencost/opencost/pkg/prom"
+	"github.com/opencost/opencost/pkg/util"
+	"testing"
+	"time"
+)
+
+const Ki = 1024
+const Mi = Ki * 1024
+const Gi = Mi * 1024
+
+const minute = 60.0
+const hour = minute * 60.0
+
+var windowStart = time.Date(2020, 6, 16, 0, 0, 0, 0, time.UTC)
+var windowEnd = time.Date(2020, 6, 17, 0, 0, 0, 0, time.UTC)
+var window = kubecost.NewWindow(&windowStart, &windowEnd)
+
+var startFloat = float64(windowStart.Unix())
+
+var podKey1 = podKey{
+	namespaceKey: namespaceKey{
+		Cluster:   "cluster1",
+		Namespace: "namespace1",
+	},
+	Pod: "pod1",
+}
+var podKey2 = podKey{
+	namespaceKey: namespaceKey{
+		Cluster:   "cluster1",
+		Namespace: "namespace1",
+	},
+	Pod: "pod2",
+}
+var podKey3 = podKey{
+	namespaceKey: namespaceKey{
+		Cluster:   "cluster2",
+		Namespace: "namespace2",
+	},
+	Pod: "pod3",
+}
+
+var podKey4 = podKey{
+	namespaceKey: namespaceKey{
+		Cluster:   "cluster2",
+		Namespace: "namespace2",
+	},
+	Pod: "pod4",
+}
+
+var podKeyUnmounted = podKey{
+	namespaceKey: namespaceKey{
+		Cluster:   "cluster2",
+		Namespace: kubecost.UnmountedSuffix,
+	},
+	Pod: kubecost.UnmountedSuffix,
+}
+
+var kcPVKey1 = kubecost.PVKey{
+	Cluster: "cluster1",
+	Name:    "pv1",
+}
+
+var kcPVKey2 = kubecost.PVKey{
+	Cluster: "cluster1",
+	Name:    "pv2",
+}
+
+var kcPVKey3 = kubecost.PVKey{
+	Cluster: "cluster2",
+	Name:    "pv3",
+}
+
+var kcPVKey4 = kubecost.PVKey{
+	Cluster: "cluster2",
+	Name:    "pv4",
+}
+
+var podMap1 = map[podKey]*pod{
+	podKey1: {
+		Window:      window.Clone(),
+		Start:       time.Date(2020, 6, 16, 0, 0, 0, 0, time.UTC),
+		End:         time.Date(2020, 6, 17, 0, 0, 0, 0, time.UTC),
+		Key:         podKey1,
+		Allocations: nil,
+	},
+	podKey2: {
+		Window:      window.Clone(),
+		Start:       time.Date(2020, 6, 16, 12, 0, 0, 0, time.UTC),
+		End:         time.Date(2020, 6, 17, 0, 0, 0, 0, time.UTC),
+		Key:         podKey2,
+		Allocations: nil,
+	},
+	podKey3: {
+		Window:      window.Clone(),
+		Start:       time.Date(2020, 6, 16, 6, 30, 0, 0, time.UTC),
+		End:         time.Date(2020, 6, 17, 18, 12, 33, 0, time.UTC),
+		Key:         podKey3,
+		Allocations: nil,
+	},
+	podKey4: {
+		Window:      window.Clone(),
+		Start:       time.Date(2020, 6, 16, 0, 0, 0, 0, time.UTC),
+		End:         time.Date(2020, 6, 17, 13, 0, 0, 0, time.UTC),
+		Key:         podKey4,
+		Allocations: nil,
+	},
+	podKeyUnmounted: {
+		Window: window.Clone(),
+		Start:  *window.Start(),
+		End:    *window.End(),
+		Key:    podKeyUnmounted,
+		Allocations: map[string]*kubecost.Allocation{
+			kubecost.UnmountedSuffix: {
+				Name: fmt.Sprintf("%s/%s/%s/%s", podKeyUnmounted.Cluster, podKeyUnmounted.Namespace, podKeyUnmounted.Pod, kubecost.UnmountedSuffix),
+				Properties: &kubecost.AllocationProperties{
+					Cluster:   podKeyUnmounted.Cluster,
+					Node:      "",
+					Container: kubecost.UnmountedSuffix,
+					Namespace: podKeyUnmounted.Namespace,
+					Pod:       podKeyUnmounted.Pod,
+					Services:  []string{"LB1"},
+				},
+				Window:                     window,
+				Start:                      *window.Start(),
+				End:                        *window.End(),
+				LoadBalancerCost:           0.60,
+				LoadBalancerCostAdjustment: 0,
+				PVs: kubecost.PVAllocations{
+					kcPVKey2: &kubecost.PVAllocation{
+						ByteHours: 24 * Gi,
+						Cost:      2.25,
+					},
+				},
+			},
+		},
+	},
+}
+
+var pvKey1 = pvKey{
+	Cluster:          "cluster1",
+	PersistentVolume: "pv1",
+}
+
+var pvKey2 = pvKey{
+	Cluster:          "cluster1",
+	PersistentVolume: "pv2",
+}
+
+var pvKey3 = pvKey{
+	Cluster:          "cluster2",
+	PersistentVolume: "pv3",
+}
+
+var pvKey4 = pvKey{
+	Cluster:          "cluster2",
+	PersistentVolume: "pv4",
+}
+
+var pvMap1 = map[pvKey]*pv{
+	pvKey1: {
+		Start:          windowStart,
+		End:            windowEnd.Add(time.Hour * -6),
+		Bytes:          20 * Gi,
+		CostPerGiBHour: 0.05,
+		Cluster:        "cluster1",
+		Name:           "pv1",
+		StorageClass:   "class1",
+	},
+	pvKey2: {
+		Start:          windowStart,
+		End:            windowEnd,
+		Bytes:          100 * Gi,
+		CostPerGiBHour: 0.05,
+		Cluster:        "cluster1",
+		Name:           "pv2",
+		StorageClass:   "class1",
+	},
+	pvKey3: {
+		Start:          windowStart.Add(time.Hour * 6),
+		End:            windowEnd.Add(time.Hour * -6),
+		Bytes:          50 * Gi,
+		CostPerGiBHour: 0.03,
+		Cluster:        "cluster2",
+		Name:           "pv3",
+		StorageClass:   "class2",
+	},
+	pvKey4: {
+		Start:          windowStart,
+		End:            windowEnd.Add(time.Hour * -6),
+		Bytes:          30 * Gi,
+		CostPerGiBHour: 0.05,
+		Cluster:        "cluster2",
+		Name:           "pv4",
+		StorageClass:   "class1",
+	},
+}
+
+/* pv/pvc Helpers */
+func TestBuildPVMap(t *testing.T) {
+	pvMap1NoBytes := make(map[pvKey]*pv, len(pvMap1))
+	for thisPVKey, thisPV := range pvMap1 {
+		clonePV := thisPV.clone()
+		clonePV.Bytes = 0.0
+		clonePV.StorageClass = ""
+		pvMap1NoBytes[thisPVKey] = clonePV
+	}
+
+	testCases := map[string]struct {
+		resolution              time.Duration
+		resultsPVCostPerGiBHour []*prom.QueryResult
+		resultsActiveMinutes    []*prom.QueryResult
+		expected                map[pvKey]*pv
+	}{
+		"pvMap1": {
+			resolution: time.Hour * 6,
+			resultsPVCostPerGiBHour: []*prom.QueryResult{
+				{
+					Metric: map[string]interface{}{
+						"cluster_id": "cluster1",
+						"volumename": "pv1",
+					},
+					Values: []*util.Vector{
+						{
+							Value: 0.05,
+						},
+					},
+				},
+				{
+					Metric: map[string]interface{}{
+						"cluster_id": "cluster1",
+						"volumename": "pv2",
+					},
+					Values: []*util.Vector{
+						{
+							Value: 0.05,
+						},
+					},
+				},
+				{
+					Metric: map[string]interface{}{
+						"cluster_id": "cluster2",
+						"volumename": "pv3",
+					},
+					Values: []*util.Vector{
+						{
+							Value: 0.03,
+						},
+					},
+				},
+				{
+					Metric: map[string]interface{}{
+						"cluster_id": "cluster2",
+						"volumename": "pv4",
+					},
+					Values: []*util.Vector{
+						{
+							Value: 0.05,
+						},
+					},
+				},
+			},
+			resultsActiveMinutes: []*prom.QueryResult{
+				{
+					Metric: map[string]interface{}{
+						"cluster_id":       "cluster1",
+						"persistentvolume": "pv1",
+					},
+					Values: []*util.Vector{
+						{
+							Timestamp: startFloat + (hour * 6),
+						},
+						{
+							Timestamp: startFloat + (hour * 12),
+						},
+						{
+							Timestamp: startFloat + (hour * 18),
+						},
+					},
+				},
+				{
+					Metric: map[string]interface{}{
+						"cluster_id":       "cluster1",
+						"persistentvolume": "pv2",
+					},
+					Values: []*util.Vector{
+						{
+							Timestamp: startFloat + (hour * 6),
+						},
+						{
+							Timestamp: startFloat + (hour * 12),
+						},
+						{
+							Timestamp: startFloat + (hour * 18),
+						},
+						{
+							Timestamp: startFloat + (hour * 24),
+						},
+					},
+				},
+				{
+					Metric: map[string]interface{}{
+						"cluster_id":       "cluster2",
+						"persistentvolume": "pv3",
+					},
+					Values: []*util.Vector{
+						{
+							Timestamp: startFloat + (hour * 12),
+						},
+						{
+							Timestamp: startFloat + (hour * 18),
+						},
+					},
+				},
+				{
+					Metric: map[string]interface{}{
+						"cluster_id":       "cluster2",
+						"persistentvolume": "pv4",
+					},
+					Values: []*util.Vector{
+						{
+							Timestamp: startFloat + (hour * 6),
+						},
+						{
+							Timestamp: startFloat + (hour * 12),
+						},
+						{
+							Timestamp: startFloat + (hour * 18),
+						},
+					},
+				},
+			},
+			expected: pvMap1NoBytes,
+		},
+	}
+
+	for name, testCase := range testCases {
+		t.Run(name, func(t *testing.T) {
+			pvMap := make(map[pvKey]*pv)
+			buildPVMap(testCase.resolution, pvMap, testCase.resultsPVCostPerGiBHour, testCase.resultsActiveMinutes)
+			if len(pvMap) != len(testCase.expected) {
+				t.Errorf("pv map does not have the expected length %d : %d", len(pvMap), len(testCase.expected))
+			}
+
+			for thisPVKey, expectedPV := range testCase.expected {
+				actualPV, ok := pvMap[thisPVKey]
+				if !ok {
+					t.Errorf("pv map is missing key %s", thisPVKey)
+				}
+				if !actualPV.equal(expectedPV) {
+					t.Errorf("pv does not match with key %s", thisPVKey)
+				}
+			}
+		})
+	}
+}
+
+/* Helper Helpers */
+
+func TestGetUnmountedPodForCluster(t *testing.T) {
+	testCases := map[string]struct {
+		window   kubecost.Window
+		podMap   map[podKey]*pod
+		cluster  string
+		expected *pod
+	}{
+		"create new": {
+			window:  window.Clone(),
+			podMap:  podMap1,
+			cluster: "cluster1",
+			expected: &pod{
+				Window: window.Clone(),
+				Start:  *window.Start(),
+				End:    *window.End(),
+				Key:    getUnmountedPodKey("cluster1"),
+				Allocations: map[string]*kubecost.Allocation{
+					kubecost.UnmountedSuffix: {
+						Name: fmt.Sprintf("%s/%s/%s/%s", "cluster1", kubecost.UnmountedSuffix, kubecost.UnmountedSuffix, kubecost.UnmountedSuffix),
+						Properties: &kubecost.AllocationProperties{
+							Cluster:   "cluster1",
+							Node:      "",
+							Container: kubecost.UnmountedSuffix,
+							Namespace: kubecost.UnmountedSuffix,
+							Pod:       kubecost.UnmountedSuffix,
+						},
+						Window: window,
+						Start:  *window.Start(),
+						End:    *window.End(),
+					},
+				},
+			},
+		},
+		"get existing": {
+			window:  window.Clone(),
+			podMap:  podMap1,
+			cluster: "cluster2",
+			expected: &pod{
+				Window: window.Clone(),
+				Start:  *window.Start(),
+				End:    *window.End(),
+				Key:    getUnmountedPodKey("cluster2"),
+				Allocations: map[string]*kubecost.Allocation{
+					kubecost.UnmountedSuffix: {
+						Name: fmt.Sprintf("%s/%s/%s/%s", "cluster2", kubecost.UnmountedSuffix, kubecost.UnmountedSuffix, kubecost.UnmountedSuffix),
+						Properties: &kubecost.AllocationProperties{
+							Cluster:   "cluster2",
+							Node:      "",
+							Container: kubecost.UnmountedSuffix,
+							Namespace: kubecost.UnmountedSuffix,
+							Pod:       kubecost.UnmountedSuffix,
+							Services:  []string{"LB1"},
+						},
+						Window:                     window,
+						Start:                      *window.Start(),
+						End:                        *window.End(),
+						LoadBalancerCost:           .60,
+						LoadBalancerCostAdjustment: 0,
+						PVs: kubecost.PVAllocations{
+							kcPVKey2: &kubecost.PVAllocation{
+								ByteHours: 24 * Gi,
+								Cost:      2.25,
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+
+	for name, testCase := range testCases {
+		t.Run(name, func(t *testing.T) {
+			actual := getUnmountedPodForCluster(testCase.window, testCase.podMap, testCase.cluster)
+			if !actual.equal(testCase.expected) {
+				t.Errorf("Unmounted pod does not match expectation")
+			}
+		})
+	}
+}
+
+func TestCalculateStartAndEnd(t *testing.T) {
+
+	testCases := map[string]struct {
+		resolution    time.Duration
+		expectedStart time.Time
+		expectedEnd   time.Time
+		result        *prom.QueryResult
+	}{
+		"1 hour resolution, 1 hour window": {
+			resolution:    time.Hour,
+			expectedStart: windowStart,
+			expectedEnd:   windowStart.Add(time.Hour),
+			result: &prom.QueryResult{
+				Values: []*util.Vector{
+					{
+						Timestamp: startFloat + (minute * 60),
+					},
+				},
+			},
+		},
+		"30 minute resolution, 1 hour window": {
+			resolution:    time.Minute * 30,
+			expectedStart: windowStart,
+			expectedEnd:   windowStart.Add(time.Hour),
+			result: &prom.QueryResult{
+				Values: []*util.Vector{
+					{
+						Timestamp: startFloat + (minute * 30),
+					},
+					{
+						Timestamp: startFloat + (minute * 60),
+					},
+				},
+			},
+		},
+		"15 minute resolution, 45 minute window": {
+			resolution:    time.Minute * 15,
+			expectedStart: windowStart.Add(time.Minute * -15),
+			expectedEnd:   windowStart.Add(time.Minute * 30),
+			result: &prom.QueryResult{
+				Values: []*util.Vector{
+					{
+						Timestamp: startFloat + (minute * 0),
+					},
+					{
+						Timestamp: startFloat + (minute * 15),
+					},
+					{
+						Timestamp: startFloat + (minute * 30),
+					},
+				},
+			},
+		},
+	}
+
+	for name, testCase := range testCases {
+		t.Run(name, func(t *testing.T) {
+			start, end := calculateStartAndEnd(testCase.result, testCase.resolution)
+			if !start.Equal(testCase.expectedStart) {
+				t.Errorf("start to not match expected %v : %v", start, testCase.expectedStart)
+			}
+			if !end.Equal(testCase.expectedEnd) {
+				t.Errorf("end to not match expected %v : %v", end, testCase.expectedEnd)
+			}
+		})
+	}
+}

pkg/costmodel/allocation_types.go

@@ -0,0 +1,226 @@
+package costmodel
+
+import (
+	"fmt"
+	"github.com/opencost/opencost/pkg/kubecost"
+	"time"
+)
+
+// pod describes a running pod's start and end time within a Window and
+// all the Allocations (i.e. containers) contained within it.
+type pod struct {
+	Window      kubecost.Window
+	Start       time.Time
+	End         time.Time
+	Key         podKey
+	Allocations map[string]*kubecost.Allocation
+}
+
+func (p *pod) equal(that *pod) bool {
+	if p == nil {
+		return that == nil
+	}
+
+	if !p.Window.Equal(that.Window) {
+		return false
+	}
+
+	if !p.Start.Equal(that.Start) {
+		return false
+	}
+
+	if !p.End.Equal(that.End) {
+		return false
+	}
+
+	if p.Key != that.Key {
+		return false
+	}
+
+	if p.Key != that.Key {
+		return false
+	}
+
+	if len(p.Allocations) != len(that.Allocations) {
+		return false
+	}
+
+	for container, thisAlloc := range p.Allocations {
+		thatAlloc, ok := that.Allocations[container]
+		if !ok || !thisAlloc.Equal(thatAlloc) {
+			return false
+		}
+	}
+	return true
+}
+
+// appendContainer adds an entry for the given container name to the pod.
+func (p *pod) appendContainer(container string) {
+	name := fmt.Sprintf("%s/%s/%s/%s", p.Key.Cluster, p.Key.Namespace, p.Key.Pod, container)
+
+	alloc := &kubecost.Allocation{
+		Name:       name,
+		Properties: &kubecost.AllocationProperties{},
+		Window:     p.Window.Clone(),
+		Start:      p.Start,
+		End:        p.End,
+	}
+	alloc.Properties.Container = container
+	alloc.Properties.Pod = p.Key.Pod
+	alloc.Properties.Namespace = p.Key.Namespace
+	alloc.Properties.Cluster = p.Key.Cluster
+
+	p.Allocations[container] = alloc
+}
+
+// pvc describes a PersistentVolumeClaim
+// TODO:CLEANUP move to pkg/kubecost?
+// TODO:CLEANUP add PersistentVolumeClaims field to type Allocation?
+type pvc struct {
+	Bytes     float64   `json:"bytes"`
+	Name      string    `json:"name"`
+	Cluster   string    `json:"cluster"`
+	Namespace string    `json:"namespace"`
+	Volume    *pv       `json:"persistentVolume"`
+	Mounted   bool      `json:"mounted"`
+	Start     time.Time `json:"start"`
+	End       time.Time `json:"end"`
+}
+
+// Cost computes the cumulative cost of the pvc
+func (p *pvc) Cost() float64 {
+	if p == nil || p.Volume == nil {
+		return 0.0
+	}
+
+	gib := p.Bytes / 1024 / 1024 / 1024
+	hrs := p.minutes() / 60.0
+
+	return p.Volume.CostPerGiBHour * gib * hrs
+}
+
+// Minutes computes the number of minutes over which the pvc is defined
+func (p *pvc) minutes() float64 {
+	if p == nil {
+		return 0.0
+	}
+
+	return p.End.Sub(p.Start).Minutes()
+}
+
+// String returns a string representation of the pvc
+func (p *pvc) String() string {
+	if p == nil {
+		return "<nil>"
+	}
+	return fmt.Sprintf("%s/%s/%s{Bytes:%.2f, Cost:%.6f, Start,End:%s}", p.Cluster, p.Namespace, p.Name, p.Bytes, p.Cost(), kubecost.NewWindow(&p.Start, &p.End))
+}
+
+// Key returns the pvcKey for the calling pvc
+func (p *pvc) key() pvcKey {
+	return newPVCKey(p.Cluster, p.Namespace, p.Name)
+}
+
+// pv describes a PersistentVolume
+type pv struct {
+	Start          time.Time `json:"start"`
+	End            time.Time `json:"end"`
+	Bytes          float64   `json:"bytes"`
+	CostPerGiBHour float64   `json:"costPerGiBHour"`
+	Cluster        string    `json:"cluster"`
+	Name           string    `json:"name"`
+	StorageClass   string    `json:"storageClass"`
+}
+
+func (p *pv) clone() *pv {
+	if p == nil {
+		return nil
+	}
+	return &pv{
+		Start:          p.Start,
+		End:            p.End,
+		Bytes:          p.Bytes,
+		CostPerGiBHour: p.CostPerGiBHour,
+		Cluster:        p.Cluster,
+		Name:           p.Name,
+		StorageClass:   p.StorageClass,
+	}
+}
+
+func (p *pv) equal(that *pv) bool {
+	if p == nil {
+		return that == nil
+	}
+
+	if !p.Start.Equal(that.Start) {
+		return false
+	}
+
+	if !p.End.Equal(that.End) {
+		return false
+	}
+
+	if p.Bytes != that.Bytes {
+		return false
+	}
+
+	if p.CostPerGiBHour != that.CostPerGiBHour {
+		return false
+	}
+
+	if p.Cluster != that.Cluster {
+		return false
+	}
+
+	if p.Name != that.Name {
+		return false
+	}
+
+	if p.StorageClass != that.StorageClass {
+		return false
+	}
+
+	return true
+}
+
+// String returns a string representation of the pv
+func (p *pv) String() string {
+	if p == nil {
+		return "<nil>"
+	}
+	return fmt.Sprintf("%s/%s{Bytes:%.2f, Cost/GiB*Hr:%.6f, StorageClass:%s}", p.Cluster, p.Name, p.Bytes, p.CostPerGiBHour, p.StorageClass)
+}
+
+func (p *pv) minutes() float64 {
+	if p == nil {
+		return 0.0
+	}
+
+	return p.End.Sub(p.Start).Minutes()
+}
+
+// key returns the pvKey for the calling pvc
+func (p *pv) key() pvKey {
+	return newPVKey(p.Cluster, p.Name)
+}
+
+// lbCost describes the start and end time of a Load Balancer along with cost
+type lbCost struct {
+	TotalCost float64
+	Start     time.Time
+	End       time.Time
+}
+
+// NodePricing describes the resource costs associated with a given node, as
+// well as the source of the information (e.g. prometheus, custom)
+type nodePricing struct {
+	Name            string
+	NodeType        string
+	ProviderID      string
+	Preemptible     bool
+	CostPerCPUHr    float64
+	CostPerRAMGiBHr float64
+	CostPerGPUHr    float64
+	Discount        float64
+	Source          string
+}

pkg/costmodel/cluster.go

@@ -7,6 +7,7 @@ import (
 
 	"github.com/opencost/opencost/pkg/kubecost"
 	"github.com/opencost/opencost/pkg/util/timeutil"
+	"golang.org/x/exp/slices"
 
 	"github.com/opencost/opencost/pkg/cloud"
 	"github.com/opencost/opencost/pkg/env"
@@ -106,16 +107,33 @@ func NewClusterCostsFromCumulative(cpu, gpu, ram, storage float64, window, offse
 }
 
 type Disk struct {
-	Cluster    string
-	Name       string
-	ProviderID string
-	Cost       float64
-	Bytes      float64
-	Local      bool
-	Start      time.Time
-	End        time.Time
-	Minutes    float64
-	Breakdown  *ClusterCostsBreakdown
+	Cluster        string
+	Name           string
+	ProviderID     string
+	StorageClass   string
+	VolumeName     string
+	ClaimName      string
+	ClaimNamespace string
+	Cost           float64
+	Bytes          float64
+
+	// These two fields may not be available at all times because they rely on
+	// a new set of metrics that may or may not be available. Thus, they must
+	// be nilable to represent the complete absence of the data.
+	//
+	// In other words, nilability here lets us distinguish between
+	// "metric is not available" and "metric is available but is 0".
+	//
+	// They end in "Ptr" to distinguish from an earlier version in order to
+	// ensure that all usages are checked for nil.
+	BytesUsedAvgPtr *float64
+	BytesUsedMaxPtr *float64
+
+	Local     bool
+	Start     time.Time
+	End       time.Time
+	Minutes   float64
+	Breakdown *ClusterCostsBreakdown
 }
 
 type DiskIdentifier struct {
@@ -155,35 +173,89 @@ func ClusterDisks(client prometheus.Client, provider cloud.Provider, start, end
 	ctx := prom.NewNamedContext(client, prom.ClusterContextName)
 	queryPVCost := fmt.Sprintf(`avg(avg_over_time(pv_hourly_cost[%s])) by (%s, persistentvolume,provider_id)`, durStr, env.GetPromClusterLabel())
 	queryPVSize := fmt.Sprintf(`avg(avg_over_time(kube_persistentvolume_capacity_bytes[%s])) by (%s, persistentvolume)`, durStr, env.GetPromClusterLabel())
-	queryActiveMins := fmt.Sprintf(`count(pv_hourly_cost) by (%s, persistentvolume)[%s:%dm]`, env.GetPromClusterLabel(), durStr, minsPerResolution)
-
+	queryActiveMins := fmt.Sprintf(`avg(kube_persistentvolume_capacity_bytes) by (%s, persistentvolume)[%s:%dm]`, env.GetPromClusterLabel(), durStr, minsPerResolution)
+	queryPVStorageClass := fmt.Sprintf(`avg(avg_over_time(kubecost_pv_info[%s])) by (%s, persistentvolume, storageclass)`, durStr, env.GetPromClusterLabel())
+	queryPVUsedAvg := fmt.Sprintf(`avg(avg_over_time(kubelet_volume_stats_used_bytes[%s])) by (%s, persistentvolumeclaim, namespace)`, durStr, env.GetPromClusterLabel())
+	queryPVUsedMax := fmt.Sprintf(`max(max_over_time(kubelet_volume_stats_used_bytes[%s])) by (%s, persistentvolumeclaim, namespace)`, durStr, env.GetPromClusterLabel())
+	queryPVCInfo := fmt.Sprintf(`avg(avg_over_time(kube_persistentvolumeclaim_info[%s])) by (%s, volumename, persistentvolumeclaim, namespace)`, durStr, env.GetPromClusterLabel())
 	queryLocalStorageCost := fmt.Sprintf(`sum_over_time(sum(container_fs_limit_bytes{device!="tmpfs", id="/"}) by (instance, %s)[%s:%dm]) / 1024 / 1024 / 1024 * %f * %f`, env.GetPromClusterLabel(), durStr, minsPerResolution, hourlyToCumulative, costPerGBHr)
 	queryLocalStorageUsedCost := fmt.Sprintf(`sum_over_time(sum(container_fs_usage_bytes{device!="tmpfs", id="/"}) by (instance, %s)[%s:%dm]) / 1024 / 1024 / 1024 * %f * %f`, env.GetPromClusterLabel(), durStr, minsPerResolution, hourlyToCumulative, costPerGBHr)
+	queryLocalStorageUsedAvg := fmt.Sprintf(`avg(avg_over_time(container_fs_usage_bytes{device!="tmpfs", id="/"}[%s])) by (instance, %s)`, durStr, env.GetPromClusterLabel())
+	queryLocalStorageUsedMax := fmt.Sprintf(`max(max_over_time(container_fs_usage_bytes{device!="tmpfs", id="/"}[%s])) by (instance, %s)`, durStr, env.GetPromClusterLabel())
 	queryLocalStorageBytes := fmt.Sprintf(`avg_over_time(sum(container_fs_limit_bytes{device!="tmpfs", id="/"}) by (instance, %s)[%s:%dm])`, env.GetPromClusterLabel(), durStr, minsPerResolution)
 	queryLocalActiveMins := fmt.Sprintf(`count(node_total_hourly_cost) by (%s, node)[%s:%dm]`, env.GetPromClusterLabel(), durStr, minsPerResolution)
 
 	resChPVCost := ctx.QueryAtTime(queryPVCost, t)
 	resChPVSize := ctx.QueryAtTime(queryPVSize, t)
 	resChActiveMins := ctx.QueryAtTime(queryActiveMins, t)
+	resChPVStorageClass := ctx.QueryAtTime(queryPVStorageClass, t)
+	resChPVUsedAvg := ctx.QueryAtTime(queryPVUsedAvg, t)
+	resChPVUsedMax := ctx.QueryAtTime(queryPVUsedMax, t)
+	resChPVCInfo := ctx.QueryAtTime(queryPVCInfo, t)
 	resChLocalStorageCost := ctx.QueryAtTime(queryLocalStorageCost, t)
 	resChLocalStorageUsedCost := ctx.QueryAtTime(queryLocalStorageUsedCost, t)
+	resChLocalStoreageUsedAvg := ctx.QueryAtTime(queryLocalStorageUsedAvg, t)
+	resChLocalStoreageUsedMax := ctx.QueryAtTime(queryLocalStorageUsedMax, t)
 	resChLocalStorageBytes := ctx.QueryAtTime(queryLocalStorageBytes, t)
 	resChLocalActiveMins := ctx.QueryAtTime(queryLocalActiveMins, t)
 
 	resPVCost, _ := resChPVCost.Await()
 	resPVSize, _ := resChPVSize.Await()
 	resActiveMins, _ := resChActiveMins.Await()
+	resPVStorageClass, _ := resChPVStorageClass.Await()
+	resPVUsedAvg, _ := resChPVUsedAvg.Await()
+	resPVUsedMax, _ := resChPVUsedMax.Await()
+	resPVCInfo, _ := resChPVCInfo.Await()
 	resLocalStorageCost, _ := resChLocalStorageCost.Await()
 	resLocalStorageUsedCost, _ := resChLocalStorageUsedCost.Await()
+	resLocalStorageUsedAvg, _ := resChLocalStoreageUsedAvg.Await()
+	resLocalStorageUsedMax, _ := resChLocalStoreageUsedMax.Await()
 	resLocalStorageBytes, _ := resChLocalStorageBytes.Await()
 	resLocalActiveMins, _ := resChLocalActiveMins.Await()
+
 	if ctx.HasErrors() {
 		return nil, ctx.ErrorCollection()
 	}
 
 	diskMap := map[DiskIdentifier]*Disk{}
 
-	pvCosts(diskMap, resolution, resActiveMins, resPVSize, resPVCost, provider)
+	for _, result := range resPVCInfo {
+		cluster, err := result.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		volumeName, err := result.GetString("volumename")
+		if err != nil {
+			log.Debugf("ClusterDisks: pv claim data missing volumename")
+			continue
+		}
+		claimName, err := result.GetString("persistentvolumeclaim")
+		if err != nil {
+			log.Debugf("ClusterDisks: pv claim data missing persistentvolumeclaim")
+			continue
+		}
+		claimNamespace, err := result.GetString("namespace")
+		if err != nil {
+			log.Debugf("ClusterDisks: pv claim data missing namespace")
+			continue
+		}
+
+		key := DiskIdentifier{cluster, volumeName}
+		if _, ok := diskMap[key]; !ok {
+			diskMap[key] = &Disk{
+				Cluster:   cluster,
+				Name:      volumeName,
+				Breakdown: &ClusterCostsBreakdown{},
+			}
+		}
+
+		diskMap[key].VolumeName = volumeName
+		diskMap[key].ClaimName = claimName
+		diskMap[key].ClaimNamespace = claimNamespace
+	}
+
+	pvCosts(diskMap, resolution, resActiveMins, resPVSize, resPVCost, resPVUsedAvg, resPVUsedMax, resPVCInfo, provider)
 
 	for _, result := range resLocalStorageCost {
 		cluster, err := result.GetString(env.GetPromClusterLabel())
@@ -208,6 +280,9 @@ func ClusterDisks(client prometheus.Client, provider cloud.Provider, start, end
 			}
 		}
 		diskMap[key].Cost += cost
+
+		//Assigning explicitly the storage class of local storage to local
+		diskMap[key].StorageClass = kubecost.LocalStorageClass
 	}
 
 	for _, result := range resLocalStorageUsedCost {
@@ -235,6 +310,56 @@ func ClusterDisks(client prometheus.Client, provider cloud.Provider, start, end
 		diskMap[key].Breakdown.System = cost / diskMap[key].Cost
 	}
 
+	for _, result := range resLocalStorageUsedAvg {
+		cluster, err := result.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("instance")
+		if err != nil {
+			log.Warnf("ClusterDisks: local storage data missing instance")
+			continue
+		}
+
+		bytesAvg := result.Values[0].Value
+		key := DiskIdentifier{cluster, name}
+		if _, ok := diskMap[key]; !ok {
+			diskMap[key] = &Disk{
+				Cluster:   cluster,
+				Name:      name,
+				Breakdown: &ClusterCostsBreakdown{},
+				Local:     true,
+			}
+		}
+		diskMap[key].BytesUsedAvgPtr = &bytesAvg
+	}
+
+	for _, result := range resLocalStorageUsedMax {
+		cluster, err := result.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, err := result.GetString("instance")
+		if err != nil {
+			log.Warnf("ClusterDisks: local storage data missing instance")
+			continue
+		}
+
+		bytesMax := result.Values[0].Value
+		key := DiskIdentifier{cluster, name}
+		if _, ok := diskMap[key]; !ok {
+			diskMap[key] = &Disk{
+				Cluster:   cluster,
+				Name:      name,
+				Breakdown: &ClusterCostsBreakdown{},
+				Local:     true,
+			}
+		}
+		diskMap[key].BytesUsedMaxPtr = &bytesMax
+	}
+
 	for _, result := range resLocalStorageBytes {
 		cluster, err := result.GetString(env.GetPromClusterLabel())
 		if err != nil {
@@ -297,6 +422,43 @@ func ClusterDisks(client prometheus.Client, provider cloud.Provider, start, end
 		diskMap[key].Minutes = mins
 	}
 
+	var unTracedDiskLogData []DiskIdentifier
+	//Iterating through Persistent Volume given by custom metrics kubecost_pv_info and assign the storage class if known and __unknown__ if not populated.
+	for _, result := range resPVStorageClass {
+		cluster, err := result.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		name, _ := result.GetString("persistentvolume")
+
+		key := DiskIdentifier{cluster, name}
+		if _, ok := diskMap[key]; !ok {
+			if !slices.Contains(unTracedDiskLogData, key) {
+				unTracedDiskLogData = append(unTracedDiskLogData, key)
+			}
+			continue
+		}
+
+		if len(result.Values) == 0 {
+			continue
+		}
+
+		storageClass, err := result.GetString("storageclass")
+
+		if err != nil {
+			diskMap[key].StorageClass = kubecost.UnknownStorageClass
+		} else {
+			diskMap[key].StorageClass = storageClass
+		}
+	}
+
+	// Logging the unidentified disk information outside the loop
+
+	for _, unIdentifiedDisk := range unTracedDiskLogData {
+		log.Warnf("ClusterDisks: Cluster %s has Storage Class information for unidentified disk %s or disk deleted from analysis", unIdentifiedDisk.Cluster, unIdentifiedDisk.Name)
+	}
+
 	for _, disk := range diskMap {
 		// Apply all remaining RAM to Idle
 		disk.Breakdown.Idle = 1.0 - (disk.Breakdown.System + disk.Breakdown.Other + disk.Breakdown.User)
@@ -1136,7 +1298,7 @@ func ClusterCostsOverTime(cli prometheus.Client, provider cloud.Provider, startS
 	}, nil
 }
 
-func pvCosts(diskMap map[DiskIdentifier]*Disk, resolution time.Duration, resActiveMins, resPVSize, resPVCost []*prom.QueryResult, cp cloud.Provider) {
+func pvCosts(diskMap map[DiskIdentifier]*Disk, resolution time.Duration, resActiveMins, resPVSize, resPVCost, resPVUsedAvg, resPVUsedMax, resPVCInfo []*prom.QueryResult, cp cloud.Provider) {
 	for _, result := range resActiveMins {
 		cluster, err := result.GetString(env.GetPromClusterLabel())
 		if err != nil {
@@ -1246,4 +1408,126 @@ func pvCosts(diskMap map[DiskIdentifier]*Disk, resolution time.Duration, resActi
 			diskMap[key].ProviderID = cloud.ParsePVID(providerID)
 		}
 	}
+
+	for _, result := range resPVUsedAvg {
+		cluster, err := result.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		claimName, err := result.GetString("persistentvolumeclaim")
+		if err != nil {
+			log.Debugf("ClusterDisks: pv usage data missing persistentvolumeclaim")
+			continue
+		}
+		claimNamespace, err := result.GetString("namespace")
+		if err != nil {
+			log.Debugf("ClusterDisks: pv usage data missing namespace")
+			continue
+		}
+
+		var volumeName string
+
+		for _, thatRes := range resPVCInfo {
+
+			thatCluster, err := thatRes.GetString(env.GetPromClusterLabel())
+			if err != nil {
+				thatCluster = env.GetClusterID()
+			}
+
+			thatVolumeName, err := thatRes.GetString("volumename")
+			if err != nil {
+				log.Debugf("ClusterDisks: pv claim data missing volumename")
+				continue
+			}
+			thatClaimName, err := thatRes.GetString("persistentvolumeclaim")
+			if err != nil {
+				log.Debugf("ClusterDisks: pv claim data missing persistentvolumeclaim")
+				continue
+			}
+			thatClaimNamespace, err := thatRes.GetString("namespace")
+			if err != nil {
+				log.Debugf("ClusterDisks: pv claim data missing namespace")
+				continue
+			}
+
+			if cluster == thatCluster && claimName == thatClaimName && claimNamespace == thatClaimNamespace {
+				volumeName = thatVolumeName
+			}
+		}
+
+		usage := result.Values[0].Value
+
+		key := DiskIdentifier{cluster, volumeName}
+
+		if _, ok := diskMap[key]; !ok {
+			diskMap[key] = &Disk{
+				Cluster:   cluster,
+				Name:      volumeName,
+				Breakdown: &ClusterCostsBreakdown{},
+			}
+		}
+		diskMap[key].BytesUsedAvgPtr = &usage
+	}
+
+	for _, result := range resPVUsedMax {
+		cluster, err := result.GetString(env.GetPromClusterLabel())
+		if err != nil {
+			cluster = env.GetClusterID()
+		}
+
+		claimName, err := result.GetString("persistentvolumeclaim")
+		if err != nil {
+			log.Debugf("ClusterDisks: pv usage data missing persistentvolumeclaim")
+			continue
+		}
+		claimNamespace, err := result.GetString("namespace")
+		if err != nil {
+			log.Debugf("ClusterDisks: pv usage data missing namespace")
+			continue
+		}
+
+		var volumeName string
+
+		for _, thatRes := range resPVCInfo {
+
+			thatCluster, err := thatRes.GetString(env.GetPromClusterLabel())
+			if err != nil {
+				thatCluster = env.GetClusterID()
+			}
+
+			thatVolumeName, err := thatRes.GetString("volumename")
+			if err != nil {
+				log.Debugf("ClusterDisks: pv claim data missing volumename")
+				continue
+			}
+			thatClaimName, err := thatRes.GetString("persistentvolumeclaim")
+			if err != nil {
+				log.Debugf("ClusterDisks: pv claim data missing persistentvolumeclaim")
+				continue
+			}
+			thatClaimNamespace, err := thatRes.GetString("namespace")
+			if err != nil {
+				log.Debugf("ClusterDisks: pv claim data missing namespace")
+				continue
+			}
+
+			if cluster == thatCluster && claimName == thatClaimName && claimNamespace == thatClaimNamespace {
+				volumeName = thatVolumeName
+			}
+		}
+
+		usage := result.Values[0].Value
+
+		key := DiskIdentifier{cluster, volumeName}
+
+		if _, ok := diskMap[key]; !ok {
+			diskMap[key] = &Disk{
+				Cluster:   cluster,
+				Name:      volumeName,
+				Breakdown: &ClusterCostsBreakdown{},
+			}
+		}
+		diskMap[key].BytesUsedMaxPtr = &usage
+	}
 }

pkg/costmodel/cluster_helpers_test.go

@@ -936,6 +936,63 @@ func TestAssetCustompricing(t *testing.T) {
 		},
 	}
 
+	pvAvgUsagePromResult := []*prom.QueryResult{
+		{
+			Metric: map[string]interface{}{
+				"cluster_id":            "cluster1",
+				"persistentvolumeclaim": "pv-claim1",
+				"namespace":             "ns1",
+			},
+			Values: []*util.Vector{
+				&util.Vector{
+					Timestamp: 0,
+					Value:     1.0,
+				},
+				&util.Vector{
+					Timestamp: 3600.0,
+					Value:     1.0,
+				},
+			},
+		},
+	}
+
+	pvMaxUsagePromResult := []*prom.QueryResult{
+		{
+			Metric: map[string]interface{}{
+				"cluster_id":            "cluster1",
+				"persistentvolumeclaim": "pv-claim1",
+				"namespace":             "ns1",
+			},
+			Values: []*util.Vector{
+				&util.Vector{
+					Timestamp: 0,
+					Value:     1.0,
+				},
+				&util.Vector{
+					Timestamp: 3600.0,
+					Value:     1.0,
+				},
+			},
+		},
+	}
+
+	pvInfoPromResult := []*prom.QueryResult{
+		{
+			Metric: map[string]interface{}{
+				"cluster_id":            "cluster1",
+				"persistentvolumeclaim": "pv-claim1",
+				"volumename":            "pvc1",
+				"namespace":             "ns1",
+			},
+			Values: []*util.Vector{
+				&util.Vector{
+					Timestamp: 0,
+					Value:     1.0,
+				},
+			},
+		},
+	}
+
 	gpuCountMap := map[NodeIdentifier]float64{
 		NodeIdentifier{
 			Cluster:    "cluster1",
@@ -1000,7 +1057,7 @@ func TestAssetCustompricing(t *testing.T) {
 			gpuResult := gpuMap[nodeKey]
 
 			diskMap := map[DiskIdentifier]*Disk{}
-			pvCosts(diskMap, time.Hour, pvMinsPromResult, pvSizePromResult, pvCostPromResult, testProvider)
+			pvCosts(diskMap, time.Hour, pvMinsPromResult, pvSizePromResult, pvCostPromResult, pvAvgUsagePromResult, pvMaxUsagePromResult, pvInfoPromResult, testProvider)
 
 			diskResult := diskMap[DiskIdentifier{"cluster1", "pvc1"}].Cost
 

pkg/costmodel/intervals.go

@@ -61,10 +61,10 @@ func getIntervalPointsFromWindows(windows map[podKey]kubecost.Window) IntervalPo
 
 	var intervals IntervalPoints
 
-	for podKey, podInterval := range windows {
+	for podKey, podWindow := range windows {
 
-		start := NewIntervalPoint(*podInterval.Start(), "start", podKey)
-		end := NewIntervalPoint(*podInterval.End(), "end", podKey)
+		start := NewIntervalPoint(*podWindow.Start(), "start", podKey)
+		end := NewIntervalPoint(*podWindow.End(), "end", podKey)
 
 		intervals = append(intervals, []IntervalPoint{start, end}...)
 
@@ -79,43 +79,49 @@ func getIntervalPointsFromWindows(windows map[podKey]kubecost.Window) IntervalPo
 // getPVCCostCoefficients gets a coefficient which represents the scale
 // factor that each PVC in a pvcIntervalMap and corresponding slice of
 // IntervalPoints intervals uses to calculate a cost for that PVC's PV.
-func getPVCCostCoefficients(intervals IntervalPoints, pvcIntervalMap map[podKey]kubecost.Window) map[podKey][]CoefficientComponent {
-
+func getPVCCostCoefficients(intervals IntervalPoints, thisPVC *pvc) map[podKey][]CoefficientComponent {
+	// pvcCostCoefficientMap has a format such that the individual coefficient
+	// components are preserved for testing purposes.
 	pvcCostCoefficientMap := make(map[podKey][]CoefficientComponent)
 
-	// pvcCostCoefficientMap is mutated in this function. The format is
-	// such that the individual coefficient components are preserved for
-	// testing purposes.
+	pvcWindow := kubecost.NewWindow(&thisPVC.Start, &thisPVC.End)
 
-	activeKeys := map[podKey]struct{}{
-		intervals[0].Key: struct{}{},
-	}
+	unmountedKey := getUnmountedPodKey(thisPVC.Cluster)
 
-	// For each interval i.e. for any time a pod-PVC relation ends or starts...
-	for i := 1; i < len(intervals); i++ {
+	var void struct{}
+	activeKeys := map[podKey]struct{}{}
 
-		// intervals will always have at least two IntervalPoints (one start/end)
-		point := intervals[i]
-		prevPoint := intervals[i-1]
+	currentTime := thisPVC.Start
 
+	// For each interval i.e. for any time a pod-PVC relation ends or starts...
+	for _, point := range intervals {
 		// If the current point happens at a later time than the previous point
-		if !point.Time.Equal(prevPoint.Time) {
+		if !point.Time.Equal(currentTime) {
 			for key := range activeKeys {
-				if pvcIntervalMap[key].Duration().Minutes() != 0 {
-					pvcCostCoefficientMap[key] = append(
-						pvcCostCoefficientMap[key],
-						CoefficientComponent{
-							Time:       point.Time.Sub(prevPoint.Time).Minutes() / pvcIntervalMap[key].Duration().Minutes(),
-							Proportion: 1.0 / float64(len(activeKeys)),
-						},
-					)
-				}
+				pvcCostCoefficientMap[key] = append(
+					pvcCostCoefficientMap[key],
+					CoefficientComponent{
+						Time:       point.Time.Sub(currentTime).Minutes() / pvcWindow.Duration().Minutes(),
+						Proportion: 1.0 / float64(len(activeKeys)),
+					},
+				)
+
+			}
+			// If there are no active keys attribute all cost to the unmounted pv
+			if len(activeKeys) == 0 {
+				pvcCostCoefficientMap[unmountedKey] = append(
+					pvcCostCoefficientMap[unmountedKey],
+					CoefficientComponent{
+						Time:       point.Time.Sub(currentTime).Minutes() / pvcWindow.Duration().Minutes(),
+						Proportion: 1.0,
+					},
+				)
 			}
 		}
 
 		// If the point was a start, increment and track
 		if point.PointType == "start" {
-			activeKeys[point.Key] = struct{}{}
+			activeKeys[point.Key] = void
 		}
 
 		// If the point was an end, decrement and stop tracking
@@ -123,6 +129,18 @@ func getPVCCostCoefficients(intervals IntervalPoints, pvcIntervalMap map[podKey]
 			delete(activeKeys, point.Key)
 		}
 
+		currentTime = point.Time
+	}
+
+	// If all pod intervals end before the end of the PVC attribute the remaining cost to unmounted
+	if currentTime.Before(thisPVC.End) {
+		pvcCostCoefficientMap[unmountedKey] = append(
+			pvcCostCoefficientMap[unmountedKey],
+			CoefficientComponent{
+				Time:       thisPVC.End.Sub(currentTime).Minutes() / pvcWindow.Duration().Minutes(),
+				Proportion: 1.0,
+			},
+		)
 	}
 
 	return pvcCostCoefficientMap

pkg/costmodel/intervals_test.go

@@ -150,187 +150,167 @@ func TestGetIntervalPointsFromWindows(t *testing.T) {
 }
 
 func TestGetPVCCostCoefficients(t *testing.T) {
+	pvc1 := &pvc{
+		Bytes:     0,
+		Name:      "pvc1",
+		Cluster:   "cluster1",
+		Namespace: "namespace1",
+		Start:     time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC),
+		End:       time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC),
+	}
+	pod1Key := newPodKey("cluster1", "namespace1", "pod1")
+	pod2Key := newPodKey("cluster1", "namespace1", "pod2")
+	pod3Key := newPodKey("cluster1", "namespace1", "pod3")
+	pod4Key := newPodKey("cluster1", "namespace1", "pod4")
+	ummountedPodKey := newPodKey("cluster1", kubecost.UnmountedSuffix, kubecost.UnmountedSuffix)
+
 	cases := []struct {
 		name           string
+		pvc            *pvc
 		pvcIntervalMap map[podKey]kubecost.Window
 		intervals      []IntervalPoint
 		expected       map[podKey][]CoefficientComponent
 	}{
 		{
 			name: "four pods w/ various overlaps",
-			pvcIntervalMap: map[podKey]kubecost.Window{
-				// Pod running from 8 am to 9 am
-				podKey{
-					Pod: "Pod1",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC),
-				)),
-				// Pod running from 8:30 am to 9 am
-				podKey{
-					Pod: "Pod2",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC),
-				)),
-				// Pod running from 8:45 am to 9 am
-				podKey{
-					Pod: "Pod3",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 45, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC),
-				)),
-				// Pod running from 8 am to 8:15 am
-				podKey{
-					Pod: "Pod4",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 8, 15, 0, 0, time.UTC),
-				)),
-			},
+			pvc:  pvc1,
 			intervals: []IntervalPoint{
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", podKey{Pod: "Pod1"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", podKey{Pod: "Pod4"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 15, 0, 0, time.UTC), "end", podKey{Pod: "Pod4"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "start", podKey{Pod: "Pod2"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 45, 0, 0, time.UTC), "start", podKey{Pod: "Pod3"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", podKey{Pod: "Pod2"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", podKey{Pod: "Pod3"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", podKey{Pod: "Pod1"}),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", pod4Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 15, 0, 0, time.UTC), "end", pod4Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "start", pod2Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 45, 0, 0, time.UTC), "start", pod3Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", pod2Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", pod3Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", pod1Key),
 			},
 			expected: map[podKey][]CoefficientComponent{
-				podKey{
-					Pod: "Pod1",
-				}: []CoefficientComponent{
+				pod1Key: []CoefficientComponent{
 					CoefficientComponent{0.5, 0.25},
 					CoefficientComponent{1, 0.25},
 					CoefficientComponent{0.5, 0.25},
 					CoefficientComponent{1.0 / 3.0, 0.25},
 				},
-				podKey{
-					Pod: "Pod2",
-				}: []CoefficientComponent{
-					CoefficientComponent{0.5, 0.50},
-					CoefficientComponent{1.0 / 3.0, 0.50},
+				pod2Key: []CoefficientComponent{
+					CoefficientComponent{0.5, 0.25},
+					CoefficientComponent{1.0 / 3.0, 0.25},
 				},
-				podKey{
-					Pod: "Pod3",
-				}: []CoefficientComponent{
-					CoefficientComponent{1.0 / 3.0, 1.0},
+				pod3Key: []CoefficientComponent{
+					CoefficientComponent{1.0 / 3.0, 0.25},
 				},
-				podKey{
-					Pod: "Pod4",
-				}: []CoefficientComponent{
-					CoefficientComponent{0.5, 1.0},
+				pod4Key: []CoefficientComponent{
+					CoefficientComponent{0.5, 0.25},
 				},
 			},
 		},
 		{
 			name: "two pods no overlap",
-			pvcIntervalMap: map[podKey]kubecost.Window{
-				// Pod running from 8 am to 8:30 am
-				podKey{
-					Pod: "Pod1",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC),
-				)),
-				// Pod running from 8:30 am to 9 am
-				podKey{
-					Pod: "Pod2",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC),
-				)),
-			},
+			pvc:  pvc1,
 			intervals: []IntervalPoint{
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", podKey{Pod: "Pod1"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "start", podKey{Pod: "Pod2"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "end", podKey{Pod: "Pod1"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", podKey{Pod: "Pod2"}),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "start", pod2Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "end", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", pod2Key),
 			},
 			expected: map[podKey][]CoefficientComponent{
-				podKey{
-					Pod: "Pod1",
-				}: []CoefficientComponent{
-					CoefficientComponent{1.0, 1.0},
+				pod1Key: []CoefficientComponent{
+					CoefficientComponent{1.0, 0.5},
 				},
-				podKey{
-					Pod: "Pod2",
-				}: []CoefficientComponent{
-					CoefficientComponent{1.0, 1.0},
+				pod2Key: []CoefficientComponent{
+					CoefficientComponent{1.0, 0.5},
 				},
 			},
 		},
 		{
 			name: "two pods total overlap",
-			pvcIntervalMap: map[podKey]kubecost.Window{
-				// Pod running from 8:30 am to 9 am
-				podKey{
-					Pod: "Pod1",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC),
-				)),
-				// Pod running from 8:30 am to 9 am
-				podKey{
-					Pod: "Pod2",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC),
-				)),
-			},
+			pvc:  pvc1,
 			intervals: []IntervalPoint{
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "start", podKey{Pod: "Pod1"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "start", podKey{Pod: "Pod2"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", podKey{Pod: "Pod1"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", podKey{Pod: "Pod2"}),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "start", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 30, 0, 0, time.UTC), "start", pod2Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", pod2Key),
 			},
 			expected: map[podKey][]CoefficientComponent{
-				podKey{
-					Pod: "Pod1",
-				}: []CoefficientComponent{
-					CoefficientComponent{0.5, 1.0},
+				pod1Key: []CoefficientComponent{
+					CoefficientComponent{0.5, 0.5},
 				},
-				podKey{
-					Pod: "Pod2",
-				}: []CoefficientComponent{
-					CoefficientComponent{0.5, 1.0},
+				pod2Key: []CoefficientComponent{
+					CoefficientComponent{0.5, 0.5},
+				},
+				ummountedPodKey: []CoefficientComponent{
+					CoefficientComponent{1.0, 0.5},
 				},
 			},
 		},
 		{
 			name: "one pod",
-			pvcIntervalMap: map[podKey]kubecost.Window{
-				// Pod running from 8 am to 9 am
-				podKey{
-					Pod: "Pod1",
-				}: kubecost.Window(kubecost.NewClosedWindow(
-					time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC),
-					time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC),
-				)),
-			},
+			pvc:  pvc1,
 			intervals: []IntervalPoint{
-				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", podKey{Pod: "Pod1"}),
-				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", podKey{Pod: "Pod1"}),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", pod1Key),
 			},
 			expected: map[podKey][]CoefficientComponent{
-				podKey{
-					Pod: "Pod1",
-				}: []CoefficientComponent{
+				pod1Key: []CoefficientComponent{
 					CoefficientComponent{1.0, 1.0},
 				},
 			},
 		},
+		{
+			name: "two pods with gap",
+			pvc:  pvc1,
+			intervals: []IntervalPoint{
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 0, 0, 0, time.UTC), "start", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 15, 0, 0, time.UTC), "end", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 45, 0, 0, time.UTC), "start", pod2Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 9, 0, 0, 0, time.UTC), "end", pod2Key),
+			},
+			expected: map[podKey][]CoefficientComponent{
+				pod1Key: []CoefficientComponent{
+					CoefficientComponent{1.0, 0.25},
+				},
+				pod2Key: []CoefficientComponent{
+					CoefficientComponent{1.0, 0.25},
+				},
+				ummountedPodKey: []CoefficientComponent{
+					CoefficientComponent{1.0, 0.5},
+				},
+			},
+		},
+		{
+			name: "one pods start and end in window",
+			pvc:  pvc1,
+			intervals: []IntervalPoint{
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 15, 0, 0, time.UTC), "start", pod1Key),
+				NewIntervalPoint(time.Date(2021, 2, 19, 8, 45, 0, 0, time.UTC), "end", pod1Key),
+			},
+			expected: map[podKey][]CoefficientComponent{
+				pod1Key: []CoefficientComponent{
+					CoefficientComponent{1.0, 0.5},
+				},
+				ummountedPodKey: []CoefficientComponent{
+					CoefficientComponent{1.0, 0.25},
+					CoefficientComponent{1.0, 0.25},
+				},
+			},
+		},
 	}
 
 	for _, testCase := range cases {
 		t.Run(testCase.name, func(t *testing.T) {
-			result := getPVCCostCoefficients(testCase.intervals, testCase.pvcIntervalMap)
+			result := getPVCCostCoefficients(testCase.intervals, testCase.pvc)
 
 			if !reflect.DeepEqual(result, testCase.expected) {
 				t.Errorf("getPVCCostCoefficients test failed: %s: Got %+v but expected %+v", testCase.name, result, testCase.expected)
 			}
+
+			// check that coefficients sum to 1, to ensure that 100% of PVC cost is being distributed
+			sum := 0.0
+			for _, coefs := range result {
+				sum += getCoefficientFromComponents(coefs)
+			}
+			if sum != 1.0 {
+				t.Errorf("getPVCCostCoefficients test failed: coefficient totals did not sum to 1.0: %f", sum)
+			}
 		})
 	}
 }

pkg/costmodel/key.go

@@ -2,6 +2,7 @@ package costmodel
 
 import (
 	"fmt"
+	"github.com/opencost/opencost/pkg/kubecost"
 
 	"github.com/opencost/opencost/pkg/env"
 	"github.com/opencost/opencost/pkg/prom"
@@ -82,6 +83,11 @@ func newPodKey(cluster, namespace, pod string) podKey {
 	}
 }
 
+// getUnmountedPodKey while certain Unmounted costs can have a namespace, all unmounted costs for a single cluster will be represented by the same asset
+func getUnmountedPodKey(cluster string) podKey {
+	return newPodKey(cluster, kubecost.UnmountedSuffix, kubecost.UnmountedSuffix)
+}
+
 // resultPodKey converts a Prometheus query result to a podKey by looking
 // up values associated with the given label names. For example, passing
 // "cluster_id" for clusterLabel will use the value of the label "cluster_id"

pkg/costmodel/router.go

@@ -4,8 +4,9 @@ import (
 	"context"
 	"encoding/base64"
 	"fmt"
-	"io/ioutil"
+	"io"
 	"net/http"
+	"os"
 	"path"
 	"reflect"
 	"regexp"
@@ -1251,7 +1252,7 @@ func logsFor(c kubernetes.Interface, namespace string, pod string, container str
 		return "", err
 	}
 
-	podLogs, err := ioutil.ReadAll(reader)
+	podLogs, err := io.ReadAll(reader)
 	if err != nil {
 		return "", err
 	}
@@ -1354,7 +1355,7 @@ func (a *Accesses) AddServiceKey(w http.ResponseWriter, r *http.Request, ps http
 
 	key := r.PostForm.Get("key")
 	k := []byte(key)
-	err := ioutil.WriteFile(path.Join(env.GetConfigPathWithDefault("/var/configs/"), "key.json"), k, 0644)
+	err := os.WriteFile(path.Join(env.GetConfigPathWithDefault("/var/configs/"), "key.json"), k, 0644)
 	if err != nil {
 		fmt.Fprintf(w, "Error writing service key: "+err.Error())
 	}

pkg/env/costmodelenv.go

@@ -15,6 +15,10 @@ const (
 	AWSAccessKeyIDEnvVar     = "AWS_ACCESS_KEY_ID"
 	AWSAccessKeySecretEnvVar = "AWS_SECRET_ACCESS_KEY"
 	AWSClusterIDEnvVar       = "AWS_CLUSTER_ID"
+	AWSPricingURL            = "AWS_PRICING_URL"
+
+	AlibabaAccessKeyIDEnvVar     = "ALIBABA_ACCESS_KEY_ID"
+	AlibabaAccessKeySecretEnvVar = "ALIBABA_SECRET_ACCESS_KEY"
 
 	KubecostNamespaceEnvVar        = "KUBECOST_NAMESPACE"
 	PodNameEnvVar                  = "POD_NAME"
@@ -32,6 +36,7 @@ const (
 	CSVPathEnvVar                  = "CSV_PATH"
 	ConfigPathEnvVar               = "CONFIG_PATH"
 	CloudProviderAPIKeyEnvVar      = "CLOUD_PROVIDER_API_KEY"
+	DisableAggregateCostModelCache = "DISABLE_AGGREGATE_COST_MODEL_CACHE"
 
 	EmitPodAnnotationsMetricEnvVar       = "EMIT_POD_ANNOTATIONS_METRIC"
 	EmitNamespaceAnnotationsMetricEnvVar = "EMIT_NAMESPACE_ANNOTATIONS_METRIC"
@@ -205,6 +210,23 @@ func GetAWSClusterID() string {
 	return Get(AWSClusterIDEnvVar, "")
 }
 
+// GetAWSPricingURL returns an optional alternative URL to fetch AWS pricing data from; for use in airgapped environments
+func GetAWSPricingURL() string {
+	return Get(AWSPricingURL, "")
+}
+
+// GetAlibabaAccessKeyID returns the environment variable value for AlibabaAccessKeyIDEnvVar which represents
+// the Alibaba access key for authentication
+func GetAlibabaAccessKeyID() string {
+	return Get(AlibabaAccessKeyIDEnvVar, "")
+}
+
+// GetAlibabaAccessKeySecret returns the environment variable value for AlibabaAccessKeySecretEnvVar which represents
+// the Alibaba access key secret for authentication
+func GetAlibabaAccessKeySecret() string {
+	return Get(AlibabaAccessKeySecretEnvVar, "")
+}
+
 // GetKubecostNamespace returns the environment variable value for KubecostNamespaceEnvVar which
 // represents the namespace the cost model exists in.
 func GetKubecostNamespace() string {
@@ -239,6 +261,12 @@ func GetInsecureSkipVerify() bool {
 	return GetBool(InsecureSkipVerify, false)
 }
 
+// IsAggregateCostModelCacheDisabled returns the environment variable value for DisableAggregateCostModelCache which
+// will inform the aggregator on whether to load cached data. Defaults to false
+func IsAggregateCostModelCacheDisabled() bool {
+	return GetBool(DisableAggregateCostModelCache, false)
+}
+
 // IsRemoteEnabled returns the environment variable value for RemoteEnabledEnvVar which represents whether
 // or not remote write is enabled for prometheus for use with SQL backed persistent storage.
 func IsRemoteEnabled() bool {

pkg/env/costmodelenv_test.go

@@ -0,0 +1,43 @@
+package env
+
+import (
+	"os"
+	"testing"
+)
+
+func TestIsCacheDisabled(t *testing.T) {
+	tests := []struct {
+		name string
+		want bool
+		pre  func()
+	}{
+		{
+			name: "Ensure the default value is false",
+			want: false,
+		},
+		{
+			name: "Ensure the value is false when DISABLE_AGGREGATE_COST_MODEL_CACHE is set to false",
+			want: false,
+			pre: func() {
+				os.Setenv("DISABLE_AGGREGATE_COST_MODEL_CACHE", "false")
+			},
+		},
+		{
+			name: "Ensure the value is true when DISABLE_AGGREGATE_COST_MODEL_CACHE is set to true",
+			want: true,
+			pre: func() {
+				os.Setenv("DISABLE_AGGREGATE_COST_MODEL_CACHE", "true")
+			},
+		},
+	}
+	for _, tt := range tests {
+		if tt.pre != nil {
+			tt.pre()
+		}
+		t.Run(tt.name, func(t *testing.T) {
+			if got := IsAggregateCostModelCacheDisabled(); got != tt.want {
+				t.Errorf("IsAggregateCostModelCacheDisabled() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}

pkg/env/env.go

@@ -123,6 +123,12 @@ func GetDuration(key string, defaultValue time.Duration) time.Duration {
 	return envMapper.GetDuration(key, defaultValue)
 }
 
+// GetList parses a []string from the enviroment variable key parameter.  If the environment
+// // variable is empty or fails to parse, nil is returned.
+func GetList(key, delimiter string) []string {
+	return envMapper.GetList(key, delimiter)
+}
+
 // Set sets the environment variable for the key provided using the value provided.
 func Set(key string, value string) error {
 	return envMapper.Set(key, value)

pkg/filter/allcut.go

@@ -0,0 +1,10 @@
+package filter
+
+// AllCut is a filter that matches nothing. This is useful
+// for applications like authorization, where a user/group/role may be disallowed
+// from viewing data entirely.
+type AllCut[T any] struct{}
+
+func (ac AllCut[T]) String() string { return "(AllCut)" }
+
+func (ac AllCut[T]) Matches(T) bool { return false }

pkg/filter/allpass.go

@@ -0,0 +1,9 @@
+package filter
+
+// AllPass is a filter that matches everything and is the same as no filter. It is implemented here as a guard
+// against universal operations occurring in the absence of filters.
+type AllPass[T any] struct{}
+
+func (n AllPass[T]) String() string { return "(AllPass)" }
+
+func (n AllPass[T]) Matches(T) bool { return true }

pkg/filter/and.go

@@ -0,0 +1,36 @@
+package filter
+
+import (
+	"fmt"
+)
+
+// And is a set of filters that should be evaluated as a logical
+// AND.
+type And[T any] struct {
+	Filters []Filter[T]
+}
+
+func (a And[T]) String() string {
+	s := "(and"
+	for _, f := range a.Filters {
+		s += fmt.Sprintf(" %s", f)
+	}
+
+	s += ")"
+	return s
+}
+
+func (a And[T]) Matches(that T) bool {
+	filters := a.Filters
+	if len(filters) == 0 {
+		return true
+	}
+
+	for _, filter := range filters {
+		if !filter.Matches(that) {
+			return false
+		}
+	}
+
+	return true
+}

pkg/filter/filter.go

@@ -0,0 +1,20 @@
+package filter
+
+// Filter represents anything that can be used to filter given generic type T.
+//
+// Implement this interface with caution. While it is generic, it
+// is intended to be introspectable so query handlers can perform various
+// optimizations. These optimizations include:
+// - Routing a query to the most optimal cache
+// - Querying backing data stores efficiently (e.g. translation to SQL)
+//
+// Custom implementations of this interface outside of this package should not
+// expect to receive these benefits. Passing a custom implementation to a
+// handler may in errors.
+type Filter[T any] interface {
+	String() string
+
+	// Matches is the canonical in-Go function for determining if T
+	// matches a filter.
+	Matches(T) bool
+}

pkg/filter/filter_test.go

@@ -0,0 +1,1073 @@
+package filter_test
+
+import (
+	"testing"
+
+	"github.com/opencost/opencost/pkg/filter"
+	"github.com/opencost/opencost/pkg/kubecost"
+)
+
+func Test_String_Matches(t *testing.T) {
+	cases := []struct {
+		name   string
+		a      *kubecost.Allocation
+		filter filter.Filter[*kubecost.Allocation]
+
+		expected bool
+	}{
+		{
+			name: "ClusterID Equals -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Cluster: "cluster-one",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationClusterProp,
+				Op:    filter.StringEquals,
+				Value: "cluster-one",
+			},
+
+			expected: true,
+		},
+		{
+			name: "ClusterID StartsWith -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Cluster: "cluster-one",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationClusterProp,
+				Op:    filter.StringStartsWith,
+				Value: "cluster",
+			},
+
+			expected: true,
+		},
+		{
+			name: "ClusterID StartsWith -> false",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Cluster: "k8s-one",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationClusterProp,
+				Op:    filter.StringStartsWith,
+				Value: "cluster",
+			},
+
+			expected: false,
+		},
+		{
+			name: "ClusterID empty StartsWith '' -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Cluster: "",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationClusterProp,
+				Op:    filter.StringStartsWith,
+				Value: "",
+			},
+
+			expected: true,
+		},
+		{
+			name: "ClusterID nonempty StartsWith '' -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Cluster: "abc",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationClusterProp,
+				Op:    filter.StringStartsWith,
+				Value: "",
+			},
+
+			expected: true,
+		},
+		{
+			name: "Node Equals -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Node: "node123",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationNodeProp,
+				Op:    filter.StringEquals,
+				Value: "node123",
+			},
+
+			expected: true,
+		},
+		{
+			name: "Namespace Equals Unallocated -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationNamespaceProp,
+				Op:    filter.StringEquals,
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: true,
+		},
+		{
+			name: "ControllerKind Equals -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					ControllerKind: "deployment", // We generally store controller kinds as all lowercase
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationControllerKindProp,
+				Op:    filter.StringEquals,
+				Value: "deployment",
+			},
+
+			expected: true,
+		},
+		{
+			name: "ControllerName Equals -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Controller: "kc-cost-analyzer",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationControllerProp,
+				Op:    filter.StringEquals,
+				Value: "kc-cost-analyzer",
+			},
+
+			expected: true,
+		},
+		{
+			name: "Pod (with UID) Equals -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Pod: "pod-123 UID-ABC",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationPodProp,
+				Op:    filter.StringEquals,
+				Value: "pod-123 UID-ABC",
+			},
+
+			expected: true,
+		},
+		{
+			name: "Container Equals -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Container: "cost-model",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationContainerProp,
+				Op:    filter.StringEquals,
+				Value: "cost-model",
+			},
+
+			expected: true,
+		},
+		{
+			name: `namespace unallocated -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationNamespaceProp,
+				Op:    filter.StringEquals,
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: true,
+		},
+	}
+
+	for _, c := range cases {
+		result := c.filter.Matches(c.a)
+
+		if result != c.expected {
+			t.Errorf("%s: expected %t, got %t", c.name, c.expected, result)
+		}
+	}
+}
+
+func Test_StringSlice_Matches(t *testing.T) {
+	cases := []struct {
+		name   string
+		a      *kubecost.Allocation
+		filter filter.Filter[*kubecost.Allocation]
+
+		expected bool
+	}{
+		{
+			name: `services contains -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContains,
+				Value: "serv2",
+			},
+
+			expected: true,
+		},
+		{
+			name: `services contains -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContains,
+				Value: "serv3",
+			},
+
+			expected: false,
+		},
+		{
+			name: `services contains unallocated -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContains,
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: false,
+		},
+		{
+			name: `services contains unallocated -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContains,
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: true,
+		},
+	}
+
+	for _, c := range cases {
+		result := c.filter.Matches(c.a)
+
+		if result != c.expected {
+			t.Errorf("%s: expected %t, got %t", c.name, c.expected, result)
+		}
+	}
+}
+
+func Test_StringMap_Matches(t *testing.T) {
+	cases := []struct {
+		name   string
+		a      *kubecost.Allocation
+		filter filter.Filter[*kubecost.Allocation]
+
+		expected bool
+	}{
+		{
+			name: `label[app]="foo" -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"app": "foo",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationLabelProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: "foo",
+			},
+
+			expected: true,
+		},
+		{
+			name: `label[app]="foo" -> different value -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"app": "bar",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationLabelProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: "foo",
+			},
+
+			expected: false,
+		},
+		{
+			name: `label[app]="foo" -> label missing -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"someotherlabel": "someothervalue",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationLabelProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: "foo",
+			},
+
+			expected: false,
+		},
+		{
+			name: `label[app]=Unallocated -> label missing -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"someotherlabel": "someothervalue",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationLabelProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: true,
+		},
+		{
+			name: `label[app]=Unallocated -> label present -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"app": "test",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationLabelProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: false,
+		},
+		{
+			name: `annotation[prom_modified_name]="testing123" -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Annotations: map[string]string{
+						"prom_modified_name": "testing123",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationAnnotationProp,
+				Op:    filter.StringMapEquals,
+				Key:   "prom_modified_name",
+				Value: "testing123",
+			},
+
+			expected: true,
+		},
+		{
+			name: `annotation[app]="foo" -> different value -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Annotations: map[string]string{
+						"app": "bar",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationAnnotationProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: "foo",
+			},
+
+			expected: false,
+		},
+		{
+			name: `annotation[app]="foo" -> annotation missing -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Annotations: map[string]string{
+						"someotherannotation": "someothervalue",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationAnnotationProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: "foo",
+			},
+
+			expected: false,
+		},
+	}
+
+	for _, c := range cases {
+		result := c.filter.Matches(c.a)
+
+		if result != c.expected {
+			t.Errorf("%s: expected %t, got %t", c.name, c.expected, result)
+		}
+	}
+}
+
+func Test_Not_Matches(t *testing.T) {
+	cases := []struct {
+		name   string
+		a      *kubecost.Allocation
+		filter filter.Filter[*kubecost.Allocation]
+
+		expected bool
+	}{
+		{
+			name: "Namespace NotEquals -> false",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "kube-system",
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationNamespaceProp,
+					Op:    filter.StringEquals,
+					Value: "kube-system",
+				},
+			},
+
+			expected: false,
+		},
+		{
+			name: "Namespace NotEquals Unallocated -> true",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "kube-system",
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationNamespaceProp,
+					Op:    filter.StringEquals,
+					Value: kubecost.UnallocatedSuffix,
+				},
+			},
+			expected: true,
+		},
+		{
+			name: "Namespace NotEquals Unallocated -> false",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "",
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationNamespaceProp,
+					Op:    filter.StringEquals,
+					Value: kubecost.UnallocatedSuffix,
+				},
+			},
+
+			expected: false,
+		},
+
+		{
+			name: `label[app]!=Unallocated -> label missing -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"someotherlabel": "someothervalue",
+					},
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringMapProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationLabelProp,
+					Op:    filter.StringMapEquals,
+					Key:   "app",
+					Value: kubecost.UnallocatedSuffix,
+				},
+			},
+			expected: false,
+		},
+		{
+			name: `label[app]!=Unallocated -> label present -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"app": "test",
+					},
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringMapProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationLabelProp,
+					Op:    filter.StringMapEquals,
+					Key:   "app",
+					Value: kubecost.UnallocatedSuffix,
+				},
+			},
+			expected: true,
+		},
+		{
+			name: `label[app]!="foo" -> label missing -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"someotherlabel": "someothervalue",
+					},
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringMapProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationLabelProp,
+					Op:    filter.StringMapEquals,
+					Key:   "app",
+					Value: "foo",
+				},
+			},
+
+			expected: true,
+		},
+		{
+			name: `annotation[prom_modified_name]="testing123" -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Annotations: map[string]string{
+						"prom_modified_name": "testing123",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationAnnotationProp,
+				Op:    filter.StringMapEquals,
+				Key:   "prom_modified_name",
+				Value: "testing123",
+			},
+
+			expected: true,
+		},
+		{
+			name: `annotation[app]="foo" -> different value -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Annotations: map[string]string{
+						"app": "bar",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationAnnotationProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: "foo",
+			},
+
+			expected: false,
+		},
+		{
+			name: `annotation[app]="foo" -> annotation missing -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Annotations: map[string]string{
+						"someotherannotation": "someothervalue",
+					},
+				},
+			},
+			filter: filter.StringMapProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationAnnotationProp,
+				Op:    filter.StringMapEquals,
+				Key:   "app",
+				Value: "foo",
+			},
+
+			expected: false,
+		},
+		{
+			name: `annotation[app]!="foo" -> annotation missing -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Annotations: map[string]string{
+						"someotherannotation": "someothervalue",
+					},
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringMapProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationAnnotationProp,
+					Op:    filter.StringMapEquals,
+					Key:   "app",
+					Value: "foo",
+				},
+			},
+
+			expected: true,
+		},
+		{
+			name: `namespace unallocated -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "",
+				},
+			},
+			filter: filter.StringProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationNamespaceProp,
+				Op:    filter.StringEquals,
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: true,
+		},
+		{
+			name: `services contains -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContains,
+				Value: "serv2",
+			},
+
+			expected: true,
+		},
+		{
+			name: `services contains -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContains,
+				Value: "serv3",
+			},
+
+			expected: false,
+		},
+		{
+			name: `services notcontains -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringSliceProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationServiceProp,
+					Op:    filter.StringSliceContains,
+					Value: "serv3",
+				},
+			},
+			expected: true,
+		},
+		{
+			name: `services notcontains -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringSliceProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationServiceProp,
+					Op:    filter.StringSliceContains,
+					Value: "serv2",
+				},
+			},
+
+			expected: false,
+		},
+		{
+			name: `services notcontains unallocated -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringSliceProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationServiceProp,
+					Op:    filter.StringSliceContains,
+					Value: kubecost.UnallocatedSuffix,
+				},
+			},
+
+			expected: true,
+		},
+		{
+			name: `services notcontains unallocated -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{},
+				},
+			},
+			filter: filter.Not[*kubecost.Allocation]{
+				Filter: filter.StringSliceProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationServiceProp,
+					Op:    filter.StringSliceContains,
+					Value: kubecost.UnallocatedSuffix,
+				},
+			},
+
+			expected: false,
+		},
+		{
+			name: `services containsprefix -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContainsPrefix,
+				Value: "serv",
+			},
+
+			expected: true,
+		},
+		{
+			name: `services containsprefix -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"foo", "bar"},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContainsPrefix,
+				Value: "serv",
+			},
+
+			expected: false,
+		},
+		{
+			name: `services contains unallocated -> false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContains,
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: false,
+		},
+		{
+			name: `services contains unallocated -> true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{},
+				},
+			},
+			filter: filter.StringSliceProperty[*kubecost.Allocation]{
+				Field: kubecost.AllocationServiceProp,
+				Op:    filter.StringSliceContains,
+				Value: kubecost.UnallocatedSuffix,
+			},
+
+			expected: true,
+		},
+	}
+
+	for _, c := range cases {
+		result := c.filter.Matches(c.a)
+
+		if result != c.expected {
+			t.Errorf("%s: expected %t, got %t", c.name, c.expected, result)
+		}
+	}
+}
+
+func Test_None_Matches(t *testing.T) {
+	cases := []struct {
+		name string
+		a    *kubecost.Allocation
+	}{
+		{
+			name: "nil",
+			a:    nil,
+		},
+		{
+			name: "nil properties",
+			a: &kubecost.Allocation{
+				Properties: nil,
+			},
+		},
+		{
+			name: "empty properties",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{},
+			},
+		},
+		{
+			name: "ClusterID",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Cluster: "cluster-one",
+				},
+			},
+		},
+		{
+			name: "Node",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Node: "node123",
+				},
+			},
+		},
+		{
+			name: "Namespace",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "kube-system",
+				},
+			},
+		},
+		{
+			name: "ControllerKind",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					ControllerKind: "deployment", // We generally store controller kinds as all lowercase
+				},
+			},
+		},
+		{
+			name: "ControllerName",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Controller: "kc-cost-analyzer",
+				},
+			},
+		},
+		{
+			name: "Pod",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Pod: "pod-123 UID-ABC",
+				},
+			},
+		},
+		{
+			name: "Container",
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Container: "cost-model",
+				},
+			},
+		},
+		{
+			name: `label`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Labels: map[string]string{
+						"app": "foo",
+					},
+				},
+			},
+		},
+		{
+			name: `annotation`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Annotations: map[string]string{
+						"prom_modified_name": "testing123",
+					},
+				},
+			},
+		},
+		{
+			name: `services`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Services: []string{"serv1", "serv2"},
+				},
+			},
+		},
+	}
+
+	for _, c := range cases {
+		result := filter.AllCut[*kubecost.Allocation]{}.Matches(c.a)
+
+		if result {
+			t.Errorf("%s: should have been rejected", c.name)
+		}
+	}
+}
+
+func Test_And_Matches(t *testing.T) {
+	cases := []struct {
+		name   string
+		a      *kubecost.Allocation
+		filter filter.Filter[*kubecost.Allocation]
+
+		expected bool
+	}{
+		{
+			name: `label[app]="foo" and namespace="kubecost" -> both true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "kubecost",
+					Labels: map[string]string{
+						"app": "foo",
+					},
+				},
+			},
+			filter: filter.And[*kubecost.Allocation]{[]filter.Filter[*kubecost.Allocation]{
+				filter.StringMapProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationLabelProp,
+					Op:    filter.StringMapEquals,
+					Key:   "app",
+					Value: "foo",
+				},
+				filter.StringProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationNamespaceProp,
+					Op:    filter.StringEquals,
+					Value: "kubecost",
+				},
+			}},
+			expected: true,
+		},
+		{
+			name: `label[app]="foo" and namespace="kubecost" -> first true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "kube-system",
+					Labels: map[string]string{
+						"app": "foo",
+					},
+				},
+			},
+			filter: filter.And[*kubecost.Allocation]{[]filter.Filter[*kubecost.Allocation]{
+				filter.StringMapProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationLabelProp,
+					Op:    filter.StringMapEquals,
+					Key:   "app",
+					Value: "foo",
+				},
+				filter.StringProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationNamespaceProp,
+					Op:    filter.StringEquals,
+					Value: "kubecost",
+				},
+			}},
+			expected: false,
+		},
+		{
+			name: `label[app]="foo" and namespace="kubecost" -> second true`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "kubecost",
+					Labels: map[string]string{
+						"app": "bar",
+					},
+				},
+			},
+			filter: filter.And[*kubecost.Allocation]{[]filter.Filter[*kubecost.Allocation]{
+				filter.StringMapProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationLabelProp,
+					Op:    filter.StringMapEquals,
+					Key:   "app",
+					Value: "foo",
+				},
+				filter.StringProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationNamespaceProp,
+					Op:    filter.StringEquals,
+					Value: "kubecost",
+				},
+			}},
+			expected: false,
+		},
+		{
+			name: `label[app]="foo" and namespace="kubecost" -> both false`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "kube-system",
+					Labels: map[string]string{
+						"app": "bar",
+					},
+				},
+			},
+			filter: filter.And[*kubecost.Allocation]{[]filter.Filter[*kubecost.Allocation]{
+				filter.StringMapProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationLabelProp,
+					Op:    filter.StringMapEquals,
+					Key:   "app",
+					Value: "foo",
+				},
+				filter.StringProperty[*kubecost.Allocation]{
+					Field: kubecost.AllocationNamespaceProp,
+					Op:    filter.StringEquals,
+					Value: "kubecost",
+				},
+			}},
+			expected: false,
+		},
+		{
+			name: `(and none) matches nothing`,
+			a: &kubecost.Allocation{
+				Properties: &kubecost.AllocationProperties{
+					Namespace: "kube-system",
+					Labels: map[string]string{
+						"app": "bar",
+					},
+				},
+			},
+			filter: filter.And[*kubecost.Allocation]{[]filter.Filter[*kubecost.Allocation]{
+				filter.AllCut[*kubecost.Allocation]{},
+			}},
+			expected: false,
+		},
+	}
+
+	for _, c := range cases {
+		result := c.filter.Matches(c.a)
+
+		if result != c.expected {
+			t.Errorf("%s: expected %t, got %t", c.name, c.expected, result)
+		}
+	}
+}

pkg/filter/not.go

@@ -0,0 +1,17 @@
+package filter
+
+import "fmt"
+
+// Not negates any filter contained within it
+type Not[T any] struct {
+	Filter Filter[T]
+}
+
+func (n Not[T]) String() string {
+	return fmt.Sprintf("(not %s)", n.Filter.String())
+}
+
+// Matches inverts the result of the child filter
+func (n Not[T]) Matches(that T) bool {
+	return !n.Filter.Matches(that)
+}

pkg/filter/or.go

@@ -0,0 +1,36 @@
+package filter
+
+import (
+	"fmt"
+)
+
+// Or is a set of filters that should be evaluated as a logical
+// OR.
+type Or[T any] struct {
+	Filters []Filter[T]
+}
+
+func (o Or[T]) String() string {
+	s := "(or"
+	for _, f := range o.Filters {
+		s += fmt.Sprintf(" %s", f)
+	}
+
+	s += ")"
+	return s
+}
+
+func (o Or[T]) Matches(that T) bool {
+	filters := o.Filters
+	if len(filters) == 0 {
+		return true
+	}
+
+	for _, filter := range filters {
+		if filter.Matches(that) {
+			return true
+		}
+	}
+
+	return false
+}

pkg/filter/stringmapproperty.go

@@ -0,0 +1,83 @@
+package filter
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/opencost/opencost/pkg/log"
+)
+
+const unallocatedSuffix = "__unallocated__"
+
+type StringMapPropertied interface {
+	StringMapProperty(string) (map[string]string, error)
+}
+
+// StringMapOperation is an enum that represents operations that can be performed
+// when filtering (equality, inequality, etc.)
+type StringMapOperation string
+
+const (
+	// StringMapHasKey passes if the map has the provided key
+	StringMapHasKey StringMapOperation = "stringmapcontains"
+
+	StringMapStartsWith = "stringmapstartswith"
+
+	// StringMapEquals when the given key and value match
+	StringMapEquals = "stringmapequals"
+)
+
+// StringMapProperty is the lowest-level type of filter. It represents
+// a filter operation (equality, inequality, etc.) on a property that contains a string map
+type StringMapProperty[T StringMapPropertied] struct {
+	Field string
+	Op    StringMapOperation
+	Key   string
+	Value string
+}
+
+func (smp StringMapProperty[T]) String() string {
+	return fmt.Sprintf(`(%s %s[%s] "%s")`, smp.Op, smp.Field, smp.Key, smp.Value)
+}
+
+func (smp StringMapProperty[T]) Matches(that T) bool {
+
+	thatMap, err := that.StringMapProperty(smp.Field)
+	if err != nil {
+		log.Errorf("Filter: StringMapProperty: could not retrieve field %s: %s", smp.Field, err.Error())
+		return false
+	}
+
+	valueToCompare, keyIsPresent := thatMap[smp.Key]
+
+	switch smp.Op {
+	case StringMapHasKey:
+		return keyIsPresent
+	case StringMapEquals:
+		// namespace:"__unallocated__" should match a.Properties.Namespace = ""
+		// label[app]:"__unallocated__" should match _, ok := Labels[app]; !ok
+		if !keyIsPresent || valueToCompare == "" {
+			return smp.Value == unallocatedSuffix
+		}
+
+		if valueToCompare == smp.Value {
+			return true
+		}
+
+	case StringMapStartsWith:
+		if !keyIsPresent {
+			return false
+		}
+
+		// We don't need special __unallocated__ logic here because a query
+		// asking for "__unallocated__" won't have a wildcard and unallocated
+		// properties are the empty string.
+
+		return strings.HasPrefix(valueToCompare, smp.Value)
+	default:
+		log.Errorf("Filter: StringMapProperty: Unhandled filter op. This is a filter implementation error and requires immediate patching. Op: %s", smp.Op)
+		return false
+	}
+
+	return false
+}

pkg/filter/stringproperty.go

@@ -0,0 +1,83 @@
+package filter
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/opencost/opencost/pkg/log"
+)
+
+// StringPropertied is used to validate the name of a property field and return its value
+type StringPropertied interface {
+	// StringProperty acts as a validator and getter for a structs string properties
+	StringProperty(string) (string, error)
+}
+
+// StringOperation is an enum that represents operations that can be performed
+// when filtering (equality, inequality, etc.)
+type StringOperation string
+
+// If you add a FilterOp, MAKE SURE TO UPDATE ALL FILTER IMPLEMENTATIONS! Go
+// does not enforce exhaustive pattern matching on "enum" types.
+const (
+	// StringEquals is the equality operator
+	// "kube-system" FilterEquals "kube-system" = true
+	// "kube-syste" FilterEquals "kube-system" = false
+	StringEquals StringOperation = "stringequals"
+
+	// StringStartsWith matches strings with the given prefix.
+	// "kube-system" StartsWith "kube" = true
+	//
+	// When comparing with a field represented by an array/slice, this is like
+	// applying FilterContains to every element of the slice.
+	StringStartsWith = "stringstartswith"
+)
+
+// StringProperty is the lowest-level type of filter. It represents
+// a filter operation (equality, inequality, etc.) on a field with a string value (namespace,
+// node, pod, etc.).
+type StringProperty[T StringPropertied] struct {
+	Field string
+	Op    StringOperation
+
+	// Value is for _all_ filters. A filter of 'namespace:"kubecost"' has
+	// Value="kubecost"
+	Value string
+}
+
+func (sp StringProperty[T]) String() string {
+	return fmt.Sprintf(`(%s %s "%s")`, sp.Op, sp.Field, sp.Value)
+}
+
+func (sp StringProperty[T]) Matches(that T) bool {
+
+	thatString, err := that.StringProperty(sp.Field)
+	if err != nil {
+		log.Errorf("Filter: StringProperty: could not retrieve field %s: %s", sp.Field, err.Error())
+		return false
+	}
+
+	switch sp.Op {
+	case StringEquals:
+		// namespace:"__unallocated__" should match a.Properties.Namespace = ""
+		if thatString == "" {
+			return sp.Value == unallocatedSuffix
+		}
+
+		if thatString == sp.Value {
+			return true
+		}
+	case StringStartsWith:
+
+		// We don't need special __unallocated__ logic here because a query
+		// asking for "__unallocated__" won't have a wildcard and unallocated
+		// properties are the empty string.
+
+		return strings.HasPrefix(thatString, sp.Value)
+	default:
+		log.Errorf("Filter: StringProperty: Unhandled filter op. This is a filter implementation error and requires immediate patching. Op: %s", sp.Op)
+		return false
+	}
+
+	return false
+}

pkg/filter/stringsliceproperty.go

@@ -0,0 +1,80 @@
+package filter
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/opencost/opencost/pkg/log"
+)
+
+type StringSlicePropertied interface {
+	StringSliceProperty(string) ([]string, error)
+}
+
+// StringSliceOperation is an enum that represents operations that can be performed
+// when filtering (equality, inequality, etc.)
+type StringSliceOperation string
+
+const (
+	// StringSliceContains is an array/slice membership operator
+	// ["a", "b", "c"] FilterContains "a" = true
+	StringSliceContains StringSliceOperation = "stringslicecontains"
+
+	// StringSliceContainsPrefix is like FilterContains, but using StartsWith instead
+	// of Equals.
+	// ["kube-system", "abc123"] ContainsPrefix ["kube"] = true
+	StringSliceContainsPrefix = "stringslicecontainsprefix"
+)
+
+// StringSliceProperty is the lowest-level type of filter. It represents
+// a filter operation (equality, inequality, etc.) on a property that contains a string slice
+type StringSliceProperty[T StringSlicePropertied] struct {
+	Field string
+	Op    StringSliceOperation
+
+	Value string
+}
+
+func (ssp StringSliceProperty[T]) String() string {
+	return fmt.Sprintf(`(%s %s "%s")`, ssp.Op, ssp.Field, ssp.Value)
+}
+
+func (ssp StringSliceProperty[T]) Matches(that T) bool {
+
+	thatSlice, err := that.StringSliceProperty(ssp.Field)
+	if err != nil {
+		log.Errorf("Filter: StringSliceProperty: could not retrieve field %s: %s", ssp.Field, err.Error())
+		return false
+	}
+
+	switch ssp.Op {
+
+	case StringSliceContains:
+		if len(thatSlice) == 0 {
+			return ssp.Value == unallocatedSuffix
+		}
+
+		for _, s := range thatSlice {
+			if s == ssp.Value {
+				return true
+			}
+		}
+	case StringSliceContainsPrefix:
+		// We don't need special __unallocated__ logic here because a query
+		// asking for "__unallocated__" won't have a wildcard and unallocated
+		// properties are the empty string.
+
+		for _, s := range thatSlice {
+			if strings.HasPrefix(s, ssp.Value) {
+				return true
+			}
+		}
+
+		return false
+	default:
+		log.Errorf("Filter: StringSliceProperty: Unhandled filter op. This is a filter implementation error and requires immediate patching. Op: %s", ssp.Op)
+		return false
+	}
+
+	return false
+}

pkg/filter/util/cloudcostaggregate.go

@@ -0,0 +1,70 @@
+package util
+
+import (
+	"strings"
+
+	"github.com/opencost/opencost/pkg/filter"
+	"github.com/opencost/opencost/pkg/kubecost"
+	"github.com/opencost/opencost/pkg/util/mapper"
+)
+
+func parseWildcardEnd(rawFilterValue string) (string, bool) {
+	return strings.TrimSuffix(rawFilterValue, "*"), strings.HasSuffix(rawFilterValue, "*")
+}
+
+func CloudCostAggregateFilterFromParams(pmr mapper.PrimitiveMapReader) filter.Filter[*kubecost.CloudCostAggregate] {
+	filter := filter.And[*kubecost.CloudCostAggregate]{
+		Filters: []filter.Filter[*kubecost.CloudCostAggregate]{},
+	}
+
+	if raw := pmr.GetList("filterAccounts", ","); len(raw) > 0 {
+		filter.Filters = append(filter.Filters, filterV1SingleValueFromList(raw, kubecost.CloudCostAccountProp))
+	}
+
+	if raw := pmr.GetList("filterProjects", ","); len(raw) > 0 {
+		filter.Filters = append(filter.Filters, filterV1SingleValueFromList(raw, kubecost.CloudCostProjectProp))
+	}
+
+	if raw := pmr.GetList("filterProviders", ","); len(raw) > 0 {
+		filter.Filters = append(filter.Filters, filterV1SingleValueFromList(raw, kubecost.CloudCostProviderProp))
+	}
+
+	if raw := pmr.GetList("filterServices", ","); len(raw) > 0 {
+		filter.Filters = append(filter.Filters, filterV1SingleValueFromList(raw, kubecost.CloudCostServiceProp))
+	}
+
+	if raw := pmr.GetList("filterLabelValues", ","); len(raw) > 0 {
+		filter.Filters = append(filter.Filters, filterV1SingleValueFromList(raw, kubecost.CloudCostLabelProp))
+	}
+
+	if len(filter.Filters) == 0 {
+		return nil
+	}
+
+	return filter
+}
+
+func filterV1SingleValueFromList(rawFilterValues []string, field string) filter.Filter[*kubecost.CloudCostAggregate] {
+	result := filter.Or[*kubecost.CloudCostAggregate]{
+		Filters: []filter.Filter[*kubecost.CloudCostAggregate]{},
+	}
+
+	for _, filterValue := range rawFilterValues {
+		filterValue = strings.TrimSpace(filterValue)
+		filterValue, wildcard := parseWildcardEnd(filterValue)
+
+		subFilter := filter.StringProperty[*kubecost.CloudCostAggregate]{
+			Field: field,
+			Op:    filter.StringEquals,
+			Value: filterValue,
+		}
+
+		if wildcard {
+			subFilter.Op = kubecost.FilterStartsWith
+		}
+
+		result.Filters = append(result.Filters, subFilter)
+	}
+
+	return result
+}

pkg/filter/window.go

@@ -0,0 +1,40 @@
+package filter
+
+//
+//import (
+//	"fmt"
+//	"github.com/opencost/opencost/pkg/kubecost"
+//	"github.com/opencost/opencost/pkg/log"
+//)
+//
+//type Windowed interface {
+//	GetWindow() kubecost.Window
+//}
+//
+//// WindowOperation are operations that can be performed on types that have windows
+//type WindowOperation string
+//
+//const (
+//	WindowContains WindowOperation = "windowcontains"
+//)
+//
+//// WindowCondition is a filter can be used on any type that has a window and implements GetWindow()
+//type WindowCondition[T Windowed] struct {
+//	Window kubecost.Window
+//	Op     WindowOperation
+//}
+//
+//func (wc WindowCondition[T]) String() string {
+//	return fmt.Sprintf(`(%s "%s")`, wc.Op, wc.Window.String())
+//}
+//
+//func (wc WindowCondition[T]) Matches(that T) bool {
+//	thatWindow := that.GetWindow()
+//	switch wc.Op {
+//	case WindowContains:
+//		return wc.Window.ContainsWindow(thatWindow)
+//	default:
+//		log.Errorf("Filter: Window: Unhandled filter operation. This is a filter implementation error and requires immediate patching. Op: %s", wc.Op)
+//		return false
+//	}
+//}

pkg/filter/window_test.go

@@ -0,0 +1,112 @@
+package filter_test
+
+// import (
+// 	"github.com/opencost/opencost/pkg/kubecost"
+// 	"testing"
+// 	"time"
+// )
+
+// type windowedImpl struct {
+// 	kubecost.Window
+// }
+
+// func (w *windowedImpl) GetWindow() kubecost.Window {
+// 	return w.Window
+// }
+
+// func newWindowedImpl(start, end *time.Time) *windowedImpl {
+// 	return &windowedImpl{kubecost.NewWindow(start, end)}
+// }
+
+// func Test_WindowContains_Matches(t *testing.T) {
+// 	noon := time.Date(2022, 9, 29, 12, 0, 0, 0, time.UTC)
+// 	one := noon.Add(time.Hour)
+// 	two := one.Add(time.Hour)
+// 	three := two.Add(time.Hour)
+// 	cases := map[string]struct {
+// 		windowed *windowedImpl
+// 		filter   Filter[*windowedImpl]
+// 		expected bool
+// 	}{
+// 		"fully contains": {
+// 			windowed: newWindowedImpl(&one, &two),
+// 			filter: WindowCondition[*windowedImpl]{
+// 				Window: kubecost.NewWindow(&noon, &three),
+// 				Op:     WindowContains,
+// 			},
+
+// 			expected: true,
+// 		},
+// 		"window matches": {
+// 			windowed: newWindowedImpl(&one, &two),
+// 			filter: WindowCondition[*windowedImpl]{
+// 				Window: kubecost.NewWindow(&one, &two),
+// 				Op:     WindowContains,
+// 			},
+
+// 			expected: true,
+// 		},
+// 		"contains start": {
+// 			windowed: newWindowedImpl(&one, &three),
+// 			filter: WindowCondition[*windowedImpl]{
+// 				Window: kubecost.NewWindow(&noon, &two),
+// 				Op:     WindowContains,
+// 			},
+
+// 			expected: false,
+// 		},
+// 		"contains end": {
+// 			windowed: newWindowedImpl(&noon, &two),
+// 			filter: WindowCondition[*windowedImpl]{
+// 				Window: kubecost.NewWindow(&one, &three),
+// 				Op:     WindowContains,
+// 			},
+
+// 			expected: false,
+// 		},
+// 		"window start = filter end": {
+// 			windowed: newWindowedImpl(&one, &two),
+// 			filter: WindowCondition[*windowedImpl]{
+// 				Window: kubecost.NewWindow(&noon, &one),
+// 				Op:     WindowContains,
+// 			},
+
+// 			expected: false,
+// 		},
+// 		"window end = filter start": {
+// 			windowed: newWindowedImpl(&noon, &one),
+// 			filter: WindowCondition[*windowedImpl]{
+// 				Window: kubecost.NewWindow(&one, &two),
+// 				Op:     WindowContains,
+// 			},
+
+// 			expected: false,
+// 		},
+// 		"window before": {
+// 			windowed: newWindowedImpl(&noon, &one),
+// 			filter: WindowCondition[*windowedImpl]{
+// 				Window: kubecost.NewWindow(&two, &three),
+// 				Op:     WindowContains,
+// 			},
+
+// 			expected: false,
+// 		},
+// 		"window after": {
+// 			windowed: newWindowedImpl(&two, &three),
+// 			filter: WindowCondition[*windowedImpl]{
+// 				Window: kubecost.NewWindow(&noon, &one),
+// 				Op:     WindowContains,
+// 			},
+
+// 			expected: false,
+// 		},
+// 	}
+
+// 	for name, c := range cases {
+// 		result := c.filter.Matches(c.windowed)
+
+// 		if result != c.expected {
+// 			t.Errorf("%s: expected %t, got %t", name, c.expected, result)
+// 		}
+// 	}
+// }

pkg/kubecost/allocation_json.go

@@ -0,0 +1,110 @@
+package kubecost
+
+import (
+	"bytes"
+	"encoding/json"
+	"time"
+)
+
+// MarshalJSON implements json.Marshaler interface
+func (a *Allocation) MarshalJSON() ([]byte, error) {
+	buffer := bytes.NewBufferString("{")
+	jsonEncodeString(buffer, "name", a.Name, ",")
+	jsonEncode(buffer, "properties", a.Properties, ",")
+	jsonEncode(buffer, "window", a.Window, ",")
+	jsonEncodeString(buffer, "start", a.Start.Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", a.End.Format(time.RFC3339), ",")
+	jsonEncodeFloat64(buffer, "minutes", a.Minutes(), ",")
+	jsonEncodeFloat64(buffer, "cpuCores", a.CPUCores(), ",")
+	jsonEncodeFloat64(buffer, "cpuCoreRequestAverage", a.CPUCoreRequestAverage, ",")
+	jsonEncodeFloat64(buffer, "cpuCoreUsageAverage", a.CPUCoreUsageAverage, ",")
+	jsonEncodeFloat64(buffer, "cpuCoreHours", a.CPUCoreHours, ",")
+	jsonEncodeFloat64(buffer, "cpuCost", a.CPUCost, ",")
+	jsonEncodeFloat64(buffer, "cpuCostAdjustment", a.CPUCostAdjustment, ",")
+	jsonEncodeFloat64(buffer, "cpuEfficiency", a.CPUEfficiency(), ",")
+	jsonEncodeFloat64(buffer, "gpuCount", a.GPUs(), ",")
+	jsonEncodeFloat64(buffer, "gpuHours", a.GPUHours, ",")
+	jsonEncodeFloat64(buffer, "gpuCost", a.GPUCost, ",")
+	jsonEncodeFloat64(buffer, "gpuCostAdjustment", a.GPUCostAdjustment, ",")
+	jsonEncodeFloat64(buffer, "networkTransferBytes", a.NetworkTransferBytes, ",")
+	jsonEncodeFloat64(buffer, "networkReceiveBytes", a.NetworkReceiveBytes, ",")
+	jsonEncodeFloat64(buffer, "networkCost", a.NetworkCost, ",")
+	jsonEncodeFloat64(buffer, "networkCostAdjustment", a.NetworkCostAdjustment, ",")
+	jsonEncodeFloat64(buffer, "loadBalancerCost", a.LoadBalancerCost, ",")
+	jsonEncodeFloat64(buffer, "loadBalancerCostAdjustment", a.LoadBalancerCostAdjustment, ",")
+	jsonEncodeFloat64(buffer, "pvBytes", a.PVBytes(), ",")
+	jsonEncodeFloat64(buffer, "pvByteHours", a.PVByteHours(), ",")
+	jsonEncodeFloat64(buffer, "pvCost", a.PVCost(), ",")
+	jsonEncode(buffer, "pvs", a.PVs, ",")
+	jsonEncodeFloat64(buffer, "pvCostAdjustment", a.PVCostAdjustment, ",")
+	jsonEncodeFloat64(buffer, "ramBytes", a.RAMBytes(), ",")
+	jsonEncodeFloat64(buffer, "ramByteRequestAverage", a.RAMBytesRequestAverage, ",")
+	jsonEncodeFloat64(buffer, "ramByteUsageAverage", a.RAMBytesUsageAverage, ",")
+	jsonEncodeFloat64(buffer, "ramByteHours", a.RAMByteHours, ",")
+	jsonEncodeFloat64(buffer, "ramCost", a.RAMCost, ",")
+	jsonEncodeFloat64(buffer, "ramCostAdjustment", a.RAMCostAdjustment, ",")
+	jsonEncodeFloat64(buffer, "ramEfficiency", a.RAMEfficiency(), ",")
+	jsonEncodeFloat64(buffer, "sharedCost", a.SharedCost, ",")
+	jsonEncodeFloat64(buffer, "externalCost", a.ExternalCost, ",")
+	jsonEncodeFloat64(buffer, "totalCost", a.TotalCost(), ",")
+	jsonEncodeFloat64(buffer, "totalEfficiency", a.TotalEfficiency(), ",")
+	jsonEncode(buffer, "rawAllocationOnly", a.RawAllocationOnly, "")
+	buffer.WriteString("}")
+	return buffer.Bytes(), nil
+}
+
+// UnmarshalJSON prevent nil pointer on PVAllocations
+func (a *Allocation) UnmarshalJSON(b []byte) error {
+	// initialize PV to prevent nil panic
+	a.PVs = PVAllocations{}
+	// Aliasing Allocation and casting to alias gives access to the default unmarshaller
+	type alloc Allocation
+	err := json.Unmarshal(b, (*alloc)(a))
+	if err != nil {
+		return err
+	}
+	// clear PVs if they are empty, it is not initialized when empty
+	if len(a.PVs) == 0 {
+		a.PVs = nil
+	}
+	return nil
+}
+
+// MarshalJSON marshals PVAllocation as map[*PVKey]*PVAllocation this allows PVKey to retain its values through marshalling
+func (pv PVAllocations) MarshalJSON() (b []byte, err error) {
+	pointerMap := make(map[*PVKey]*PVAllocation)
+	for pvKey, pvAlloc := range pv {
+		kp := pvKey
+		pointerMap[&kp] = pvAlloc
+	}
+	return json.Marshal(pointerMap)
+}
+
+// MarshalText converts PVKey to string to make it compatible with JSON Marshaller as an Object key
+// this function is required to have a value caller for the actual values to be saved
+func (pvk PVKey) MarshalText() (text []byte, err error) {
+	return []byte(pvk.String()), nil
+}
+
+// UnmarshalText converts JSON key string to PVKey it compatible with JSON Unmarshaller from an Object key
+// this function is required to have a pointer caller for values to be pulled into marshalling struct
+func (pvk *PVKey) UnmarshalText(text []byte) error {
+	return pvk.FromString(string(text))
+}
+
+// MarshalJSON JSON-encodes the AllocationSet
+func (as *AllocationSet) MarshalJSON() ([]byte, error) {
+	if as == nil {
+		return json.Marshal(map[string]*Allocation{})
+	}
+	return json.Marshal(as.Allocations)
+}
+
+// MarshalJSON JSON-encodes the range
+func (asr *AllocationSetRange) MarshalJSON() ([]byte, error) {
+	if asr == nil {
+		return json.Marshal([]*AllocationSet{})
+	}
+
+	return json.Marshal(asr.Allocations)
+}

pkg/kubecost/allocation_json_test.go

@@ -0,0 +1,154 @@
+package kubecost
+
+import (
+	"encoding/json"
+	"testing"
+	"time"
+)
+
+func TestAllocation_MarshalJSON(t *testing.T) {
+	start := time.Date(2021, time.January, 1, 0, 0, 0, 0, time.UTC)
+	end := time.Date(2021, time.January, 2, 0, 0, 0, 0, time.UTC)
+	hrs := 24.0
+
+	gib := 1024.0 * 1024.0 * 1024.0
+
+	cpuPrice := 0.02
+	gpuPrice := 2.00
+	ramPrice := 0.01
+	pvPrice := 0.00005
+
+	before := &Allocation{
+		Name: "cluster1/namespace1/node1/pod1/container1",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Node:      "node1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container1",
+		},
+		Window:                NewWindow(&start, &end),
+		Start:                 start,
+		End:                   end,
+		CPUCoreHours:          2.0 * hrs,
+		CPUCoreRequestAverage: 2.0,
+		CPUCoreUsageAverage:   1.0,
+		CPUCost:               2.0 * hrs * cpuPrice,
+		CPUCostAdjustment:     3.0,
+		GPUHours:              1.0 * hrs,
+		GPUCost:               1.0 * hrs * gpuPrice,
+		GPUCostAdjustment:     2.0,
+		NetworkCost:           0.05,
+		LoadBalancerCost:      0.02,
+		PVs: PVAllocations{
+			disk: {
+				ByteHours: 100.0 * gib * hrs,
+				Cost:      100.0 * hrs * pvPrice,
+			},
+		},
+		PVCostAdjustment:       4.0,
+		RAMByteHours:           8.0 * gib * hrs,
+		RAMBytesRequestAverage: 8.0 * gib,
+		RAMBytesUsageAverage:   4.0 * gib,
+		RAMCost:                8.0 * hrs * ramPrice,
+		RAMCostAdjustment:      1.0,
+		SharedCost:             2.00,
+		ExternalCost:           1.00,
+		RawAllocationOnly:      &RawAllocationOnlyData{},
+	}
+
+	data, err := json.Marshal(before)
+	if err != nil {
+		t.Fatalf("Allocation.MarshalJSON: unexpected error: %s", err)
+	}
+
+	after := &Allocation{}
+	err = json.Unmarshal(data, after)
+	if err != nil {
+		t.Fatalf("Allocation.UnmarshalJSON: unexpected error: %s", err)
+	}
+
+	// TODO:CLEANUP fix json marshaling of Window so that all of this works.
+	// In the meantime, just set the Window so that we can test the rest.
+	after.Window = before.Window.Clone()
+	if !after.Equal(before) {
+		t.Fatalf("Allocation.MarshalJSON: before and after are not equal")
+	}
+}
+
+func TestPVAllocations_MarshalJSON(t *testing.T) {
+	testCases := map[string]PVAllocations{
+		"empty": {},
+		"single": {
+			{
+				Cluster: "cluster1",
+				Name:    "pv1",
+			}: {
+				ByteHours: 100,
+				Cost:      1,
+			},
+		},
+		"multi": {
+			{
+				Cluster: "cluster1",
+				Name:    "pv1",
+			}: {
+				ByteHours: 100,
+				Cost:      1,
+			},
+			{
+				Cluster: "cluster1",
+				Name:    "pv2",
+			}: {
+				ByteHours: 200,
+				Cost:      2,
+			},
+		},
+		"emptyPV": {
+			{
+				Cluster: "cluster1",
+				Name:    "pv1",
+			}: {},
+		},
+		"emptyKey": {
+			{}: {
+				ByteHours: 100,
+				Cost:      1,
+			},
+		},
+	}
+	for name, before := range testCases {
+		t.Run(name, func(t *testing.T) {
+			data, err := json.Marshal(before)
+			if err != nil {
+				t.Fatalf("PVAllocations.MarshalJSON: unexpected error: %s", err)
+			}
+
+			after := PVAllocations{}
+			err = json.Unmarshal(data, &after)
+			if err != nil {
+				t.Fatalf("PVAllocations.UnmarshalJSON: unexpected error: %s", err)
+			}
+
+			if len(before) != len(after) {
+				t.Fatalf("PVAllocations.MarshalJSON: before and after are not equal")
+			}
+
+			for pvKey, beforePV := range before {
+				afterPV, ok := after[pvKey]
+				if !ok {
+					t.Fatalf("PVAllocations.MarshalJSON: after missing PVKey %s", pvKey)
+				}
+				if beforePV.Cost != afterPV.Cost {
+					t.Fatalf("PVAllocations.MarshalJSON: PVAllocation Cost not equal for PVKey %s", pvKey)
+				}
+
+				if beforePV.ByteHours != afterPV.ByteHours {
+					t.Fatalf("PVAllocations.MarshalJSON: PVAllocation ByteHours not equal for PVKey %s", pvKey)
+				}
+			}
+
+		})
+	}
+
+}

pkg/kubecost/allocation_test.go

@@ -379,78 +379,6 @@ func TestAllocation_AddDifferentController(t *testing.T) {
 
 }
 
-func TestAllocation_MarshalJSON(t *testing.T) {
-	start := time.Date(2021, time.January, 1, 0, 0, 0, 0, time.UTC)
-	end := time.Date(2021, time.January, 2, 0, 0, 0, 0, time.UTC)
-	hrs := 24.0
-
-	gib := 1024.0 * 1024.0 * 1024.0
-
-	cpuPrice := 0.02
-	gpuPrice := 2.00
-	ramPrice := 0.01
-	pvPrice := 0.00005
-
-	before := &Allocation{
-		Name: "cluster1/namespace1/node1/pod1/container1",
-		Properties: &AllocationProperties{
-			Cluster:   "cluster1",
-			Node:      "node1",
-			Namespace: "namespace1",
-			Pod:       "pod1",
-			Container: "container1",
-		},
-		Window:                NewWindow(&start, &end),
-		Start:                 start,
-		End:                   end,
-		CPUCoreHours:          2.0 * hrs,
-		CPUCoreRequestAverage: 2.0,
-		CPUCoreUsageAverage:   1.0,
-		CPUCost:               2.0 * hrs * cpuPrice,
-		CPUCostAdjustment:     3.0,
-		GPUHours:              1.0 * hrs,
-		GPUCost:               1.0 * hrs * gpuPrice,
-		GPUCostAdjustment:     2.0,
-		NetworkCost:           0.05,
-		LoadBalancerCost:      0.02,
-		PVs: PVAllocations{
-			disk: {
-				ByteHours: 100.0 * gib * hrs,
-				Cost:      100.0 * hrs * pvPrice,
-			},
-		},
-		PVCostAdjustment:       4.0,
-		RAMByteHours:           8.0 * gib * hrs,
-		RAMBytesRequestAverage: 8.0 * gib,
-		RAMBytesUsageAverage:   4.0 * gib,
-		RAMCost:                8.0 * hrs * ramPrice,
-		RAMCostAdjustment:      1.0,
-		SharedCost:             2.00,
-		ExternalCost:           1.00,
-		RawAllocationOnly:      &RawAllocationOnlyData{},
-	}
-
-	data, err := json.Marshal(before)
-	if err != nil {
-		t.Fatalf("Allocation.MarshalJSON: unexpected error: %s", err)
-	}
-
-	after := &Allocation{}
-	err = json.Unmarshal(data, after)
-	if err != nil {
-		t.Fatalf("Allocation.UnmarshalJSON: unexpected error: %s", err)
-	}
-
-	// TODO:CLEANUP fix json marshaling of Window so that all of this works.
-	// In the meantime, just set the Window so that we can test the rest.
-	after.Window = before.Window.Clone()
-	// TODO Sean: fix JSON marshaling of PVs
-	after.PVs = before.PVs
-	if !after.Equal(before) {
-		t.Fatalf("Allocation.MarshalJSON: before and after are not equal")
-	}
-}
-
 func TestAllocationSet_generateKey(t *testing.T) {
 	var alloc *Allocation
 	var key string
@@ -582,7 +510,7 @@ func assertAllocationSetTotals(t *testing.T, as *AllocationSet, msg string, err
 }
 
 func assertAllocationTotals(t *testing.T, as *AllocationSet, msg string, exps map[string]float64) {
-	as.Each(func(k string, a *Allocation) {
+	for _, a := range as.Allocations {
 		if exp, ok := exps[a.Name]; ok {
 			if math.Round(a.TotalCost()*100) != math.Round(exp*100) {
 				t.Fatalf("AllocationSet.AggregateBy[%s]: expected total cost %f, actual %f", msg, exp, a.TotalCost())
@@ -590,11 +518,11 @@ func assertAllocationTotals(t *testing.T, as *AllocationSet, msg string, exps ma
 		} else {
 			t.Fatalf("AllocationSet.AggregateBy[%s]: unexpected allocation: %s", msg, a.Name)
 		}
-	})
+	}
 }
 
 func assertAllocationWindow(t *testing.T, as *AllocationSet, msg string, expStart, expEnd time.Time, expMinutes float64) {
-	as.Each(func(k string, a *Allocation) {
+	for _, a := range as.Allocations {
 		if !a.Start.Equal(expStart) {
 			t.Fatalf("AllocationSet.AggregateBy[%s]: expected start %s, actual %s", msg, expStart, a.Start)
 		}
@@ -604,14 +532,14 @@ func assertAllocationWindow(t *testing.T, as *AllocationSet, msg string, expStar
 		if a.Minutes() != expMinutes {
 			t.Fatalf("AllocationSet.AggregateBy[%s]: expected minutes %f, actual %f", msg, expMinutes, a.Minutes())
 		}
-	})
+	}
 }
 
 func printAllocationSet(msg string, as *AllocationSet) {
 	fmt.Printf("--- %s ---\n", msg)
-	as.Each(func(k string, a *Allocation) {
+	for _, a := range as.Allocations {
 		fmt.Printf(" > %s\n", a)
-	})
+	}
 }
 
 func TestAllocationSet_AggregateBy(t *testing.T) {
@@ -1639,21 +1567,21 @@ func TestAllocationSet_insertMatchingWindow(t *testing.T) {
 	}
 
 	as := NewAllocationSet(setStart, setEnd)
-	as.insert(a1)
-	as.insert(a2)
+	as.Insert(a1)
+	as.Insert(a2)
 
 	if as.Length() != 2 {
 		t.Errorf("AS length got %d, expected %d", as.Length(), 2)
 	}
 
-	as.Each(func(k string, a *Allocation) {
+	for _, a := range as.Allocations {
 		if !(*a.Window.Start()).Equal(setStart) {
 			t.Errorf("Allocation %s window start is %s, expected %s", a.Name, *a.Window.Start(), setStart)
 		}
 		if !(*a.Window.End()).Equal(setEnd) {
 			t.Errorf("Allocation %s window end is %s, expected %s", a.Name, *a.Window.End(), setEnd)
 		}
-	})
+	}
 }
 
 // TODO niko/etl
@@ -1686,6 +1614,48 @@ func TestAllocationSet_insertMatchingWindow(t *testing.T) {
 // TODO niko/etl
 //func TestNewAllocationSetRange(t *testing.T) {}
 
+func TestAllocationSetRange_AccumulateRepeat(t *testing.T) {
+	ago2d := time.Now().UTC().Truncate(day).Add(-2 * day)
+	yesterday := time.Now().UTC().Truncate(day).Add(-day)
+	today := time.Now().UTC().Truncate(day)
+	tomorrow := time.Now().UTC().Truncate(day).Add(day)
+
+	a := GenerateMockAllocationSet(ago2d)
+	b := GenerateMockAllocationSet(yesterday)
+	c := GenerateMockAllocationSet(today)
+	d := GenerateMockAllocationSet(tomorrow)
+
+	asr := NewAllocationSetRange(a, b, c, d)
+
+	// Take Total Cost
+	totalCost := asr.TotalCost()
+
+	// NewAccumulation does not mutate
+	result, err := asr.NewAccumulation()
+	if err != nil {
+		t.Fatal(err)
+	}
+	asr2 := NewAllocationSetRange(result)
+
+	// Ensure Costs Match
+	if totalCost != asr2.TotalCost() {
+		t.Fatalf("Accumulated Total Cost does not match original Total Cost")
+	}
+
+	// Next NewAccumulation() call should prove that there is no mutation of inner data
+	result, err = asr.NewAccumulation()
+	if err != nil {
+		t.Fatal(err)
+	}
+	asr3 := NewAllocationSetRange(result)
+
+	// Costs should be correct, as multiple calls to NewAccumulation() should not alter
+	// the internals of the AllocationSetRange
+	if totalCost != asr3.TotalCost() {
+		t.Fatalf("Accumulated Total Cost does not match original Total Cost. %f != %f", totalCost, asr3.TotalCost())
+	}
+}
+
 func TestAllocationSetRange_Accumulate(t *testing.T) {
 	ago2d := time.Now().UTC().Truncate(day).Add(-2 * day)
 	yesterday := time.Now().UTC().Truncate(day).Add(-day)
@@ -1776,7 +1746,7 @@ func TestAllocationSetRange_Accumulate(t *testing.T) {
 	if result.TotalCost() != 12.0 {
 		t.Fatalf("accumulating AllocationSetRange: expected total cost 12.0; actual %f", result.TotalCost())
 	}
-	allocMap := result.Map()
+	allocMap := result.Allocations
 	if len(allocMap) != 1 {
 		t.Fatalf("accumulating AllocationSetRange: expected length 1; actual length %d", len(allocMap))
 	}
@@ -1874,7 +1844,7 @@ func TestAllocationSetRange_AccumulateBy_Nils(t *testing.T) {
 
 	for _, c := range nilEmptycases {
 		result, err = c.asr.AccumulateBy(c.resolution)
-		for _, as := range result.allocations {
+		for _, as := range result.Allocations {
 			if !as.IsEmpty() {
 				t.Errorf("accumulating nil AllocationSetRange: expected empty; actual %s; TestId: %s", result, c.testId)
 			}
@@ -1927,7 +1897,7 @@ func TestAllocationSetRange_AccumulateBy_Nils(t *testing.T) {
 			t.Errorf("accumulating AllocationSetRange: expected AllocationSet; actual %s; TestId: %s", result, c.testId)
 		}
 
-		for _, as := range result.allocations {
+		for _, as := range result.Allocations {
 			sumCost += as.TotalCost()
 		}
 
@@ -2096,7 +2066,7 @@ func TestAllocationSetRange_AccumulateBy(t *testing.T) {
 			t.Errorf("accumulating AllocationSetRange: expected %v number of allocation sets; actual %v; TestId: %s", c.expectedSets, result.Length(), c.testId)
 		}
 
-		for _, as := range result.allocations {
+		for _, as := range result.Allocations {
 			sumCost += as.TotalCost()
 		}
 		if sumCost != c.expectedCost {
@@ -2118,11 +2088,11 @@ func TestAllocationSetRange_AccumulateBy(t *testing.T) {
 	}
 
 	sumCost = 0.0
-	for _, as := range result.allocations {
+	for _, as := range result.Allocations {
 		sumCost += as.TotalCost()
 	}
 
-	allocMap := result.allocations[0].Map()
+	allocMap := result.Allocations[0].Allocations
 	if len(allocMap) != 1 {
 		t.Errorf("accumulating AllocationSetRange: expected length 1; actual length %d", len(allocMap))
 	}
@@ -2236,8 +2206,8 @@ func TestAllocationSetRange_InsertRange(t *testing.T) {
 	if err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}
-	thisASR.Each(func(i int, as *AllocationSet) {
-		as.Each(func(k string, a *Allocation) {
+	for _, as := range thisASR.Allocations {
+		for k, a := range as.Allocations {
 			if !util.IsApproximately(a.CPUCoreHours, unit.CPUCoreHours) {
 				t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCoreHours, a.CPUCoreHours)
 			}
@@ -2271,8 +2241,8 @@ func TestAllocationSetRange_InsertRange(t *testing.T) {
 			if !util.IsApproximately(a.TotalCost(), unit.TotalCost()) {
 				t.Fatalf("allocation %s: expected %f; got %f", k, unit.TotalCost(), a.TotalCost())
 			}
-		})
-	})
+		}
+	}
 
 	// Expect an error calling InsertRange with a range exceeding the receiver
 	err = thisASR.InsertRange(longASR)
@@ -2288,7 +2258,7 @@ func TestAllocationSetRange_InsertRange(t *testing.T) {
 		t.Fatalf("unexpected error: %s", err)
 	}
 	yAS, err := thisASR.Get(0)
-	yAS.Each(func(k string, a *Allocation) {
+	for k, a := range yAS.Allocations {
 		if !util.IsApproximately(a.CPUCoreHours, 2*unit.CPUCoreHours) {
 			t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCoreHours, a.CPUCoreHours)
 		}
@@ -2323,9 +2293,9 @@ func TestAllocationSetRange_InsertRange(t *testing.T) {
 		if !util.IsApproximately(a.TotalCost(), 2*unit.TotalCost()) {
 			t.Fatalf("allocation %s: expected %f; got %f", k, unit.TotalCost(), a.TotalCost())
 		}
-	})
+	}
 	tAS, err := thisASR.Get(1)
-	tAS.Each(func(k string, a *Allocation) {
+	for k, a := range tAS.Allocations {
 		if !util.IsApproximately(a.CPUCoreHours, unit.CPUCoreHours) {
 			t.Fatalf("allocation %s: expected %f; got %f", k, unit.CPUCoreHours, a.CPUCoreHours)
 		}
@@ -2359,7 +2329,7 @@ func TestAllocationSetRange_InsertRange(t *testing.T) {
 		if !util.IsApproximately(a.TotalCost(), unit.TotalCost()) {
 			t.Fatalf("allocation %s: expected %f; got %f", k, unit.TotalCost(), a.TotalCost())
 		}
-	})
+	}
 }
 
 // TODO niko/etl
@@ -2383,9 +2353,9 @@ func TestAllocationSetRange_MarshalJSON(t *testing.T) {
 		{
 			name: "Normal ASR",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								Start: time.Now().UTC().Truncate(day),
 							},
@@ -2443,9 +2413,9 @@ func TestAllocationSetRange_Start(t *testing.T) {
 		{
 			name: "Single allocation",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
@@ -2459,9 +2429,9 @@ func TestAllocationSetRange_Start(t *testing.T) {
 		{
 			name: "Two allocations",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
@@ -2478,16 +2448,16 @@ func TestAllocationSetRange_Start(t *testing.T) {
 		{
 			name: "Two AllocationSets",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"b": {
 								Start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
@@ -2531,9 +2501,9 @@ func TestAllocationSetRange_End(t *testing.T) {
 		{
 			name: "Single allocation",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								End: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
@@ -2547,9 +2517,9 @@ func TestAllocationSetRange_End(t *testing.T) {
 		{
 			name: "Two allocations",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								End: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
@@ -2566,16 +2536,16 @@ func TestAllocationSetRange_End(t *testing.T) {
 		{
 			name: "Two AllocationSets",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								End: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"b": {
 								End: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
@@ -2618,9 +2588,9 @@ func TestAllocationSetRange_Minutes(t *testing.T) {
 		{
 			name: "Single allocation",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 								End:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
@@ -2635,9 +2605,9 @@ func TestAllocationSetRange_Minutes(t *testing.T) {
 		{
 			name: "Two allocations",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 								End:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
@@ -2656,9 +2626,9 @@ func TestAllocationSetRange_Minutes(t *testing.T) {
 		{
 			name: "Two AllocationSets",
 			arg: &AllocationSetRange{
-				allocations: []*AllocationSet{
+				Allocations: []*AllocationSet{
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"a": {
 								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 								End:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
@@ -2666,7 +2636,7 @@ func TestAllocationSetRange_Minutes(t *testing.T) {
 						},
 					},
 					{
-						allocations: map[string]*Allocation{
+						Allocations: map[string]*Allocation{
 							"b": {
 								Start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 								End:   time.Date(1970, 1, 3, 0, 0, 0, 0, time.UTC),
@@ -2687,3 +2657,78 @@ func TestAllocationSetRange_Minutes(t *testing.T) {
 		}
 	}
 }
+
+func TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate(t *testing.T) {
+
+	today := time.Now().Round(day)
+	start := today.AddDate(0, 0, -4)
+
+	var allocationSets []*AllocationSet
+
+	for i := 0; i < 4; i++ {
+		allocationSets = append(allocationSets, GenerateMockAllocationSet(start))
+		start = start.AddDate(0, 0, 1)
+	}
+
+	var originalAllocationSets []*AllocationSet
+
+	for _, as := range allocationSets {
+		originalAllocationSets = append(originalAllocationSets, as.Clone())
+	}
+
+	asr := NewAllocationSetRange()
+	for _, as := range allocationSets {
+		asr.Append(as.Clone())
+	}
+
+	expected, err := asr.Accumulate()
+	if err != nil {
+		t.Errorf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: AllocationSetRange.Accumulate() returned an error\n")
+	}
+
+	var got *AllocationSet
+
+	for i := 0; i < len(allocationSets); i++ {
+		got, err = got.Accumulate(allocationSets[i])
+		if err != nil {
+			t.Errorf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: got.Accumulate(allocationSets[%d]) returned an error\n", i)
+		}
+	}
+
+	// compare the got and expected Allocation sets, ensure that they match
+	if len(got.Allocations) != len(expected.Allocations) {
+		t.Fatalf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: length of got.Allocations does not match length of expected.Allocations\n")
+	}
+	for key, a := range got.Allocations {
+		if _, ok := expected.Allocations[key]; !ok {
+			t.Fatalf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: got.Allocations[%s] not found in expected.Allocations\n", key)
+		}
+
+		if !a.Equal(expected.Allocations[key]) {
+			t.Fatalf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: got.Allocations[%s] did not match expected.Allocations[%[1]s]", key)
+		}
+	}
+
+	if len(got.ExternalKeys) != len(expected.ExternalKeys) {
+		t.Fatalf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: length of got.ExternalKeys does not match length of expected.ExternalKeys\n")
+	}
+
+	if len(got.IdleKeys) != len(expected.IdleKeys) {
+		t.Fatalf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: length of got.IdleKeys does not match length of expected.IdleKeys\n")
+	}
+
+	if !got.Window.Start().UTC().Equal(expected.Window.Start().UTC()) {
+		t.Fatalf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: Window.start: got:%s, expected:%s\n", got.Window.Start(), expected.Window.Start())
+	}
+	if !got.Window.End().UTC().Equal(expected.Window.End().UTC()) {
+		t.Fatalf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: Window.end: got:%s, expected:%s\n", got.Window.End(), expected.Window.End())
+	}
+
+	for i := range allocationSets {
+		for key, allocation := range allocationSets[i].Allocations {
+			if !allocation.Equal(originalAllocationSets[i].Allocations[key]) {
+				t.Fatalf("TestAllocationSet_Accumulate_Equals_AllocationSetRange_Accumulate: allocationSet has been mutated in Accumulate; allocationSet: %d, allocation: %s\n", i, key)
+			}
+		}
+	}
+}

pkg/kubecost/allocationfilter.go

@@ -30,6 +30,7 @@ const (
 
 	FilterLabel      = "label"
 	FilterAnnotation = "annotation"
+	FilterAlias      = "alias"
 
 	FilterServices = "services"
 )
@@ -362,6 +363,15 @@ func (filter AllocationFilterCondition) Matches(a *Allocation) bool {
 		} else {
 			valueToCompare = val
 		}
+	case FilterAlias:
+		var ok bool
+		valueToCompare, ok = a.Properties.Labels[filter.Key]
+		if !ok {
+			valueToCompare, ok = a.Properties.Annotations[filter.Key]
+			if !ok {
+				toCompareMissing = true
+			}
+		}
 	case FilterServices:
 		valueToCompare = a.Properties.Services
 	default:

pkg/kubecost/allocationprops_test.go

@@ -7,9 +7,13 @@ import (
 
 func TestGenerateKey(t *testing.T) {
 
+	customOwnerLabelConfig := NewLabelConfig()
+	customOwnerLabelConfig.OwnerLabel = "example_com_project"
+
 	cases := map[string]struct {
 		aggregate       []string
 		allocationProps *AllocationProperties
+		labelConfig     *LabelConfig
 		expected        string
 	}{
 		"aggregate by owner without owner labels": {
@@ -75,6 +79,15 @@ func TestGenerateKey(t *testing.T) {
 			},
 			expected: "product-label/owner-label",
 		},
+		"user test": {
+			aggregate: []string{"owner"},
+			allocationProps: &AllocationProperties{
+				Labels:      map[string]string{"app_kubernetes_io_name": "x-mongo", "example_com_service_owner": "x", "component": "primary", "controller_revision_hash": "x-mongo-primary-x", "kubernetes_io_metadata_name": "app-microservices", "name": "app-microservices", "statefulset_kubernetes_io_pod_name": "x-mongo-primary-0"},
+				Annotations: map[string]string{"example_com_project": "redacted"},
+			},
+			labelConfig: customOwnerLabelConfig,
+			expected:    "redacted",
+		},
 	}
 
 	for name, tc := range cases {
@@ -82,6 +95,10 @@ func TestGenerateKey(t *testing.T) {
 
 			lc := NewLabelConfig()
 
+			if tc.labelConfig != nil {
+				lc = tc.labelConfig
+			}
+
 			result := tc.allocationProps.GenerateKey(tc.aggregate, lc)
 
 			if !reflect.DeepEqual(result, tc.expected) {

pkg/kubecost/asset_unmarshal.go → pkg/kubecost/asset_json.go

@@ -16,13 +16,13 @@ import (
 // MarshalJSON implements json.Marshaler
 func (a *Any) MarshalJSON() ([]byte, error) {
 	buffer := bytes.NewBufferString("{")
-	jsonEncode(buffer, "properties", a.Properties(), ",")
-	jsonEncode(buffer, "labels", a.Labels(), ",")
-	jsonEncode(buffer, "window", a.Window(), ",")
-	jsonEncodeString(buffer, "start", a.Start().Format(time.RFC3339), ",")
-	jsonEncodeString(buffer, "end", a.End().Format(time.RFC3339), ",")
+	jsonEncode(buffer, "properties", a.Properties, ",")
+	jsonEncode(buffer, "labels", a.Labels, ",")
+	jsonEncode(buffer, "window", a.Window, ",")
+	jsonEncodeString(buffer, "start", a.Start.Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", a.End.Format(time.RFC3339), ",")
 	jsonEncodeFloat64(buffer, "minutes", a.Minutes(), ",")
-	jsonEncodeFloat64(buffer, "adjustment", a.Adjustment(), ",")
+	jsonEncodeFloat64(buffer, "adjustment", a.Adjustment, ",")
 	jsonEncodeFloat64(buffer, "totalCost", a.TotalCost(), "")
 	buffer.WriteString("}")
 	return buffer.Bytes(), nil
@@ -70,20 +70,20 @@ func (a *Any) InterfaceToAny(itf interface{}) error {
 		return err
 	}
 
-	a.properties = &properties
-	a.labels = labels
-	a.start = start
-	a.end = end
-	a.window = Window{
+	a.Properties = &properties
+	a.Labels = labels
+	a.Start = start
+	a.End = end
+	a.Window = Window{
 		start: &start,
 		end:   &end,
 	}
 
 	if adjustment, err := getTypedVal(fmap["adjustment"]); err == nil {
-		a.adjustment = adjustment.(float64)
+		a.Adjustment = adjustment.(float64)
 	}
 	if Cost, err := getTypedVal(fmap["totalCost"]); err == nil {
-		a.Cost = Cost.(float64) - a.adjustment
+		a.Cost = Cost.(float64) - a.Adjustment
 	}
 
 	return nil
@@ -95,13 +95,13 @@ func (a *Any) InterfaceToAny(itf interface{}) error {
 func (ca *Cloud) MarshalJSON() ([]byte, error) {
 	buffer := bytes.NewBufferString("{")
 	jsonEncodeString(buffer, "type", ca.Type().String(), ",")
-	jsonEncode(buffer, "properties", ca.Properties(), ",")
-	jsonEncode(buffer, "labels", ca.Labels(), ",")
-	jsonEncode(buffer, "window", ca.Window(), ",")
-	jsonEncodeString(buffer, "start", ca.Start().Format(time.RFC3339), ",")
-	jsonEncodeString(buffer, "end", ca.End().Format(time.RFC3339), ",")
+	jsonEncode(buffer, "properties", ca.Properties, ",")
+	jsonEncode(buffer, "labels", ca.Labels, ",")
+	jsonEncode(buffer, "window", ca.Window, ",")
+	jsonEncodeString(buffer, "start", ca.Start.Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", ca.End.Format(time.RFC3339), ",")
 	jsonEncodeFloat64(buffer, "minutes", ca.Minutes(), ",")
-	jsonEncodeFloat64(buffer, "adjustment", ca.Adjustment(), ",")
+	jsonEncodeFloat64(buffer, "adjustment", ca.Adjustment, ",")
 	jsonEncodeFloat64(buffer, "credit", ca.Credit, ",")
 	jsonEncodeFloat64(buffer, "totalCost", ca.TotalCost(), "")
 	buffer.WriteString("}")
@@ -150,23 +150,23 @@ func (ca *Cloud) InterfaceToCloud(itf interface{}) error {
 		return err
 	}
 
-	ca.properties = &properties
-	ca.labels = labels
-	ca.start = start
-	ca.end = end
-	ca.window = Window{
+	ca.Properties = &properties
+	ca.Labels = labels
+	ca.Start = start
+	ca.End = end
+	ca.Window = Window{
 		start: &start,
 		end:   &end,
 	}
 
 	if adjustment, err := getTypedVal(fmap["adjustment"]); err == nil {
-		ca.adjustment = adjustment.(float64)
+		ca.Adjustment = adjustment.(float64)
 	}
 	if Credit, err := getTypedVal(fmap["credit"]); err == nil {
 		ca.Credit = Credit.(float64)
 	}
 	if Cost, err := getTypedVal(fmap["totalCost"]); err == nil {
-		ca.Cost = Cost.(float64) - ca.adjustment - ca.Credit
+		ca.Cost = Cost.(float64) - ca.Adjustment - ca.Credit
 	}
 
 	return nil
@@ -178,11 +178,11 @@ func (ca *Cloud) InterfaceToCloud(itf interface{}) error {
 func (cm *ClusterManagement) MarshalJSON() ([]byte, error) {
 	buffer := bytes.NewBufferString("{")
 	jsonEncodeString(buffer, "type", cm.Type().String(), ",")
-	jsonEncode(buffer, "properties", cm.Properties(), ",")
-	jsonEncode(buffer, "labels", cm.Labels(), ",")
-	jsonEncode(buffer, "window", cm.Window(), ",")
-	jsonEncodeString(buffer, "start", cm.Start().Format(time.RFC3339), ",")
-	jsonEncodeString(buffer, "end", cm.End().Format(time.RFC3339), ",")
+	jsonEncode(buffer, "properties", cm.Properties, ",")
+	jsonEncode(buffer, "labels", cm.Labels, ",")
+	jsonEncode(buffer, "window", cm.Window, ",")
+	jsonEncodeString(buffer, "start", cm.GetStart().Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", cm.GetEnd().Format(time.RFC3339), ",")
 	jsonEncodeFloat64(buffer, "minutes", cm.Minutes(), ",")
 	jsonEncodeFloat64(buffer, "totalCost", cm.TotalCost(), "")
 	buffer.WriteString("}")
@@ -231,9 +231,9 @@ func (cm *ClusterManagement) InterfaceToClusterManagement(itf interface{}) error
 		return err
 	}
 
-	cm.properties = &properties
-	cm.labels = labels
-	cm.window = Window{
+	cm.Properties = &properties
+	cm.Labels = labels
+	cm.Window = Window{
 		start: &start,
 		end:   &end,
 	}
@@ -251,17 +251,31 @@ func (cm *ClusterManagement) InterfaceToClusterManagement(itf interface{}) error
 func (d *Disk) MarshalJSON() ([]byte, error) {
 	buffer := bytes.NewBufferString("{")
 	jsonEncodeString(buffer, "type", d.Type().String(), ",")
-	jsonEncode(buffer, "properties", d.Properties(), ",")
-	jsonEncode(buffer, "labels", d.Labels(), ",")
-	jsonEncode(buffer, "window", d.Window(), ",")
-	jsonEncodeString(buffer, "start", d.Start().Format(time.RFC3339), ",")
-	jsonEncodeString(buffer, "end", d.End().Format(time.RFC3339), ",")
+	jsonEncode(buffer, "properties", d.Properties, ",")
+	jsonEncode(buffer, "labels", d.Labels, ",")
+	jsonEncode(buffer, "window", d.Window, ",")
+	jsonEncodeString(buffer, "start", d.Start.Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", d.End.Format(time.RFC3339), ",")
 	jsonEncodeFloat64(buffer, "minutes", d.Minutes(), ",")
 	jsonEncodeFloat64(buffer, "byteHours", d.ByteHours, ",")
 	jsonEncodeFloat64(buffer, "bytes", d.Bytes(), ",")
+	if d.ByteHoursUsed == nil {
+		jsonEncode(buffer, "byteHoursUsed", nil, ",")
+	} else {
+		jsonEncodeFloat64(buffer, "byteHoursUsed", *d.ByteHoursUsed, ",")
+	}
+	if d.ByteUsageMax == nil {
+		jsonEncode(buffer, "byteUsageMax", nil, ",")
+	} else {
+		jsonEncodeFloat64(buffer, "byteUsageMax", *d.ByteUsageMax, ",")
+	}
 	jsonEncode(buffer, "breakdown", d.Breakdown, ",")
-	jsonEncodeFloat64(buffer, "adjustment", d.Adjustment(), ",")
-	jsonEncodeFloat64(buffer, "totalCost", d.TotalCost(), "")
+	jsonEncodeFloat64(buffer, "adjustment", d.Adjustment, ",")
+	jsonEncodeFloat64(buffer, "totalCost", d.TotalCost(), ",")
+	jsonEncodeString(buffer, "storageClass", d.StorageClass, ",")
+	jsonEncodeString(buffer, "volumeName", d.VolumeName, ",")
+	jsonEncodeString(buffer, "claimName", d.ClaimName, ",")
+	jsonEncodeString(buffer, "claimNamespace", d.ClaimNamespace, "")
 	buffer.WriteString("}")
 	return buffer.Bytes(), nil
 }
@@ -312,25 +326,54 @@ func (d *Disk) InterfaceToDisk(itf interface{}) error {
 
 	breakdown := toBreakdown(fbreakdown)
 
-	d.properties = &properties
-	d.labels = labels
-	d.start = start
-	d.end = end
-	d.window = Window{
+	d.Properties = &properties
+	d.Labels = labels
+	d.Start = start
+	d.End = end
+	d.Window = Window{
 		start: &start,
 		end:   &end,
 	}
 	d.Breakdown = &breakdown
 
 	if adjustment, err := getTypedVal(fmap["adjustment"]); err == nil {
-		d.adjustment = adjustment.(float64)
+		d.Adjustment = adjustment.(float64)
 	}
 	if Cost, err := getTypedVal(fmap["totalCost"]); err == nil {
-		d.Cost = Cost.(float64) - d.adjustment
+		d.Cost = Cost.(float64) - d.Adjustment
 	}
 	if ByteHours, err := getTypedVal(fmap["byteHours"]); err == nil {
 		d.ByteHours = ByteHours.(float64)
 	}
+	if ByteHoursUsed, err := getTypedVal(fmap["byteHoursUsed"]); err == nil {
+		if ByteHoursUsed == nil {
+			d.ByteHoursUsed = nil
+		} else {
+			byteHours := ByteHoursUsed.(float64)
+			d.ByteHoursUsed = &byteHours
+		}
+	}
+	if ByteUsageMax, err := getTypedVal(fmap["byteUsageMax"]); err == nil {
+		if ByteUsageMax == nil {
+			d.ByteUsageMax = nil
+		} else {
+			max := ByteUsageMax.(float64)
+			d.ByteUsageMax = &max
+		}
+	}
+
+	if StorageClass, err := getTypedVal(fmap["storageClass"]); err == nil {
+		d.StorageClass = StorageClass.(string)
+	}
+	if VolumeName, err := getTypedVal(fmap["volumeName"]); err == nil {
+		d.VolumeName = VolumeName.(string)
+	}
+	if ClaimName, err := getTypedVal(fmap["claimName"]); err == nil {
+		d.ClaimName = ClaimName.(string)
+	}
+	if ClaimNamespace, err := getTypedVal(fmap["claimNamespace"]); err == nil {
+		d.ClaimNamespace = ClaimNamespace.(string)
+	}
 
 	// d.Local is not marhsaled, and cannot be calculated from marshaled values.
 	// Currently, it is just ignored and not set in the resulting unmarshal to Disk
@@ -347,13 +390,13 @@ func (d *Disk) InterfaceToDisk(itf interface{}) error {
 func (n *Network) MarshalJSON() ([]byte, error) {
 	buffer := bytes.NewBufferString("{")
 	jsonEncodeString(buffer, "type", n.Type().String(), ",")
-	jsonEncode(buffer, "properties", n.Properties(), ",")
-	jsonEncode(buffer, "labels", n.Labels(), ",")
-	jsonEncode(buffer, "window", n.Window(), ",")
-	jsonEncodeString(buffer, "start", n.Start().Format(time.RFC3339), ",")
-	jsonEncodeString(buffer, "end", n.End().Format(time.RFC3339), ",")
+	jsonEncode(buffer, "properties", n.Properties, ",")
+	jsonEncode(buffer, "labels", n.Labels, ",")
+	jsonEncode(buffer, "window", n.Window, ",")
+	jsonEncodeString(buffer, "start", n.Start.Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", n.End.Format(time.RFC3339), ",")
 	jsonEncodeFloat64(buffer, "minutes", n.Minutes(), ",")
-	jsonEncodeFloat64(buffer, "adjustment", n.Adjustment(), ",")
+	jsonEncodeFloat64(buffer, "adjustment", n.Adjustment, ",")
 	jsonEncodeFloat64(buffer, "totalCost", n.TotalCost(), "")
 	buffer.WriteString("}")
 	return buffer.Bytes(), nil
@@ -401,20 +444,20 @@ func (n *Network) InterfaceToNetwork(itf interface{}) error {
 		return err
 	}
 
-	n.properties = &properties
-	n.labels = labels
-	n.start = start
-	n.end = end
-	n.window = Window{
+	n.Properties = &properties
+	n.Labels = labels
+	n.Start = start
+	n.End = end
+	n.Window = Window{
 		start: &start,
 		end:   &end,
 	}
 
 	if adjustment, err := getTypedVal(fmap["adjustment"]); err == nil {
-		n.adjustment = adjustment.(float64)
+		n.Adjustment = adjustment.(float64)
 	}
 	if Cost, err := getTypedVal(fmap["totalCost"]); err == nil {
-		n.Cost = Cost.(float64) - n.adjustment
+		n.Cost = Cost.(float64) - n.Adjustment
 	}
 
 	return nil
@@ -427,11 +470,11 @@ func (n *Network) InterfaceToNetwork(itf interface{}) error {
 func (n *Node) MarshalJSON() ([]byte, error) {
 	buffer := bytes.NewBufferString("{")
 	jsonEncodeString(buffer, "type", n.Type().String(), ",")
-	jsonEncode(buffer, "properties", n.Properties(), ",")
-	jsonEncode(buffer, "labels", n.Labels(), ",")
-	jsonEncode(buffer, "window", n.Window(), ",")
-	jsonEncodeString(buffer, "start", n.Start().Format(time.RFC3339), ",")
-	jsonEncodeString(buffer, "end", n.End().Format(time.RFC3339), ",")
+	jsonEncode(buffer, "properties", n.Properties, ",")
+	jsonEncode(buffer, "labels", n.Labels, ",")
+	jsonEncode(buffer, "window", n.Window, ",")
+	jsonEncodeString(buffer, "start", n.Start.Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", n.End.Format(time.RFC3339), ",")
 	jsonEncodeFloat64(buffer, "minutes", n.Minutes(), ",")
 	jsonEncodeString(buffer, "nodeType", n.NodeType, ",")
 	jsonEncodeFloat64(buffer, "cpuCores", n.CPUCores(), ",")
@@ -447,7 +490,7 @@ func (n *Node) MarshalJSON() ([]byte, error) {
 	jsonEncodeFloat64(buffer, "gpuCost", n.GPUCost, ",")
 	jsonEncodeFloat64(buffer, "gpuCount", n.GPUs(), ",")
 	jsonEncodeFloat64(buffer, "ramCost", n.RAMCost, ",")
-	jsonEncodeFloat64(buffer, "adjustment", n.Adjustment(), ",")
+	jsonEncodeFloat64(buffer, "adjustment", n.Adjustment, ",")
 	jsonEncodeFloat64(buffer, "totalCost", n.TotalCost(), "")
 	buffer.WriteString("}")
 	return buffer.Bytes(), nil
@@ -501,11 +544,11 @@ func (n *Node) InterfaceToNode(itf interface{}) error {
 	cpuBreakdown := toBreakdown(fcpuBreakdown)
 	ramBreakdown := toBreakdown(framBreakdown)
 
-	n.properties = &properties
-	n.labels = labels
-	n.start = start
-	n.end = end
-	n.window = Window{
+	n.Properties = &properties
+	n.Labels = labels
+	n.Start = start
+	n.End = end
+	n.Window = Window{
 		start: &start,
 		end:   &end,
 	}
@@ -513,7 +556,7 @@ func (n *Node) InterfaceToNode(itf interface{}) error {
 	n.RAMBreakdown = &ramBreakdown
 
 	if adjustment, err := getTypedVal(fmap["adjustment"]); err == nil {
-		n.adjustment = adjustment.(float64)
+		n.Adjustment = adjustment.(float64)
 	}
 	if NodeType, err := getTypedVal(fmap["nodeType"]); err == nil {
 		n.NodeType = NodeType.(string)
@@ -555,13 +598,13 @@ func (n *Node) InterfaceToNode(itf interface{}) error {
 func (lb *LoadBalancer) MarshalJSON() ([]byte, error) {
 	buffer := bytes.NewBufferString("{")
 	jsonEncodeString(buffer, "type", lb.Type().String(), ",")
-	jsonEncode(buffer, "properties", lb.Properties(), ",")
-	jsonEncode(buffer, "labels", lb.Labels(), ",")
-	jsonEncode(buffer, "window", lb.Window(), ",")
-	jsonEncodeString(buffer, "start", lb.Start().Format(time.RFC3339), ",")
-	jsonEncodeString(buffer, "end", lb.End().Format(time.RFC3339), ",")
+	jsonEncode(buffer, "properties", lb.Properties, ",")
+	jsonEncode(buffer, "labels", lb.Labels, ",")
+	jsonEncode(buffer, "window", lb.Window, ",")
+	jsonEncodeString(buffer, "start", lb.Start.Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", lb.End.Format(time.RFC3339), ",")
 	jsonEncodeFloat64(buffer, "minutes", lb.Minutes(), ",")
-	jsonEncodeFloat64(buffer, "adjustment", lb.Adjustment(), ",")
+	jsonEncodeFloat64(buffer, "adjustment", lb.Adjustment, ",")
 	jsonEncodeFloat64(buffer, "totalCost", lb.TotalCost(), "")
 	buffer.WriteString("}")
 	return buffer.Bytes(), nil
@@ -609,20 +652,20 @@ func (lb *LoadBalancer) InterfaceToLoadBalancer(itf interface{}) error {
 		return err
 	}
 
-	lb.properties = &properties
-	lb.labels = labels
-	lb.start = start
-	lb.end = end
-	lb.window = Window{
+	lb.Properties = &properties
+	lb.Labels = labels
+	lb.Start = start
+	lb.End = end
+	lb.Window = Window{
 		start: &start,
 		end:   &end,
 	}
 
 	if adjustment, err := getTypedVal(fmap["adjustment"]); err == nil {
-		lb.adjustment = adjustment.(float64)
+		lb.Adjustment = adjustment.(float64)
 	}
 	if Cost, err := getTypedVal(fmap["totalCost"]); err == nil {
-		lb.Cost = Cost.(float64) - lb.adjustment
+		lb.Cost = Cost.(float64) - lb.Adjustment
 	}
 
 	return nil
@@ -635,13 +678,11 @@ func (lb *LoadBalancer) InterfaceToLoadBalancer(itf interface{}) error {
 func (sa *SharedAsset) MarshalJSON() ([]byte, error) {
 	buffer := bytes.NewBufferString("{")
 	jsonEncodeString(buffer, "type", sa.Type().String(), ",")
-	jsonEncode(buffer, "properties", sa.Properties(), ",")
-	jsonEncode(buffer, "labels", sa.Labels(), ",")
-	jsonEncode(buffer, "properties", sa.Properties(), ",")
-	jsonEncode(buffer, "labels", sa.Labels(), ",")
-	jsonEncode(buffer, "window", sa.Window(), ",")
-	jsonEncodeString(buffer, "start", sa.Start().Format(time.RFC3339), ",")
-	jsonEncodeString(buffer, "end", sa.End().Format(time.RFC3339), ",")
+	jsonEncode(buffer, "properties", sa.Properties, ",")
+	jsonEncode(buffer, "labels", sa.Labels, ",")
+	jsonEncode(buffer, "window", sa.Window, ",")
+	jsonEncodeString(buffer, "start", sa.GetStart().Format(time.RFC3339), ",")
+	jsonEncodeString(buffer, "end", sa.GetEnd().Format(time.RFC3339), ",")
 	jsonEncodeFloat64(buffer, "minutes", sa.Minutes(), ",")
 	jsonEncodeFloat64(buffer, "totalCost", sa.TotalCost(), "")
 	buffer.WriteString("}")
@@ -690,9 +731,9 @@ func (sa *SharedAsset) InterfaceToSharedAsset(itf interface{}) error {
 		return err
 	}
 
-	sa.properties = &properties
-	sa.labels = labels
-	sa.window = Window{
+	sa.Properties = &properties
+	sa.Labels = labels
+	sa.Window = Window{
 		start: &start,
 		end:   &end,
 	}
@@ -712,9 +753,8 @@ func (as *AssetSet) MarshalJSON() ([]byte, error) {
 	if as == nil {
 		return json.Marshal(map[string]Asset{})
 	}
-	as.RLock()
-	defer as.RUnlock()
-	return json.Marshal(as.assets)
+
+	return json.Marshal(as.Assets)
 }
 
 // AssetSetResponse for unmarshaling of AssetSet.assets into AssetSet

pkg/kubecost/asset_unmarshal_test.go → pkg/kubecost/asset_json_test.go

@@ -35,22 +35,22 @@ func TestAny_Unmarshal(t *testing.T) {
 	}
 
 	// Check if all fields in initial Any equal those in Any from unmarshal
-	if !any1.properties.Equal(any2.properties) {
+	if !any1.Properties.Equal(any2.Properties) {
 		t.Fatalf("Any Unmarshal: properties mutated in unmarshal")
 	}
-	if !any1.labels.Equal(any2.labels) {
+	if !any1.Labels.Equal(any2.Labels) {
 		t.Fatalf("Any Unmarshal: labels mutated in unmarshal")
 	}
-	if !any1.window.Equal(any2.window) {
+	if !any1.Window.Equal(any2.Window) {
 		t.Fatalf("Any Unmarshal: window mutated in unmarshal")
 	}
-	if !any1.start.Equal(any2.start) {
+	if !any1.Start.Equal(any2.Start) {
 		t.Fatalf("Any Unmarshal: start mutated in unmarshal")
 	}
-	if !any1.end.Equal(any2.end) {
+	if !any1.End.Equal(any2.End) {
 		t.Fatalf("Any Unmarshal: end mutated in unmarshal")
 	}
-	if any1.adjustment != any2.adjustment {
+	if any1.Adjustment != any2.Adjustment {
 		t.Fatalf("Any Unmarshal: adjustment mutated in unmarshal")
 	}
 	if any1.Cost != any2.Cost {
@@ -88,22 +88,22 @@ func TestCloud_Unmarshal(t *testing.T) {
 	}
 
 	// Check if all fields in initial Cloud equal those in Cloud from unmarshal
-	if !cloud1.properties.Equal(cloud2.properties) {
+	if !cloud1.Properties.Equal(cloud2.Properties) {
 		t.Fatalf("Cloud Unmarshal: properties mutated in unmarshal")
 	}
-	if !cloud1.labels.Equal(cloud2.labels) {
+	if !cloud1.Labels.Equal(cloud2.Labels) {
 		t.Fatalf("Cloud Unmarshal: labels mutated in unmarshal")
 	}
-	if !cloud1.window.Equal(cloud2.window) {
+	if !cloud1.Window.Equal(cloud2.Window) {
 		t.Fatalf("Cloud Unmarshal: window mutated in unmarshal")
 	}
-	if !cloud1.start.Equal(cloud2.start) {
+	if !cloud1.Start.Equal(cloud2.Start) {
 		t.Fatalf("Cloud Unmarshal: start mutated in unmarshal")
 	}
-	if !cloud1.end.Equal(cloud2.end) {
+	if !cloud1.End.Equal(cloud2.End) {
 		t.Fatalf("Cloud Unmarshal: end mutated in unmarshal")
 	}
-	if cloud1.adjustment != cloud2.adjustment {
+	if cloud1.Adjustment != cloud2.Adjustment {
 		t.Fatalf("Cloud Unmarshal: adjustment mutated in unmarshal")
 	}
 	if cloud1.Cost != cloud2.Cost {
@@ -138,13 +138,13 @@ func TestClusterManagement_Unmarshal(t *testing.T) {
 	}
 
 	// Check if all fields in initial ClusterManagement equal those in ClusterManagement from unmarshal
-	if !cm1.properties.Equal(cm2.properties) {
+	if !cm1.Properties.Equal(cm2.Properties) {
 		t.Fatalf("ClusterManagement Unmarshal: properties mutated in unmarshal")
 	}
-	if !cm1.labels.Equal(cm2.labels) {
+	if !cm1.Labels.Equal(cm2.Labels) {
 		t.Fatalf("ClusterManagement Unmarshal: labels mutated in unmarshal")
 	}
-	if !cm1.window.Equal(cm2.window) {
+	if !cm1.Window.Equal(cm2.Window) {
 		t.Fatalf("ClusterManagement Unmarshal: window mutated in unmarshal")
 	}
 	if cm1.Cost != cm2.Cost {
@@ -164,6 +164,10 @@ func TestDisk_Unmarshal(t *testing.T) {
 
 	disk1 := NewDisk("disk1", "cluster1", "disk1", *unmarshalWindow.start, *unmarshalWindow.end, unmarshalWindow)
 	disk1.ByteHours = 60.0 * gb * hours
+	used := 40.0 * gb * hours
+	disk1.ByteHoursUsed = &used
+	max := 50.0 * gb * hours
+	disk1.ByteUsageMax = &max
 	disk1.Cost = 4.0
 	disk1.Local = 1.0
 	disk1.SetAdjustment(1.0)
@@ -187,30 +191,36 @@ func TestDisk_Unmarshal(t *testing.T) {
 	}
 
 	// Check if all fields in initial Disk equal those in Disk from unmarshal
-	if !disk1.properties.Equal(disk2.properties) {
+	if !disk1.Properties.Equal(disk2.Properties) {
 		t.Fatalf("Disk Unmarshal: properties mutated in unmarshal")
 	}
-	if !disk1.labels.Equal(disk2.labels) {
+	if !disk1.Labels.Equal(disk2.Labels) {
 		t.Fatalf("Disk Unmarshal: labels mutated in unmarshal")
 	}
-	if !disk1.window.Equal(disk2.window) {
+	if !disk1.Window.Equal(disk2.Window) {
 		t.Fatalf("Disk Unmarshal: window mutated in unmarshal")
 	}
 	if !disk1.Breakdown.Equal(disk2.Breakdown) {
 		t.Fatalf("Disk Unmarshal: Breakdown mutated in unmarshal")
 	}
-	if !disk1.start.Equal(disk2.start) {
+	if !disk1.Start.Equal(disk2.Start) {
 		t.Fatalf("Disk Unmarshal: start mutated in unmarshal")
 	}
-	if !disk1.end.Equal(disk2.end) {
+	if !disk1.End.Equal(disk2.End) {
 		t.Fatalf("Disk Unmarshal: end mutated in unmarshal")
 	}
-	if disk1.adjustment != disk2.adjustment {
+	if disk1.Adjustment != disk2.Adjustment {
 		t.Fatalf("Disk Unmarshal: adjustment mutated in unmarshal")
 	}
 	if disk1.ByteHours != disk2.ByteHours {
 		t.Fatalf("Disk Unmarshal: ByteHours mutated in unmarshal")
 	}
+	if *disk1.ByteHoursUsed != *disk2.ByteHoursUsed {
+		t.Fatalf("Disk Unmarshal: ByteHoursUsed mutated in unmarshal")
+	}
+	if *disk1.ByteUsageMax != *disk2.ByteUsageMax {
+		t.Fatalf("Disk Unmarshal: ByteUsageMax mutated in unmarshal")
+	}
 	if disk1.Cost != disk2.Cost {
 		t.Fatalf("Disk Unmarshal: cost mutated in unmarshal")
 	}
@@ -220,6 +230,42 @@ func TestDisk_Unmarshal(t *testing.T) {
 	// it is also ignored in this test; be aware that this means a resulting Disk from an
 	// unmarshal is therefore NOT equal to the originally marshaled Disk.
 
+	disk3 := NewDisk("disk3", "cluster1", "disk3", *unmarshalWindow.start, *unmarshalWindow.end, unmarshalWindow)
+
+	disk3.ByteHours = 60.0 * gb * hours
+	disk3.ByteHoursUsed = nil
+	disk3.ByteUsageMax = nil
+	disk3.Cost = 4.0
+	disk3.Local = 1.0
+	disk3.SetAdjustment(1.0)
+	disk3.Breakdown = &Breakdown{
+		Idle:   0.1,
+		System: 0.2,
+		User:   0.3,
+		Other:  0.4,
+	}
+
+	bytes, _ = json.Marshal(disk3)
+
+	var testdisk2 Disk
+	disk4 := &testdisk2
+
+	err = json.Unmarshal(bytes, disk4)
+
+	// Check if unmarshal was successful
+	if err != nil {
+		t.Fatalf("Disk Unmarshal: unexpected error: %s", err)
+	}
+
+	// Check that both disks have nil usage
+	if disk3.ByteHoursUsed != disk4.ByteHoursUsed {
+		t.Fatalf("Disk Unmarshal: ByteHoursUsed mutated in unmarshal")
+	}
+	// Check that both disks have nil max usage
+	if disk3.ByteUsageMax != disk4.ByteUsageMax {
+		t.Fatalf("Disk Unmarshal: ByteUsageMax mutated in unmarshal")
+	}
+
 }
 
 func TestNetwork_Unmarshal(t *testing.T) {
@@ -241,22 +287,22 @@ func TestNetwork_Unmarshal(t *testing.T) {
 	}
 
 	// Check if all fields in initial Network equal those in Network from unmarshal
-	if !network1.properties.Equal(network2.properties) {
+	if !network1.Properties.Equal(network2.Properties) {
 		t.Fatalf("Network Unmarshal: properties mutated in unmarshal")
 	}
-	if !network1.labels.Equal(network2.labels) {
+	if !network1.Labels.Equal(network2.Labels) {
 		t.Fatalf("Network Unmarshal: labels mutated in unmarshal")
 	}
-	if !network1.window.Equal(network2.window) {
+	if !network1.Window.Equal(network2.Window) {
 		t.Fatalf("Network Unmarshal: window mutated in unmarshal")
 	}
-	if !network1.start.Equal(network2.start) {
+	if !network1.Start.Equal(network2.Start) {
 		t.Fatalf("Network Unmarshal: start mutated in unmarshal")
 	}
-	if !network1.end.Equal(network2.end) {
+	if !network1.End.Equal(network2.End) {
 		t.Fatalf("Network Unmarshal: end mutated in unmarshal")
 	}
-	if network1.adjustment != network2.adjustment {
+	if network1.Adjustment != network2.Adjustment {
 		t.Fatalf("Network Unmarshal: adjustment mutated in unmarshal")
 	}
 	if network1.Cost != network2.Cost {
@@ -309,13 +355,13 @@ func TestNode_Unmarshal(t *testing.T) {
 	}
 
 	// Check if all fields in initial Node equal those in Node from unmarshal
-	if !node1.properties.Equal(node2.properties) {
+	if !node1.Properties.Equal(node2.Properties) {
 		t.Fatalf("Node Unmarshal: properties mutated in unmarshal")
 	}
-	if !node1.labels.Equal(node2.labels) {
+	if !node1.Labels.Equal(node2.Labels) {
 		t.Fatalf("Node Unmarshal: labels mutated in unmarshal")
 	}
-	if !node1.window.Equal(node2.window) {
+	if !node1.Window.Equal(node2.Window) {
 		t.Fatalf("Node Unmarshal: window mutated in unmarshal")
 	}
 	if !node1.CPUBreakdown.Equal(node2.CPUBreakdown) {
@@ -324,13 +370,13 @@ func TestNode_Unmarshal(t *testing.T) {
 	if !node1.RAMBreakdown.Equal(node2.RAMBreakdown) {
 		t.Fatalf("Node Unmarshal: RAMBreakdown mutated in unmarshal")
 	}
-	if !node1.start.Equal(node2.start) {
+	if !node1.Start.Equal(node2.Start) {
 		t.Fatalf("Node Unmarshal: start mutated in unmarshal")
 	}
-	if !node1.end.Equal(node2.end) {
+	if !node1.End.Equal(node2.End) {
 		t.Fatalf("Node Unmarshal: end mutated in unmarshal")
 	}
-	if node1.adjustment != node2.adjustment {
+	if node1.Adjustment != node2.Adjustment {
 		t.Fatalf("Node Unmarshal: adjustment mutated in unmarshal")
 	}
 	if node1.NodeType != node2.NodeType {
@@ -390,22 +436,22 @@ func TestLoadBalancer_Unmarshal(t *testing.T) {
 	}
 
 	// Check if all fields in initial LoadBalancer equal those in LoadBalancer from unmarshal
-	if !lb1.properties.Equal(lb2.properties) {
+	if !lb1.Properties.Equal(lb2.Properties) {
 		t.Fatalf("LoadBalancer Unmarshal: properties mutated in unmarshal")
 	}
-	if !lb1.labels.Equal(lb2.labels) {
+	if !lb1.Labels.Equal(lb2.Labels) {
 		t.Fatalf("LoadBalancer Unmarshal: labels mutated in unmarshal")
 	}
-	if !lb1.window.Equal(lb2.window) {
+	if !lb1.Window.Equal(lb2.Window) {
 		t.Fatalf("LoadBalancer Unmarshal: window mutated in unmarshal")
 	}
-	if !lb1.start.Equal(lb2.start) {
+	if !lb1.Start.Equal(lb2.Start) {
 		t.Fatalf("LoadBalancer Unmarshal: start mutated in unmarshal")
 	}
-	if !lb1.end.Equal(lb2.end) {
+	if !lb1.End.Equal(lb2.End) {
 		t.Fatalf("LoadBalancer Unmarshal: end mutated in unmarshal")
 	}
-	if lb1.adjustment != lb2.adjustment {
+	if lb1.Adjustment != lb2.Adjustment {
 		t.Fatalf("LoadBalancer Unmarshal: adjustment mutated in unmarshal")
 	}
 	if lb1.Cost != lb2.Cost {
@@ -437,13 +483,13 @@ func TestSharedAsset_Unmarshal(t *testing.T) {
 	}
 
 	// Check if all fields in initial SharedAsset equal those in SharedAsset from unmarshal
-	if !sa1.properties.Equal(sa2.properties) {
+	if !sa1.Properties.Equal(sa2.Properties) {
 		t.Fatalf("SharedAsset Unmarshal: properties mutated in unmarshal")
 	}
-	if !sa1.labels.Equal(sa2.labels) {
+	if !sa1.Labels.Equal(sa2.Labels) {
 		t.Fatalf("SharedAsset Unmarshal: labels mutated in unmarshal")
 	}
-	if !sa1.window.Equal(sa2.window) {
+	if !sa1.Window.Equal(sa2.Window) {
 		t.Fatalf("SharedAsset Unmarshal: window mutated in unmarshal")
 	}
 	if sa1.Cost != sa2.Cost {
@@ -488,7 +534,7 @@ func TestAssetset_Unmarshal(t *testing.T) {
 	}
 
 	// For each asset in unmarshaled AssetSetResponse, check if it is equal to the corresponding AssetSet asset
-	for key, asset := range assetset.assets {
+	for key, asset := range assetset.Assets {
 
 		if unmarshaledAsset, exists := assetUnmarshalResponse.Assets[key]; exists {
 
@@ -500,23 +546,23 @@ func TestAssetset_Unmarshal(t *testing.T) {
 					asset, _ := asset.(*Disk)
 					unmarshaledAsset, _ := unmarshaledAsset.(*Disk)
 
-					if !asset.Labels().Equal(unmarshaledAsset.Labels()) {
+					if !asset.GetLabels().Equal(unmarshaledAsset.Labels) {
 						return false
 					}
-					if !asset.Properties().Equal(unmarshaledAsset.Properties()) {
+					if !asset.Properties.Equal(unmarshaledAsset.Properties) {
 						return false
 					}
 
-					if !asset.Start().Equal(unmarshaledAsset.Start()) {
+					if !asset.GetStart().Equal(unmarshaledAsset.Start) {
 						return false
 					}
-					if !asset.End().Equal(unmarshaledAsset.End()) {
+					if !asset.End.Equal(unmarshaledAsset.End) {
 						return false
 					}
-					if !asset.window.Equal(unmarshaledAsset.window) {
+					if !asset.Window.Equal(unmarshaledAsset.Window) {
 						return false
 					}
-					if asset.adjustment != unmarshaledAsset.adjustment {
+					if asset.Adjustment != unmarshaledAsset.Adjustment {
 						return false
 					}
 					if asset.Cost != unmarshaledAsset.Cost {

pkg/kubecost/asset_test.go

@@ -31,25 +31,25 @@ func assertAssetSet(t *testing.T, as *AssetSet, msg string, window Window, exps
 	if !as.Window.Equal(window) {
 		t.Fatalf("AssetSet.AggregateBy[%s]: expected window %s, actual %s", msg, window, as.Window)
 	}
-	as.Each(func(key string, a Asset) {
+	for key, a := range as.Assets {
 		if exp, ok := exps[key]; ok {
 			if math.Round(a.TotalCost()*100) != math.Round(exp*100) {
 				t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected total cost %.2f, actual %.2f", msg, key, exp, a.TotalCost())
 			}
-			if !a.Window().Equal(window) {
-				t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected window %s, actual %s", msg, key, window, a.Window())
+			if !a.GetWindow().Equal(window) {
+				t.Fatalf("AssetSet.AggregateBy[%s]: key %s expected window %s, actual %s", msg, key, window, a.GetWindow())
 			}
 		} else {
 			t.Fatalf("AssetSet.AggregateBy[%s]: unexpected asset: %s", msg, key)
 		}
-	})
+	}
 }
 
 func printAssetSet(msg string, as *AssetSet) {
 	fmt.Printf("--- %s ---\n", msg)
-	as.Each(func(key string, a Asset) {
+	for key, a := range as.Assets {
 		fmt.Printf(" > %s: %s\n", key, a)
-	})
+	}
 }
 
 func TestAny_Add(t *testing.T) {
@@ -77,34 +77,34 @@ func TestAny_Add(t *testing.T) {
 	if any3.TotalCost() != 15.0 {
 		t.Fatalf("Any.Add: expected %f; got %f", 15.0, any3.TotalCost())
 	}
-	if any3.Adjustment() != 2.0 {
-		t.Fatalf("Any.Add: expected %f; got %f", 2.0, any3.Adjustment())
+	if any3.GetAdjustment() != 2.0 {
+		t.Fatalf("Any.Add: expected %f; got %f", 2.0, any3.GetAdjustment())
 	}
-	if any3.Properties().Cluster != "cluster1" {
-		t.Fatalf("Any.Add: expected %s; got %s", "cluster1", any3.Properties().Cluster)
+	if any3.GetProperties().Cluster != "cluster1" {
+		t.Fatalf("Any.Add: expected %s; got %s", "cluster1", any3.GetProperties().Cluster)
 	}
 	if any3.Type() != AnyAssetType {
 		t.Fatalf("Any.Add: expected %s; got %s", AnyAssetType, any3.Type())
 	}
-	if any3.Properties().ProviderID != "" {
-		t.Fatalf("Any.Add: expected %s; got %s", "", any3.Properties().ProviderID)
+	if any3.GetProperties().ProviderID != "" {
+		t.Fatalf("Any.Add: expected %s; got %s", "", any3.GetProperties().ProviderID)
 	}
-	if any3.Properties().Name != "" {
-		t.Fatalf("Any.Add: expected %s; got %s", "", any3.Properties().Name)
+	if any3.GetProperties().Name != "" {
+		t.Fatalf("Any.Add: expected %s; got %s", "", any3.GetProperties().Name)
 	}
 
 	// Check that the original assets are unchanged
 	if any1.TotalCost() != 10.0 {
 		t.Fatalf("Any.Add: expected %f; got %f", 10.0, any1.TotalCost())
 	}
-	if any1.Adjustment() != 1.0 {
-		t.Fatalf("Any.Add: expected %f; got %f", 1.0, any1.Adjustment())
+	if any1.Adjustment != 1.0 {
+		t.Fatalf("Any.Add: expected %f; got %f", 1.0, any1.Adjustment)
 	}
 	if any2.TotalCost() != 5.0 {
 		t.Fatalf("Any.Add: expected %f; got %f", 5.0, any2.TotalCost())
 	}
-	if any2.Adjustment() != 1.0 {
-		t.Fatalf("Any.Add: expected %f; got %f", 1.0, any2.Adjustment())
+	if any2.Adjustment != 1.0 {
+		t.Fatalf("Any.Add: expected %f; got %f", 1.0, any2.Adjustment)
 	}
 }
 
@@ -127,8 +127,8 @@ func TestAny_Clone(t *testing.T) {
 	if any2.TotalCost() != 10.0 {
 		t.Fatalf("Any.Clone: expected %f; got %f", 10.0, any2.TotalCost())
 	}
-	if any2.Adjustment() != 1.0 {
-		t.Fatalf("Any.Clone: expected %f; got %f", 1.0, any2.Adjustment())
+	if any2.GetAdjustment() != 1.0 {
+		t.Fatalf("Any.Clone: expected %f; got %f", 1.0, any2.GetAdjustment())
 	}
 }
 
@@ -194,20 +194,20 @@ func TestDisk_Add(t *testing.T) {
 	if diskT.TotalCost() != 15.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 15.0, diskT.TotalCost())
 	}
-	if diskT.Adjustment() != 2.0 {
-		t.Fatalf("Disk.Add: expected %f; got %f", 2.0, diskT.Adjustment())
+	if diskT.Adjustment != 2.0 {
+		t.Fatalf("Disk.Add: expected %f; got %f", 2.0, diskT.Adjustment)
 	}
-	if diskT.Properties().Cluster != "cluster1" {
-		t.Fatalf("Disk.Add: expected %s; got %s", "cluster1", diskT.Properties().Cluster)
+	if diskT.Properties.Cluster != "cluster1" {
+		t.Fatalf("Disk.Add: expected %s; got %s", "cluster1", diskT.Properties.Cluster)
 	}
 	if diskT.Type() != DiskAssetType {
 		t.Fatalf("Disk.Add: expected %s; got %s", AnyAssetType, diskT.Type())
 	}
-	if diskT.Properties().ProviderID != "" {
-		t.Fatalf("Disk.Add: expected %s; got %s", "", diskT.Properties().ProviderID)
+	if diskT.Properties.ProviderID != "" {
+		t.Fatalf("Disk.Add: expected %s; got %s", "", diskT.Properties.ProviderID)
 	}
-	if diskT.Properties().Name != "" {
-		t.Fatalf("Disk.Add: expected %s; got %s", "", diskT.Properties().Name)
+	if diskT.Properties.Name != "" {
+		t.Fatalf("Disk.Add: expected %s; got %s", "", diskT.Properties.Name)
 	}
 	if diskT.Bytes() != 160.0*gb {
 		t.Fatalf("Disk.Add: expected %f; got %f", 160.0*gb, diskT.Bytes())
@@ -220,8 +220,8 @@ func TestDisk_Add(t *testing.T) {
 	if disk1.TotalCost() != 10.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 10.0, disk1.TotalCost())
 	}
-	if disk1.Adjustment() != 1.0 {
-		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, disk1.Adjustment())
+	if disk1.Adjustment != 1.0 {
+		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, disk1.Adjustment)
 	}
 	if disk1.Local != 0.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 0.0, disk1.Local)
@@ -229,8 +229,8 @@ func TestDisk_Add(t *testing.T) {
 	if disk2.TotalCost() != 5.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 5.0, disk2.TotalCost())
 	}
-	if disk2.Adjustment() != 1.0 {
-		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, disk2.Adjustment())
+	if disk2.Adjustment != 1.0 {
+		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, disk2.Adjustment)
 	}
 	if disk2.Local != 1.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, disk2.Local)
@@ -274,20 +274,20 @@ func TestDisk_Add(t *testing.T) {
 	if diskAT.TotalCost() != 20.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 20.0, diskAT.TotalCost())
 	}
-	if diskAT.Adjustment() != 2.0 {
-		t.Fatalf("Disk.Add: expected %f; got %f", 2.0, diskAT.Adjustment())
+	if diskAT.Adjustment != 2.0 {
+		t.Fatalf("Disk.Add: expected %f; got %f", 2.0, diskAT.Adjustment)
 	}
-	if diskAT.Properties().Cluster != "cluster1" {
-		t.Fatalf("Disk.Add: expected %s; got %s", "cluster1", diskAT.Properties().Cluster)
+	if diskAT.Properties.Cluster != "cluster1" {
+		t.Fatalf("Disk.Add: expected %s; got %s", "cluster1", diskAT.Properties.Cluster)
 	}
 	if diskAT.Type() != DiskAssetType {
 		t.Fatalf("Disk.Add: expected %s; got %s", AnyAssetType, diskAT.Type())
 	}
-	if diskAT.Properties().ProviderID != "" {
-		t.Fatalf("Disk.Add: expected %s; got %s", "", diskAT.Properties().ProviderID)
+	if diskAT.Properties.ProviderID != "" {
+		t.Fatalf("Disk.Add: expected %s; got %s", "", diskAT.Properties.ProviderID)
 	}
-	if diskAT.Properties().Name != "" {
-		t.Fatalf("Disk.Add: expected %s; got %s", "", diskAT.Properties().Name)
+	if diskAT.Properties.Name != "" {
+		t.Fatalf("Disk.Add: expected %s; got %s", "", diskAT.Properties.Name)
 	}
 	if diskAT.Bytes() != 100.0*gb {
 		t.Fatalf("Disk.Add: expected %f; got %f", 100.0*gb, diskT.Bytes())
@@ -300,8 +300,8 @@ func TestDisk_Add(t *testing.T) {
 	if diskA1.TotalCost() != 10.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 10.0, diskA1.TotalCost())
 	}
-	if diskA1.Adjustment() != 1.0 {
-		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, diskA1.Adjustment())
+	if diskA1.Adjustment != 1.0 {
+		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, diskA1.Adjustment)
 	}
 	if diskA1.Local != 0.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 0.0, diskA1.Local)
@@ -309,8 +309,8 @@ func TestDisk_Add(t *testing.T) {
 	if diskA2.TotalCost() != 10.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 10.0, diskA2.TotalCost())
 	}
-	if diskA2.Adjustment() != 1.0 {
-		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, diskA2.Adjustment())
+	if diskA2.Adjustment != 1.0 {
+		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, diskA2.Adjustment)
 	}
 	if diskA2.Local != 0.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 0.0, diskA2.Local)
@@ -333,8 +333,8 @@ func TestDisk_Clone(t *testing.T) {
 	if disk2.TotalCost() != 10.0 {
 		t.Fatalf("Any.Clone: expected %f; got %f", 10.0, disk2.TotalCost())
 	}
-	if disk2.Adjustment() != 1.0 {
-		t.Fatalf("Any.Clone: expected %f; got %f", 1.0, disk2.Adjustment())
+	if disk2.Adjustment != 1.0 {
+		t.Fatalf("Any.Clone: expected %f; got %f", 1.0, disk2.Adjustment)
 	}
 	if disk2.Local != 1.0 {
 		t.Fatalf("Disk.Add: expected %f; got %f", 1.0, disk2.Local)
@@ -412,20 +412,20 @@ func TestNode_Add(t *testing.T) {
 	if !util.IsApproximately(nodeT.TotalCost(), 15.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 15.0, nodeT.TotalCost())
 	}
-	if nodeT.Adjustment() != 2.6 {
-		t.Fatalf("Node.Add: expected %f; got %f", 2.6, nodeT.Adjustment())
+	if nodeT.Adjustment != 2.6 {
+		t.Fatalf("Node.Add: expected %f; got %f", 2.6, nodeT.Adjustment)
 	}
-	if nodeT.Properties().Cluster != "cluster1" {
-		t.Fatalf("Node.Add: expected %s; got %s", "cluster1", nodeT.Properties().Cluster)
+	if nodeT.Properties.Cluster != "cluster1" {
+		t.Fatalf("Node.Add: expected %s; got %s", "cluster1", nodeT.Properties.Cluster)
 	}
 	if nodeT.Type() != NodeAssetType {
 		t.Fatalf("Node.Add: expected %s; got %s", AnyAssetType, nodeT.Type())
 	}
-	if nodeT.Properties().ProviderID != "" {
-		t.Fatalf("Node.Add: expected %s; got %s", "", nodeT.Properties().ProviderID)
+	if nodeT.Properties.ProviderID != "" {
+		t.Fatalf("Node.Add: expected %s; got %s", "", nodeT.Properties.ProviderID)
 	}
-	if nodeT.Properties().Name != "" {
-		t.Fatalf("Node.Add: expected %s; got %s", "", nodeT.Properties().Name)
+	if nodeT.Properties.Name != "" {
+		t.Fatalf("Node.Add: expected %s; got %s", "", nodeT.Properties.Name)
 	}
 	if nodeT.CPUCores() != 2.0 {
 		t.Fatalf("Node.Add: expected %f; got %f", 2.0, nodeT.CPUCores())
@@ -438,14 +438,14 @@ func TestNode_Add(t *testing.T) {
 	if !util.IsApproximately(node1.TotalCost(), 10.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 10.0, node1.TotalCost())
 	}
-	if node1.Adjustment() != 1.6 {
-		t.Fatalf("Node.Add: expected %f; got %f", 1.0, node1.Adjustment())
+	if node1.Adjustment != 1.6 {
+		t.Fatalf("Node.Add: expected %f; got %f", 1.0, node1.Adjustment)
 	}
 	if !util.IsApproximately(node2.TotalCost(), 5.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 5.0, node2.TotalCost())
 	}
-	if node2.Adjustment() != 1.0 {
-		t.Fatalf("Node.Add: expected %f; got %f", 1.0, node2.Adjustment())
+	if node2.Adjustment != 1.0 {
+		t.Fatalf("Node.Add: expected %f; got %f", 1.0, node2.Adjustment)
 	}
 
 	// Check that we don't divide by zero computing Local
@@ -506,20 +506,20 @@ func TestNode_Add(t *testing.T) {
 	if !util.IsApproximately(nodeAT.TotalCost(), 15.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 15.0, nodeAT.TotalCost())
 	}
-	if nodeAT.Adjustment() != 2.6 {
-		t.Fatalf("Node.Add: expected %f; got %f", 2.6, nodeAT.Adjustment())
+	if nodeAT.Adjustment != 2.6 {
+		t.Fatalf("Node.Add: expected %f; got %f", 2.6, nodeAT.Adjustment)
 	}
-	if nodeAT.Properties().Cluster != "cluster1" {
-		t.Fatalf("Node.Add: expected %s; got %s", "cluster1", nodeAT.Properties().Cluster)
+	if nodeAT.Properties.Cluster != "cluster1" {
+		t.Fatalf("Node.Add: expected %s; got %s", "cluster1", nodeAT.Properties.Cluster)
 	}
 	if nodeAT.Type() != NodeAssetType {
 		t.Fatalf("Node.Add: expected %s; got %s", AnyAssetType, nodeAT.Type())
 	}
-	if nodeAT.Properties().ProviderID != "" {
-		t.Fatalf("Node.Add: expected %s; got %s", "", nodeAT.Properties().ProviderID)
+	if nodeAT.Properties.ProviderID != "" {
+		t.Fatalf("Node.Add: expected %s; got %s", "", nodeAT.Properties.ProviderID)
 	}
-	if nodeAT.Properties().Name != "" {
-		t.Fatalf("Node.Add: expected %s; got %s", "", nodeAT.Properties().Name)
+	if nodeAT.Properties.Name != "" {
+		t.Fatalf("Node.Add: expected %s; got %s", "", nodeAT.Properties.Name)
 	}
 	if nodeAT.CPUCores() != 1.0 {
 		t.Fatalf("Node.Add: expected %f; got %f", 1.0, nodeAT.CPUCores())
@@ -535,14 +535,14 @@ func TestNode_Add(t *testing.T) {
 	if !util.IsApproximately(nodeA1.TotalCost(), 10.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 10.0, nodeA1.TotalCost())
 	}
-	if nodeA1.Adjustment() != 1.6 {
-		t.Fatalf("Node.Add: expected %f; got %f", 1.0, nodeA1.Adjustment())
+	if nodeA1.Adjustment != 1.6 {
+		t.Fatalf("Node.Add: expected %f; got %f", 1.0, nodeA1.Adjustment)
 	}
 	if !util.IsApproximately(nodeA2.TotalCost(), 5.0) {
 		t.Fatalf("Node.Add: expected %f; got %f", 5.0, nodeA2.TotalCost())
 	}
-	if nodeA2.Adjustment() != 1.0 {
-		t.Fatalf("Node.Add: expected %f; got %f", 1.0, nodeA2.Adjustment())
+	if nodeA2.Adjustment != 1.0 {
+		t.Fatalf("Node.Add: expected %f; got %f", 1.0, nodeA2.Adjustment)
 	}
 }
 
@@ -582,8 +582,8 @@ func TestClusterManagement_Add(t *testing.T) {
 	if cm3.TotalCost() != 13.0 {
 		t.Fatalf("ClusterManagement.Add: expected %f; got %f", 13.0, cm3.TotalCost())
 	}
-	if cm3.Properties().Cluster != "cluster1" {
-		t.Fatalf("ClusterManagement.Add: expected %s; got %s", "cluster1", cm3.Properties().Cluster)
+	if cm3.GetProperties().Cluster != "cluster1" {
+		t.Fatalf("ClusterManagement.Add: expected %s; got %s", "cluster1", cm3.GetProperties().Cluster)
 	}
 	if cm3.Type() != ClusterManagementAssetType {
 		t.Fatalf("ClusterManagement.Add: expected %s; got %s", ClusterManagementAssetType, cm3.Type())
@@ -617,8 +617,8 @@ func TestCloudAny_Add(t *testing.T) {
 	if ca3.TotalCost() != 15.0 {
 		t.Fatalf("Any.Add: expected %f; got %f", 15.0, ca3.TotalCost())
 	}
-	if ca3.Adjustment() != 2.0 {
-		t.Fatalf("Any.Add: expected %f; got %f", 2.0, ca3.Adjustment())
+	if ca3.GetAdjustment() != 2.0 {
+		t.Fatalf("Any.Add: expected %f; got %f", 2.0, ca3.GetAdjustment())
 	}
 	if ca3.Type() != CloudAssetType {
 		t.Fatalf("Any.Add: expected %s; got %s", CloudAssetType, ca3.Type())
@@ -628,14 +628,14 @@ func TestCloudAny_Add(t *testing.T) {
 	if ca1.TotalCost() != 10.0 {
 		t.Fatalf("Any.Add: expected %f; got %f", 10.0, ca1.TotalCost())
 	}
-	if ca1.Adjustment() != 1.0 {
-		t.Fatalf("Any.Add: expected %f; got %f", 1.0, ca1.Adjustment())
+	if ca1.Adjustment != 1.0 {
+		t.Fatalf("Any.Add: expected %f; got %f", 1.0, ca1.Adjustment)
 	}
 	if ca2.TotalCost() != 5.0 {
 		t.Fatalf("Any.Add: expected %f; got %f", 5.0, ca2.TotalCost())
 	}
-	if ca2.Adjustment() != 1.0 {
-		t.Fatalf("Any.Add: expected %f; got %f", 1.0, ca2.Adjustment())
+	if ca2.Adjustment != 1.0 {
+		t.Fatalf("Any.Add: expected %f; got %f", 1.0, ca2.Adjustment)
 	}
 }
 
@@ -786,7 +786,7 @@ func TestAssetSet_FindMatch(t *testing.T) {
 	// Assert success of a simple match of Type and ProviderID
 	as = GenerateMockAssetSet(startYesterday)
 	query = NewNode("", "", "gcp-node3", s, e, w)
-	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)})
+	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)}, nil)
 	if err != nil {
 		t.Fatalf("AssetSet.FindMatch: unexpected error: %s", err)
 	}
@@ -794,7 +794,7 @@ func TestAssetSet_FindMatch(t *testing.T) {
 	// Assert error of a simple non-match of Type and ProviderID
 	as = GenerateMockAssetSet(startYesterday)
 	query = NewNode("", "", "aws-node3", s, e, w)
-	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)})
+	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)}, nil)
 	if err == nil {
 		t.Fatalf("AssetSet.FindMatch: expected error (no match); found %s", match)
 	}
@@ -802,7 +802,7 @@ func TestAssetSet_FindMatch(t *testing.T) {
 	// Assert error of matching ProviderID, but not Type
 	as = GenerateMockAssetSet(startYesterday)
 	query = NewCloud(ComputeCategory, "gcp-node3", s, e, w)
-	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)})
+	match, err = as.FindMatch(query, []string{string(AssetTypeProp), string(AssetProviderIDProp)}, nil)
 	if err == nil {
 		t.Fatalf("AssetSet.FindMatch: expected error (no match); found %s", match)
 	}
@@ -824,30 +824,30 @@ func TestAssetSet_InsertMatchingWindow(t *testing.T) {
 	a1.SetProperties(&AssetProperties{
 		Name: "asset-1",
 	})
-	a1.window = NewClosedWindow(a1WindowStart, a1WindowEnd)
+	a1.Window = NewClosedWindow(a1WindowStart, a1WindowEnd)
 
 	a2 := &Disk{}
 	a2.SetProperties(&AssetProperties{
 		Name: "asset-2",
 	})
-	a2.window = NewClosedWindow(a2WindowStart, a2WindowEnd)
+	a2.Window = NewClosedWindow(a2WindowStart, a2WindowEnd)
 
 	as := NewAssetSet(setStart, setEnd)
-	as.Insert(a1)
-	as.Insert(a2)
+	as.Insert(a1, nil)
+	as.Insert(a2, nil)
 
 	if as.Length() != 2 {
 		t.Errorf("AS length got %d, expected %d", as.Length(), 2)
 	}
 
-	as.Each(func(k string, a Asset) {
-		if !(*a.Window().Start()).Equal(setStart) {
-			t.Errorf("Asset %s window start is %s, expected %s", a.Properties().Name, *a.Window().Start(), setStart)
+	for _, a := range as.Assets {
+		if !(*a.GetWindow().Start()).Equal(setStart) {
+			t.Errorf("Asset %s window start is %s, expected %s", a.GetProperties().Name, *a.GetWindow().Start(), setStart)
 		}
-		if !(*a.Window().End()).Equal(setEnd) {
-			t.Errorf("Asset %s window end is %s, expected %s", a.Properties().Name, *a.Window().End(), setEnd)
+		if !(*a.GetWindow().End()).Equal(setEnd) {
+			t.Errorf("Asset %s window end is %s, expected %s", a.GetProperties().Name, *a.GetWindow().End(), setEnd)
 		}
-	})
+	}
 }
 
 func TestAssetSet_ReconciliationMatchMap(t *testing.T) {
@@ -859,15 +859,15 @@ func TestAssetSet_ReconciliationMatchMap(t *testing.T) {
 
 	// Determine the number of assets by provider ID
 	assetCountByProviderId := make(map[string]int, len(matchMap))
-	as.Each(func(key string, a Asset) {
-		if a == nil || a.Properties() == nil || a.Properties().ProviderID == "" {
+	for _, a := range as.Assets {
+		if a == nil || a.GetProperties() == nil || a.GetProperties().ProviderID == "" {
 			return
 		}
-		if _, ok := assetCountByProviderId[a.Properties().ProviderID]; !ok {
-			assetCountByProviderId[a.Properties().ProviderID] = 0
+		if _, ok := assetCountByProviderId[a.GetProperties().ProviderID]; !ok {
+			assetCountByProviderId[a.GetProperties().ProviderID] = 0
 		}
-		assetCountByProviderId[a.Properties().ProviderID] += 1
-	})
+		assetCountByProviderId[a.GetProperties().ProviderID] += 1
+	}
 
 	for k, count := range assetCountByProviderId {
 		if len(matchMap[k]) != count {
@@ -1180,11 +1180,11 @@ func TestAssetSetRange_Start(t *testing.T) {
 		{
 			name: "Single asset",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1196,14 +1196,14 @@ func TestAssetSetRange_Start(t *testing.T) {
 		{
 			name: "Two assets",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
 							"b": &Node{
-								start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1215,18 +1215,18 @@ func TestAssetSetRange_Start(t *testing.T) {
 		{
 			name: "Two AssetSets",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
-					&AssetSet{
-						assets: map[string]Asset{
+					{
+						Assets: map[string]Asset{
 							"b": &Node{
-								start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1268,11 +1268,11 @@ func TestAssetSetRange_End(t *testing.T) {
 		{
 			name: "Single asset",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								end: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								End: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1284,14 +1284,14 @@ func TestAssetSetRange_End(t *testing.T) {
 		{
 			name: "Two assets",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								end: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								End: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
 							"b": &Node{
-								end: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								End: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1303,18 +1303,18 @@ func TestAssetSetRange_End(t *testing.T) {
 		{
 			name: "Two AssetSets",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								end: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								End: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
-					&AssetSet{
-						assets: map[string]Asset{
+					{
+						Assets: map[string]Asset{
 							"b": &Node{
-								end: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								End: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1355,12 +1355,12 @@ func TestAssetSetRange_Minutes(t *testing.T) {
 		{
 			name: "Single asset",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
-								end:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								End:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1372,16 +1372,16 @@ func TestAssetSetRange_Minutes(t *testing.T) {
 		{
 			name: "Two assets",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
-								end:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								End:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
 							"b": &Node{
-								start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
-								end:   time.Date(1970, 1, 3, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								End:   time.Date(1970, 1, 3, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1393,20 +1393,20 @@ func TestAssetSetRange_Minutes(t *testing.T) {
 		{
 			name: "Two AssetSets",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
-					&AssetSet{
-						assets: map[string]Asset{
+				Assets: []*AssetSet{
+					{
+						Assets: map[string]Asset{
 							"a": &Node{
-								start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
-								end:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC),
+								End:   time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
-					&AssetSet{
-						assets: map[string]Asset{
+					{
+						Assets: map[string]Asset{
 							"b": &Node{
-								start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
-								end:   time.Date(1970, 1, 3, 0, 0, 0, 0, time.UTC),
+								Start: time.Date(1970, 1, 2, 0, 0, 0, 0, time.UTC),
+								End:   time.Date(1970, 1, 3, 0, 0, 0, 0, time.UTC),
 							},
 						},
 					},
@@ -1443,11 +1443,11 @@ func TestAssetSetRange_MarshalJSON(t *testing.T) {
 		{
 			name: "Normal ASR",
 			arg: &AssetSetRange{
-				assets: []*AssetSet{
+				Assets: []*AssetSet{
 					{
-						assets: map[string]Asset{
+						Assets: map[string]Asset{
 							"a": &Any{
-								start: time.Now().UTC().Truncate(day),
+								Start: time.Now().UTC().Truncate(day),
 							},
 						},
 					},

pkg/kubecost/assetprops.go

@@ -41,6 +41,21 @@ const (
 
 	// AssetTypeProp describes the type of the Asset
 	AssetTypeProp AssetProperty = "type"
+
+	// AssetDepartmentProp describes the department of the Asset
+	AssetDepartmentProp AssetProperty = "department"
+
+	// AssetEnvironmentProp describes the environment of the Asset
+	AssetEnvironmentProp AssetProperty = "environment"
+
+	// AssetOwnerProp describes the owner of the Asset
+	AssetOwnerProp AssetProperty = "owner"
+
+	// AssetProductProp describes the product of the Asset
+	AssetProductProp AssetProperty = "product"
+
+	// AssetTeamProp describes the team of the Asset
+	AssetTeamProp AssetProperty = "team"
 )
 
 // ParseAssetProperty attempts to parse a string into an AssetProperty
@@ -64,6 +79,16 @@ func ParseAssetProperty(text string) (AssetProperty, error) {
 		return AssetServiceProp, nil
 	case "type":
 		return AssetTypeProp, nil
+	case "department":
+		return AssetDepartmentProp, nil
+	case "environment":
+		return AssetEnvironmentProp, nil
+	case "owner":
+		return AssetOwnerProp, nil
+	case "product":
+		return AssetProductProp, nil
+	case "team":
+		return AssetTeamProp, nil
 	}
 	return AssetNilProp, fmt.Errorf("invalid asset property: %s", text)
 }
@@ -99,9 +124,15 @@ const GCPProvider = "GCP"
 // AzureProvider describes the provider Azure
 const AzureProvider = "Azure"
 
+// AlibabaProvider describes the provider for Alibaba Cloud
+const AlibabaProvider = "Alibaba"
+
 // CSVProvider describes the provider a CSV
 const CSVProvider = "CSV"
 
+// CustomProvider describes a custom provider
+const CustomProvider = "custom"
+
 // ScalewayProvider describes the provider Scaleway
 const ScalewayProvider = "Scaleway"
 

pkg/kubecost/audit.go

@@ -1,9 +1,10 @@
 package kubecost
 
 import (
-	"golang.org/x/exp/slices"
 	"sync"
 	"time"
+
+	"golang.org/x/exp/slices"
 )
 
 // AuditType the types of Audits, each of which should be contained in an AuditSet
@@ -37,8 +38,8 @@ func ToAuditType(check string) AuditType {
 		return AuditAssetTotalStore
 	case string(AuditAssetAggStore):
 		return AuditAssetAggStore
-	//case string(AuditClusterEquality):
-	//	return AuditClusterEquality
+	case string(AuditClusterEquality):
+		return AuditClusterEquality
 	case string(AuditAll):
 		return AuditAll
 	default:
@@ -235,7 +236,6 @@ func (ea *EqualityAudit) Clone() *EqualityAudit {
 
 // AuditCoverage tracks coverage of each audit type
 type AuditCoverage struct {
-	sync.RWMutex
 	AllocationReconciliation Window `json:"allocationReconciliation"`
 	AllocationAgg            Window `json:"allocationAgg"`
 	AllocationTotal          Window `json:"allocationTotal"`

pkg/kubecost/bingen.go

@@ -22,15 +22,17 @@ package kubecost
 
 // Default Version Set (uses -version flag passed) includes shared resources
 // @bingen:generate:Window
+// @bingen:generate:Coverage
+// @bingen:generate:CoverageSet
 
 // Asset Version Set: Includes Asset pipeline specific resources
-// @bingen:set[name=Assets,version=16]
+// @bingen:set[name=Assets,version=18]
 // @bingen:generate:Any
 // @bingen:generate:Asset
 // @bingen:generate:AssetLabels
 // @bingen:generate:AssetProperties
 // @bingen:generate:AssetProperty
-// @bingen:generate[stringtable]:AssetSet
+// @bingen:generate[stringtable,preprocess,postprocess]:AssetSet
 // @bingen:generate:AssetSetRange
 // @bingen:generate:Breakdown
 // @bingen:generate:Cloud
@@ -71,4 +73,20 @@ package kubecost
 // @bingen:generate:AuditSetRange
 // @bingen:end
 
-//go:generate bingen -package=kubecost -version=15 -buffer=github.com/opencost/opencost/pkg/util
+// @bingen:set[name=CloudCostAggregate,version=1]
+// @bingen:generate:CloudCostAggregate
+// @bingen:generate[stringtable]:CloudCostAggregateSet
+// @bingen:generate:CloudCostAggregateSetRange
+// @bingen:generate:CloudCostAggregateProperties
+// @bingen:generate:CloudCostAggregateLabels
+// @bingen:end
+
+// @bingen:set[name=CloudCostItem,version=1]
+// @bingen:generate:CloudCostItem
+// @bingen:generate[stringtable]:CloudCostItemSet
+// @bingen:generate:CloudCostItemSetRange
+// @bingen:generate:CloudCostItemProperties
+// @bingen:generate:CloudCostItemLabels
+// @bingen:end
+
+//go:generate bingen -package=kubecost -version=17 -buffer=github.com/opencost/opencost/pkg/util

pkg/kubecost/cloudcostaggregate.go

@@ -0,0 +1,422 @@
+package kubecost
+
+import (
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	"github.com/opencost/opencost/pkg/filter"
+	"github.com/opencost/opencost/pkg/log"
+)
+
+const (
+	CloudCostAccountProp  string = "account"
+	CloudCostProjectProp  string = "project"
+	CloudCostProviderProp string = "provider"
+	CloudCostServiceProp  string = "service"
+	CloudCostLabelProp    string = "label"
+)
+
+// CloudCostAggregateProperties unique property set for CloudCostAggregate within a window
+type CloudCostAggregateProperties struct {
+	Provider   string `json:"provider"`
+	Account    string `json:"account"`
+	Project    string `json:"project"`
+	Service    string `json:"service"`
+	LabelValue string `json:"label"`
+}
+
+func (ccap CloudCostAggregateProperties) Equal(that CloudCostAggregateProperties) bool {
+	return ccap.Provider == that.Provider &&
+		ccap.Account == that.Account &&
+		ccap.Project == that.Project &&
+		ccap.Service == that.Service &&
+		ccap.LabelValue == that.LabelValue
+}
+
+func (ccap CloudCostAggregateProperties) Key(props []string) string {
+	if len(props) == 0 {
+		return fmt.Sprintf("%s/%s/%s/%s/%s", ccap.Provider, ccap.Account, ccap.Project, ccap.Service, ccap.LabelValue)
+	}
+
+	keys := make([]string, len(props))
+	for i, prop := range props {
+		key := UnallocatedSuffix
+
+		switch prop {
+		case CloudCostProviderProp:
+			if ccap.Provider != "" {
+				key = ccap.Provider
+			}
+		case CloudCostAccountProp:
+			if ccap.Account != "" {
+				key = ccap.Account
+			}
+		case CloudCostProjectProp:
+			if ccap.Project != "" {
+				key = ccap.Project
+			}
+		case CloudCostServiceProp:
+			if ccap.Service != "" {
+				key = ccap.Service
+			}
+		case CloudCostLabelProp:
+			if ccap.LabelValue != "" {
+				key = ccap.LabelValue
+			}
+		}
+
+		keys[i] = key
+	}
+
+	return strings.Join(keys, "/")
+}
+
+// CloudCostAggregate represents an aggregation of Billing Integration data on the properties listed
+// - KubernetesPercent is the percent of the CloudCostAggregates cost which was from an item which could be identified
+//   as coming from a kubernetes resources.
+// - Cost is the sum of the cost of each item in the CloudCostAggregate
+// - Credit is the sum of credits applied to each item in the CloudCostAggregate
+
+type CloudCostAggregate struct {
+	Properties        CloudCostAggregateProperties `json:"properties"`
+	KubernetesPercent float64                      `json:"kubernetesPercent"`
+	Cost              float64                      `json:"cost"`
+	Credit            float64                      `json:"credit"`
+}
+
+func (cca *CloudCostAggregate) Clone() *CloudCostAggregate {
+	return &CloudCostAggregate{
+		Properties:        cca.Properties,
+		KubernetesPercent: cca.KubernetesPercent,
+		Cost:              cca.Cost,
+		Credit:            cca.Credit,
+	}
+}
+
+func (cca *CloudCostAggregate) Equal(that *CloudCostAggregate) bool {
+	if that == nil {
+		return false
+	}
+
+	return cca.Cost == that.Cost &&
+		cca.Credit == that.Credit &&
+		cca.Properties.Equal(that.Properties)
+}
+
+func (cca *CloudCostAggregate) Key(props []string) string {
+	return cca.Properties.Key(props)
+}
+
+func (cca *CloudCostAggregate) StringProperty(prop string) (string, error) {
+	if cca == nil {
+		return "", nil
+	}
+
+	switch prop {
+	case CloudCostAccountProp:
+		return cca.Properties.Account, nil
+	case CloudCostProjectProp:
+		return cca.Properties.Project, nil
+	case CloudCostProviderProp:
+		return cca.Properties.Provider, nil
+	case CloudCostServiceProp:
+		return cca.Properties.Service, nil
+	case CloudCostLabelProp:
+		return cca.Properties.LabelValue, nil
+	default:
+		return "", fmt.Errorf("invalid property name: %s", prop)
+	}
+}
+
+func (cca *CloudCostAggregate) add(that *CloudCostAggregate) {
+	if cca == nil {
+		log.Warnf("cannot add to nil CloudCostAggregate")
+		return
+	}
+
+	// Compute KubernetesPercent for sum
+	k8sPct := 0.0
+	sumCost := cca.Cost + that.Cost
+	if sumCost > 0.0 {
+		thisK8sCost := cca.Cost * cca.KubernetesPercent
+		thatK8sCost := that.Cost * that.KubernetesPercent
+		k8sPct = (thisK8sCost + thatK8sCost) / sumCost
+	}
+
+	cca.Cost = sumCost
+	cca.Credit += that.Credit
+	cca.KubernetesPercent = k8sPct
+}
+
+type CloudCostAggregateSet struct {
+	CloudCostAggregates   map[string]*CloudCostAggregate `json:"items"`
+	AggregationProperties []string                       `json:"-"`
+	Integration           string                         `json:"-"`
+	LabelName             string                         `json:"labelName,omitempty"`
+	Window                Window                         `json:"window"`
+}
+
+func NewCloudCostAggregateSet(start, end time.Time, cloudCostAggregates ...*CloudCostAggregate) *CloudCostAggregateSet {
+	ccas := &CloudCostAggregateSet{
+		CloudCostAggregates: map[string]*CloudCostAggregate{},
+		Window:              NewWindow(&start, &end),
+	}
+
+	for _, cca := range cloudCostAggregates {
+		ccas.insertByProperty(cca, nil)
+	}
+
+	return ccas
+}
+
+func (ccas *CloudCostAggregateSet) Aggregate(props []string) (*CloudCostAggregateSet, error) {
+	if ccas == nil {
+		return nil, errors.New("cannot aggregate a nil CloudCostAggregateSet")
+	}
+
+	if ccas.Window.IsOpen() {
+		return nil, fmt.Errorf("cannot aggregate a CloudCostAggregateSet with an open window: %s", ccas.Window)
+	}
+
+	// Create a new result set, with the given aggregation property
+	result := NewCloudCostAggregateSet(*ccas.Window.Start(), *ccas.Window.End())
+	result.AggregationProperties = props
+	result.LabelName = ccas.LabelName
+	result.Integration = ccas.Integration
+
+	// Insert clones of each item in the set, keyed by the given property.
+	// The underlying insert logic will add binned items together.
+	for name, cca := range ccas.CloudCostAggregates {
+		ccaClone := cca.Clone()
+		err := result.insertByProperty(ccaClone, props)
+		if err != nil {
+			return nil, fmt.Errorf("error aggregating %s by %v: %s", name, props, err)
+		}
+	}
+
+	return result, nil
+}
+
+func (ccas *CloudCostAggregateSet) Filter(filters filter.Filter[*CloudCostAggregate]) *CloudCostAggregateSet {
+	if ccas == nil {
+		return nil
+	}
+
+	result := ccas.Clone()
+	result.filter(filters)
+
+	return result
+}
+
+func (ccas *CloudCostAggregateSet) filter(filters filter.Filter[*CloudCostAggregate]) {
+	if ccas == nil {
+		return
+	}
+
+	if filters == nil {
+		return
+	}
+
+	for name, cca := range ccas.CloudCostAggregates {
+		if !filters.Matches(cca) {
+			delete(ccas.CloudCostAggregates, name)
+		}
+	}
+}
+
+func (ccas *CloudCostAggregateSet) Insert(that *CloudCostAggregate) error {
+	// Publicly, only allow Inserting as a basic operation (i.e. without causing
+	// an aggregation on a property).
+	return ccas.insertByProperty(that, nil)
+}
+
+func (ccas *CloudCostAggregateSet) insertByProperty(that *CloudCostAggregate, props []string) error {
+	if ccas == nil {
+		return fmt.Errorf("cannot insert into nil CloudCostAggregateSet")
+	}
+
+	if ccas.CloudCostAggregates == nil {
+		ccas.CloudCostAggregates = map[string]*CloudCostAggregate{}
+	}
+
+	// Add the given CloudCostAggregate to the existing entry, if there is one;
+	// otherwise just set directly into allocations
+	if _, ok := ccas.CloudCostAggregates[that.Key(props)]; !ok {
+		ccas.CloudCostAggregates[that.Key(props)] = that
+	} else {
+		ccas.CloudCostAggregates[that.Key(props)].add(that)
+	}
+
+	return nil
+}
+
+func (ccas *CloudCostAggregateSet) Clone() *CloudCostAggregateSet {
+	aggs := make(map[string]*CloudCostAggregate, len(ccas.CloudCostAggregates))
+	for k, v := range ccas.CloudCostAggregates {
+		aggs[k] = v.Clone()
+	}
+
+	return &CloudCostAggregateSet{
+		CloudCostAggregates: aggs,
+		Integration:         ccas.Integration,
+		LabelName:           ccas.LabelName,
+		Window:              ccas.Window.Clone(),
+	}
+}
+
+func (ccas *CloudCostAggregateSet) Equal(that *CloudCostAggregateSet) bool {
+	if ccas.Integration != that.Integration {
+		return false
+	}
+
+	if ccas.LabelName != that.LabelName {
+		return false
+	}
+
+	if !ccas.Window.Equal(that.Window) {
+		return false
+	}
+
+	if len(ccas.CloudCostAggregates) != len(that.CloudCostAggregates) {
+		return false
+	}
+
+	for k, cca := range ccas.CloudCostAggregates {
+		tcca, ok := that.CloudCostAggregates[k]
+		if !ok {
+			return false
+		}
+		if !cca.Equal(tcca) {
+			return false
+		}
+	}
+
+	return true
+}
+
+func (ccas *CloudCostAggregateSet) IsEmpty() bool {
+	if ccas == nil {
+		return true
+	}
+
+	if len(ccas.CloudCostAggregates) == 0 {
+		return true
+	}
+
+	return false
+}
+
+func (ccas *CloudCostAggregateSet) Length() int {
+	if ccas == nil {
+		return 0
+	}
+	return len(ccas.CloudCostAggregates)
+}
+
+func (ccas *CloudCostAggregateSet) GetWindow() Window {
+	return ccas.Window
+}
+
+func (ccas *CloudCostAggregateSet) Merge(that *CloudCostAggregateSet) (*CloudCostAggregateSet, error) {
+	if ccas == nil || that == nil {
+		return nil, fmt.Errorf("cannot merge nil CloudCostAggregateSets")
+	}
+
+	if that.IsEmpty() {
+		return ccas.Clone(), nil
+	}
+
+	if !ccas.Window.Equal(that.Window) {
+		return nil, fmt.Errorf("cannot merge CloudCostAggregateSets with different windows")
+	}
+
+	if ccas.LabelName != that.LabelName {
+		return nil, fmt.Errorf("cannot merge CloudCostAggregateSets with different label names: '%s' != '%s'", ccas.LabelName, that.LabelName)
+	}
+
+	start, end := *ccas.Window.Start(), *ccas.Window.End()
+	result := NewCloudCostAggregateSet(start, end)
+	result.LabelName = ccas.LabelName
+
+	for _, cca := range ccas.CloudCostAggregates {
+		result.insertByProperty(cca, nil)
+	}
+
+	for _, cca := range that.CloudCostAggregates {
+		result.insertByProperty(cca, nil)
+	}
+
+	return result, nil
+}
+
+func GetCloudCostAggregateSets(start, end time.Time, windowDuration time.Duration, integration string, labelName string) ([]*CloudCostAggregateSet, error) {
+	windows, err := GetWindows(start, end, windowDuration)
+	if err != nil {
+		return nil, err
+	}
+
+	// Build slice of CloudCostAggregateSet to cover the range
+	CloudCostAggregateSets := []*CloudCostAggregateSet{}
+	for _, w := range windows {
+		ccas := NewCloudCostAggregateSet(*w.Start(), *w.End())
+		ccas.Integration = integration
+		ccas.LabelName = labelName
+		CloudCostAggregateSets = append(CloudCostAggregateSets, ccas)
+	}
+	return CloudCostAggregateSets, nil
+}
+
+// LoadCloudCostAggregateSets creates and loads CloudCostAggregates into provided CloudCostAggregateSets. This method makes it so
+// that the input windows do not have to match the one day frame of the Athena queries. CloudCostAggregates being generated from a
+// CUR which may be the identical except for the pricing model used (default, RI or savings plan)
+// are accumulated here so that the resulting CloudCostAggregate with the 1d window has the correct price for the entire day.
+func LoadCloudCostAggregateSets(itemStart time.Time, itemEnd time.Time, properties CloudCostAggregateProperties, K8sPercent, cost, credit float64, CloudCostAggregateSets []*CloudCostAggregateSet) {
+	// Disperse cost of the current item across one or more CloudCostAggregates in
+	// across each relevant CloudCostAggregateSet. Stop when the end of the current
+	// block reaches the item's end time or the end of the range.
+	for _, ccas := range CloudCostAggregateSets {
+		pct := ccas.GetWindow().GetPercentInWindow(itemStart, itemEnd)
+
+		// Insert an CloudCostAggregate with that cost into the CloudCostAggregateSet at the given index
+		cca := &CloudCostAggregate{
+			Properties:        properties,
+			KubernetesPercent: K8sPercent * pct,
+			Cost:              cost * pct,
+			Credit:            credit * pct,
+		}
+		err := ccas.insertByProperty(cca, nil)
+		if err != nil {
+			log.Errorf("LoadCloudCostAggregateSets: failed to load CloudCostAggregate with key %s and window %s", cca.Key(nil), ccas.GetWindow().String())
+		}
+	}
+}
+
+type CloudCostAggregateSetRange struct {
+	CloudCostAggregateSets []*CloudCostAggregateSet `json:"sets"`
+	Window                 Window                   `json:"window"`
+}
+
+func (ccasr *CloudCostAggregateSetRange) Accumulate() (*CloudCostAggregateSet, error) {
+	if ccasr == nil {
+		return nil, errors.New("cannot accumulate a nil CloudCostAggregateSetRange")
+	}
+
+	if ccasr.Window.IsOpen() {
+		return nil, fmt.Errorf("cannot accumulate a CloudCostAggregateSetRange with an open window: %s", ccasr.Window)
+	}
+
+	result := NewCloudCostAggregateSet(*ccasr.Window.Start(), *ccasr.Window.End())
+
+	for _, ccas := range ccasr.CloudCostAggregateSets {
+		for name, cca := range ccas.CloudCostAggregates {
+			err := result.insertByProperty(cca.Clone(), ccas.AggregationProperties)
+			if err != nil {
+				return nil, fmt.Errorf("error accumulating CloudCostAggregateSetRange[%s][%s]: %s", ccas.Window.String(), name, err)
+			}
+		}
+	}
+
+	return result, nil
+}

pkg/kubecost/cloudcostitem.go

@@ -0,0 +1,321 @@
+package kubecost
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/opencost/opencost/pkg/filter"
+	"github.com/opencost/opencost/pkg/log"
+)
+
+type CloudCostItemLabels map[string]string
+
+func (ccil CloudCostItemLabels) Clone() CloudCostItemLabels {
+	result := make(map[string]string, len(ccil))
+	for k, v := range ccil {
+		result[k] = v
+	}
+	return result
+}
+
+func (ccil CloudCostItemLabels) Equal(that CloudCostItemLabels) bool {
+	if len(ccil) != len(that) {
+		return false
+	}
+
+	// Maps are of equal length, so if all keys are in both maps, we don't
+	// have to check the keys of the other map.
+	for k, v := range ccil {
+		if tv, ok := that[k]; !ok || v != tv {
+			return false
+		}
+	}
+
+	return true
+}
+
+type CloudCostItemProperties struct {
+	ProviderID string              `json:"providerID,omitempty"`
+	Provider   string              `json:"provider,omitempty"`
+	Account    string              `json:"account,omitempty"`
+	Project    string              `json:"project,omitempty"`
+	Service    string              `json:"service,omitempty"`
+	Category   string              `json:"category,omitempty"`
+	Labels     CloudCostItemLabels `json:"labels,omitempty"`
+}
+
+func (ccip CloudCostItemProperties) Equal(that CloudCostItemProperties) bool {
+	return ccip.ProviderID == that.ProviderID &&
+		ccip.Provider == that.Provider &&
+		ccip.Account == that.Account &&
+		ccip.Project == that.Project &&
+		ccip.Service == that.Service &&
+		ccip.Category == that.Category &&
+		ccip.Labels.Equal(that.Labels)
+}
+
+func (ccip CloudCostItemProperties) Clone() CloudCostItemProperties {
+	return CloudCostItemProperties{
+		ProviderID: ccip.ProviderID,
+		Provider:   ccip.Provider,
+		Account:    ccip.Account,
+		Project:    ccip.Project,
+		Service:    ccip.Service,
+		Category:   ccip.Category,
+		Labels:     ccip.Labels.Clone(),
+	}
+}
+
+func (ccip CloudCostItemProperties) Key() string {
+	return fmt.Sprintf("%s/%s/%s/%s/%s/%s", ccip.Provider, ccip.Account, ccip.Project, ccip.Category, ccip.Service, ccip.ProviderID)
+}
+
+// CloudCostItem represents a CUR line item, identifying a cloud resource and
+// its cost over some period of time.
+type CloudCostItem struct {
+	Properties   CloudCostItemProperties
+	IsKubernetes bool
+	Window       Window
+	Cost         float64
+	Credit       float64
+}
+
+func (cci *CloudCostItem) Clone() *CloudCostItem {
+	return &CloudCostItem{
+		Properties:   cci.Properties.Clone(),
+		IsKubernetes: cci.IsKubernetes,
+		Window:       cci.Window.Clone(),
+		Cost:         cci.Cost,
+		Credit:       cci.Credit,
+	}
+}
+
+func (cci *CloudCostItem) Equal(that *CloudCostItem) bool {
+	if that == nil {
+		return false
+	}
+
+	return cci.Properties.Equal(that.Properties) &&
+		cci.IsKubernetes == that.IsKubernetes &&
+		cci.Window.Equal(that.Window) &&
+		cci.Cost == that.Cost &&
+		cci.Credit == that.Credit
+}
+
+func (cci *CloudCostItem) Key() string {
+	return cci.Properties.Key()
+}
+
+func (cci *CloudCostItem) add(that *CloudCostItem) {
+	if cci == nil {
+		log.Warnf("cannot add to nil CloudCostItem")
+		return
+	}
+
+	cci.Cost += that.Cost
+	cci.Credit += that.Credit
+	cci.Window = cci.Window.Expand(that.Window)
+}
+
+type CloudCostItemSet struct {
+	CloudCostItems map[string]*CloudCostItem
+	Window         Window
+	Integration    string
+}
+
+// NewAssetSet instantiates a new AssetSet and, optionally, inserts
+// the given list of Assets
+func NewCloudCostItemSet(start, end time.Time, cloudCostItems ...*CloudCostItem) *CloudCostItemSet {
+	ccis := &CloudCostItemSet{
+		CloudCostItems: map[string]*CloudCostItem{},
+		Window:         NewWindow(&start, &end),
+	}
+
+	for _, cci := range cloudCostItems {
+		ccis.Insert(cci)
+	}
+
+	return ccis
+}
+
+func (ccis *CloudCostItemSet) Equal(that *CloudCostItemSet) bool {
+	if ccis.Integration != that.Integration {
+		return false
+	}
+
+	if !ccis.Window.Equal(that.Window) {
+		return false
+	}
+
+	if len(ccis.CloudCostItems) != len(that.CloudCostItems) {
+		return false
+	}
+
+	for k, cci := range ccis.CloudCostItems {
+		tcci, ok := that.CloudCostItems[k]
+		if !ok {
+			return false
+		}
+		if !cci.Equal(tcci) {
+			return false
+		}
+	}
+
+	return true
+}
+
+func (ccis *CloudCostItemSet) Filter(filters filter.Filter[*CloudCostItem]) *CloudCostItemSet {
+	if ccis == nil {
+		return nil
+	}
+
+	if filters == nil {
+		return ccis.Clone()
+	}
+
+	result := NewCloudCostItemSet(*ccis.Window.start, *ccis.Window.end)
+
+	for _, cci := range ccis.CloudCostItems {
+		if filters.Matches(cci) {
+			result.Insert(cci.Clone())
+		}
+	}
+
+	return result
+}
+
+func (ccis *CloudCostItemSet) Insert(that *CloudCostItem) error {
+	if ccis == nil {
+		return fmt.Errorf("cannot insert into nil CloudCostItemSet")
+	}
+
+	if that == nil {
+		return fmt.Errorf("cannot insert nil CloudCostItem into CloudCostItemSet")
+	}
+
+	if ccis.CloudCostItems == nil {
+		ccis.CloudCostItems = map[string]*CloudCostItem{}
+	}
+
+	// Add the given CloudCostItem to the existing entry, if there is one;
+	// otherwise just set directly into allocations
+	if _, ok := ccis.CloudCostItems[that.Key()]; !ok {
+		ccis.CloudCostItems[that.Key()] = that.Clone()
+	} else {
+		ccis.CloudCostItems[that.Key()].add(that)
+	}
+
+	return nil
+}
+
+func (ccis *CloudCostItemSet) Clone() *CloudCostItemSet {
+	items := make(map[string]*CloudCostItem, len(ccis.CloudCostItems))
+	for k, v := range ccis.CloudCostItems {
+		items[k] = v.Clone()
+	}
+
+	return &CloudCostItemSet{
+		CloudCostItems: items,
+		Integration:    ccis.Integration,
+		Window:         ccis.Window.Clone(),
+	}
+}
+
+func (ccis *CloudCostItemSet) IsEmpty() bool {
+	if ccis == nil {
+		return true
+	}
+
+	if len(ccis.CloudCostItems) == 0 {
+		return true
+	}
+
+	return false
+}
+
+func (ccis *CloudCostItemSet) Length() int {
+	if ccis == nil {
+		return 0
+	}
+	return len(ccis.CloudCostItems)
+}
+
+func (ccis *CloudCostItemSet) GetWindow() Window {
+	return ccis.Window
+}
+
+func (ccis *CloudCostItemSet) Merge(that *CloudCostItemSet) (*CloudCostItemSet, error) {
+	if ccis == nil {
+		return nil, fmt.Errorf("cannot merge nil CloudCostItemSets")
+	}
+
+	if that.IsEmpty() {
+		return ccis.Clone(), nil
+	}
+
+	if !ccis.Window.Equal(that.Window) {
+		return nil, fmt.Errorf("cannot merge CloudCostItemSets with different windows")
+	}
+
+	start, end := *ccis.Window.Start(), *ccis.Window.End()
+	result := NewCloudCostItemSet(start, end)
+
+	for _, cci := range ccis.CloudCostItems {
+		result.Insert(cci)
+	}
+
+	for _, cci := range that.CloudCostItems {
+		result.Insert(cci)
+	}
+
+	return result, nil
+}
+
+// GetCloudCostItemSets
+func GetCloudCostItemSets(start time.Time, end time.Time, window time.Duration, integration string) ([]*CloudCostItemSet, error) {
+	windows, err := GetWindows(start, end, window)
+	if err != nil {
+		return nil, err
+	}
+
+	// Build slice of CloudCostItemSet to cover the range
+	CloudCostItemSets := []*CloudCostItemSet{}
+	for _, w := range windows {
+		ccis := NewCloudCostItemSet(*w.Start(), *w.End())
+		ccis.Integration = integration
+		CloudCostItemSets = append(CloudCostItemSets, ccis)
+	}
+	return CloudCostItemSets, nil
+}
+
+// LoadCloudCostItemSets creates and loads CloudCostItems into provided CloudCostItemSets. This method makes it so
+// that the input windows do not have to match the one day frame of the Athena queries. CloudCostItems being generated from a
+// CUR which may be the identical except for the pricing model used (default, RI or savings plan)
+// are accumulated here so that the resulting CloudCostItem with the 1d window has the correct price for the entire day.
+func LoadCloudCostItemSets(itemStart time.Time, itemEnd time.Time, properties CloudCostItemProperties, isK8s bool, cost, credit float64, CloudCostItemSets []*CloudCostItemSet) {
+
+	// Disperse cost of the current item across one or more CloudCostItems in
+	// across each relevant CloudCostItemSet. Stop when the end of the current
+	// block reaches the item's end time or the end of the range.
+	for _, ccis := range CloudCostItemSets {
+		pct := ccis.GetWindow().GetPercentInWindow(itemStart, itemEnd)
+
+		// Insert an CloudCostItem with that cost into the CloudCostItemSet at the given index
+		cci := &CloudCostItem{
+			Properties:   properties,
+			IsKubernetes: isK8s,
+			Window:       ccis.GetWindow(),
+			Cost:         cost * pct,
+			Credit:       credit * pct,
+		}
+		err := ccis.Insert(cci)
+		if err != nil {
+			log.Errorf("LoadCloudCostItemSets: failed to load CloudCostItem with key %s and window %s: %s", cci.Key(), ccis.GetWindow().String(), err.Error())
+		}
+	}
+}
+
+type CloudCostItemSetRange struct {
+	CloudCostItemSets []*CloudCostItemSet `json:"sets"`
+	Window            Window              `json:"window"`
+}

pkg/kubecost/coverage.go

@@ -0,0 +1,118 @@
+package kubecost
+
+import (
+	"time"
+
+	"github.com/opencost/opencost/pkg/filter"
+)
+
+// Coverage This is a placeholder struct which can be replaced by a more specific implementation later
+type Coverage struct {
+	Window   Window    `json:"window"`
+	Type     string    `json:"type"`
+	Count    int       `json:"count"`
+	Updated  time.Time `json:"updated"`
+	Errors   []string  `json:"errors"`
+	Warnings []string  `json:"warnings"`
+}
+
+func (c *Coverage) GetWindow() Window {
+	return c.Window
+}
+
+func (c *Coverage) Key() string {
+	return c.Type
+}
+
+func (c *Coverage) IsEmpty() bool {
+	return c.Type == "" && c.Count == 0 && len(c.Errors) == 0 && len(c.Warnings) == 0 && c.Updated == time.Time{}
+}
+
+func (c *Coverage) Clone() *Coverage {
+	var errors []string
+	if len(c.Errors) > 0 {
+		errors = make([]string, len(c.Errors))
+		copy(errors, c.Errors)
+	}
+	var warnings []string
+	if len(c.Warnings) > 0 {
+		warnings = make([]string, len(c.Warnings))
+		copy(warnings, c.Warnings)
+	}
+	return &Coverage{
+		Window:   c.Window.Clone(),
+		Type:     c.Type,
+		Count:    c.Count,
+		Updated:  c.Updated,
+		Errors:   errors,
+		Warnings: warnings,
+	}
+}
+
+// Coverage This is a placeholder struct which can be replaced by a more specific implementation later
+type CoverageSet struct {
+	Window Window               `json:"window"`
+	Items  map[string]*Coverage `json:"items"`
+}
+
+func NewCoverageSet(start, end time.Time) *CoverageSet {
+	return &CoverageSet{
+		Window: NewWindow(&start, &end),
+		Items:  map[string]*Coverage{},
+	}
+}
+
+func (cs *CoverageSet) GetWindow() Window {
+	return cs.Window
+}
+
+func (cs *CoverageSet) IsEmpty() bool {
+	for _, item := range cs.Items {
+		if !item.IsEmpty() {
+			return false
+		}
+	}
+	return true
+}
+
+func (cs *CoverageSet) Clone() *CoverageSet {
+	var items map[string]*Coverage
+	if cs.Items != nil {
+		items = make(map[string]*Coverage, len(cs.Items))
+		for k, item := range cs.Items {
+			items[k] = item.Clone()
+		}
+
+	}
+	return &CoverageSet{
+		Window: cs.Window.Clone(),
+		Items:  items,
+	}
+}
+
+func (cs *CoverageSet) Insert(coverage *Coverage) {
+	if cs.Items == nil {
+		cs.Items = map[string]*Coverage{}
+	}
+	cs.Items[coverage.Key()] = coverage
+}
+
+func (cs *CoverageSet) Filter(filters filter.Filter[*Coverage]) *CoverageSet {
+	if cs == nil {
+		return nil
+	}
+
+	if filters == nil {
+		return cs.Clone()
+	}
+
+	result := NewCoverageSet(*cs.Window.start, *cs.Window.end)
+
+	for _, c := range cs.Items {
+		if filters.Matches(c) {
+			result.Insert(c.Clone())
+		}
+	}
+
+	return result
+}

pkg/kubecost/diff_test.go

@@ -16,20 +16,20 @@ func TestDiff(t *testing.T) {
 	node1.CPUCost = 10
 	node1b := node1.Clone().(*Node)
 	node1b.CPUCost = 20
-	node1Key, _ := key(node1, nil)
+	node1Key, _ := key(node1, nil, nil)
 	node2 := NewNode("node2", "cluster1", "123abc", start, end, window1)
 	node2.CPUCost = 100
 	node2b := node2.Clone().(*Node)
 	node2b.CPUCost = 105
-	node2Key, _ := key(node2, nil)
+	node2Key, _ := key(node2, nil, nil)
 	node3 := NewNode("node3", "cluster1", "123abc", start, end, window1)
-	node3Key, _ := key(node3, nil)
+	node3Key, _ := key(node3, nil, nil)
 	node4 := NewNode("node4", "cluster1", "123abc", start, end, window1)
-	node4Key, _ := key(node4, nil)
+	node4Key, _ := key(node4, nil, nil)
 	disk1 := NewDisk("disk1", "cluster1", "123abc", start, end, window1)
-	disk1Key, _ := key(disk1, nil)
+	disk1Key, _ := key(disk1, nil, nil)
 	disk2 := NewDisk("disk2", "cluster1", "123abc", start, end, window1)
-	disk2Key, _ := key(disk2, nil)
+	disk2Key, _ := key(disk2, nil, nil)
 
 	cases := map[string]struct {
 		inputAssetsBefore []Asset

pkg/kubecost/kubecost_codecs_test.go

@@ -51,7 +51,7 @@ func BenchmarkAllocationSetRange_BinaryEncoding(b *testing.B) {
 			b.Fatalf("AllocationSetRange.Binary: expected %s; found %s", asr0.Window(), asr1.Window())
 		}
 
-		asr0.Each(func(i int, as0 *AllocationSet) {
+		for i, as0 := range asr0.Allocations {
 			as1, err := asr1.Get(i)
 			if err != nil {
 				b.Fatalf("AllocationSetRange.Binary: unexpected error: %s", err)
@@ -64,7 +64,7 @@ func BenchmarkAllocationSetRange_BinaryEncoding(b *testing.B) {
 				b.Fatalf("AllocationSetRange.Binary: expected %s; found %s", as0.Window, as1.Window)
 			}
 
-			as0.Each(func(k string, a0 *Allocation) {
+			for k, a0 := range as0.Allocations {
 				a1 := as1.Get(k)
 				if a1 == nil {
 					b.Fatalf("AllocationSetRange.Binary: missing Allocation: %s", a0)
@@ -73,8 +73,8 @@ func BenchmarkAllocationSetRange_BinaryEncoding(b *testing.B) {
 				if !a0.Equal(a1) {
 					b.Fatalf("AllocationSetRange.Binary: unequal Allocations \"%s\": expected %s; found %s", k, a0, a1)
 				}
-			})
-		})
+			}
+		}
 	}
 }
 
@@ -115,7 +115,7 @@ func TestAllocationSetRange_BinaryEncoding(t *testing.T) {
 		t.Fatalf("AllocationSetRange.Binary: expected %s; found %s", asr0.Window(), asr1.Window())
 	}
 
-	asr0.Each(func(i int, as0 *AllocationSet) {
+	for i, as0 := range asr0.Allocations {
 		as1, err := asr1.Get(i)
 		if err != nil {
 			t.Fatalf("AllocationSetRange.Binary: unexpected error: %s", err)
@@ -128,7 +128,7 @@ func TestAllocationSetRange_BinaryEncoding(t *testing.T) {
 			t.Fatalf("AllocationSetRange.Binary: expected %s; found %s", as0.Window, as1.Window)
 		}
 
-		as0.Each(func(k string, a0 *Allocation) {
+		for k, a0 := range as0.Allocations {
 			a1 := as1.Get(k)
 			if a1 == nil {
 				t.Fatalf("AllocationSetRange.Binary: missing Allocation: %s", a0)
@@ -137,10 +137,10 @@ func TestAllocationSetRange_BinaryEncoding(t *testing.T) {
 			// TODO Sean: fix JSON marshaling of PVs
 			a1.PVs = a0.PVs
 			if !a0.Equal(a1) {
-				t.Fatalf("AllocationSetRange.Binary: unequal Allocations \"%s\": expected %s; found %s", k, a0, a1)
+				t.Fatalf("AllocationSetRange.Binary: unequal Allocations \"%s\": expected \"%s\"; found \"%s\"", k, a0, a1)
 			}
-		})
-	})
+		}
+	}
 }
 
 func TestAny_BinaryEncoding(t *testing.T) {
@@ -172,23 +172,23 @@ func TestAny_BinaryEncoding(t *testing.T) {
 		t.Fatalf("Any.Binary: unexpected error: %s", err)
 	}
 
-	if a1.Properties().Name != a0.Properties().Name {
-		t.Fatalf("Any.Binary: expected %s, found %s", a0.Properties().Name, a1.Properties().Name)
+	if a1.Properties.Name != a0.Properties.Name {
+		t.Fatalf("Any.Binary: expected %s, found %s", a0.Properties.Name, a1.Properties.Name)
 	}
-	if a1.Properties().Cluster != a0.Properties().Cluster {
-		t.Fatalf("Any.Binary: expected %s, found %s", a0.Properties().Cluster, a1.Properties().Cluster)
+	if a1.Properties.Cluster != a0.Properties.Cluster {
+		t.Fatalf("Any.Binary: expected %s, found %s", a0.Properties.Cluster, a1.Properties.Cluster)
 	}
-	if a1.Properties().ProviderID != a0.Properties().ProviderID {
-		t.Fatalf("Any.Binary: expected %s, found %s", a0.Properties().ProviderID, a1.Properties().ProviderID)
+	if a1.Properties.ProviderID != a0.Properties.ProviderID {
+		t.Fatalf("Any.Binary: expected %s, found %s", a0.Properties.ProviderID, a1.Properties.ProviderID)
 	}
-	if a1.Adjustment() != a0.Adjustment() {
-		t.Fatalf("Any.Binary: expected %f, found %f", a0.Adjustment(), a1.Adjustment())
+	if a1.Adjustment != a0.Adjustment {
+		t.Fatalf("Any.Binary: expected %f, found %f", a0.Adjustment, a1.Adjustment)
 	}
 	if a1.TotalCost() != a0.TotalCost() {
 		t.Fatalf("Any.Binary: expected %f, found %f", a0.TotalCost(), a1.TotalCost())
 	}
-	if !a1.Window().Equal(a0.Window()) {
-		t.Fatalf("Any.Binary: expected %s, found %s", a0.Window(), a1.Window())
+	if !a1.Window.Equal(a0.Window) {
+		t.Fatalf("Any.Binary: expected %s, found %s", a0.Window, a1.Window)
 	}
 }
 
@@ -237,7 +237,7 @@ func TestAssetSetRange_BinaryEncoding(t *testing.T) {
 		t.Fatalf("AssetSetRange.Binary: expected %s; found %s", asr0.Window(), asr1.Window())
 	}
 
-	asr0.Each(func(i int, as0 *AssetSet) {
+	for i, as0 := range asr0.Assets {
 		as1, err := asr1.Get(i)
 		if err != nil {
 			t.Fatalf("AssetSetRange.Binary: unexpected error: %s", err)
@@ -250,7 +250,7 @@ func TestAssetSetRange_BinaryEncoding(t *testing.T) {
 			t.Fatalf("AssetSetRange.Binary: expected %s; found %s", as0.Window, as1.Window)
 		}
 
-		as0.Each(func(k string, a0 Asset) {
+		for k, a0 := range as0.Assets {
 			a1, ok := as1.Get(k)
 			if !ok {
 				t.Fatalf("AssetSetRange.Binary: missing Asset: %s", a0)
@@ -259,8 +259,8 @@ func TestAssetSetRange_BinaryEncoding(t *testing.T) {
 			if !a0.Equal(a1) {
 				t.Fatalf("AssetSetRange.Binary: unequal Assets \"%s\": expected %s; found %s", k, a0, a1)
 			}
-		})
-	})
+		}
+	}
 }
 
 func TestBreakdown_BinaryEncoding(t *testing.T) {

pkg/kubecost/mock.go

@@ -8,7 +8,10 @@ import (
 const gb = 1024 * 1024 * 1024
 const day = 24 * time.Hour
 
-var disk = PVKey{}
+var disk = PVKey{
+	Cluster: "cluster1",
+	Name:    "pv1",
+}
 
 // NewMockUnitAllocation creates an *Allocation with all of its float64 values set to 1 and generic properties if not provided in arg
 func NewMockUnitAllocation(name string, start time.Time, resolution time.Duration, props *AllocationProperties) *Allocation {
@@ -328,7 +331,7 @@ func GenerateMockAllocationSetWithAssetProperties(start time.Time) *AllocationSe
 		Cluster: "cluster2",
 		Name:    "disk2",
 	}
-	for _, a := range as.allocations {
+	for _, a := range as.Allocations {
 		// add reconcilable pvs to pod-mno
 		if a.Properties.Pod == "pod-mno" {
 			a.PVs = a.PVs.Add(PVAllocations{
@@ -398,7 +401,7 @@ func GenerateMockAssetSets(start, end time.Time) []*AssetSet {
 	cluster1Nodes.CPUCost = 55.0
 	cluster1Nodes.RAMCost = 44.0
 	cluster1Nodes.GPUCost = 11.0
-	cluster1Nodes.adjustment = -10.00
+	cluster1Nodes.Adjustment = -10.00
 	cluster1Nodes.CPUCoreHours = 8
 	cluster1Nodes.RAMByteHours = 6
 	cluster1Nodes.GPUHours = 24
@@ -430,12 +433,12 @@ func GenerateMockAssetSets(start, end time.Time) []*AssetSet {
 	// Add PVs
 	cluster2Disk1 := NewDisk("disk1", "cluster2", "disk1", start, end, NewWindow(&start, &end))
 	cluster2Disk1.Cost = 5.0
-	cluster2Disk1.adjustment = 1.0
+	cluster2Disk1.Adjustment = 1.0
 	cluster2Disk1.ByteHours = 5 * gb
 
 	cluster2Disk2 := NewDisk("disk2", "cluster2", "disk2", start, end, NewWindow(&start, &end))
 	cluster2Disk2.Cost = 10.0
-	cluster2Disk2.adjustment = 3.0
+	cluster2Disk2.Adjustment = 3.0
 	cluster2Disk2.ByteHours = 10 * gb
 
 	cluster2Node1Disk := NewDisk("node1", "cluster2", "node1", start, end, NewWindow(&start, &end))
@@ -514,7 +517,7 @@ func GenerateMockAssetSets(start, end time.Time) []*AssetSet {
 	cluster1Nodes.CPUCost = 5.0
 	cluster1Nodes.RAMCost = 4.0
 	cluster1Nodes.GPUCost = 1.0
-	cluster1Nodes.adjustment = 90.00
+	cluster1Nodes.Adjustment = 90.00
 	cluster1Nodes.CPUCoreHours = 8
 	cluster1Nodes.RAMByteHours = 6
 	cluster1Nodes.GPUHours = 24
@@ -546,12 +549,12 @@ func GenerateMockAssetSets(start, end time.Time) []*AssetSet {
 	// Add PVs
 	cluster2Disk1 = NewDisk("disk1", "cluster2", "disk1", start, end, NewWindow(&start, &end))
 	cluster2Disk1.Cost = 5.0
-	cluster2Disk1.adjustment = 1.0
+	cluster2Disk1.Adjustment = 1.0
 	cluster2Disk1.ByteHours = 5 * gb
 
 	cluster2Disk2 = NewDisk("disk2", "cluster2", "disk2", start, end, NewWindow(&start, &end))
 	cluster2Disk2.Cost = 12.0
-	cluster2Disk2.adjustment = 4.0
+	cluster2Disk2.Adjustment = 4.0
 	cluster2Disk2.ByteHours = 20 * gb
 
 	assetSet2 := NewAssetSet(start, end, cluster1Nodes, cluster2Node1, cluster2Node2, cluster2Node3, cluster2Disk1,

pkg/kubecost/query.go

@@ -67,6 +67,7 @@ type AssetQueryOptions struct {
 	IncludeCloud            bool
 	SharedHourlyCosts       map[string]float64
 	Step                    time.Duration
+	LabelConfig             *LabelConfig
 }
 
 // CloudUsageQueryOptions define optional parameters for querying a Store
@@ -76,6 +77,7 @@ type CloudUsageQueryOptions struct {
 	Compute      bool
 	FilterFuncs  []CloudUsageMatchFunc
 	FilterValues CloudUsageFilter
+	LabelConfig  *LabelConfig
 }
 
 type CloudUsageFilter struct {

pkg/kubecost/summaryallocation.go

@@ -170,7 +170,7 @@ func (sa *SummaryAllocation) Clone() *SummaryAllocation {
 // no usage or cost, then efficiency is zero. If there is no request, but there
 // is usage or cost, then efficiency is 100%.
 func (sa *SummaryAllocation) CPUEfficiency() float64 {
-	if sa == nil {
+	if sa == nil || sa.IsIdle() {
 		return 0.0
 	}
 
@@ -245,7 +245,7 @@ func (sa *SummaryAllocation) Minutes() float64 {
 // no usage or cost, then efficiency is zero. If there is no request, but there
 // is usage or cost, then efficiency is 100%.
 func (sa *SummaryAllocation) RAMEfficiency() float64 {
-	if sa == nil {
+	if sa == nil || sa.IsIdle() {
 		return 0.0
 	}
 
@@ -272,7 +272,7 @@ func (sa *SummaryAllocation) TotalCost() float64 {
 // TotalEfficiency is the cost-weighted average of CPU and RAM efficiency. If
 // there is no cost at all, then efficiency is zero.
 func (sa *SummaryAllocation) TotalEfficiency() float64 {
-	if sa == nil {
+	if sa == nil || sa.IsIdle() {
 		return 0.0
 	}
 
@@ -318,7 +318,7 @@ func NewSummaryAllocationSet(as *AllocationSet, filter AllocationFilter, kfs []A
 	if filter == nil && len(kfs) == 0 {
 		// No filters, so make the map of summary allocations exactly the size
 		// of the origin allocation set.
-		sasMap = make(map[string]*SummaryAllocation, len(as.allocations))
+		sasMap = make(map[string]*SummaryAllocation, len(as.Allocations))
 	} else {
 		// There are filters, so start with a standard map
 		sasMap = make(map[string]*SummaryAllocation)
@@ -329,7 +329,7 @@ func NewSummaryAllocationSet(as *AllocationSet, filter AllocationFilter, kfs []A
 		Window:             as.Window.Clone(),
 	}
 
-	for _, alloc := range as.allocations {
+	for _, alloc := range as.Allocations {
 		// First, detect if the allocation should be kept. If so, mark it as
 		// such, insert it, and continue.
 		shouldKeep := false
@@ -358,11 +358,11 @@ func NewSummaryAllocationSet(as *AllocationSet, filter AllocationFilter, kfs []A
 		}
 	}
 
-	for key := range as.externalKeys {
+	for key := range as.ExternalKeys {
 		sas.externalKeys[key] = true
 	}
 
-	for key := range as.idleKeys {
+	for key := range as.IdleKeys {
 		sas.idleKeys[key] = true
 	}
 
@@ -1170,6 +1170,96 @@ func (sas *SummaryAllocationSet) TotalCost() float64 {
 	return tc
 }
 
+// RAMEfficiency func to calculate average RAM efficiency over SummaryAllocationSet
+func (sas *SummaryAllocationSet) RAMEfficiency() float64 {
+	if sas == nil {
+		return 0.0
+	}
+
+	sas.RLock()
+	defer sas.RUnlock()
+
+	totalRAMBytesMinutesUsage := 0.0
+	totalRAMBytesMinutesRequest := 0.0
+	totalRAMCost := 0.0
+	for _, sa := range sas.SummaryAllocations {
+		if sa.IsIdle() {
+			continue
+		}
+		totalRAMBytesMinutesUsage += sa.RAMBytesUsageAverage * sa.Minutes()
+		totalRAMBytesMinutesRequest += sa.RAMBytesRequestAverage * sa.Minutes()
+		totalRAMCost += sa.RAMCost
+	}
+
+	if totalRAMBytesMinutesRequest > 0 {
+		return totalRAMBytesMinutesUsage / totalRAMBytesMinutesRequest
+	}
+
+	if totalRAMBytesMinutesUsage == 0.0 || totalRAMCost == 0.0 {
+		return 0.0
+	}
+
+	return 1.0
+}
+
+// CPUEfficiency func to calculate average CPU efficiency over SummaryAllocationSet
+func (sas *SummaryAllocationSet) CPUEfficiency() float64 {
+	if sas == nil {
+		return 0.0
+	}
+
+	sas.RLock()
+	defer sas.RUnlock()
+
+	totalCPUCoreMinutesUsage := 0.0
+	totalCPUCoreMinutesRequest := 0.0
+	totalCPUCost := 0.0
+	for _, sa := range sas.SummaryAllocations {
+		if sa.IsIdle() {
+			continue
+		}
+		totalCPUCoreMinutesUsage += sa.CPUCoreUsageAverage * sa.Minutes()
+		totalCPUCoreMinutesRequest += sa.CPUCoreRequestAverage * sa.Minutes()
+		totalCPUCost += sa.CPUCost
+	}
+
+	if totalCPUCoreMinutesRequest > 0 {
+		return totalCPUCoreMinutesUsage / totalCPUCoreMinutesRequest
+	}
+
+	if totalCPUCoreMinutesUsage == 0.0 || totalCPUCost == 0.0 {
+		return 0.0
+	}
+
+	return 1.0
+}
+
+// TotalEfficiency func to calculate average Total efficiency over SummaryAllocationSet
+func (sas *SummaryAllocationSet) TotalEfficiency() float64 {
+	if sas == nil {
+		return 0.0
+	}
+
+	sas.RLock()
+	defer sas.RUnlock()
+
+	totalRAMCost := 0.0
+	totalCPUCost := 0.0
+	for _, sa := range sas.SummaryAllocations {
+		if sa.IsIdle() {
+			continue
+		}
+		totalRAMCost += sa.RAMCost
+		totalCPUCost += sa.CPUCost
+	}
+
+	if totalRAMCost+totalCPUCost > 0 {
+		return (totalRAMCost*sas.RAMEfficiency() + totalCPUCost*sas.CPUEfficiency()) / (totalRAMCost + totalCPUCost)
+	}
+
+	return 0.0
+}
+
 // SummaryAllocationSetRange is a thread-safe slice of SummaryAllocationSets.
 type SummaryAllocationSetRange struct {
 	sync.RWMutex
@@ -1230,6 +1320,39 @@ func (sasr *SummaryAllocationSetRange) Accumulate() (*SummaryAllocationSet, erro
 	return result, nil
 }
 
+// NewAccumulation clones the first available SummaryAllocationSet to use as the data structure to
+// accumulate the remaining data. This leaves the original SummaryAllocationSetRange intact.
+func (sasr *SummaryAllocationSetRange) NewAccumulation() (*SummaryAllocationSet, error) {
+	var result *SummaryAllocationSet
+	var err error
+
+	sasr.RLock()
+	defer sasr.RUnlock()
+
+	for _, sas := range sasr.SummaryAllocationSets {
+		// we want to clone the first summary allocation set, then just Add the others
+		// to the clone. We will eventually use the clone to create the set range.
+		if result == nil {
+			result = sas.Clone()
+			continue
+		}
+
+		// Copy if sas is non-nil
+		var sasCopy *SummaryAllocationSet = nil
+		if sas != nil {
+			sasCopy = sas.Clone()
+		}
+
+		// nil is ok to pass into Add
+		result, err = result.Add(sasCopy)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return result, nil
+}
+
 // AggregateBy aggregates each AllocationSet in the range by the given
 // properties and options.
 func (sasr *SummaryAllocationSetRange) AggregateBy(aggregateBy []string, options *AllocationAggregationOptions) error {
@@ -1313,27 +1436,27 @@ func (sasr *SummaryAllocationSetRange) InsertExternalAllocations(that *Allocatio
 	}
 
 	var err error
-	that.Each(func(j int, thatAS *AllocationSet) {
+	for _, thatAS := range that.Allocations {
 		if thatAS == nil || err != nil {
-			return
+			continue
 		}
 
 		// Find matching AllocationSet in asr
 		i, ok := keys[thatAS.Window.String()]
 		if !ok {
 			err = fmt.Errorf("cannot merge AllocationSet into window that does not exist: %s", thatAS.Window.String())
-			return
+			continue
 		}
 		sas := sasr.SummaryAllocationSets[i]
 
 		// Insert each Allocation from the given set
-		thatAS.Each(func(k string, alloc *Allocation) {
+		for _, alloc := range thatAS.Allocations {
 			externalSA := NewSummaryAllocation(alloc, true, true)
 			// This error will be returned below
 			// TODO:CLEANUP should Each have early-error-return functionality?
 			err = sas.Insert(externalSA)
-		})
-	})
+		}
+	}
 
 	// err might be nil
 	return err

pkg/kubecost/summaryallocation_test.go

@@ -210,3 +210,670 @@ func TestSummaryAllocation_Add(t *testing.T) {
 		}
 	})
 }
+
+func TestSummaryAllocationSet_RAMEfficiency(t *testing.T) {
+	// Generating 6 sample summary allocations for testing
+	var sa1, sa2, sa3, sa4, sa5, sa6, idlesa *SummaryAllocation
+
+	// Generating accumulated summary allocation sets for testing
+	var sas1, sas2, sas3, sas4, sas5, sas6 *SummaryAllocationSet
+
+	window, _ := ParseWindowUTC("7d")
+
+	saStart := *window.Start()
+
+	saEnd := *window.End()
+
+	sa1 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container1",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container1",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		RAMBytesRequestAverage: 50.0 * 1024.0 * 1024.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                0.05,
+	}
+
+	sa2 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container2",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container2",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		RAMBytesRequestAverage: 50.0 * 1024.0 * 1024.0,
+		RAMBytesUsageAverage:   15.0 * 1024.0 * 1024.0,
+		RAMCost:                0.10,
+	}
+
+	sa3 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container3",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container3",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		RAMBytesRequestAverage: 0.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                0.0,
+	}
+
+	sa4 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container4",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		RAMBytesRequestAverage: 0.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                0.0,
+	}
+
+	sa5 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container5",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		RAMBytesRequestAverage: 0.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                0.10,
+	}
+
+	sa6 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container6",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		RAMBytesRequestAverage: 0.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                0.10,
+	}
+
+	idlesa = &SummaryAllocation{
+		Name: IdleSuffix,
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container7",
+		},
+		Start:   saStart,
+		End:     saEnd,
+		CPUCost: 1.0,
+		RAMCost: 1.0,
+	}
+
+	testcase1Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container1": sa1,
+		"cluster1/namespace1/pod1/container2": sa2,
+	}
+
+	testcase2Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container3": sa3,
+		"cluster1/namespace1/pod1/container4": sa4,
+	}
+
+	testcase3Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container5": sa5,
+		"cluster1/namespace1/pod1/container6": sa6,
+	}
+
+	testcase4Map := map[string]*SummaryAllocation{}
+
+	testcase5Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container1": sa1,
+		"cluster1/namespace1/pod1/container2": sa2,
+		"cluster1/namespace1/pod1/container3": sa3,
+		"cluster1/namespace1/pod1/container4": sa4,
+		"cluster1/namespace1/pod1/container5": sa5,
+		"cluster1/namespace1/pod1/container6": sa6,
+	}
+
+	testcase6Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container1": sa1,
+		"cluster1/namespace1/pod1/container2": sa2,
+		"cluster1/__idle__":                   idlesa,
+	}
+
+	sas1 = &SummaryAllocationSet{
+		SummaryAllocations: testcase1Map,
+		Window:             window,
+	}
+
+	sas2 = &SummaryAllocationSet{
+		SummaryAllocations: testcase2Map,
+		Window:             window,
+	}
+
+	sas3 = &SummaryAllocationSet{
+		SummaryAllocations: testcase3Map,
+		Window:             window,
+	}
+
+	sas4 = &SummaryAllocationSet{
+		SummaryAllocations: testcase4Map,
+		Window:             window,
+	}
+
+	sas5 = &SummaryAllocationSet{
+		SummaryAllocations: testcase5Map,
+		Window:             window,
+	}
+
+	sas6 = &SummaryAllocationSet{
+		SummaryAllocations: testcase6Map,
+		Window:             window,
+	}
+
+	cases := []struct {
+		name               string
+		testsas            *SummaryAllocationSet
+		expectedEfficiency float64
+	}{
+		{
+			name:               "Check RAMEfficiency when totalRAMBytesRequest over allocation summary set is greater than 0",
+			testsas:            sas1,
+			expectedEfficiency: 0.25,
+		},
+		{
+			name:               "Check RAMEfficiency when totalRAMBytesRequest is 0 and totalRAMCost or totalRAMBytesUsage equal to 0",
+			testsas:            sas2,
+			expectedEfficiency: 0.0,
+		},
+		{
+			name:               "Check RAMEfficiency when totalRAMBytesRequest is 0 and totalRAMCost or totalRAMBytesUsage is not 0",
+			testsas:            sas3,
+			expectedEfficiency: 1.0,
+		},
+		{
+			name:               "Check RAMEfficiency when allocation summary set is empty",
+			testsas:            sas4,
+			expectedEfficiency: 0.0,
+		},
+		{
+			name:               "Check RAMEfficiency over combination of all allocation summaries",
+			testsas:            sas5,
+			expectedEfficiency: 0.65,
+		},
+		{
+			name:               "Check RAMEfficiency in presense of an idle allocation",
+			testsas:            sas6,
+			expectedEfficiency: 0.25,
+		},
+	}
+
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnEfficiency := c.testsas.RAMEfficiency()
+			if !util.IsApproximately(c.expectedEfficiency, returnEfficiency) {
+				t.Errorf("Case %s failed: Expected RAM Efficiency %.2f but got RAM Efficiency of as %.2f", c.name, c.expectedEfficiency, returnEfficiency)
+				t.Fail()
+			}
+		})
+	}
+}
+
+func TestSummaryAllocationSet_CPUEfficiency(t *testing.T) {
+	// Generating 6 sample summary allocations for testing
+	var sa1, sa2, sa3, sa4, sa5, sa6, idlesa *SummaryAllocation
+
+	// Generating accumulated summary allocation sets for testing
+	var sas1, sas2, sas3, sas4, sas5, sas6 *SummaryAllocationSet
+
+	window, _ := ParseWindowUTC("7d")
+
+	saStart := *window.Start()
+
+	saEnd := *window.End()
+
+	sa1 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container1",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container1",
+		},
+		Start:                 saStart,
+		End:                   saEnd,
+		CPUCoreRequestAverage: 0.5,
+		CPUCoreUsageAverage:   0.1,
+		CPUCost:               0.2,
+	}
+
+	sa2 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container2",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container2",
+		},
+		Start:                 saStart,
+		End:                   saEnd,
+		CPUCoreRequestAverage: 0.5,
+		CPUCoreUsageAverage:   0.2,
+		CPUCost:               0.2,
+	}
+
+	sa3 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container3",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container3",
+		},
+		Start:                 saStart,
+		End:                   saEnd,
+		CPUCoreRequestAverage: 0.0,
+		CPUCoreUsageAverage:   0.0,
+		CPUCost:               1.0,
+	}
+
+	sa4 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container4",
+		},
+		Start:                 saStart,
+		End:                   saEnd,
+		CPUCoreRequestAverage: 0.0,
+		CPUCoreUsageAverage:   0.0,
+		CPUCost:               2.0,
+	}
+
+	sa5 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container5",
+		},
+		Start:                 saStart,
+		End:                   saEnd,
+		CPUCoreRequestAverage: 0.0,
+		CPUCoreUsageAverage:   0.1,
+		CPUCost:               0.2,
+	}
+
+	sa6 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container6",
+		},
+		Start:                 saStart,
+		End:                   saEnd,
+		CPUCoreRequestAverage: 0.0,
+		CPUCoreUsageAverage:   0.1,
+		CPUCost:               0.2,
+	}
+
+	idlesa = &SummaryAllocation{
+		Name: IdleSuffix,
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container7",
+		},
+		Start:   saStart,
+		End:     saEnd,
+		CPUCost: 1.0,
+		RAMCost: 1.0,
+	}
+
+	testcase1Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container1": sa1,
+		"cluster1/namespace1/pod1/container2": sa2,
+	}
+
+	testcase2Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container3": sa3,
+		"cluster1/namespace1/pod1/container4": sa4,
+	}
+
+	testcase3Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container5": sa5,
+		"cluster1/namespace1/pod1/container6": sa6,
+	}
+
+	testcase4Map := map[string]*SummaryAllocation{}
+
+	testcase5Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container1": sa1,
+		"cluster1/namespace1/pod1/container2": sa2,
+		"cluster1/namespace1/pod1/container3": sa3,
+		"cluster1/namespace1/pod1/container4": sa4,
+		"cluster1/namespace1/pod1/container5": sa5,
+		"cluster1/namespace1/pod1/container6": sa6,
+	}
+
+	testcase6Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container1": sa1,
+		"cluster1/namespace1/pod1/container2": sa2,
+		"cluster1/__idle__":                   idlesa,
+	}
+
+	sas1 = &SummaryAllocationSet{
+		SummaryAllocations: testcase1Map,
+		Window:             window,
+	}
+
+	sas2 = &SummaryAllocationSet{
+		SummaryAllocations: testcase2Map,
+		Window:             window,
+	}
+
+	sas3 = &SummaryAllocationSet{
+		SummaryAllocations: testcase3Map,
+		Window:             window,
+	}
+
+	sas4 = &SummaryAllocationSet{
+		SummaryAllocations: testcase4Map,
+		Window:             window,
+	}
+
+	sas5 = &SummaryAllocationSet{
+		SummaryAllocations: testcase5Map,
+		Window:             window,
+	}
+
+	sas6 = &SummaryAllocationSet{
+		SummaryAllocations: testcase6Map,
+		Window:             window,
+	}
+
+	cases := []struct {
+		name               string
+		testsas            *SummaryAllocationSet
+		expectedEfficiency float64
+	}{
+		{
+			name:               "Check CPUEfficiency when totalCPUCoreRequest is greater than 0 over allocation summary set",
+			testsas:            sas1,
+			expectedEfficiency: 0.30,
+		},
+		{
+			name:               "Check CPUEfficiency when totalCPUCoreRequest is 0 and totalCPUCost or totalCPUCoreUsage equal to 0",
+			testsas:            sas2,
+			expectedEfficiency: 0.0,
+		},
+		{
+			name:               "Check CPUEfficiency when totalCPUCoreRequest is 0 and totalCPUCost or totalCPUCoreUsage is not 0",
+			testsas:            sas3,
+			expectedEfficiency: 1.0,
+		},
+		{
+			name:               "Check CPUEfficiency when allocation summary set is empty",
+			testsas:            sas4,
+			expectedEfficiency: 0.0,
+		},
+		{
+			name:               "Check CPUEfficiency over combination of all allocation summaries",
+			testsas:            sas5,
+			expectedEfficiency: 0.50,
+		},
+		{
+			name:               "Check CPUEfficiency in presence of an idle allocation",
+			testsas:            sas6,
+			expectedEfficiency: 0.30,
+		},
+	}
+
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnEfficiency := c.testsas.CPUEfficiency()
+			if !util.IsApproximately(c.expectedEfficiency, returnEfficiency) {
+				t.Errorf("Case %s failed: Expected CPU Efficiency %.2f but got CPU Efficiency of as %.2f", c.name, c.expectedEfficiency, returnEfficiency)
+				t.Fail()
+			}
+		})
+	}
+}
+
+func TestSummaryAllocationSet_TotalEfficiency(t *testing.T) {
+	// Generating 6 sample summary allocations for testing
+	var sa1, sa2, sa3, sa4, sa5, sa6, idlesa *SummaryAllocation
+
+	// Generating accumulated summary allocation sets for testing
+	var sas1, sas2, sas3, sas4 *SummaryAllocationSet
+
+	window, _ := ParseWindowUTC("7d")
+
+	saStart := *window.Start()
+
+	saEnd := *window.End()
+
+	sa1 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container1",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container1",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		CPUCoreRequestAverage:  0.5,
+		CPUCoreUsageAverage:    0.1,
+		CPUCost:                0.0,
+		RAMBytesRequestAverage: 0.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                0.0,
+	}
+
+	sa2 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container2",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container2",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		CPUCoreRequestAverage:  0.5,
+		CPUCoreUsageAverage:    0.2,
+		CPUCost:                0.0,
+		RAMBytesRequestAverage: 0.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                0.0,
+	}
+
+	sa3 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container3",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container3",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		CPUCoreRequestAverage:  0.5,
+		CPUCoreUsageAverage:    0.2,
+		CPUCost:                1.0,
+		RAMBytesRequestAverage: 50.0 * 1024.0 * 1024.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                1.0,
+	}
+
+	sa4 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container4",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		CPUCoreRequestAverage:  0.5,
+		CPUCoreUsageAverage:    0.1,
+		CPUCost:                1.0,
+		RAMBytesRequestAverage: 50.0 * 1024.0 * 1024.0,
+		RAMBytesUsageAverage:   20.0 * 1024.0 * 1024.0,
+		RAMCost:                1.0,
+	}
+
+	sa5 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container5",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		CPUCoreRequestAverage:  0.5,
+		CPUCoreUsageAverage:    0.1,
+		CPUCost:                1.0,
+		RAMBytesRequestAverage: 50.0 * 1024.0 * 1024.0,
+		RAMBytesUsageAverage:   10.0 * 1024.0 * 1024.0,
+		RAMCost:                1.0,
+	}
+
+	sa6 = &SummaryAllocation{
+		Name: "cluster1/namespace1/pod1/container4",
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container6",
+		},
+		Start:                  saStart,
+		End:                    saEnd,
+		CPUCoreRequestAverage:  0.5,
+		CPUCoreUsageAverage:    0.2,
+		CPUCost:                1.0,
+		RAMBytesRequestAverage: 50.0 * 1024.0 * 1024.0,
+		RAMBytesUsageAverage:   20.0 * 1024.0 * 1024.0,
+		RAMCost:                1.0,
+	}
+
+	idlesa = &SummaryAllocation{
+		Name: IdleSuffix,
+		Properties: &AllocationProperties{
+			Cluster:   "cluster1",
+			Namespace: "namespace1",
+			Pod:       "pod1",
+			Container: "container7",
+		},
+		Start:   saStart,
+		End:     saEnd,
+		CPUCost: 1.0,
+		RAMCost: 1.0,
+	}
+
+	testcase1Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container1": sa1,
+		"cluster1/namespace1/pod1/container2": sa2,
+	}
+
+	testcase2Map := map[string]*SummaryAllocation{}
+
+	testcase3Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container3": sa3,
+		"cluster1/namespace1/pod1/container4": sa4,
+		"cluster1/namespace1/pod1/container5": sa5,
+		"cluster1/namespace1/pod1/container6": sa6,
+	}
+
+	testcase4Map := map[string]*SummaryAllocation{
+		"cluster1/namespace1/pod1/container5": sa5,
+		"cluster1/namespace1/pod1/container6": sa6,
+		"cluster1/__idle__":                   idlesa,
+	}
+
+	sas1 = &SummaryAllocationSet{
+		SummaryAllocations: testcase1Map,
+		Window:             window,
+	}
+
+	sas2 = &SummaryAllocationSet{
+		SummaryAllocations: testcase2Map,
+		Window:             window,
+	}
+
+	sas3 = &SummaryAllocationSet{
+		SummaryAllocations: testcase3Map,
+		Window:             window,
+	}
+
+	sas4 = &SummaryAllocationSet{
+		SummaryAllocations: testcase4Map,
+		Window:             window,
+	}
+
+	cases := []struct {
+		name               string
+		testsas            *SummaryAllocationSet
+		expectedEfficiency float64
+	}{
+		{
+			name:               "When TotalEfficiency when sum of TotalRAMCost and TotalCPUCost is 0",
+			testsas:            sas1,
+			expectedEfficiency: 0.0,
+		},
+		{
+			name:               "Check TotalEfficiency when allocation summary set is empty",
+			testsas:            sas2,
+			expectedEfficiency: 0.0,
+		},
+		{
+			name:               "Check TotalEfficiency over all 4 allocation summaries",
+			testsas:            sas3,
+			expectedEfficiency: 0.30,
+		},
+		{
+			name:               "Check TotalEfficiency with idle cost",
+			testsas:            sas4,
+			expectedEfficiency: 0.30,
+		},
+	}
+
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			returnEfficiency := c.testsas.TotalEfficiency()
+			if !util.IsApproximately(c.expectedEfficiency, returnEfficiency) {
+				t.Errorf("Case %s failed: Expected Total Efficiency %.2f but got Total Efficiency of as %.2f", c.name, c.expectedEfficiency, returnEfficiency)
+				t.Fail()
+			}
+		})
+	}
+}

pkg/kubecost/totals.go

@@ -112,10 +112,10 @@ func (art *AllocationTotals) TotalCost() float64 {
 func ComputeAllocationTotals(as *AllocationSet, prop string) map[string]*AllocationTotals {
 	arts := map[string]*AllocationTotals{}
 
-	as.Each(func(name string, alloc *Allocation) {
+	for _, alloc := range as.Allocations {
 		// Do not count idle or unmounted allocations
 		if alloc.IsIdle() || alloc.IsUnmounted() {
-			return
+			continue
 		}
 
 		// Default to computing totals by Cluster, but allow override to use Node.
@@ -163,7 +163,7 @@ func ComputeAllocationTotals(as *AllocationSet, prop string) map[string]*Allocat
 
 		arts[key].RAMCost += alloc.RAMCost
 		arts[key].RAMCostAdjustment += alloc.RAMCostAdjustment
-	})
+	}
 
 	return arts
 }
@@ -303,139 +303,143 @@ func ComputeAssetTotals(as *AssetSet, prop AssetProperty) map[string]*AssetTotal
 	nodeNames := map[string]bool{}
 	disks := map[string]*Disk{}
 
-	as.Each(func(name string, asset Asset) {
-		if node, ok := asset.(*Node); ok {
-			// Default to computing totals by Cluster, but allow override to use Node.
-			key := node.Properties().Cluster
-			if prop == AssetNodeProp {
-				key = fmt.Sprintf("%s/%s", node.Properties().Cluster, node.Properties().Name)
-			}
+	for _, node := range as.Nodes {
+		// Default to computing totals by Cluster, but allow override to use Node.
+		key := node.Properties.Cluster
+		if prop == AssetNodeProp {
+			key = fmt.Sprintf("%s/%s", node.Properties.Cluster, node.Properties.Name)
+		}
 
-			// Add node name to list of node names. (These are to be used later
-			// for attached volumes.)
-			nodeNames[fmt.Sprintf("%s/%s", node.Properties().Cluster, node.Properties().Name)] = true
-
-			// adjustmentRate is used to scale resource costs proportionally
-			// by the adjustment. This is necessary because we only get one
-			// adjustment per Node, not one per-resource-per-Node.
-			//
-			// e.g. total cost =  $90 (cost = $100, adjustment = -$10)  => 0.9000 ( 90 / 100)
-			// e.g. total cost = $150 (cost = $450, adjustment = -$300) => 0.3333 (150 / 450)
-			// e.g. total cost = $150 (cost = $100, adjustment = $50)   => 1.5000 (150 / 100)
-			adjustmentRate := 1.0
-			if node.TotalCost()-node.Adjustment() == 0 {
-				// If (totalCost - adjustment) is 0.0 then adjustment cancels
-				// the entire node cost and we should make everything 0
-				// without dividing by 0.
-				adjustmentRate = 0.0
-				log.DedupedWarningf(5, "ComputeTotals: node cost adjusted to $0.00 for %s", node.Properties().Name)
-			} else if node.Adjustment() != 0.0 {
-				// adjustmentRate is the ratio of cost-with-adjustment (i.e. TotalCost)
-				// to cost-without-adjustment (i.e. TotalCost - Adjustment).
-				adjustmentRate = node.TotalCost() / (node.TotalCost() - node.Adjustment())
-			}
+		// Add node name to list of node names. (These are to be used later
+		// for attached volumes.)
+		nodeNames[fmt.Sprintf("%s/%s", node.Properties.Cluster, node.Properties.Name)] = true
+
+		// adjustmentRate is used to scale resource costs proportionally
+		// by the adjustment. This is necessary because we only get one
+		// adjustment per Node, not one per-resource-per-Node.
+		//
+		// e.g. total cost =  $90 (cost = $100, adjustment = -$10)  => 0.9000 ( 90 / 100)
+		// e.g. total cost = $150 (cost = $450, adjustment = -$300) => 0.3333 (150 / 450)
+		// e.g. total cost = $150 (cost = $100, adjustment = $50)   => 1.5000 (150 / 100)
+		adjustmentRate := 1.0
+		if node.TotalCost()-node.Adjustment == 0 {
+			// If (totalCost - adjustment) is 0.0 then adjustment cancels
+			// the entire node cost and we should make everything 0
+			// without dividing by 0.
+			adjustmentRate = 0.0
+			log.DedupedWarningf(5, "ComputeTotals: node cost adjusted to $0.00 for %s", node.Properties.Name)
+		} else if node.Adjustment != 0.0 {
+			// adjustmentRate is the ratio of cost-with-adjustment (i.e. TotalCost)
+			// to cost-without-adjustment (i.e. TotalCost - Adjustment).
+			adjustmentRate = node.TotalCost() / (node.TotalCost() - node.Adjustment)
+		}
 
-			// 1. Start with raw, measured resource cost
-			// 2. Apply discount to get discounted resource cost
-			// 3. Apply adjustment to get final "adjusted" resource cost
-			// 4. Subtract (3 - 2) to get adjustment in doller-terms
-			// 5. Use (2 + 4) as total cost, so (2) is "cost" and (4) is "adjustment"
-
-			// Example:
-			// - node.CPUCost   = 10.00
-			// - node.Discount  =  0.20  // We assume a 20% discount
-			// - adjustmentRate =  0.75  // CUR says we need to reduce to 75% of our post-discount node cost
-			//
-			// 1. See above
-			// 2. discountedCPUCost = 10.00 * (1.0 - 0.2) =  8.00
-			// 3. adjustedCPUCost   =  8.00 * 0.75        =  6.00  // this is the actual cost according to the CUR
-			// 4. adjustment        =  6.00 - 8.00        = -2.00
-			// 5. totalCost = 6.00, which is the sum of (2) cost = 8.00 and (4) adjustment = -2.00
-
-			discountedCPUCost := node.CPUCost * (1.0 - node.Discount)
-			adjustedCPUCost := discountedCPUCost * adjustmentRate
-			cpuCostAdjustment := adjustedCPUCost - discountedCPUCost
-
-			discountedRAMCost := node.RAMCost * (1.0 - node.Discount)
-			adjustedRAMCost := discountedRAMCost * adjustmentRate
-			ramCostAdjustment := adjustedRAMCost - discountedRAMCost
-
-			adjustedGPUCost := node.GPUCost * adjustmentRate
-			gpuCostAdjustment := adjustedGPUCost - node.GPUCost
+		// 1. Start with raw, measured resource cost
+		// 2. Apply discount to get discounted resource cost
+		// 3. Apply adjustment to get final "adjusted" resource cost
+		// 4. Subtract (3 - 2) to get adjustment in doller-terms
+		// 5. Use (2 + 4) as total cost, so (2) is "cost" and (4) is "adjustment"
+
+		// Example:
+		// - node.CPUCost   = 10.00
+		// - node.Discount  =  0.20  // We assume a 20% discount
+		// - adjustmentRate =  0.75  // CUR says we need to reduce to 75% of our post-discount node cost
+		//
+		// 1. See above
+		// 2. discountedCPUCost = 10.00 * (1.0 - 0.2) =  8.00
+		// 3. adjustedCPUCost   =  8.00 * 0.75        =  6.00  // this is the actual cost according to the CUR
+		// 4. adjustment        =  6.00 - 8.00        = -2.00
+		// 5. totalCost = 6.00, which is the sum of (2) cost = 8.00 and (4) adjustment = -2.00
+
+		discountedCPUCost := node.CPUCost * (1.0 - node.Discount)
+		adjustedCPUCost := discountedCPUCost * adjustmentRate
+		cpuCostAdjustment := adjustedCPUCost - discountedCPUCost
+
+		discountedRAMCost := node.RAMCost * (1.0 - node.Discount)
+		adjustedRAMCost := discountedRAMCost * adjustmentRate
+		ramCostAdjustment := adjustedRAMCost - discountedRAMCost
+
+		adjustedGPUCost := node.GPUCost * adjustmentRate
+		gpuCostAdjustment := adjustedGPUCost - node.GPUCost
 
-			if _, ok := arts[key]; !ok {
-				arts[key] = &AssetTotals{
-					Start:   node.Start(),
-					End:     node.End(),
-					Cluster: node.Properties().Cluster,
-					Node:    node.Properties().Name,
-				}
+		if _, ok := arts[key]; !ok {
+			arts[key] = &AssetTotals{
+				Start:   node.Start,
+				End:     node.End,
+				Cluster: node.Properties.Cluster,
+				Node:    node.Properties.Name,
 			}
+		}
 
-			if arts[key].Start.After(node.Start()) {
-				arts[key].Start = node.Start()
-			}
-			if arts[key].End.Before(node.End()) {
-				arts[key].End = node.End()
-			}
+		if arts[key].Start.After(node.Start) {
+			arts[key].Start = node.Start
+		}
+		if arts[key].End.Before(node.End) {
+			arts[key].End = node.End
+		}
 
-			if arts[key].Node != node.Properties().Name {
-				arts[key].Node = ""
-			}
+		if arts[key].Node != node.Properties.Name {
+			arts[key].Node = ""
+		}
 
-			arts[key].Count++
+		arts[key].Count++
 
-			// TotalCPUCost will be discounted cost + adjustment
-			arts[key].CPUCost += discountedCPUCost
-			arts[key].CPUCostAdjustment += cpuCostAdjustment
+		// TotalCPUCost will be discounted cost + adjustment
+		arts[key].CPUCost += discountedCPUCost
+		arts[key].CPUCostAdjustment += cpuCostAdjustment
 
-			// TotalRAMCost will be discounted cost + adjustment
-			arts[key].RAMCost += discountedRAMCost
-			arts[key].RAMCostAdjustment += ramCostAdjustment
+		// TotalRAMCost will be discounted cost + adjustment
+		arts[key].RAMCost += discountedRAMCost
+		arts[key].RAMCostAdjustment += ramCostAdjustment
+
+		// TotalGPUCost will be discounted cost + adjustment
+		arts[key].GPUCost += node.GPUCost
+		arts[key].GPUCostAdjustment += gpuCostAdjustment
+	}
 
-			// TotalGPUCost will be discounted cost + adjustment
-			arts[key].GPUCost += node.GPUCost
-			arts[key].GPUCostAdjustment += gpuCostAdjustment
-		} else if lb, ok := asset.(*LoadBalancer); ok && prop == AssetClusterProp {
-			// Only record load balancers when prop is Cluster because we
-			// can't break down LoadBalancer by node.
-			key := lb.Properties().Cluster
+	// Only record LoadBalancer and ClusterManagement when prop
+	// is cluster. We can't breakdown these types by Node.
+	if prop == AssetClusterProp {
+		for _, lb := range as.LoadBalancers {
+			key := lb.Properties.Cluster
 
 			if _, ok := arts[key]; !ok {
 				arts[key] = &AssetTotals{
-					Start:   lb.Start(),
-					End:     lb.End(),
-					Cluster: lb.Properties().Cluster,
+					Start:   lb.Start,
+					End:     lb.End,
+					Cluster: lb.Properties.Cluster,
 				}
 			}
 
 			arts[key].Count++
 			arts[key].LoadBalancerCost += lb.Cost
-			arts[key].LoadBalancerCostAdjustment += lb.adjustment
-		} else if cm, ok := asset.(*ClusterManagement); ok && prop == AssetClusterProp {
-			// Only record cluster management when prop is Cluster because we
-			// can't break down ClusterManagement by node.
-			key := cm.Properties().Cluster
+			arts[key].LoadBalancerCostAdjustment += lb.Adjustment
+		}
+
+		for _, cm := range as.ClusterManagement {
+			key := cm.Properties.Cluster
 
 			if _, ok := arts[key]; !ok {
 				arts[key] = &AssetTotals{
-					Start:   cm.Start(),
-					End:     cm.End(),
-					Cluster: cm.Properties().Cluster,
+					Start:   cm.GetStart(),
+					End:     cm.GetEnd(),
+					Cluster: cm.Properties.Cluster,
 				}
 			}
 
 			arts[key].Count++
 			arts[key].ClusterManagementCost += cm.Cost
-			arts[key].ClusterManagementCostAdjustment += cm.adjustment
-		} else if disk, ok := asset.(*Disk); ok {
-			// Record disks in an intermediate structure, which will be
-			// processed after all assets have been seen.
-			key := fmt.Sprintf("%s/%s", disk.Properties().Cluster, disk.Properties().Name)
-
-			disks[key] = disk
+			arts[key].ClusterManagementCostAdjustment += cm.Adjustment
 		}
-	})
+	}
+
+	// Record disks in an intermediate structure, which will be
+	// processed after all assets have been seen.
+	for _, disk := range as.Disks {
+		key := fmt.Sprintf("%s/%s", disk.Properties.Cluster, disk.Properties.Name)
+
+		disks[key] = disk
+	}
 
 	// Record all disks as either attached volumes or persistent volumes.
 	for name, disk := range disks {
@@ -444,18 +448,18 @@ func ComputeAssetTotals(as *AssetSet, prop AssetProperty) map[string]*AssetTotal
 		// reset the key to just the cluster.
 		key := name
 		if prop == AssetClusterProp {
-			key = disk.Properties().Cluster
+			key = disk.Properties.Cluster
 		}
 
 		if _, ok := arts[key]; !ok {
 			arts[key] = &AssetTotals{
-				Start:   disk.Start(),
-				End:     disk.End(),
-				Cluster: disk.Properties().Cluster,
+				Start:   disk.Start,
+				End:     disk.End,
+				Cluster: disk.Properties.Cluster,
 			}
 
 			if prop == AssetNodeProp {
-				arts[key].Node = disk.Properties().Name
+				arts[key].Node = disk.Properties.Name
 			}
 		}
 
@@ -466,14 +470,14 @@ func ComputeAssetTotals(as *AssetSet, prop AssetProperty) map[string]*AssetTotal
 			// TODO can we make a stronger match at the underlying ETL layer?
 			arts[key].Count++
 			arts[key].AttachedVolumeCost += disk.Cost
-			arts[key].AttachedVolumeCostAdjustment += disk.adjustment
+			arts[key].AttachedVolumeCostAdjustment += disk.Adjustment
 		} else if prop == AssetClusterProp {
 			// Here, we're looking at a PersistentVolume because we're not
 			// looking at an AttachedVolume. Only record PersistentVolume data
 			// at the cluster level (i.e. prop == AssetClusterProp).
 			arts[key].Count++
 			arts[key].PersistentVolumeCost += disk.Cost
-			arts[key].PersistentVolumeCostAdjustment += disk.adjustment
+			arts[key].PersistentVolumeCostAdjustment += disk.Adjustment
 		}
 	}
 

pkg/kubecost/window.go

@@ -278,8 +278,8 @@ func parseWindow(window string, now time.Time) (Window, error) {
 	if match != nil {
 		s, _ := strconv.ParseInt(match[1], 10, 64)
 		e, _ := strconv.ParseInt(match[2], 10, 64)
-		start := time.Unix(s, 0)
-		end := time.Unix(e, 0)
+		start := time.Unix(s, 0).UTC()
+		end := time.Unix(e, 0).UTC()
 		return NewWindow(&start, &end), nil
 	}
 
@@ -458,22 +458,22 @@ func (w Window) Hours() float64 {
 	return w.end.Sub(*w.start).Hours()
 }
 
-//IsEmpty a Window is empty if it does not have a start and an end
+// IsEmpty a Window is empty if it does not have a start and an end
 func (w Window) IsEmpty() bool {
 	return w.start == nil && w.end == nil
 }
 
-//HasDuration a Window has duration if neither start and end are not nil and not equal
+// HasDuration a Window has duration if neither start and end are not nil and not equal
 func (w Window) HasDuration() bool {
 	return !w.IsOpen() && !w.end.Equal(*w.Start())
 }
 
-//IsNegative a Window is negative if start and end are not null and end is before start
+// IsNegative a Window is negative if start and end are not null and end is before start
 func (w Window) IsNegative() bool {
 	return !w.IsOpen() && w.end.Before(*w.Start())
 }
 
-//IsOpen a Window is open if it has a nil start or end
+// IsOpen a Window is open if it has a nil start or end
 func (w Window) IsOpen() bool {
 	return w.start == nil || w.end == nil
 }
@@ -689,6 +689,118 @@ func (w Window) DurationOffsetStrings() (string, string) {
 	return timeutil.DurationOffsetStrings(dur, off)
 }
 
+// GetPercentInWindow Determine pct of item time contained the window.
+// determined by the overlap of the start/end with the given
+// window, which will be negative if there is no overlap. If
+// there is positive overlap, compare it with the total mins.
+//
+// e.g. here are the two possible scenarios as simplidied
+// 10m windows with dashes representing item's time running:
+//
+//  1. item falls entirely within one CloudCostItemSet window
+//     |     ---- |          |          |
+//     totalMins = 4.0
+//     pct := 4.0 / 4.0 = 1.0 for window 1
+//     pct := 0.0 / 4.0 = 0.0 for window 2
+//     pct := 0.0 / 4.0 = 0.0 for window 3
+//
+//  2. item overlaps multiple CloudCostItemSet windows
+//     |      ----|----------|--        |
+//     totalMins = 16.0
+//     pct :=  4.0 / 16.0 = 0.250 for window 1
+//     pct := 10.0 / 16.0 = 0.625 for window 2
+//     pct :=  2.0 / 16.0 = 0.125 for window 3
+func (w Window) GetPercentInWindow(itemStart time.Time, itemEnd time.Time) float64 {
+
+	s := itemStart
+	if s.Before(*w.Start()) {
+		s = *w.Start()
+	}
+
+	e := itemEnd
+	if e.After(*w.End()) {
+		e = *w.End()
+	}
+
+	mins := e.Sub(s).Minutes()
+	if mins <= 0.0 {
+		return 0.0
+	}
+
+	totalMins := itemEnd.Sub(itemStart).Minutes()
+
+	pct := mins / totalMins
+	return pct
+}
+
+// GetWindows returns a slice of Window with equal size between the given start and end. If windowSize does not evenly
+// divide the period between start and end, the last window is not added
+func GetWindows(start time.Time, end time.Time, windowSize time.Duration) ([]Window, error) {
+	// Ensure the range is evenly divisible into windows of the given duration
+	dur := end.Sub(start)
+	if int(dur.Minutes())%int(windowSize.Minutes()) != 0 {
+		return nil, fmt.Errorf("range not divisible by window: [%s, %s] by %s", start, end, windowSize)
+	}
+
+	// Ensure that provided times are multiples of the provided windowSize (e.g. midnight for daily windows, on the hour for hourly windows)
+	if start != start.Truncate(windowSize) {
+		return nil, fmt.Errorf("provided times are not divisible by provided window: [%s, %s] by %s", start, end, windowSize)
+	}
+
+	// Ensure timezones match
+	_, sz := start.Zone()
+	_, ez := end.Zone()
+	if sz != ez {
+		return nil, fmt.Errorf("range has mismatched timezones: %s, %s", start, end)
+	}
+	if sz != int(env.GetParsedUTCOffset().Seconds()) {
+		return nil, fmt.Errorf("range timezone doesn't match configured timezone: expected %s; found %ds", env.GetParsedUTCOffset(), sz)
+	}
+
+	// Build array of windows to cover the CloudCostItemSetRange
+	windows := []Window{}
+	s, e := start, start.Add(windowSize)
+	for !e.After(end) {
+		ws := s
+		we := e
+		windows = append(windows, NewWindow(&ws, &we))
+
+		s = s.Add(windowSize)
+		e = e.Add(windowSize)
+	}
+	return windows, nil
+}
+
+// GetWindowsForQueryWindow breaks up a window into an array of windows with a max size of queryWindow
+func GetWindowsForQueryWindow(start time.Time, end time.Time, queryWindow time.Duration) ([]Window, error) {
+	// Ensure timezones match
+	_, sz := start.Zone()
+	_, ez := end.Zone()
+	if sz != ez {
+		return nil, fmt.Errorf("range has mismatched timezones: %s, %s", start, end)
+	}
+	if sz != int(env.GetParsedUTCOffset().Seconds()) {
+		return nil, fmt.Errorf("range timezone doesn't match configured timezone: expected %s; found %ds", env.GetParsedUTCOffset(), sz)
+	}
+
+	// Build array of windows to cover the CloudCostItemSetRange
+	windows := []Window{}
+	s, e := start, start.Add(queryWindow)
+	for s.Before(end) {
+		ws := s
+		we := e
+		windows = append(windows, NewWindow(&ws, &we))
+
+		s = s.Add(queryWindow)
+		e = e.Add(queryWindow)
+		if e.After(end) {
+			e = end
+		}
+	}
+
+	return windows, nil
+}
+
 type BoundaryError struct {
 	Requested Window
 	Supported Window

pkg/kubecost/window_test.go

@@ -2,6 +2,7 @@ package kubecost
 
 import (
 	"fmt"
+	"github.com/opencost/opencost/pkg/util/timeutil"
 	"strings"
 	"testing"
 	"time"
@@ -845,3 +846,302 @@ func TestWindow_Expand(t *testing.T) {
 
 // TODO
 // func TestWindow_String(t *testing.T) {}
+
+func TestWindow_GetPercentInWindow(t *testing.T) {
+	dayStart := time.Date(2022, 12, 6, 0, 0, 0, 0, time.UTC)
+	dayEnd := dayStart.Add(timeutil.Day)
+	window := NewClosedWindow(dayStart, dayEnd)
+
+	testcases := map[string]struct {
+		window    Window
+		itemStart time.Time
+		itemEnd   time.Time
+		expected  float64
+	}{
+		"matching start/matching end": {
+			window:    window,
+			itemStart: dayStart,
+			itemEnd:   dayEnd,
+			expected:  1.0,
+		},
+		"matching start/contained end": {
+			window:    window,
+			itemStart: dayStart,
+			itemEnd:   dayEnd.Add(-time.Hour * 6),
+			expected:  1.0,
+		},
+		"contained start/matching end": {
+			window:    window,
+			itemStart: dayStart.Add(time.Hour * 6),
+			itemEnd:   dayEnd,
+			expected:  1.0,
+		},
+		"contained start/contained end": {
+			window:    window,
+			itemStart: dayStart.Add(time.Hour * 6),
+			itemEnd:   dayEnd.Add(-time.Hour * 6),
+			expected:  1.0,
+		},
+		"before start/contained end": {
+			window:    window,
+			itemStart: dayStart.Add(-time.Hour * 12),
+			itemEnd:   dayEnd.Add(-time.Hour * 12),
+			expected:  0.5,
+		},
+		"before start/before end": {
+			window:    window,
+			itemStart: dayStart.Add(-time.Hour * 24),
+			itemEnd:   dayEnd.Add(-time.Hour * 24),
+			expected:  0.0,
+		},
+		"contained start/after end": {
+			window:    window,
+			itemStart: dayStart.Add(time.Hour * 12),
+			itemEnd:   dayEnd.Add(time.Hour * 12),
+			expected:  0.5,
+		},
+		"after start/after end": {
+			window:    window,
+			itemStart: dayStart.Add(time.Hour * 24),
+			itemEnd:   dayEnd.Add(time.Hour * 24),
+			expected:  0.0,
+		},
+		"before start/after end": {
+			window:    window,
+			itemStart: dayStart.Add(-time.Hour * 12),
+			itemEnd:   dayEnd.Add(time.Hour * 12),
+			expected:  0.5,
+		},
+	}
+	for name, tc := range testcases {
+		t.Run(name, func(t *testing.T) {
+			if actual := tc.window.GetPercentInWindow(tc.itemStart, tc.itemEnd); actual != tc.expected {
+				t.Errorf("GetPercentInWindow() = %v, want %v", actual, tc.expected)
+			}
+		})
+	}
+}
+
+func TestWindow_GetWindows(t *testing.T) {
+	dayStart := time.Date(2022, 12, 6, 0, 0, 0, 0, time.UTC)
+	dayEnd := dayStart.Add(timeutil.Day)
+	loc, _ := time.LoadLocation("America/Vancouver")
+	testCases := map[string]struct {
+		start       time.Time
+		end         time.Time
+		windowSize  time.Duration
+		expected    []Window
+		expectedErr bool
+	}{
+		"mismatching tz": {
+			start:       dayStart,
+			end:         dayEnd.In(loc),
+			windowSize:  time.Hour,
+			expected:    nil,
+			expectedErr: true,
+		},
+		"hour windows over 1 hours": {
+			start:      dayStart,
+			end:        dayStart.Add(time.Hour),
+			windowSize: time.Hour,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayStart.Add(time.Hour)),
+			},
+			expectedErr: false,
+		},
+		"hour windows over 3 hours": {
+			start:      dayStart,
+			end:        dayStart.Add(time.Hour * 3),
+			windowSize: time.Hour,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayStart.Add(time.Hour)),
+				NewClosedWindow(dayStart.Add(time.Hour), dayStart.Add(time.Hour*2)),
+				NewClosedWindow(dayStart.Add(time.Hour*2), dayStart.Add(time.Hour*3)),
+			},
+			expectedErr: false,
+		},
+		"hour windows off hour grid": {
+			start:       dayStart.Add(time.Minute),
+			end:         dayEnd.Add(time.Minute),
+			windowSize:  time.Hour,
+			expected:    nil,
+			expectedErr: true,
+		},
+		"hour windows range not divisible by hour": {
+			start:       dayStart,
+			end:         dayStart.Add(time.Minute * 90),
+			windowSize:  time.Hour,
+			expected:    nil,
+			expectedErr: true,
+		},
+		"day windows over 1 day": {
+			start:      dayStart,
+			end:        dayEnd,
+			windowSize: timeutil.Day,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayEnd),
+			},
+			expectedErr: false,
+		},
+		"day windows over 3 days": {
+			start:      dayStart,
+			end:        dayStart.Add(timeutil.Day * 3),
+			windowSize: timeutil.Day,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayStart.Add(timeutil.Day)),
+				NewClosedWindow(dayStart.Add(timeutil.Day), dayStart.Add(timeutil.Day*2)),
+				NewClosedWindow(dayStart.Add(timeutil.Day*2), dayStart.Add(timeutil.Day*3)),
+			},
+			expectedErr: false,
+		},
+		"day windows off day grid": {
+			start:       dayStart.Add(time.Hour),
+			end:         dayEnd.Add(time.Hour),
+			windowSize:  timeutil.Day,
+			expected:    nil,
+			expectedErr: true,
+		},
+		"day windows range not divisible by day": {
+			start:       dayStart,
+			end:         dayEnd.Add(time.Hour),
+			windowSize:  timeutil.Day,
+			expected:    nil,
+			expectedErr: true,
+		},
+	}
+	for name, tc := range testCases {
+		t.Run(name, func(t *testing.T) {
+			actual, err := GetWindows(tc.start, tc.end, tc.windowSize)
+			if (err != nil) != tc.expectedErr {
+				t.Errorf("GetWindows() error = %v, expectedErr %v", err, tc.expectedErr)
+				return
+			}
+			if len(tc.expected) != len(actual) {
+				t.Errorf("GetWindows() []window has incorrect length expected: %d, actual: %d", len(tc.expected), len(actual))
+			}
+			for i, actualWindow := range actual {
+				expectedWindow := tc.expected[i]
+				if !actualWindow.Equal(expectedWindow) {
+					t.Errorf("GetWindow() window at index %d were not equal expected: %s, actual %s", i, expectedWindow.String(), actualWindow)
+				}
+			}
+		})
+	}
+}
+
+func TestWindow_GetWindowsForQueryWindow(t *testing.T) {
+	dayStart := time.Date(2022, 12, 6, 0, 0, 0, 0, time.UTC)
+	dayEnd := dayStart.Add(timeutil.Day)
+	loc, _ := time.LoadLocation("America/Vancouver")
+	testCases := map[string]struct {
+		start       time.Time
+		end         time.Time
+		windowSize  time.Duration
+		expected    []Window
+		expectedErr bool
+	}{
+		"mismatching tz": {
+			start:       dayStart,
+			end:         dayEnd.In(loc),
+			windowSize:  time.Hour,
+			expected:    nil,
+			expectedErr: true,
+		},
+		"hour windows over 1 hours": {
+			start:      dayStart,
+			end:        dayStart.Add(time.Hour),
+			windowSize: time.Hour,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayStart.Add(time.Hour)),
+			},
+			expectedErr: false,
+		},
+		"hour windows over 3 hours": {
+			start:      dayStart,
+			end:        dayStart.Add(time.Hour * 3),
+			windowSize: time.Hour,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayStart.Add(time.Hour)),
+				NewClosedWindow(dayStart.Add(time.Hour), dayStart.Add(time.Hour*2)),
+				NewClosedWindow(dayStart.Add(time.Hour*2), dayStart.Add(time.Hour*3)),
+			},
+			expectedErr: false,
+		},
+		"hour windows off hour grid": {
+			start:      dayStart.Add(time.Minute),
+			end:        dayStart.Add(time.Minute * 61),
+			windowSize: time.Hour,
+			expected: []Window{
+				NewClosedWindow(dayStart.Add(time.Minute), dayStart.Add(time.Minute*61)),
+			},
+			expectedErr: false,
+		},
+		"hour windows range not divisible by hour": {
+			start:      dayStart,
+			end:        dayStart.Add(time.Minute * 90),
+			windowSize: time.Hour,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayStart.Add(time.Hour)),
+				NewClosedWindow(dayStart.Add(time.Hour), dayStart.Add(time.Minute*90)),
+			},
+			expectedErr: false,
+		},
+		"day windows over 1 day": {
+			start:      dayStart,
+			end:        dayEnd,
+			windowSize: timeutil.Day,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayEnd),
+			},
+			expectedErr: false,
+		},
+		"day windows over 3 days": {
+			start:      dayStart,
+			end:        dayStart.Add(timeutil.Day * 3),
+			windowSize: timeutil.Day,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayStart.Add(timeutil.Day)),
+				NewClosedWindow(dayStart.Add(timeutil.Day), dayStart.Add(timeutil.Day*2)),
+				NewClosedWindow(dayStart.Add(timeutil.Day*2), dayStart.Add(timeutil.Day*3)),
+			},
+			expectedErr: false,
+		},
+		"day windows off day grid": {
+			start:      dayStart.Add(time.Hour),
+			end:        dayEnd.Add(time.Hour),
+			windowSize: timeutil.Day,
+			expected: []Window{
+				NewClosedWindow(dayStart.Add(time.Hour), dayEnd.Add(time.Hour)),
+			},
+			expectedErr: false,
+		},
+		"day windows range not divisible by day": {
+			start:      dayStart,
+			end:        dayEnd.Add(time.Hour),
+			windowSize: timeutil.Day,
+			expected: []Window{
+				NewClosedWindow(dayStart, dayEnd),
+				NewClosedWindow(dayEnd, dayEnd.Add(time.Hour)),
+			},
+			expectedErr: false,
+		},
+	}
+	for name, tc := range testCases {
+		t.Run(name, func(t *testing.T) {
+			actual, err := GetWindowsForQueryWindow(tc.start, tc.end, tc.windowSize)
+			if (err != nil) != tc.expectedErr {
+				t.Errorf("GetWindowsForQueryWindow() error = %v, expectedErr %v", err, tc.expectedErr)
+				return
+			}
+			if len(tc.expected) != len(actual) {
+				t.Errorf("GetWindowsForQueryWindow() []window has incorrect length expected: %d, actual: %d", len(tc.expected), len(actual))
+			}
+			for i, actualWindow := range actual {
+				expectedWindow := tc.expected[i]
+				if !actualWindow.Equal(expectedWindow) {
+					t.Errorf("GetWindowsForQueryWindow() window at index %d were not equal expected: %s, actual %s", i, expectedWindow.String(), actualWindow)
+				}
+			}
+		})
+	}
+}

pkg/metrics/metricsconfig.go

@@ -3,7 +3,6 @@ package metrics
 import (
 	"encoding/json"
 	"fmt"
-	"io/ioutil"
 	"os"
 	"path"
 	"sync"
@@ -37,7 +36,7 @@ func GetMetricsConfig() (*MetricsConfig, error) {
 	metricsConfigLock.Lock()
 	defer metricsConfigLock.Unlock()
 	mc := &MetricsConfig{}
-	body, err := ioutil.ReadFile(metricsFilePath)
+	body, err := os.ReadFile(metricsFilePath)
 	if os.IsNotExist(err) {
 
 		return mc, nil
@@ -63,7 +62,7 @@ func UpdateMetricsConfig(mc *MetricsConfig) (*MetricsConfig, error) {
 		return nil, fmt.Errorf("error encoding metrics config struct: %s", err)
 	}
 
-	err = ioutil.WriteFile(metricsFilePath, mcb, 0644)
+	err = os.WriteFile(metricsFilePath, mcb, 0644)
 	if err != nil {
 		return nil, fmt.Errorf("error writing to metrics config file: %s", err)
 	}

pkg/metrics/pvmetrics.go

@@ -28,6 +28,9 @@ func (kpvcb KubePVCollector) Describe(ch chan<- *prometheus.Desc) {
 	if _, disabled := disabledMetrics["kube_persistentvolume_status_phase"]; !disabled {
 		ch <- prometheus.NewDesc("kube_persistentvolume_status_phase", "The phase indicates if a volume is available, bound to a claim, or released by a claim.", []string{}, nil)
 	}
+	if _, disabled := disabledMetrics["kubecost_pv_info"]; !disabled {
+		ch <- prometheus.NewDesc("kubecost_pv_info", "The pv information", []string{}, nil)
+	}
 }
 
 // Collect is called by the Prometheus registry when collecting metrics.
@@ -59,7 +62,12 @@ func (kpvcb KubePVCollector) Collect(ch chan<- prometheus.Metric) {
 		if _, disabled := disabledMetrics["kube_persistentvolume_capacity_bytes"]; !disabled {
 			storage := pv.Spec.Capacity[v1.ResourceStorage]
 			m := newKubePVCapacityBytesMetric("kube_persistentvolume_capacity_bytes", pv.Name, float64(storage.Value()))
+			ch <- m
+		}
 
+		if _, disabled := disabledMetrics["kubecost_pv_info"]; !disabled {
+			storageClass := pv.Spec.StorageClassName
+			m := newKubecostPVInfoMetric("kubecost_pv_info", pv.Name, storageClass, float64(1))
 			ch <- m
 		}
 	}
@@ -165,3 +173,56 @@ func (kpcrr KubePVStatusPhaseMetric) Write(m *dto.Metric) error {
 	}
 	return nil
 }
+
+//--------------------------------------------------------------------------
+//  KubecostPVInfoMetric
+//--------------------------------------------------------------------------
+// KubecostPVInfoMetric is a prometheus.Metric
+type KubecostPVInfoMetric struct {
+	fqName       string
+	help         string
+	pv           string
+	storageClass string
+	value        float64
+}
+
+// Creates a new newKubecostPVInfoMetric, implementation of prometheus.Metric
+func newKubecostPVInfoMetric(fqname, pv, storageClass string, value float64) KubecostPVInfoMetric {
+	return KubecostPVInfoMetric{
+		fqName:       fqname,
+		help:         "kubecost_pv_info pv info",
+		pv:           pv,
+		storageClass: storageClass,
+		value:        value,
+	}
+}
+
+// Desc returns the descriptor for the Metric. This method idempotently
+// returns the same descriptor throughout the lifetime of the Metric.
+func (kpvim KubecostPVInfoMetric) Desc() *prometheus.Desc {
+	l := prometheus.Labels{
+		"persistentvolume": kpvim.pv,
+		"storageclass":     kpvim.storageClass,
+	}
+	return prometheus.NewDesc(kpvim.fqName, kpvim.help, []string{}, l)
+}
+
+// Write encodes the Metric into a "Metric" Protocol Buffer data
+// transmission object.
+func (kpvim KubecostPVInfoMetric) Write(m *dto.Metric) error {
+	m.Gauge = &dto.Gauge{
+		Value: &kpvim.value,
+	}
+
+	m.Label = []*dto.LabelPair{
+		{
+			Name:  toStringPtr("persistentvolume"),
+			Value: &kpvim.pv,
+		},
+		{
+			Name:  toStringPtr("storageclass"),
+			Value: &kpvim.storageClass,
+		},
+	}
+	return nil
+}

pkg/prom/contextnames.go

@@ -24,4 +24,8 @@ const (
 
 	// DiagnosticContextName is the name we assign queries that check the state of the prometheus connection
 	DiagnosticContextName = "diagnostic"
+
+	// ContainerStatsContextName is the name we assign queries that build
+	// container stats aggregations.
+	ContainerStatsContextName = "container-stats"
 )