Security in Snowflake
22 March 2023
AWS Sagemaker Studio Resource Isolation & Authentication via Azure AD
10 May 2023
Published by
BluePi
Data-Driven Business Transformation
Understanding snowflake costing
Understanding Snowflake’s Costing and Pricing Model
Snowflake’s pricing is based on a consumption-based model making it one of the best data warehouse services providers in India. It means you only pay for the resources you use. This includes the amount of data you store, the number of queries you run, and the amount of compute time you use. Furthermore, you don’t pay cash directly for Snowflake resource consumption. Instead, you pay using Snowflake credits. In other words, a Snowflake credit is a unit of measurement, similar to a virtual currency, that is only used when you use resources, such as when a virtual warehouse is running, the cloud services layer is at work, or serverless features are employed. The price of these credits varies depending on the Snowflake edition you choose.
Snowflake offers several different editions of its cloud data warehouse service to meet the needs of different types of organizations. Here is an overview of the main editions:
- Standard Edition: This is the basic edition of Snowflake and is suitable for organizations of all sizes. It includes core features such as data loading, querying, and data sharing, as well as support for popular data formats and platforms.
- Enterprise Edition: This edition is designed for large organizations with complex data requirements and includes additional features such as data governance, data encryption, and integration with other enterprise systems.
- Business Critical Edition, formerly known as Enterprise for Sensitive Data (ESD):It offers even higher levels of data protection to support the needs of organizations with extremely sensitive data. It includes all the features and services of Enterprise Edition, with the addition of enhanced security and data protection. In addition, database failover/failback adds support for business continuity and disaster recovery.
- Virtual Private Snowflake (VPS): This edition allows organizations to run Snowflake in their own virtual private cloud (VPC) on Amazon Web Services (AWS) or Microsoft Azure. This provides additional security and control over data access and management. It includes all the features and services of Business Critical Edition, but in a completely separate Snowflake environment, isolated from all other Snowflake accounts (i.e. VPS accounts do not share any resources with accounts outside the VPS). However, you may choose to enable data sharing with non-VPS customers by contacting Snowflake Support to enable listing auto-fulfillment.
In addition to the main editions, Snowflake also offers a number of specialized editions for specific use cases, such as data lake management, data integration, and data engineering. These editions include additional features and functionality to support these specific needs.
To determine which edition of Snowflake is right for your organization, you’ll need to consider your specific data needs and requirements, as well as your budget and any regulatory or security considerations. Snowflake provides detailed information on its website and offers a variety of resources and tools to help you make an informed decision. You can also work with a Snowflake sales representative to determine the best edition for your organization.
Snowflake offers several pricing plans to choose from, including a pay-as-you-go option, a commitment option, and an enterprise option. The pay-as-you-go plan allows you to pay only for the resources you use, while the commitment plan offers discounted rates for committing to a certain level of usage over a set period of time. The enterprise plan is tailored to the needs of larger organizations and includes additional features and support.
There are four major factors leading to credit consumption: Storage, Compute, Cloud services, and Data Transfer. Snowflake bases charges on the daily average data volume (in bytes) you store in its system. This includes compressed or uncompressed files you stage for bulk unloading or loading, historical data you maintain for File-safe, and data kept in database tables. Snowflake automatically compresses all the table data so that the actual physical space occupied by these tables is less than their combined raw sizes. It then calculates how much storage an account uses based on this compressed file size. This is one of the reason why Snowflake can be considered the best Data Ingestion Services Provider in India. Also, Snowflake’s monthly data storage charge is set at a flat rate per terabyte (TB). However, the precise amount per TB that you pay is based on your:
- Type of account (Pre-Purchase Capacity or On-Demand)
- Region (US or EU)
- Platform (Amazon Web Services (AWS), Azure, or Google Cloud Platform (GCP))
Snowflake offers three types of compute resources:
- Virtual Warehouses: These are the main compute resources used to run queries and perform data transformations. The cost of virtual warehouses is based on the size and number of warehouses you use, as well as the duration of their usage. Credits are billed per-second basis, with a 60-second minimum requirement. Meaning, starting or resuming a suspended warehouse incurs a fee of one minute’s worth of usage. But after a minute’s usage, all subsequent usage resumes on a per-second billing as long as you run the virtual warehouses continuously. Stopping and restarting warehouses within the first minutes leads to multiple charges because the one-minute minimum charge applies each time you restart. Warehouses are billed only for the credits they actually consume when they are actively working — not when suspended or idle. In fact, it offers a quick start/stop feature to suspend resource usage whenever you choose or automatically with user-defined rules, such as “suspend after five minutes of inactivity”.
- Serverless Compute: There are Snowflake features such as Search Optimization and Snowpipe that use Snowflake-managed compute resources rather than virtual warehouses. To minimize cost, these serverless compute resources are automatically resized and scaled up or down by Snowflake as required for each workload. The credits used by these serverless features are not monitored or controlled by resource monitors; resource monitors only work with the virtual warehouses that you create/manage in your account.
- External Functions: This is a service that allows you to run custom code within Snowflake to perform data transformations and other tasks. The cost of external functions is based on the number of function invocations and the duration of their usage.
Cloud Services are the third component of Snowflake’s pricing. A group of services that coordinate operations throughout Snowflake make up the cloud services layer of the Snowflake architecture. This layer handles user authentication, security enforcement, query compilation and optimization, request query caching, and other tasks. Cloud services connect all of Snowflake’s component parts, facilitating the utilization of virtual warehouses among them. This layer of cloud services is made up of stateless computing resources that operate across several availability zones and maintain a global state through a distributed, highly available metadata store. Only when the daily resource consumption for cloud services surpasses 10% of the daily warehouse utilization is the use of the cloud services layer subject to payment. The cost is determined each day (in the UTC time zone). By doing this, it is made sure that the 10% adjustment is always done correctly, at the credit price that day. The total of these daily calculations corresponds to the adjustment on the monthly use statement.
In addition, to compute costs, Snowflake also charges for the amount of data you store and the amount of data you transfer out of the service. Data transfer costs are based on the volume of data transferred and the destination of the transfer. Features such as External Tables, External Functions, and Data Lake Export or replicating data may incur data transfer charges. This per-byte fee for data egress depends on the region where your Snowflake account is hosted.
To determine the cost of using Snowflake, you’ll need to consider your specific usage patterns and needs. Snowflake provides a cost calculator tool on its website that can help you estimate your costs based on your usage. You can also work with a Snowflake sales representative to determine the best pricing plan for your organization.
There are several strategies you can use to optimize the cost of using Snowflake, a cloud-based data warehouse service. Here are a few tips:
- Virtual Warehouses: These are the main compute resources used to run queries and perform data transformations. The cost of virtual warehouses is based on the size and number of warehouses you use, as well as the duration of their usage. Credits are billed per-second basis, with a 60-second minimum requirement. Meaning, starting or resuming a suspended warehouse incurs a fee of one minute’s worth of usage. But after a minute’s usage, all subsequent usage resumes on a per-second billing as long as you run the virtual warehouses continuously. Stopping and restarting warehouses within the first minutes leads to multiple charges because the one-minute minimum charge applies each time you restart. Warehouses are billed only for the credits they actually consume when they are actively working — not when suspended or idle. In fact, it offers a quick start/stop feature to suspend resource usage whenever you choose or automatically with user-defined rules, such as “suspend after five minutes of inactivity”.
- Serverless Compute: There are Snowflake features such as Search Optimization and Snowpipe that use Snowflake-managed compute resources rather than virtual warehouses. To minimize cost, these serverless compute resources are automatically resized and scaled up or down by Snowflake as required for each workload. The credits used by these serverless features are not monitored or controlled by resource monitors; resource monitors only work with the virtual warehouses that you create/manage in your account.
- External Functions: This is a service that allows you to run custom code within Snowflake to perform data transformations and other tasks. The cost of external functions is based on the number of function invocations and the duration of their usage.
Cloud Services are the third component of Snowflake’s pricing. A group of services that coordinate operations throughout Snowflake make up the cloud services layer of the Snowflake architecture. This layer handles user authentication, security enforcement, query compilation and optimization, request query caching, and other tasks. Cloud services connect all of Snowflake’s component parts, facilitating the utilization of virtual warehouses among them. This layer of cloud services is made up of stateless computing resources that operate across several availability zones and maintain a global state through a distributed, highly available metadata store. Only when the daily resource consumption for cloud services surpasses 10% of the daily warehouse utilization is the use of the cloud services layer subject to payment. The cost is determined each day (in the UTC time zone). By doing this, it is made sure that the 10% adjustment is always done correctly, at the credit price that day. The total of these daily calculations corresponds to the adjustment on the monthly use statement.
In addition, to compute costs, Snowflake also charges for the amount of data you store and the amount of data you transfer out of the service. Data transfer costs are based on the volume of data transferred and the destination of the transfer. Features such as External Tables, External Functions, and Data Lake Export or replicating data may incur data transfer charges. This per-byte fee for data egress depends on the region where your Snowflake account is hosted.
To determine the cost of using Snowflake, you’ll need to consider your specific usage patterns and needs. Snowflake provides a cost calculator tool on its website that can help you estimate your costs based on your usage. You can also work with a Snowflake sales representative to determine the best pricing plan for your organization.
There are several strategies you can use to optimize the cost of using Snowflake, a cloud-based data warehouse service. Here are a few tips:
- Use pay-as-you-go pricing: Snowflake offers a pay-as-you-go pricing option that allows you to pay only for the resources you use. This can be a cost-effective option if your usage patterns are unpredictable or vary significantly over time.
- Use smaller, more efficient virtual warehouses: Snowflake’s virtual warehouses are the main compute resources used to run queries and perform data transformations. Using smaller, more efficient warehouses can help reduce costs, especially for workloads that don’t require a lot of compute power.
- Turn off virtual warehouses when not in use: Snowflake charges for virtual warehouse usage based on the duration of their usage. Turning off warehouses when they are not in use can help reduce costs. You can use Snowflake’s Automatic Suspend feature to automatically turn off warehouses when they are idle.
- Use data archiving: If you have data that is infrequently accessed, you can use Snowflake’s data archiving feature to store it in a less expensive storage tier. This can help reduce storage costs while still allowing you to retain the data for future use.
- Use a commitment pricing plan: Snowflake offers commitment pricing plans that provide discounted rates for committing to a certain level of usage over a set period of time. If you have predictable or consistent usage patterns, a commitment plan can help reduce costs.
By following these tips and regularly reviewing your Snowflake usage and costs, you can optimize your spending and get the most value out of the service.
