I have read through the documentation but I don't see much technical detail on DBFS.
Is this a hosted service or is it in the client's account? I assume it's not hosted, but I can't find it in my azure account or my aws account. I'm very interested in how this is set up and the technical details I can provide to clients. The most technical detail I can find is that there is a 2 gig file limit.
DBFS is the name for implementation of abstraction around underlying cloud storage, potentially of different types. Usually, when people are referring to the DBFS, it comes to two things:
DBFS Root - the main entry point of DBFS (/, /tmp, etc.). On AWS you need to provision it yourself as S3 Bucket. On Azure it's created during workspace creation as a dedicated & isolated storage account in a separate managed resource group. You can't update settings of that storage account after it's created, or access it directly. That's why it's recommended not to store critical data in the DBFS Root.
Other storage accounts (you can also use S3 or GCS) that are mounted into a workspace. Although it's convenient to work with mounted storage, you need to understand that these mounts are available for all in workspace (except so-called passthrough mount)
Is it a hosted service. DBFS is provisioned as a part of workspace creation process.
If you prefer you can also mount storage accounts.
You can find more details about DBFS here.
I'm working on a project for a web app where each enterprise customer will have separate resources independent to themselves (i.e. users and user data unique to a single paying customer, content unique for each paying customer, etc). It's somewhat similar to a normal CRM. We're building on AWS and planning to use RDS for the database. Given the fact that each customer does not share data across any different region, would it be most effective to:
Upon enterprise customer sign up, a new VPC is created with new RDS and EC2 instances, so that each customer has siloed databased and services
Have a single VPC with RDS and EC2 instances shared across all customers and use an indicator field in the database to determine what data belongs to each customer
We don't anticipate having over 10000 users for any given enterprise customer, and right now the project is small with only a single enterprise customer.
This answer all depends on how you anticipate your application growth (not just in terms of number of customers but also performance), however personally I would say if you have a single enterprise customer at the moment create a single VPC.
Perhaps separate out a seperate database on the RDS for each customer initially assuming that cost would become a concern, with a seperate db user for each customer to limit exposure.
If the financials allow then you would look at separating out to a seperate RDS database where the performance demands (maybe you have tiers for SLAs etc).
If you get larger customers (who may have bespoke needs, or stringent requirements such as no multi tenant hosts) I would suggest you look at an organizations setup with a account per large customer to separate out the resources and a single account for any shared or multi-customer resources.
The key is plan ahead for how you would want to but if you're still in the early midst of onboarding clients then do not over complicate your setup yet.
I enabled free trial on Google Cloud Platform and added billing account. Now billing account is closed and I have few questions:
1) Will I be able to use Firebase after trial expire without paid account?
2) Is closing billig acoount means I won't be charged for anything?
All projects are unlinked from billing account.
This is how it works:
You have a billing account linked to say 5 projects
If you want to stop billing on your resources -- you unlink that project
Now you want to close the Billing account itself -- you will not be charged for any services which fall under free tier of GCP. i.e. Firebase's free tier per month
One potential issue is that you can not create anything which is charged on the standard rates
Once you exhaust your $300 free trial subscription (keep in mind that if you cancel the billing account, you will lose access to the $300), you will still be entitled to per month free tier based on the services you are using.
Without enabling billing for a project, you can not create any resources.
If you've closed the billing account, you can re-open it. A billing account can cannot be deleted.
Here is a quote from GCP's doc regarding Closure of a Billing account:
Closing an active Cloud Billing account stops all billable services in
any projects linked to the account, such as running VM instances or
storage buckets. Projects that are not linked to an active and valid
Cloud Billing account cannot use Google Cloud or Google Maps Platform
services that aren't free.
Hope it helps!
You don't need to move google account. Your account is type free or not free' as simple as that. Once your 1 year free plan is lapsed you have to upgrade it to normal (non free) billing account.
Are there different type of a volume other than EBS volume in aws?
I see a term EBS volume but don't seem to recall any other volume mentioned.
Also, EBS is just a name for their volume?
EBS - volume
Hard - disk
It depends on your definition of "Volume". There are actually 4 types of storage available on AWS, though only Instance Storage and EBS qualify as "Volumes".
1: Instance Storage
- This is storage tied directly to an instance, like a hard drive on a physical machine. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
2: EBS
- You could think of EBS like an external drive on a physical machine. It can be detached and attached to another instance. http://aws.amazon.com/ebs/
3: S3
- This is an http based storage option. It is extremely reliable, but cannot be mapped as a volume directly, files must be retreived via an http request. http://aws.amazon.com/s3/
4: Glacier
- This is meant to be a replacement for backup tapes. It is very slow to access, but very reliable. It is only intended for backup purposes as there are heavy fees for frequent access. http://aws.amazon.com/glacier/
In terms of volumes for AWS that I know of there are two:
EBS Volumes. These are managed drives for your EC2 instances. You can boot off them, snapshot them, and move them between machines. EBS volumes come in two flavors: Standard and PIOPS. The billing between the two is slightly different, as are their performance characteristics. Unlike Ephemeral volumes the data written to EBS is durable. You can create EBS drives in a number of ways.
Ephemeral Volumes. These are volumes that come-and-go along with your EC2 instance. Anything you write to them will be lost if the EC2 instance is shutdown or moved.
AWS has a number of other storage offerings (RDS, S3, DynamoDB, Glacier) but they're not focused on offering products that act like hard disks.
Storage in AWS
Amazon S3
Amazon Elastic Block Store
Amazon Elastic File System
Amazon FSx for Lustre
Amazon FSx for Windows File Server
Amazon S3 Glacier
AWS Storage Gateway
Amazon S3
:- Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9's) of durability, and stores data for millions of applications for companies all around the world.
Amazon Elastic Block Store:-
Amazon Elastic Block Store (Amazon EBS) provides persistent block storage volumes for use with Amazon EC2 instances in the AWS Cloud. Each Amazon EBS volume is automatically replicated within its Availability Zone to protect you from component failure, offering high availability and durability. Amazon EBS volumes offer the consistent and low-latency performance needed to run your workloads. With Amazon EBS, you can scale your usage up or down within minutes—all while paying a low price for only what you provision.
Amazon Elastic File System:-
Amazon Elastic File System (Amazon EFS) provides a simple, scalable, elastic file system for Linux-based workloads for use with AWS Cloud services and on-premises resources. It is built to scale on demand to petabytes without disrupting applications, growing and shrinking automatically as you add and remove files, so your applications have the storage they need – when they need it. It is designed to provide massively parallel shared access to thousands of Amazon EC2 instances, enabling your applications to achieve high levels of aggregate throughput and IOPS with consistent low latencies. Amazon EFS is a fully managed service that requires no changes to your existing applications and tools, providing access through a standard file system interface for seamless integration. Amazon EFS is a regional service storing data within and across multiple Availability Zones (AZs) for high availability and durability. You can access your file systems across AZs and regions and share files between thousands of Amazon EC2 instances and on-premises servers via AWS Direct Connect or AWS VPN.
Amazon EFS is well suited to support a broad spectrum of use cases from highly parallelized, scale-out workloads that require the highest possible throughput to single-threaded, latency-sensitive workloads. Use cases such as lift-and-shift enterprise applications, big data analytics, web serving and content management, application development and testing, media and entertainment workflows, database backups, and container storage.
Amazon FSx for Lustre:-
Amazon FSx for Lustre is a fully managed file system that is optimized for compute-intensive workloads, such as high performance computing, machine learning, and media data processing workflows. Many of these applications require the high-performance and low latencies of scale-out, parallel file systems. Operating these file systems typically requires specialized expertise and administrative overhead, requiring you to provision storage servers and tune complex performance parameters. With Amazon FSx, you can launch and run a Lustre file system that can process massive data sets at up to hundreds of gigabytes per second of throughput, millions of IOPS, and sub-millisecond latencies.
Amazon FSx for Lustre is seamlessly integrated with Amazon S3, making it easy to link your long-term data sets with your high performance file systems to run compute-intensive workloads. You can automatically copy data from S3 to FSx for Lustre, run your workloads, and then write results back to S3. FSx for Lustre also enables you to burst your compute-intensive workloads from on-premises to AWS by allowing you to access your FSx file system over Amazon Direct Connect or VPN. FSx for Lustre helps you cost-optimize your storage for compute-intensive workloads: It provides cheap and performant non-replicated storage for processing data, with your long-term data stored durably in Amazon S3 or other low-cost data stores. With Amazon FSx, you pay for only the resources you use. There are no minimum commitments, upfront hardware or software costs, or additional fees.
Amazon FSx for Windows File Server:-
Amazon FSx for Windows File Server provides a fully managed native Microsoft Windows file system so you can easily move your Windows-based applications that require file storage to AWS. Built on Windows Server, Amazon FSx provides shared file storage with the compatibility and features that your Windows-based applications rely on, including full support for the SMB protocol and Windows NTFS, Active Directory (AD) integration, and Distributed File System (DFS). Amazon FSx uses SSD storage to provide the fast performance your Windows applications and users expect, with high levels of throughput and IOPS, and consistent sub-millisecond latencies. This compatibility and performance is particularly important when moving workloads that require Windows shared file storage, like CRM, ERP, and .NET applications, as well as home directories.
With Amazon FSx, you can launch highly durable and available Windows file systems that can be accessed from up to thousands of compute instances using the industry-standard SMB protocol. Amazon FSx eliminates the typical administrative overhead of managing Windows file servers. You pay for only the resources used, with no upfront costs, minimum commitments, or additional fees.
Amazon S3 Glacier:-
Amazon S3 Glacier is a secure, durable, and extremely low-cost storage service for data archiving and long-term backup. It is designed to deliver 99.999999999% durability, and provides comprehensive security and compliance capabilities that can help meet even the most stringent regulatory requirements. Amazon S3 Glacier provides query-in-place functionality, allowing you to run powerful analytics directly on your archive data at rest. You can store data for as little as $0.004 per gigabyte per month, a significant savings compared to on-premises solutions. To keep costs low yet suitable for varying retrieval needs, Amazon S3 Glacier provides three options for access to archives, from a few minutes to several hours.
AWS Storage Gateway:-
The AWS Storage Gateway is a hybrid storage service that enables your on-premises applications to seamlessly use AWS cloud storage. You can use the service for backup and archiving, disaster recovery, cloud data processing, storage tiering, and migration. Your applications connect to the service through a virtual machine or hardware gateway appliance using standard storage protocols, such as NFS, SMB and iSCSI. The gateway connects to AWS storage services, such as Amazon S3, Glacier, and Amazon EBS, providing storage for files, volumes, and virtual tapes in AWS. The service includes a highly-optimized data transfer mechanism, with bandwidth management, automated network resilience, and efficient data transfer, along with a local cache for low-latency on-premises access to your most active data.
I've been searching for details of this all day and cannot find any information. I'm using AWS and wanted to know if it is possible, with an amazon service or external, to monitor and log the EC2 compute usage of individual directories?
No, it is not possible to allocate Amazon EC2 usage costs to a fine-grained level, such as individual directory, application or user.
Amazon EC2 is charged on an hourly basis, with the cost varying by:
Instance Type (CPU, RAM, chipset)
Operating System (eg Windows instances include an hourly charge for Windows)
Billing type (On-Demand, Reserved Instance, Spot)
While the AWS billing system can allocate EC2 costs based on tags (eg Department), this is only done for a whole instance on an hourly basis.
Some ideas for fine-grained allocation of costs:
Record usage in CloudWatch via a Custom Metric, then extract data for billing purposes
Store usage information in log files and send them to CloudWatch Logs, then extra data for billing purposes
Track usage in a file, send it to Amazon S3 and then aggregate information each billing cycle
Each of these options would require custom coding and processing.