hbase using google cloud storage (bucket) - google-cloud-platform

Question

I'm new to hbase.
New to explore few options.
Can I use gcs file system as backing store for hbase on google cloud?
I know there's a google's big table, assuming setting up our own cluster with gcs as filesystem will reduce cost. Agree?

Answer 1

You can try the following as an experiment. I'm almost certain that it will work in principle, but I would be concerned with the impact on speed and performance, because HBase can be very IO intensive. So try this at small scale just to see if it works, and then try to stress-test it at scale:
HBase uses HDFS as the underlying file storage system. So your HBase config should point to an HDFS cluster that it writes to.
Your HDFS cluster, in turn, needs to point to a physical file directory to which it will write its files. So this really is just a physical directory on each HDFS Datanode in the HDFS cluster. Assuming you want to setup your HBase/HDFS cluster on a set of VMs in GCP, this is really just a directory on each VM that runs an HDFS Datanode
GCP has a utility called something like GCFuse (at least for Linux VMs). This is a utility that allows you to fuse a local directory on a Linux VM to a GCP cloud bucket. Once you make it work, you can read/write files directly to/from the GCP bucket as if it were just a directory on your Linux VM.
So why not do this for the directory that is configured as your underlying physical storage in HDFS? Simply fuse that directory to a GCP cloud bucket and it should work, at least in theory. I'd start with actually fusing a directory first, and writing a couple of files to it. Once you make that work, simply point you HDFS to that directory.
I don't see a reason why the above shouldn't work. But I'm not sure whether the GCFuse utility is meant for very heavy and fast IO loads. But you can give it a try!
If yo don't know how/where to configure all this at HBase and HDFS level, all you have to do is this:
In HBase, edit the hbase-site.xml file in the ../HBase/conf folder:
<property>
<name>hbase.rootdir</name>
<value>hdfs://namenode-address:9000/hbase</value>
</property>
So this entry tells HBase that it needs to write its files into HDFS, whose Namenode is running at address namenode-address, port 9000
From here, in HDFS, edit hdfs-site.xml in the ../hadoop/etc/hadoop folder:
<property>
<name>dfs.data.dir</name>
<value>file:///home/data/hdfs1/datanode</value>
</property>
This tells each HDFS Datanode where exactly to write its physical files. This could be the directory that is fused to the GCP bucket.
Another thought: If I'm not mistaken, HBase might now be available in GCP as part of Dataproc? I haven't used GCP for a while, and back in the day there was no HBase in Dataproc, but I think it's been added since.

Answer 2

Use BigTable as an alternative for HBase. Cloud BigTable open source client libraries have implemented the same set of interfaces with HBase client libraries making it HBase compliant. As for your last question, there's no way you could connect BigTable with Cloud Storage.