So we have databases like mssql and oracle and we use informatica to scan those databases for fields with personal identifiable information. I have information given like social and name and I want to use informatica to scan the databases and give me a report of where they all are. Does anyone have any suggestions.
Related
I am creating a recruitment website, and i need to search a keys skills text area field, for multiple values.
eg. I may be looking for a C++ programmer with good python, and worked in healthcare data scientist.
so my search entry form i would put c++ python healthcare data scientist.
there could be up to 20 staff performing searches against a database with a few thousand records. I need to know what is the best technology to perform this type of search with Django and postresql database hosted on digitalocean.
I can suggest to use Django with PostgreSQL Full-Text Search.
In my opinion it's the best solution because you will have the data and the search indexes directly inside PostgreSQL and you will not be forced to install and maintain additional software (such as Elasticsearch) and keep the data and indexes in sync.
This is the simplest code example you can have to perform a full-text search in Django with PostgreSQL:
search_entry = 'c++ python healthcare data scientist'
Person.objects.filter(skills__search=search_entry)
For all the basic documentation on using the full-text search in Django with PostgreSQL you can use the official documentation: "Full text search"
If you want to deepen further you can read an article that I wrote on the subject:
"Full-Text Search in Django with PostgreSQL"
You should look into using the JSONB storage in postgres. Which is NOSQL inside a SQL database.
https://www.postgresql.org/docs/9.6/static/functions-json.html
How to access in DJANGO
https://docs.djangoproject.com/en/2.1/ref/contrib/postgres/fields/#querying-jsonfield
We are using WSO2 identity server with LDAP as a user store. Now we would like to replace LDAP with database because of huge load (> 10 Millions) and growing. I have migrated the users but how tenant details in wso2 core database could be set with the new user store. That is the user store xml details are stored in their tenant table. How this could be migrated. Or what are all the changes, configurations, needs to be set in wso2 identity server level.
At the moment we don't have exact mechanism to migrate data from one DB type to another. What we have is upgrading one version to another using same DB.
But you can do this migrating data from an LDAP to any DB like MySQL manually. If you can write an shell script to convert LDAP data toa csv file, its easy for you to move to a MySQL like DB in one import command.
Furthermore, I was able to find out some similar articles which can help you to migrate content from LDAP to MySQL [2],[3],[4]
The table structures of DB types can be found from [1].
Once you migrate the data to JDBC, you can change the custom-userstore.xml file with new connection values and restart the server. If you changed your primary user store, you need to change the user store configurations in user-mgt.xml file.
Please let me know if you need further help in migrating.
[1]https://docs.wso2.com/display/IS550/Data+Dictionary
[2] https://social.msdn.microsoft.com/Forums/sqlserver/en-US/dfae020f-a3bf-4e9b-9614-eccf7890f8c6/how-to-extract-data-from-ldap-and-then-import-it-into-sql-database-for-quicker-retrieval?forum=transactsql
[3]Active Directory data into SQL table
[4] https://www.egnyte.com/blog/2014/01/how-we-migrated-millions-of-users-from-ldap-to-mysql-using-feature-flags/
I have a list of posts with a title and descrption. I have found the full text search documentation for postgres database here. But this is not useful for me as I am presently using sqlite3 database.
What are my options to have a similar functionality for my sqlite3 db?
Since full-text-search is a database(PgSQL) specific tool, Django doesn't natively support for full text search in sqlite database,
Read more on Searching in other databases
All of the searching tools provided by django.contrib.postgres are constructed entirely on public APIs such as custom lookups and database functions. Depending on your database, you should be able to construct queries to allow similar APIs. If there are specific things which cannot be achieved this way, please open a ticket.
More: https://docs.djangoproject.com/en/2.0/topics/db/search/#a-database-s-more-advanced-comparison-functions
We are using an LDAP for user administration and authorization/authentication for a set of applications.
But there are also some applications were we need to have the LDAP user/group data accessible in an sql db2.
I'm using django to connect all these datasources and I was planning to use django celery to keep the LDAP and db2 in sync, where LDAP should be leading.
The following applies:
- The db2 user/groups tables cannot be written to, except from the scheduled celery task.
- The ldap model and db2 model both use "userid" as the primary key.
- The db2 table will insert if a userid cannot be found, and update only the changed fields, which are different on ldap then are on db2.
I was thinking of hashing the fields on LDAP and DB2 into a saved field, to check if the objects are in sync?
But what are the best practices to keep data integrity in this example?
I have a relational database on MSSQL, I've created this one on a different schema (titan) than DBO. When I try to retrieve the models from my database, the composer of strongloop arc only shows those tables that are currently saved on DBO schema but not my tables saved in my custom schema.
Any help might be useful. Thanks in advance.
For relational database discovery, LoopBack API allows you to supply schema and table. But the schema is not exposed by StrongLoop Arc. As a result, only default schema names are used by Arc. Please note dbo is the default schema name for mssql.
I suggest you open a feature request on github for StrongLoop Arc.