Airflow plugins, RBAC enabled Blueprint not working - flask

We had our Airflow custom UI based on this link and it was working fine with Airflow 1.9.0.
Following this we upgraded to 1.10.1 and also enabled RBAC. Our custom UI stopped coming after this.
We followed this explanation note-on-role-based-views and tried to use our old UI templates with appbuilder_views. On the using the TestAppBuilderBaseView from /tests/plugins/test_plugin.py,
class TestAppBuilderBaseView(AppBuilderBaseView):
#expose("/")
def test(self):
return self.render("test_plugin/test.html", content="Hello galaxy!")
we get the menu and the link, but on clicking we get the error
object has no attribute 'render'
On changing this to
return self.render_template("test_plugin/test.html",content="Hello galaxy!")
we get the error
jinja2.exceptions.TemplateNotFound: test_plugin/test.html
I have tried all possible combination placing the templates folder and the html file, but still its the same error.
I do find some forums telling to enable debug on Blueprint. but I am not aware on how you can do that with Airflow
Any guidance on this please?.
Thanks in Advance
Jeenson

The version 1.10.0 when released had a bug that was not installing the plugins correctly in the new UI. This was fixed in the version 1.10.1, but the code example for plugins in Airflow documentation is broken.
I wrote a sample project to make the integration work, you can check it here: https://github.com/felipegasparini/airflow_plugin_rbac_test
But in short, you need to:
Import the BaseView from appbuilder correctly using:
from flask_appbuilder import BaseView as AppBuilderBaseView
Change the name of the method 'test' to 'list'
Set the template_folder property to point to where your templates are.
Something like this:
from airflow.plugins_manager import AirflowPlugin
from flask_appbuilder import BaseView as AppBuilderBaseView
class TestAppBuilderBaseView(AppBuilderBaseView):
template_folder = '/root/airflow/plugins/test_plugin/templates'
#expose("/")
def list(self):
return self.render_template("test.html", content="Hello galaxy!")
v_appbuilder_view = TestAppBuilderBaseView()
v_appbuilder_package = {"name": "Test View",
"category": "Test Plugin",
"view": v_appbuilder_view}
# Defining the plugin class
class AirflowTestPlugin(AirflowPlugin):
name = "test_plugin"
# operators = [PluginOperator]
# sensors = [PluginSensorOperator]
# hooks = [PluginHook]
# executors = [PluginExecutor]
# macros = [plugin_macro]
# admin_views = [v]
# flask_blueprints = [bp]
# menu_links = [ml]
appbuilder_views = [v_appbuilder_package]
# appbuilder_menu_items = [appbuilder_mitem]

I am also faced the same issue.
After including template folder in blueprint its picking up the correct folder and here is my working example.
Please keep the folder structure like below
Plugin
|_test_plugin
|_templates
|_test.html
test_plugin.py
test_plugin.py
from airflow.plugins_manager import AirflowPlugin
from flask import Blueprint
from flask_admin import BaseView, expose
from flask_admin.base import MenuLink
class TestView(BaseView):
#expose('/')
def test(self):
return self.render("test.html", content="Hello galaxy!")
v = TestView(category="Test Plugin", name="Test View")
blue_print_ = Blueprint("test_plugin",
__name__,
template_folder='templates')
class AirflowTestPlugin(AirflowPlugin):
name = "MenuLinks"
# operators = []
flask_blueprints = [blue_print_]
# hooks = []
# executors = []
admin_views = [v]
#appbuilder_views = [v_appbuilder_package]

fgasparini's answer is correct, but I also need to enable the RBAC setting
rbac = True
in airflow.cfg in order for flask_appbuilder to work with airflow, otherwise the menu won't show up.

Related

How to add global header parameter to all url paths in drf_spectacular?

I'm using drf_spectacular to be able to use swagger and it's working fine.
I define the required parameters for my API views (whether in the path or in the header) like this:
#extend_schema(
OpenApiParameter(
name='accept-language',
type=str,
location=OpenApiParameter.HEADER,
description="fa or en. The default value is en"
),
)
but I don't want to add these lines of code to all of my API views. Is there an easy way to do this? (Something like defining these lines of code in SPECTACULAR_SETTINGS)
I already found an APPEND_COMPONENTS option in drf_spectacular's documentation but I'm not familiar with it.
You can create a custom schema class from drf_spectacular.openapi.AutoSchema and override the get_override_parameters(...) method as
from drf_spectacular.openapi import AutoSchema
from drf_spectacular.utils import OpenApiParameter
class CustomAutoSchema(AutoSchema):
global_params = [
OpenApiParameter(
name="accept-language",
type=str,
location=OpenApiParameter.HEADER,
description="`fa` or `en`. The default value is en",
)
]
def get_override_parameters(self):
params = super().get_override_parameters()
return params + self.global_params
and then attach this CustomAutoSchema class in your DEFAULT_SCHEMA_CLASS setting
REST_FRAMEWORK = {
# YOUR SETTINGS
"DEFAULT_SCHEMA_CLASS": "path.to.your.custom.class.CustomAutoSchema",
}

How to click on a element through Selenium Python

I'm trying to fetch data for facebook account using selenium browser python but can't able to find the which element I can look out for clicking on an export button.
See attached screenshot
I tried but it seems giving me an error for the class.
def login_facebook(self, username, password):
chrome_options = webdriver.ChromeOptions()
preference = {"download.default_directory": self.section_value[24]}
chrome_options.add_experimental_option("prefs", preference)
self.driver = webdriver.Chrome(self.section_value[20], chrome_options=chrome_options)
self.driver.get(self.section_value[25])
username_field = self.driver.find_element_by_id("email")
password_field = self.driver.find_element_by_id("pass")
username_field.send_keys(username)
self.driver.implicitly_wait(10)
password_field.send_keys(password)
self.driver.implicitly_wait(10)
self.driver.find_element_by_id("loginbutton").click()
self.driver.implicitly_wait(10)
self.driver.get("https://business.facebook.com/select/?next=https%3A%2F%2Fbusiness.facebook.com%2F")
self.driver.get("https://business.facebook.com/home/accounts?business_id=698597566882728")
self.driver.get("https://business.facebook.com/adsmanager/reporting/view?act="
"717590098609803&business_id=698597566882728&selected_report_id=23843123660810666")
# self.driver.get("https://business.facebook.com/adsmanager/manage/campaigns?act=717590098609803&business_id"
# "=698597566882728&tool=MANAGE_ADS&date={}-{}_{}%2Clast_month".format(self.last_month,
# self.first_day_month,
# self.last_day_month))
self.driver.find_element_by_id("export_button").click()
self.driver.implicitly_wait(10)
self.driver.find_element_by_class_name("_43rl").click()
self.driver.implicitly_wait(10)
Can you please let me know how can i click on Export button?
Well, I'm able to resolve it by using xpath.
Here is the solution
self.driver.find_element_by_xpath("//*[contains(#class, '_271k _271m _1qjd layerConfirm')]").click()
The element with text as Export is a dynamically generated element so to locate the element you have to induce WebDriverWait for the element to be clickable and you can use either of the locator strategies:
Using CSS_SELECTOR:
WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.layerConfirm>div[data-hover='tooltip'][data-tooltip-display='overflow']"))).click()
Using XPATH:
WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(#class, 'layerConfirm')]/div[#data-hover='tooltip' and text()='Export']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
to run automation scripts on applications like facebook, youtube quite a hard because they are huge coporations and their web applications are developed by the worlds best developers but its not impossible to run automation scripts sometimes elements are generated dynamically sometimes hidden or inactive you cant just go and click
one solution is you can do by click action by xpath realtive or absolute their is not id specified as "export_button" in resource file i think this might help you
you can also find element by class name or css selector as i see in screen shot the class name is present "_271K _271m _1qjd layerConfirm " you can perform click action on that

Best way to update my django model coming from an external api source?

I am getting my data through requesting an api source, then I put it in my django model. However, data update daily.. so how can I update these data without rendering it everytime?
def index (request):
session = requests.Session()
df = session.get('https://api.coincap.io/v2/assets')
response= df.json()
coin = response['data']
final_result = coin.to_dict('records')
for coin in final_result:
obj, created = Coincap.objects.update_or_create(
symbol = coin['symbol'],
name = coin['name'],
defaults = {
'price': coin['priceUsd']
})
return render(request, '/home.html/')
Right now, I have to go to /home.html , if I want my data update. However, my goal is to later serialize it and make it REST api data, so I wouldn't touch django template anymore. Anyway for it to update internally once a day after i do manage.py runserver?
For those that are looking for an example:
from django.core.management.base import BaseCommand
class Command(BaseCommand):
def handle(self,*args,**kwargs):
//Your request api here
for coin in final_result:
obj, created = Coincap.objects.update_or_create(
symbol = coin['symbol'],
name = coin['name'],
defaults = {
'price': coin['priceUsd']})
Then you run in with cron just as Nikita suggested.
One simple and common solution is to create a custom Django admin command and use Cron to run it at specified intervals. You can write a command's code to your liking and it can have access to all of the models, settings and other parts of your Django project.
You would put your code making a request and writing data to the DB, using your Django models, in your new Command class's handle() method (obviously request parameter is no longer needed). And then, if for example you have named your command update_some_data, you can run it as python manage.py update_some_data.
Assuming Cron exists and is running on the machine. Then you could setup Cron to run this command for you at specified intervals, for example create a file /etc/cron.d/your_app_name and put
0 4 * * * www-data /usr/local/bin/python /path/to/your/manage.py update_some_data >> /var/log/update_some_data.log 2>&1
This would make your update be done everyday at 04:00. If your command would provide any output, it will be written to /var/log/update_some_data.log file.
Of course this is just an example, so your server user running your app (www-data here) and path to the Python executable on the server (/usr/local/bin/python here) should be adjusted for particular use.
See links for further guidance.

How to set static_url_path in Flask application

I want to do something like this:
app = Flask(__name__)
app.config.from_object(mypackage.config)
app.static_url_path = app.config['PREFIX']+"/static"
when I try:
print app.static_url_path
I get the correct static_url_path
But in my templates when I use url_for('static'), The html file generated using jinja2 still has the default static URL path /static with the missing PREFIX that I added.
If I hardcode the path like this:
app = Flask(__name__, static_url_path='PREFIX/static')
It works fine. What am I doing wrong?
Flask creates the URL route when you create the Flask() object. You'll need to re-add that route:
# remove old static map
url_map = app.url_map
try:
for rule in url_map.iter_rules('static'):
url_map._rules.remove(rule)
except ValueError;
# no static view was created yet
pass
# register new; the same view function is used
app.add_url_rule(
app.static_url_path + '/<path:filename>',
endpoint='static', view_func=app.send_static_file)
It'll be easier just to configure your Flask() object with the correct static URL path.
Demo:
>>> from flask import Flask
>>> app = Flask(__name__)
>>> app.url_map
Map([<Rule '/static/<filename>' (HEAD, OPTIONS, GET) -> static>])
>>> app.static_url_path = '/PREFIX/static'
>>> url_map = app.url_map
>>> for rule in url_map.iter_rules('static'):
... url_map._rules.remove(rule)
...
>>> app.add_url_rule(
... app.static_url_path + '/<path:filename>',
... endpoint='static', view_func=app.send_static_file)
>>> app.url_map
Map([<Rule '/PREFIX/static/<filename>' (HEAD, OPTIONS, GET) -> static>])
The accepted answer is correct, but slightly incomplete. It is true that in order to change the static_url_path one must also update the app's url_map by removing the existing Rule for the static endpoint and adding a new Rule with the modified url path. However, one must also update the _rules_by_endpoint property on the url_map.
It is instructive to examine the add() method on the underlying Map in Werkzeug. In addition to adding a new Rule to its ._rules property, the Map also indexes the Rule by adding it to ._rules_by_endpoint. This latter mapping is what is used when you call app.url_map.iter_rules('static'). It is also what is used by Flask's url_for().
Here is a working example of how to completely rewrite the static_url_path, even if it was set in the Flask app constructor.
app = Flask(__name__, static_url_path='/some/static/path')
a_new_static_path = '/some/other/path'
# Set the static_url_path property.
app.static_url_path = a_new_static_path
# Remove the old rule from Map._rules.
for rule in app.url_map.iter_rules('static'):
app.url_map._rules.remove(rule) # There is probably only one.
# Remove the old rule from Map._rules_by_endpoint. In this case we can just
# start fresh.
app.url_map._rules_by_endpoint['static'] = []
# Add the updated rule.
app.add_url_rule(f'{a_new_static_path}/<path:filename>',
endpoint='static',
view_func=app.send_static_file)
Just to complete the answer above, we also need to clear the view_functions that maps the endpoint with the delegate:
app.view_functions["static"] = None
I think you missed static_folder
app = Flask(__name__)
app.config.from_object(mypackage.config)
app.static_url_path = app.config['PREFIX']+"/static"
app.static_folder = app.config['PREFIX']+"/static"

Dynamically generating Hudson custom workspace path

I'm trying to get a Hudson job to get built in a custom workspace path that is automatically generated using yyyyMMdd-HHmm. I can get the $BUILD_ID variable expanded as mentioned in bug 3997, and that seems to work fine. However, the workspace path is incorrect as it is of the format yyyy-MM-dd_HH-mm-ss. I've tried using the ZenTimestamp plugin v2.0.1, which changes the $BUILD_ID, but this only seems to take effect after the workspace is created.
Is there a method of defining a custom workspace in the manner that I want it?
You can use a groovy script to achieve that.
import hudson.model.*;
import hudson.util.*;
import java.util.*;
import java.text.*;
import java.io.*;
//Part 1 : Recover build parameter
AbstractBuild currentBuild = (AbstractBuild) Thread.currentThread().executable;
def envVars= currentBuild.properties.get("envVars");
def branchName = envVars["BRANCH_NAME"];
//Part 2 : Define new workspace Path
def newWorkspace = "C:\\Build\\"+branchName;
//Part 3 : Change current build workspace
def newWorspaceFilePath = new FilePath(new File(newWorkspace));
currentBuild.setWorkspace(newWorspaceFilePath);